Tôi thấy phương trình sau trong " Học tăng cường. Giới thiệu ", nhưng không hoàn toàn làm theo bước tôi đã tô màu xanh lam bên dưới. Làm thế nào chính xác là bước này bắt nguồn?
Tôi thấy phương trình sau trong " Học tăng cường. Giới thiệu ", nhưng không hoàn toàn làm theo bước tôi đã tô màu xanh lam bên dưới. Làm thế nào chính xác là bước này bắt nguồn?
Câu trả lời:
Đây là câu trả lời cho tất cả những ai thắc mắc về toán học có cấu trúc rõ ràng đằng sau nó (tức là nếu bạn thuộc nhóm người biết biến ngẫu nhiên là gì và bạn phải chỉ ra hoặc giả sử rằng một biến ngẫu nhiên có mật độ thì đây là câu trả lời cho bạn ;-)):
Trước hết chúng ta cần phải có rằng Quy trình Quyết định Markov chỉ có số lượng hữu hạn , nghĩa là chúng ta cần tồn tại một tập hợp hữu hạn có mật độ, mỗi biến thuộc về biến, tức là cho tất cả và bản đồ sao cho
(tức là trong automata đằng sau MDP, có thể có vô số trạng thái nhưng chỉ có nhiều phân phối chính xác gắn liền với sự chuyển tiếp vô hạn giữa các trạng thái)
Định lý 1 : Hãy (tức là một biến ngẫu nhiên thực khả tích) và để cho là một biến ngẫu nhiên như vậy mà có mật độ phổ biến sau đó
Bằng chứng : Về cơ bản đã được chứng minh ở đây bởi Stefan Hansen.
Định lý 2 : Hãy và để cho là các biến ngẫu nhiên tiếp tục như vậy mà có mật độ phổ biến sau đó
Chứng minh :
Đặt và đưa sau đó người ta có thể hiển thị (sử dụng thực tế là MDP chỉ có hữu hạn nhiều -rewards) mà hội tụ và rằng kể từ khi chức năng vẫn còn trong (tức là khả tích) người ta cũng có thể hiển thị (bằng cách sử dụng sự kết hợp thông thường của định lý của đơn điệu hội tụ và sau đó bị chi phối hội tụ trên các phương trình xác định cho [các factorizations của] sự kỳ vọng có điều kiện) mà
Hoặc là không gian trạng thái là hữu hạn (sau đó và tổng là hữu hạn) hoặc tất cả các phần thưởng đều là tích cực (sau đó chúng tôi sử dụng giọng đều đều hội tụ) hoặc tất cả các phần thưởng là số âm (sau đó chúng tôi đặt một dấu trừ ở phía trước của phương trình và sử dụng hội tụ đơn điệu một lần nữa) hoặc tất cả các phần thưởng bị ràng buộc (sau đó chúng tôi sử dụng hội tụ thống trị). Sau đó (bằng cách áp dụng đến cả hai mặt của một phần / hữu hạn Bellman phương trình trên), chúng tôi có được
and then the rest is usual density manipulation.
REMARK: Even in very simple tasks the state space can be infinite! One example would be the 'balancing a pole'-task. The state is essentially the angle of the pole (a value in , an uncountably infinite set!)
REMARK: People might comment 'dough, this proof can be shortened much more if you just use the density of directly and show that ' ... BUT ... my questions would be:
Let total sum of discounted rewards after time be:
Utility value of starting in state, at time, is equivalent to expected sum of
discounted rewards of executing policy starting from state onwards.
By definition of
By law of linearity
By law of Total Expectation
By definition of
By law of linearity
Assuming that the process satisfies Markov Property:
Probability of ending up in state having started from state and taken action ,
and
Reward of ending up in state having started from state and taken action ,
Therefore we can re-write above utility equation as,
Where; : Probability of taking action when in state for a stochastic policy. For deterministic policy,
Here is my proof. It is based on the manipulation of conditional distributions, which makes it easier to follow. Hope this one helps you.
What's with the following approach?
The sums are introduced in order to retrieve , and from . After all, the possible actions and possible next states can be . With these extra conditions, the linearity of the expectation leads to the result almost directly.
I am not sure how rigorous my argument is mathematically, though. I am open for improvements.
This is just a comment/addition to the accepted answer.
I was confused at the line where law of total expectation is being applied. I don't think the main form of law of total expectation can help here. A variant of that is in fact needed here.
If are random variables and assuming all the expectation exists, then the following identity holds:
In this case, , and . Then
, which by Markov property eqauls to
From there, one could follow the rest of the proof from the answer.
usually denotes the expectation assuming the agent follows policy . In this case seems non-deterministic, i.e. returns the probability that the agent takes action when in state .
It looks like , lower-case, is replacing , a random variable. The second expectation replaces the infinite sum, to reflect the assumption that we continue to follow for all future . is then the expected immediate reward on the next time step; The second expectation—which becomes —is the expected value of the next state, weighted by the probability of winding up in state having taken from .
Thus, the expectation accounts for the policy probability as well as the transition and reward functions, here expressed together as .
even though the correct answer has already been given and some time has passed, I thought the following step by step guide might be useful:
By linearity of the Expected Value we can split
into and .
I will outline the steps only for the first part, as the second part follows by the same steps combined with the Law of Total Expectation.
Whereas (III) follows form:
I know there is already an accepted answer, but I wish to provide a probably more concrete derivation. I would also like to mention that although @Jie Shi trick somewhat makes sense, but it makes me feel very uncomfortable:(. We need to consider the time dimension to make this work. And it is important to note that, the expectation is actually taken over the entire infinite horizon, rather than just over and . Let assume we start from (in fact, the derivation is the same regardless of the starting time; I do not want to contaminate the equations with another subscript )
At this stage, I believe most of us should already have in mind how the above leads to the final expression--we just need to apply sum-product rule() painstakingly.
Let us apply the law of linearity of Expectation to each term inside the
Part 1
Well this is rather trivial, all probabilities disappear (actually sum to 1) except those related to . Therefore, we have
Part 2
Guess what, this part is even more trivial--it only involves rearranging the sequence of summations.
And Eureka!! we recover a recursive pattern in side the big parentheses. Let us combine it with , and we obtain
and part 2 becomes
Part 1 + Part 2
And now if we can tuck in the time dimension and recover the general recursive formulae
Final confession, I laughed when I saw people above mention the use of law of total expectation. So here I am
There are already a great many answers to this question, but most involve few words describing what is going on in the manipulations. I'm going to answer it using way more words, I think. To start,
is defined in equation 3.11 of Sutton and Barto, with a constant discount factor and we can have or , but not both. Since the rewards, , are random variables, so is as it is merely a linear combination of random variables.
That last line follows from the linearity of expectation values. is the reward the agent gains after taking action at time step . For simplicity, I assume that it can take on a finite number of values .
Work on the first term. In words, I need to compute the expectation values of given that we know that the current state is . The formula for this is
In other words the probability of the appearance of reward is conditioned on the state ; different states may have different rewards. This distribution is a marginal distribution of a distribution that also contained the variables and , the action taken at time and the state at time after the action, respectively:
Where I have used , following the book's convention. If that last equality is confusing, forget the sums, suppress the (the probability now looks like a joint probability), use the law of multiplication and finally reintroduce the condition on in all the new terms. It in now easy to see that the first term is
as required. On to the second term, where I assume that is a random variable that takes on a finite number of values . Just like the first term:
Once again, I "un-marginalize" the probability distribution by writing (law of multiplication again)
The last line in there follows from the Markovian property. Remember that is the sum of all the future (discounted) rewards that the agent receives after state . The Markovian property is that the process is memory-less with regards to previous states, actions and rewards. Future actions (and the rewards they reap) depend only on the state in which the action is taken, so , by assumption. Ok, so the second term in the proof is now
as required, once again. Combining the two terms completes the proof
UPDATE
I want to address what might look like a sleight of hand in the derivation of the second term. In the equation marked with , I use a term and then later in the equation marked I claim that doesn't depend on , by arguing the Markovian property. So, you might say that if this is the case, then . But this is not true. I can take because the probability on the left side of that statement says that this is the probability of conditioned on , , , and . Because we either know or assume the state , none of the other conditionals matter, because of the Markovian property. If you do not know or assume the state , then the future rewards (the meaning of ) will depend on which state you begin at, because that will determine (based on the policy) which state you start at when computing .
If that argument doesn't convince you, try to compute what is:
As can be seen in the last line, it is not true that . The expected value of depends on which state you start in (i.e. the identity of ), if you do not know or assume the state .