Lecture 7.4: Policy Gradient Methods

Last Time:

Q-Learning with Function Approximation:

• Learned to approximate $‘Q(s,a)‘$ using parameters $‘\theta ‘$
• Updated parameters using TD error and gradient
• Still focused on learning value functions

Direct Policy Search

Key Idea: Instead of learning value functions, directly optimize policy parameters Policy Representation:

• Policy $‘{\pi }_{\theta }(s,a)‘$ gives probability of taking action $‘a‘$ in state $‘s‘$
• Parameters $‘\theta ‘$ determine the policy behavior

Common Policy Forms:

• Deterministic: $‘{\pi }_{\theta }(s)={\text{argmax}}_a{Q}_{\theta }(s,a)‘$

• Stochastic (Softmax): $‘{\pi }_{\theta }(s,a)=\frac{e^{\beta Q_{\theta }(s,a)}}{{\sum }_{a^{\prime }}{e}^{\beta Q_{\theta }(s,a^{\prime })}}‘$ where $‘\beta ‘$ controls exploration-exploitation trade-off

Policy Value and Gradient

Policy Value:

• Expected return under policy $‘{\pi }_{\theta }‘$: $‘\rho (\theta )={\mathbb{E}}_{{\pi }_{\theta }}[{\sum }_{t=0}^{\infty }{\gamma }^t{r}_t]‘$

Policy Gradient:

• Goal: Find $‘{\nabla }_{\theta }\rho (\theta )‘$ to improve policy
• For episodic case with immediate rewards $‘R(s_0,a,s_0)‘$: $‘{\nabla }_{\theta }\rho (\theta )={\sum }_{a}R(s_0,a,s_0){\nabla }_{\theta }{\pi }_{\theta }(s_0,a)‘$

REINFORCE Algorithm

Key Insight: Can estimate gradient from experience Update Rule: $‘{\nabla }_{\theta }\rho (\theta )\approx \frac{1}{N}{\sum }_{j=1}^N\frac{u_j(s){\nabla }_{\theta }{\pi }_{\theta }(s,a_j)}{{\pi }_{\theta }(s,a_j)}‘$ where:

• $‘N‘$ is number of trials
• $‘u_j(s)‘$ is total reward from state $‘s‘$ in trial $‘j‘$
• $‘a_j‘$ is action taken in trial $‘j‘$

Practical Considerations

Challenges:

• High variance in gradient estimates
• Need many samples for reliable estimates

Solutions:

• Correlated sampling (PEGASUS algorithm)
  • Generate fixed set of random sequences
  • Compare policies on same sequences
  • Reduces variance in policy comparison

Advantages of Policy Gradient Methods

• Natural with Function Approximation:
  • Policy can be any differentiable function
  • Direct optimization of performance
• Stochastic Policies:
  • Better exploration properties
  • Smoother optimization landscape
• Handle Continuous Actions:
  • No need for explicit maximization over actions