Lecture 7.1: Reinforcement Learning Intro

Learning Objectives

1. Compare and contrast reinforcement learning with other AI paradigms (search, adversarial search, machine learning)
2. Define the key components of the agent-environment interaction loop
3. Identify the five components of a Markov Decision Process (MDP), explain each, explain their relationships
4. Differentiate between states and observations in reinforcement learning environments
5. Explain the structure of a “trajectory”. Explain how trajectories are generated using the MDP framework
6. Define fully observed versus partially observed environments

Artificial Intelligence paradigms so far

Search:

Developer writes heuristics

Imperative specification for how to solve the problem

Adversarial Search:

Developer writes evaluation function

Machine Learning:

Developer collects data

Declarative, implicit specification via examples

Reinforcement Learning:

Developer defines state space, actions, and reward function

Reward Function

The reward function is an abstract definition of good behavior

Different from machine learning… ML relies on concrete examples of good behavior

Different from search in Weeks 1 & 2… Those approaches relied on specifications of how to do good behavior

Agent-Environment Interaction

RL involves an agent-environment interaction loop

Markov Decision Process

$$\begin{array}{rl}& \text{An MDP is a 5-tuple, }⟨S,A,R,P,{\rho }_0⟩\\ & \begin{array}{lll}S& \text{set of valid states}& \\ A& \text{set of valid actions}& \\ R:S\times A\times S\to \mathbb{R}& \text{reward function}& \\ P:S\times A\to \mathcal{P}(S)& \text{transition probability}& P(s^{\prime }|s,a)\text{ is the probability of}\\ & \text{function}& \text{transitioning into state }{s}^{\prime }\text{ if}\\ & & \text{you start in state }s\text{ and take}\\ & & \text{action }a\\ {\rho }_0:S\to \mathbb{R}& \text{initial state distribution}& {\rho }_0(s)\text{ is the probability of}\\ & & \text{starting in state }s\end{array}\end{array}$$

States and Observations

A state is a complete description of the state of the world. There is no information about the world which is hidden from the state.

An observation is a partial description of a state, which may omit information.

When the agent is able to observe the complete state of the environment, we say that the environment is fully observed.

When the agent can only see a partial observation, we say that the environment is partially observed.

Trajectory

A trajectory τ is a sequence of states and actions in the world,

$\tau =(s_0,a_0,s_1,a_1,...)$

The very first state of the world, s₀, is randomly sampled from the start-state distribution, denoted by ρ₀:

$s_0\sim {\rho }_0(\cdot )$