Lecture 5: Reinforcement Learning

Lecturer: Lyudmila Kushnir

Introduction

Example of unsupervized learning: modeling generalization (in biology): an organism is presented with an inupt ⟶ it has to respond to it in the same way as already seen “analogous” input

Reinforcement learning (RL): between supervized and unsupervized learning.

Partial supervision: you get a reward, that tells you if you’re doing good or bad (ex: in chess: no position to learn, but a given position can enhance your chance of winning (reward))

Examples of RL:

  • navigation: you know if you’re getting closer to the goal

  • Precise motor command: moving the muscles of the end ⇒ influence the direction of the arrow if you’re using a bow

  • Social interaction: there’s some structure/predictability to how people respond to your actions

⇒ Reinforcement is often delayed

Classical conditioning paradigm

Pavlov dog (unconditioned/conditioned stimulus): cf. computational neuroscience and neuromodeling courses

⟶ Many interpretations possible: the dog is trying to learn the reward

Stimulus: $u$, Reward: $r$

\text{Estimated reward: } v = wu\\ \text{Loss: } \quad L = \sum\limits_{ i } (r_i - wu_i)^2

The learning stops when:

w = ⟨r⟩_{u=1}

Generalization:

\textbf{v} = \textbf{w} \cdot \textbf{u}

→ Not different from supervized learning

Static action choice

Bee landing on flowers ⟶ cf. computational neuroscience and neuromodeling courses

Softmax parametrization: policy

  • p_i = \cfrac{\exp(β m_i)}{\exp(β m_b) + \exp(β m_y)}
  • $m_b, m_y$: action values (internal estimates)

  • $β$: exploration-exploitation tradeoff parameter

How to update the action values?

Indirect actor framework

m_i → m_i + ε (r_i-m_i)
⟹ \text{Ultimately: } m_i = ⟨r_i⟩

Risk avoidance

Observed with real bees!

  • $r_b = 2$ with $p=1$
  • $r_y = 6$ with $p = 1/3$

So

⟨r_b⟩ = ⟨r_y⟩

But in practice: bees prefer the more reliable/stable one, i.e. the blue flower

⇒ Not explained by the indirect actor framework, since $m_b$ and $m_y$ should be the same

Fix: bees don’t use $r$ in the update rule, but a (stricly) concave utility function

Direct actor

Goal: Optimize

⟨r⟩ = p_b ⟨r_b⟩ + p_y ⟨r_y⟩
m_b → m_b + ε(1 - p_b)\underbrace{(r_b - \bar{r})}_{≝ δ} \qquad \text{when blue is selected}\\ m_b → m_b - ε p_b (r_b - \bar{r}) \qquad \text{when blue is not selected}

where $\bar{r}$ is an arbitrary parameter controling the speed of learning. Reasonable choice:

\bar{r} = ⟨r⟩

NB: so the parameter is set to be the empirical average thereof so far: $\bar{r} = ⟨r⟩$ is not really necessary, the closer you are to $⟨r⟩$, the more likely you are to converge

Sequential action choice

Ex: Rat in a maze.

Notation: $u$: current location, $u’$: next one

Expected future reward for current policy:
v(u) = \textbf{w} \cdot \underbrace{ \textbf{u}}_{\rlap{\text{stimulus at location}}}(u)

There a two theoretical “agents”

  • the critic, who evaluates $v$ (and hence $\textbf{w}$)
  • the actor, who changes the policy

Critic learning (policy evaluation):

\textbf{w} → \textbf{w} + ε δ \textbf{u}(u)

TD-learning:

δ = r_a(u) + v(u') - v(u)

Actor learning (policy improvement):

m_a(\textbf{u}) → m_a(\textbf{u}) + ε(1-p_{a, \textbf{u}}) δ \qquad \text{if action } a \text{ is chosen}\\ m_a(\textbf{u}) → m_a(\textbf{u}) - ε p_{a, \textbf{u}} δ \qquad \text{if action } a \text{ is not chosen}\\

where $δ = (r_a(u)+v(u’)) - v(u)= r_a - \bar{r}$ remains the same (direct actor algorithm with reward $r_a(u) + v(u’)$ and $\bar{r} = v(u)$)

What is $v(u)?$

In this case:

v(u) = \Big\langle \sum\limits_{ τ≥0 } r(t+τ) \Big\rangle

With a discount factor $0 < γ ≤ 1$

The TD-learning becomes:

δ = r_a(u) + γv(u') - v(u)

v(u) = \sum\limits_{ \underbrace{i}_{\text{types of stimuli}} } \sum\limits_{ τ=0 }^t w_i(τ) u_i(t-τ)

Experiments on monkeys: Dopamine signals reward prediction error

Leave a comment