# Lecture 3: Exploration-exploitation dilemma

Computational model of behavior:

• based on a feedback, the animal is able to learn something

Conditioning

• classical: Pavlovian

• instrumental

## Classical conditioning

Recall the

Rescorla-Wagner Rule:

$w ← w + ε u_i δ_i$

## Instrumental conditioning

• static action choice: reward delivered immediately after the choice

• sequential choice: reward after a series of action

### Decision strategies

Policy:

the strategy used by the animal to maximize the reward

Ex: Real experiment: bees and flowers

Flower Drops of nectar
Blue $r_b = 8$
Yellow $r_y = 2$

Expected value of the reward:

$⟨R⟩ = r_y \cdot p(a=\text{yellow}) + r_b \cdot p(a=\text{blue})$

These probabilities depend only on the policy of the animal.

### Greedy policy

• $p(a=\text{blue}) = 1$
• $p(a=\text{yellow}) = 0$

Ex: go always for the blue flower.

$⟨R⟩ = 8 × 1 + 0 = 8$

But if $r_b$ and $r_y$ change throughout time, the animal is tricked.

### $ε$-Greedy policy

For $ε « 1$:

• $p(a=\text{blue}) = 1-ε$
• $p(a=\text{yellow}) = ε$

Ex: back to our example:

$⟨R⟩ = 8 - 6 ε$

### softmax Gibbs-policy

Depends on the reward. For a fixed $β ≥ 0$:

• $p(a=\text{blue}) = \frac{\exp(βr_b)}{\exp(βr_b)+\exp(βr_y)}$
• $p(a=\text{yellow}) = \frac{\exp(βr_y)}{\exp(βr_b)+\exp(βr_y)}$

NB: Gibbs distribution comes from physics, where $β$ is proportional to the inverse of the temperature.

• $β ⟶ 0$: Exploration (physical analogy: very high temperature)

• $β ⟶ +∞$: Exploitation (physical analogy: very low temperature)

$p(b)$ is a sigmoid of the differences of the reward:

$p(b) = \frac{1}{1+ \exp(-β(r_b-r_y))} \begin{cases} \xrightarrow[r_b-r_y \to +∞]{} 1 \\ \xrightarrow[r_b-r_y \to -∞]{} 0 \end{cases}$

NB: $r_b-r_y$ can be positive of negative

But:

• the animal never knows the reward, it can only estimate it
• what if the reward changes over time?

## Internal estimates

Internal estimates:

Flower Drops of nectar Internal estimate
Blue $r_b = 8$ $m_b$
Yellow $r_y = 2$ $m_y$

### Greedy update

• $m_b = r_{b,i}$
• $m_y = r_{y,i}$

### Batch update

• $m_b = \frac 1 N \sum\limits_{ i=1 }^N r_{b,i}$
• $m_y = \frac 1 N \sum\limits_{ i=1 }^N r_{y,i}$

### Online update

Indirect actor, same as Rescorla-Wagner rule (delta-rule):

$m_b ← m_b + ε \underbrace{(r_{b,i} - m_b)}_{ ≝ \, δ}$
• $ε$: learning rate

• $δ$: prediction error

Tags:

Updated: