Lecture 3: Exploration-exploitation dilemma

Computational model of behavior:

based on a feedback, the animal is able to learn something

Conditioning

classical: Pavlovian
instrumental

Classical conditioning

Recall the

Rescorla-Wagner Rule:
$w \leftarrow w + ε u_{i} δ_{i}$

Instrumental conditioning

static action choice: reward delivered immediately after the choice
sequential choice: reward after a series of action

Decision strategies

Policy:: the strategy used by the animal to maximize the reward

Ex: Real experiment: bees and flowers

Flower	Drops of nectar
Blue	$r_{b} = 8$
Yellow	$r_{y} = 2$

Expected value of the reward:
$⟨ R ⟩ = r_{y} \cdot p (a = yellow) + r_{b} \cdot p (a = blue)$

These probabilities depend only on the policy of the animal.

Greedy policy

$p (a = blue) = 1$
$p (a = yellow) = 0$

Ex: go always for the blue flower.

⟨ R ⟩ = 8 \times 1 + 0 = 8

But if $r_{b}$ and $r_{y}$ change throughout time, the animal is tricked.

$ε$ -Greedy policy

For $ε « 1$ :

$p (a = blue) = 1 - ε$
$p (a = yellow) = ε$

Ex: back to our example:

⟨ R ⟩ = 8 - 6 ε

softmax Gibbs-policy

Depends on the reward. For a fixed $β \geq 0$ :

$p (a = blue) = \frac{\exp (β r_{b})}{\exp (β r_{b}) + \exp (β r_{y})}$
$p (a = yellow) = \frac{\exp (β r_{y})}{\exp (β r_{b}) + \exp (β r_{y})}$

NB: Gibbs distribution comes from physics, where $β$ is proportional to the inverse of the temperature.

Exploration-Exploitation trade-off:

$β ⟶ 0$ : Exploration (physical analogy: very high temperature)
$β ⟶ + \infty$ : Exploitation (physical analogy: very low temperature)

$p (b)$ is a sigmoid of the differences of the reward:

p (b) = \frac{1}{1 + \exp (- β (r_{b} - r_{y}))} {\begin{cases} \to_{r_{b} - r_{y} \to + \infty}^{} 1 \\ \to_{r_{b} - r_{y} \to - \infty}^{} 0 \end{cases}

NB: $r_{b} - r_{y}$ can be positive of negative

But:

the animal never knows the reward, it can only estimate it
what if the reward changes over time?

Internal estimates

Internal estimates:

Flower	Drops of nectar	Internal estimate
Blue	$r_{b} = 8$	$m_{b}$
Yellow	$r_{y} = 2$	$m_{y}$

Greedy update

$m_{b} = r_{b, i}$
$m_{y} = r_{y, i}$

Batch update

$m_{b} = \frac{1}{N} \sum_{i = 1}^{N} r_{b, i}$
$m_{y} = \frac{1}{N} \sum_{i = 1}^{N} r_{y, i}$

Online update

Indirect actor, same as Rescorla-Wagner rule (delta-rule):

m_{b} \leftarrow m_{b} + ε \underset{≝ δ}{\underset{⏟}{(r_{b, i} - m_{b})}}

$ε$ : learning rate
$δ$ : prediction error

Share on

Twitter Facebook Google+ LinkedIn