Lecture 5: The Binary Neuron model

Teacher: Grégory Dumont

“Information” in neurons ⟺ receiving/producing action potential ⟶ spikes

At the synapse, neurotransmitters are released

→ We’ll try to describe the neural dynamics

The Binary Neuron

Binary Neuron

• $x_1, ⋯, x_n$: synaptic inputs
• $w_1, ⋯, w_n$: synaptic weights
• $b$: biais (assumed to be $0$ here)

Output:

$y = \underbrace{H}_{\text{Heaviside function}}\left(\sum\limits_{ k=1 }^N w_k x_k - b\right)$

Binary classifier: Hyperplane separation

Ex: Alternating two choices tasks (monkey VS moving dots) ⟶ in the temporal region of the brain: there are motion-sensitive neurons

What does the binary neuron in this case? There is no proof that it happens like that in the brain, and in fact ⟶ experimentally: at the beginning, the monkey is making a lot of mistakes, but it gets better and better afterwards.

⇒ How can we make the binary neuron behave like that?

Synpatic plasticity: the brain learns by modifying the synapse weights.

Rosenblatt (1958): Perceptron learning rule

Training set of patterns:

$\lbrace (x^{(0)}, d_0), ⋯, (x^{(p)}, d_p) \rbrace$

where

• $x^{(k)}$
• $d_k = 0$ or $1$

At every step: for each pattern $k$

1. compute the output

$y_k = H\left(\sum\limits_{ i=1 }^N w_i x_i^{(k)} - b\right)$
2. If $y_k ≠ d_k$: update the weights:

$w_i(t+1) = w_i(t) + (d_k - y_k) x_i^{(k)}$

⟹ Convergence in a finite number of steps if there is a solution

$D_{blue} = \lbrace (1, 2), (1, 3), (3, 4), (2, 4) \rbrace \qquad \text{label: } 1\\ D_{red} = \lbrace (4,0), (6, 2), (2, -3), (6, -2) \rbrace \qquad \text{label: } 0$

Then if you initialize $\textbf{w} ≝ (0, 0)$:

• as $\textbf{w} \cdot \underbrace{(1, 2)}{∈ D{blue}} = 0 ≠ 1$
• $\textbf{w} → \textbf{w} + (1-0) (1, 2) = (1, 2)$

Can we compute anything with a perceptron?

Can any function:

$f: \lbrace 0, 1 \rbrace^n ⟶ \lbrace 0, 1 \rbrace$

be computed by a perceptron?

⟹ Multilayer perceptron are Turing-complete

Network dynamics in discrete time

$y_i(t+1) = H\left[\sum\limits_{ j=1 }^N w_{i,j} y_j(t) + x_i(t)\right]$
• $y_i(t+1)$: activity of neuron 1 at time step $t$
• $\sum\limits_{ j=1 }^N w_{i,j} y_j(t)$: total input from the network
• $x_i(t)$: external input (assumed to be $0$ from now on)

Dynamics:

$\vec{y}(t+1) = sign[W \vec{y}(t)]$

Fixed points = Ouput of a network

Attractor: when you always converge to a given fixed point, no matter the starting activity ⟶ this pattern of activity is said to be memorized by the pattern.

Hopfield learning rule

Set of $p$ desired outcomes:

$\lbrace ξ^{(1)}, ⋯, ξ^{(p)} \rbrace$

(i.e. desired fixed points)

Set weights to (Hopfield):

$w_{i,j} = \frac 1 N \sum\limits_{ k=1 }^p ξ_i^{(k)} ξ_j^{(k)}$

When you memorize a pattern: the inverse one is also!

Tags:

Updated: