II.

1.

a).

In the general case:

$$ℛ(f) =𝔼_{X,Y}(l(f(X), Y))$$

For a quadratic loss:

$$ℛ(f) =𝔼_{X,Y}(f(X) - Y)^2$$

Machine Learning model:

• data: $(x_i, \underbrace{y_i}_{\text{label}})_i$
• goal: find an $f$ s.t. $$f(X) = Y$$

b).

$$F^\ast = {\rm argmin}_{f ∈ F} ℛ(f)$$

Model:

The "real" function:

$$Y = f(X) + ε$$

---

$$ℛ(f) = 𝔼_{X,Y} \Big[ (f(X) - Y)^2 \Big] \\ = 𝔼_{X,Y} \Big[ f(X)^2 + g(X)^2 + ε^2 + 2 g(X) ε - 2 f(X) g(X) - 2 f(X) ε\Big] \\ = σ^2 + 𝔼_{x,y} (f(X) - g(X))^2$$

Since the gaussian noise is centered and independent of $f(X), g(X)$

So $f^\ast = g$ if $F ⊆ 𝒞^∞([0,1])$ as $g = \exp(3 \bullet) ∈ F$

import plotly
import plotly.plotly as py
import plotly.graph_objs as go
%pylab inline

Populating the interactive namespace from numpy and matplotlib

plotly.offline.init_notebook_mode()