# Probably Approximately Correct

PAC:

A theoritical framework to analyze machine learning algos. (Leslie Valiant 1984)

# Introduction

$D_n ≝ \lbrace (X_i, Y_i) \rbrace_{1 ≤ i ≤ n} ⇝ ℙ \text{ iid }$
• $X_i ∈ 𝒳$ (input space)
• $Y_i ∈ 𝒴$ (ouput)
Learning algo:
$𝒜: \bigcup\limits_{n ≥ 1} \underbrace{(𝒳 × 𝒴)^n}_{\text{training data}} ⟶ \underbrace{𝒴^𝒳}_{\text{estimator of } Y \mid X}$
Loss function:
$l: \begin{cases} 𝒴 × 𝒴 ⟶ ℝ\\ \hat{y}, y ⟼ l(\hat{y}, y) = 1_{\hat{y}≠ y} \end{cases}$
Satistical risk:
$ℛ(f) ≝ 𝔼_{(X, Y) \sim ℙ}( l(f(X, Y)))$
Estimator:
$\hat{f}_n = 𝒜(D_n) : \text{ random}$

NB: $ℛ(\hat{f}_n)$ is also random

⟹ Solution we analyze: $\underbrace{𝔼}_{\text{randomness of} \hat{f}_n : D_n}(ℛ(\hat{f}_n))$

Classification:
$ℛ(\hat{f}_n) ≝ 𝔼_{X, Y}(1_{\hat{f}_n ≠ Y}) = ℙ_{X, Y}(\hat{f}_n ≠ Y)$

Goal: minimize $ℛ(\hat{f}_n)$ ⟶ $𝔼(ℛ(\hat{f}_n))$

$ℛ^\ast ≝ \min_{f \text{ mesurable } ∈ 𝒴^𝒳} ℛ(f)$ $𝔼(ℛ(\hat{f}_n)) - ℛ^\ast = \Big(𝔼(ℛ(\hat{f}_n)) - \min_{f∈F} ℛ(f)\Big) + \Big(\min_{f∈F} ℛ(f) - ℛ^\ast \Big)$
• the first term is the estimation error of $F$

• here: we will analyze this term, we will prove results independently of $P$ (just wrt $\hat{f}_n, F$)
• the second one is the approximation error of $F$

## PAC Bounds

PAC-Bound:

$\hat{f}_n$ is said to be $ε$-accurate with proba $1-δ$ (aka $(ε, δ)$-PAC) $ℙ\Big(𝔼(ℛ(\hat{f}_n)) - \min_{f∈F} ℛ(f) > ε\Big) < δ$

NB: this is better than a guarantee in expectation:

$\underbrace{𝔼(ℛ(\hat{f}_n)) - \min_{f∈F} ℛ(f)}_{A(\hat{f}_n)} ≤ 𝔼(A(\hat{f}_n) \mid A(\hat{f}_n) > ε) \underbrace{ℙ(A(\hat{f}_n > ε))}_{≤ δ} + \underbrace{𝔼(A(\hat{f}_n) \mid A(\hat{f}_n) < ε)}_{≤ ε} ℙ(A(\hat{f}_n < ε)) \\ ≤ δ + ε$

A simple example of PAC-bound for binary classification:

• $𝒴 ≝ \lbrace 0, 1 \rbrace$
• $l(\hat{y}, y) = 1_{\hat{y}≠y}$
• $F$: finite set of functions from $𝒳 ⟶ 𝒴$

Assumption: $\min_{f∈ F} ℛ(f) = 0$

Th:

Let $\hat{f}n ∈ argmin{f∈F} \Big\lbrace \hat{ℛ}n(f) = \frac 1 n \sum\limits{ i=1 }^n 1_{f(X_i) ≠ Y_i} \Big\rbrace$ then for any $ε>0$:

$ℙ(ℛ(\hat{f}_n) > ε) ≤ \underbrace{\vert F \vert \exp(-n ε)}_{δ}$

Proof: first we note that $\hat{ℛ}(\hat{f}_n) = 0$

Then, we show that

$ℙ(ℛ(\hat{f}_n) > ε) ≤ \vert F \vert (1 - ε)^n ≤ \vert F \vert \exp(-nε) \overset{n ⟶ ∞}{⟶} 0$

NB: $δ = \vert F \vert \exp(-nε) \overset{n ⟶ ∞}{⟶} 0$

For a fixed $δ$:

$n = \frac{\log \vert F \vert + \log(1/δ)}{ε}$

Corollary:

$𝔼(ℛ(\hat{f}_n)) ≤ \frac{1+\log \vert F \vert + \log n}{n}$

Proof:

$𝔼(ℛ(\hat{f}_n)) ≤ δ + ε ≤ \vert F \vert \exp(-n ε) + ε$

Then, take $ε = \frac{\log \vert F \vert + \log n}{n}$

# General PAC-Bounds

Goal: with no assumption, $\min_{f∈ F} ℛ(f) = 0$

We still focus on $\hat{f}n ∈ argmin{f∈F}(\hat{ℛ}_n(f))$

### How good is $\hat{f}_n$?

Law of large numbers:
$\hat{ℛ}_n(f) = \frac 1 n \sum\limits_{ i=1 }^n l(f(X_i), Y_i) \overset{n⟶∞}{⟶} ℛ(f)$

Assume that we have the following inequality with proba $1 - δ$:

$∀f ∈ F, \; \vert \hat{ℛ}_n(f) - ℛ(f) \vert ≤ ε \qquad ⊛$

Roughly true because of law of large numbers.

With $⊛$:

$ℛ(\hat{f}_n) \overset{⊛}{≤} \hat{ℛ}(\hat{f}_n) + ε \\ ≤ \hat{ℛ}(f^\ast) + ε \\ \overset{⊛}{≤} ℛ(f^\ast) + 2ε$

⟹ Bounds like $⊛$ is enough to get PAC-bounds: why?

We want the rate of convergence of $\hat{ℛ}_n(f) ⟶ ℛ(f)$

NB: Central-limit theorem not enough ⟶ because it’s only asymptotical.

Solution: we use a concentration bound (the analogous of the central-limit theorem, but which holds for each $n$)

## Chernoff bound

$∀ε> 0, ℙ(\frac 1 n \sum\limits_{ i=1 }^n Z_i ≥ p+ε ) ≤ \exp(-2n ε^2)$

and

$∀ε> 0, ℙ(\frac 1 n \sum\limits_{ i=1 }^n Z_i ≤ p-ε ) ≤ \exp(-2n ε^2)$

where $Z_i \overset{\text{iid}}{⇝} 𝔹(p)$

Other concentration inequalities, such as Azuma-Hoeffding (for any bounded $Z_i$)

NB: a concentration bound (the analogous of the central-limit theorem, but for each $n$

But… the want the result for all $f∈F$, not just one fixed.

Solution: use a union bound:

$ℙ(∃f∈F, Property(f)) ≤ \sum\limits_{ f ∈ F } ℙ(Property(f))$

### With a finite class $F$ for binary classification

Th:

with probability $1 - δ$:

$∀f∈F, \; ℛ(\hat{f}_n) ≤ ℛ(f) + \sqrt{\frac{2 \log \vert F \vert + \log(2/δ)}{n}}$

## Application to histograms

Histogram predicts by majority voting in each cell ($\hat{f}_n^m$)

• $𝒴 = \lbrace 0, 1 \rbrace$
• $𝒳 ≝ [0, 1]^d$
• $(Q_j)_{1 ≤ j ≤ n}$ partition of $𝒳$

$ℱ_n$: class of classification rules predicting $0$ or $1$ in each cell

⟹ $\vert ℱ_n \vert = 2^m$

NB: the histogram is the emprical risk minimizer over $ℱ_n$

$\hat{f}_n^m ∈ argmin_{f∈ℱ_n} \hat{ℛ}_n(f)$ $ℛ(\hat{f}_n) ≤ \min_{f∈F} ℛ(f) + \sqrt{\frac{2 m \log 2 + \log(2/δ)}{n}}$

Compromise:

• for the $\min$: $m$ needs to be big
• to minimize the square root: $m$ needs to be small

Th:

$ℛ(\hat{f}_n^m) ⟶ ℛ^\ast \text{ if } m ⟶ ∞ \text{ and } \frac{m}{n} ⟶ 0$

## Contably infinite set $F$

We need to assign a positive number $c(f)$ to each $f∈F$ s.t.

$\sum\limits_{ f∈F }\exp(-c(f)) ≤ 1$

$c(f)$ can be interpreted as:

• a measure of complexity of $f$
• obtained by an a priori: suppose we know in advance what may be the good functions $π: \text{probability distribution over } F$ Then, we have $\sum\limits_{ f∈F } π(f) = 1$ Thus $\sum\limits_{ f∈F } \exp(-\underbrace{\log{1/π(f)}}_{ ≝ \; c(f)}) ≤ 1$

### Information theory view of $c(f)$

encode each function $f∈F$ in a binary language: $c(f)$ is the length of the code word for $f$.

Then, $\sum\limits_{ f∈F }\exp(-c(f)) ≤ 1$ is Kraft inequality

Th:

with probability $1 - δ$:

$∀f∈F, \; ℛ(f) ≤ \hat{ℛ}_n(f) + \sqrt{\frac{2 (c(f) + \log(1/δ))}{2n}}$

Algorithm:

Penalized empirical risk minimizer:
$\hat{f}_n ∈ argmin_{f∈ F} \left\lbrace \hat{ℛ}_n(f) + \sqrt{\frac{2 (c(f) + \log(1/δ))}{2n}}\right\rbrace$

For this penalized ERM:

$∀f∈F, \; ℛ(\hat{f}_n) ≤ ℛ(f) + \sqrt{\frac{2 (c(f) + \log(2/δ)))}{n}} \text{ with proba } 1-δ$

To target a good bound, we need $f∈F$ with small $ℛ(f)$ and small $c(f)$ ⟶ no free lunch

Warning: the algorithm can be NP-hard to compute ⟹ we need convex loss functions to compute it efficiently

Algorithm:

$F_m = 2^m \text{ histograms}$

if we have $f ∈ \bigcup_{m≥1} F_m$: we can use

• $m$ bits to encode in which $F_n$ it belongs
• $m$ bits to encode which $f∈ F_n$

⟹ We can encode all histograms in $\bigcup_{m≥1} F_m$ in a language with $2m$ bits for any $f∈F_m$.

⟹ $c(f) = 2m$

⟹ we just lose a $\sqrt{2}$ factor when we compute for $\bigcup_{m≥1} F_m$ instead of $F_n$

Tags:

Updated: