Limit Theorems and Estimation

Overview

This section covers the fundamental limit theorems of probability theory and statistical estimation methods.

Key Definitions

Def. (Sample Mean)

For a sample of $n$ observations $X_1, X_2, \ldots, X_n$, the sample mean is:

$\bar{X}_n = \frac{1}{n} \sum_{i=1}^{n} X_i$

If the observations are i.i.d. with mean $\mu$ and variance $\sigma^2$, then:

• $E[\bar{X}_n] = \mu$ (unbiased)
• $\text{Var}(\bar{X}_n) = \frac{\sigma^2}{n}$

Def. (Law of Large Numbers)

The Law of Large Numbers states that as the sample size increases, the sample mean converges to the population mean.

For i.i.d. random variables $X_1, X_2, \ldots$ with mean $\mu$ and finite variance, the sample mean $\bar{X}_n = \frac{1}{n} \sum_{i=1}^{n} X_i$ converges to $\mu$ as $n \to \infty$.

• Weak LLN: Convergence in probability: $P(|\bar{X}_n - \mu| > \epsilon) \to 0$ as $n \to \infty$
• Strong LLN: Almost sure convergence: $P(\lim_{n \to \infty} \bar{X}_n = \mu) = 1$
• Applications: Monte Carlo simulations, statistical inference

Def. (Central Limit Theorem)

The Central Limit Theorem states that the distribution of the sample mean approaches a normal distribution as the sample size increases, regardless of the population distribution (under certain conditions).

For i.i.d. random variables $X_1, X_2, \ldots, X_n$ with mean $\mu$ and variance $\sigma^2$:

$\frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} N(0,1)$

Equivalently: $\bar{X}_n \sim N\left(\mu, \frac{\sigma^2}{n}\right)$ for large $n$.

• Standard Error: $SE = \frac{\sigma}{\sqrt{n}}$
• Applications: Constructing confidence intervals, hypothesis testing

Def. (Likelihood Function)

For a sample $x_1, x_2, \ldots, x_n$ from a distribution with parameter $\theta$, the likelihood function is:

$L(\theta) = \prod_{i=1}^{n} f(x_i; \theta)$

where $f(x; \theta)$ is the PMF or PDF. The likelihood measures how likely the observed data is for different parameter values.

Def. (Maximum Likelihood Estimation)

The Maximum Likelihood Estimator (MLE) is the parameter value $\hat{\theta}$ that maximizes the likelihood function:

$\hat{\theta}_{MLE} = \arg\max_{\theta} L(\theta)$

In practice, we often maximize the log-likelihood:

$\ell(\theta) = \log L(\theta) = \sum_{i=1}^{n} \log f(x_i; \theta)$

To find the MLE, solve: $\frac{d\ell(\theta)}{d\theta} = 0$

Def. (Unbiased Estimator)

An estimator $\hat{\theta}$ is unbiased for parameter $\theta$ if:

$E[\hat{\theta}] = \theta$

The sample mean $\bar{X}$ is an unbiased estimator of the population mean $\mu$.

Examples of MLEs

• Bernoulli($p$): $\hat{p} = \frac{1}{n}\sum_{i=1}^{n} X_i$ (sample proportion)
• Poisson($\lambda$): $\hat{\lambda} = \bar{X}$ (sample mean)
• Normal($\mu, \sigma^2$): $\hat{\mu} = \bar{X}$, $\hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^{n}(X_i - \bar{X})^2$