Cramér–Rao bound

Cramer-Rao bound illustration variance

Cramér–Rao Bound (CRB) is a fundamental theorem in the field of statistics and estimation theory that sets a lower bound on the variance of unbiased estimators. Named after the Swedish mathematician Harald Cramér and the Indian-born American mathematician Calyampudi Radhakrishna Rao, the Cramér–Rao bound provides a benchmark for evaluating the efficiency of an unbiased estimator, indicating the best possible accuracy that any unbiased estimator can achieve for a given parameter.

Definition

The Cramér–Rao bound is mathematically defined for an unbiased estimator. If \(\theta\) is an unknown deterministic parameter to be estimated, and \(\hat{\theta}\) is an unbiased estimator of \(\theta\), then the variance of \(\hat{\theta}\) is bounded as follows:

\[ \mathrm{Var}(\hat{\theta}) \geq \frac{1}{nI(\theta)} \]

where \(n\) is the sample size and \(I(\theta)\) is the Fisher information of the sample, which measures the amount of information that an observable random variable carries about an unknown parameter \(\theta\) upon which the probability of the random variable depends.

Fisher Information

The Fisher information \(I(\theta)\) is a key concept in the definition of the Cramér–Rao bound. It is defined for a single observation as:

\[ I(\theta) = E\left[ \left( \frac{\partial}{\partial \theta} \log f(X; \theta) \right)^2 \right] \]

where \(E\) denotes the expectation, \(f(X; \theta)\) is the probability density function of the observation \(X\), and \(\frac{\partial}{\partial \theta}\) represents the partial derivative with respect to \(\theta\).

Applications and Implications

The Cramér–Rao bound has significant implications in the fields of statistics, signal processing, and econometrics. It is used to assess the efficiency of estimators, with an estimator being considered efficient if it achieves the Cramér–Rao bound. In practical terms, the CRB provides a theoretical limit on the precision and reliability of parameter estimates, guiding the design and evaluation of estimation procedures.

Limitations

While the Cramér–Rao bound offers valuable insights into the limits of estimation accuracy, it has limitations. It applies only to unbiased estimators, and many practical estimators are biased. Additionally, achieving the Cramér–Rao bound is not always possible, especially in complex or high-dimensional problems.

Extensions

The Cramér–Rao bound has been extended in several ways to accommodate a wider range of scenarios. These include the Bayesian Cramér–Rao bound, which incorporates prior information about the parameter, and the generalized Cramér–Rao bound for biased estimators.

See Also

• Estimator
• Bias of an estimator
• Maximum likelihood estimation
• Bayesian statistics