Bayesian information criterion

Bayesian Information Criterion (BIC), also known as Schwarz Information Criterion (SIC), is a criterion for model selection among a finite set of models; the model with the lowest BIC is preferred. It is based on the likelihood function and is closely related to the Akaike Information Criterion (AIC). However, BIC introduces a penalty term for the number of parameters in the model to prevent overfitting, which is more stringent than that of AIC.

Overview

The BIC is an asymptotic result derived under the assumption that the data distribution is in the exponential family. It is defined as:

\[ \text{BIC} = -2 \cdot \ln(\hat{L}) + k \cdot \ln(n) \]

where:

• \( \hat{L} \) is the maximum value of the likelihood function for the model,
• \( k \) is the number of parameters to be estimated in the model,
• \( n \) is the number of observations or sample size.

The term \( -2 \cdot \ln(\hat{L}) \) penalizes lack of fit, while the term \( k \cdot \ln(n) \) penalizes complexity. The BIC is particularly useful in model selection where the goal is to select the model that balances a good fit with simplicity.

Comparison with AIC

Both AIC and BIC aim to resolve the trade-off between model fit and complexity but do so in slightly different ways. The key difference lies in the penalty term for the number of parameters: BIC's penalty term is larger, especially as the sample size \( n \) increases. This means BIC tends to select simpler models than AIC, particularly as the sample size grows.

Applications

BIC is widely used in many areas of statistical analysis, including:

• Regression analysis
• Structural equation modeling
• Machine learning for model selection
• Factor analysis

It is particularly popular in the fields of econometrics, sociology, and psychology, where complex models are common, and the risk of overfitting is a concern.

Limitations

While BIC is a powerful tool for model selection, it has limitations:

• It assumes that the true model is among the set of candidate models, which may not always be the case.
• BIC can be overly simplistic in situations where model complexity does not linearly relate to the number of parameters.
• It relies on the assumption of the data being identically and independently distributed, which may not hold in all scenarios.

See Also

• Akaike Information Criterion
• Model selection
• Overfitting
• Underfitting

References

‎