Akaike information criterion

Akaike Information Criterion (AIC) is a measure used in statistics for model selection among a set of models. The AIC is founded on information theory and is a means of evaluating the relative quality of a statistical model for a given set of data. As a methodology, it deals with the trade-off between the goodness of fit of the model and the complexity of the model itself.

Overview

The AIC was developed by Hirotsugu Akaike in 1974 and is formulated to estimate the information lost when a model is used to represent the process that generates the data. In essence, AIC provides a means for model comparison: a lower AIC score indicates a better model relative to others for a given dataset. It is important to note that AIC does not provide a test for a model in the absolute sense; rather, it is a tool for model selection.

Formula

The formula for the AIC is:

AIC = 2k - 2ln(L)

where:

• k is the number of parameters in the statistical model, and
• ln(L) is the natural logarithm of the likelihood function for the model.

The term 2k penalizes the complexity of the model (to discourage overfitting), and -2ln(L) rewards the goodness of fit.

Application

AIC is widely used in many areas of statistics, including regression analysis, time series analysis, and more. It is particularly useful in the context of model selection, where several competing models are being considered. However, it assumes that the models are nested and identically distributed.

Comparison with Other Criteria

AIC is often compared with other model selection criteria such as the Bayesian Information Criterion (BIC). While both criteria aim to select models that balance goodness of fit and simplicity, they do so in slightly different ways, leading to different selections in practice. BIC introduces a stronger penalty for the number of parameters in the model, which can lead to the selection of simpler models compared to AIC.

Limitations

One limitation of AIC is that it relies on the assumption that the true model is among those being compared. However, in practice, this may not always be the case. Additionally, AIC can only be used to compare models that have the same outcome variable and are estimated by maximum likelihood.

Conclusion

The Akaike Information Criterion is a valuable tool in statistical model selection, offering a balance between model complexity and goodness of fit. Despite its limitations, AIC remains widely used across various statistical applications for its simplicity and effectiveness in identifying models that are likely to predict or explain future data best.