Errors and residuals

Errors and residuals in statistics and in the context of regression analysis are two closely related but distinct measures that quantify how well a model represents the data being modeled. Understanding the difference between an error and a residual is fundamental in statistical modeling, as it helps in diagnosing and improving models.

Definition

An error refers to the difference between the observed value and the true value of a quantity of interest. Errors are not directly observable because the true value is unknown in most practical situations. In contrast, a residual is the difference between the observed value and the estimated value (the value predicted by a model). Residuals are observable and are used to assess the goodness-of-fit of a model.

Mathematical Representation

Mathematically, if \(y_i\) is the observed value and \(\mu_i\) is the true value for the \(i\)th observation, then the error (\(e_i\)) is given by: \[e_i = y_i - \mu_i\]

Similarly, if \(\hat{y}_i\) is the estimated or predicted value for the \(i\)th observation, then the residual (\(r_i\)) is: \[r_i = y_i - \hat{y}_i\]

Importance in Regression Analysis

In regression analysis, residuals are used to assess the fit of a regression model. If a model is well-fitted, the residuals will be randomly scattered around zero, indicating that the model does not systematically over- or under-predict the observed values. Various diagnostic plots and tests, such as the residual plot and the Durbin-Watson statistic, are used to examine residuals for patterns that suggest problems with the model fit.

Assumptions about Errors and Residuals

Several assumptions are made about errors and residuals in regression models, including:

• Errors are independently and identically distributed (i.i.d.).
• Errors have a mean of zero.
• Errors have constant variance (homoscedasticity).
• Errors are normally distributed (in many, but not all, regression analyses).

Violations of these assumptions can lead to issues with the model, such as biased estimates and invalid inference.

See Also

• Regression analysis
• Model selection
• Goodness of fit
• Diagnostic plot
• Durbin-Watson statistic