Iteratively reweighted least squares

Iteratively Reweighted Least Squares (IRLS) is a numerical optimization technique used to solve linear or nonlinear least squares problems where the error terms are heteroscedastic or have non-normal distributions. This method iteratively adjusts the weights of data points to minimize the residual sum of squares, making it particularly useful in robust regression and logistic regression.

Overview

The IRLS algorithm is an iterative method to find the minimum of a function that is a sum of per-observation loss functions, possibly subject to constraints. It is especially suited for problems where the residuals (differences between observed and predicted values) do not follow a normal distribution or have varying variances. By reweighting the residuals at each iteration, IRLS minimizes the impact of outliers and achieves a more robust fit than ordinary least squares (OLS).

Algorithm

The basic steps of the IRLS algorithm are as follows:

1. Start with an initial guess for the parameters.
2. Compute the residuals based on the current parameter estimates.
3. Calculate weights for each data point, typically inversely proportional to the variance of the residuals.
4. Solve the weighted least squares problem to update the parameter estimates.
5. Repeat steps 2-4 until convergence, i.e., until the change in parameter estimates falls below a predetermined threshold.

Applications

IRLS is widely used in various fields such as statistics, signal processing, and machine learning. Its applications include, but are not limited to:

• Robust regression, where the goal is to fit a model that is not unduly influenced by outliers.
• Logistic regression, particularly for binary and multinomial logistic models where the error distribution is inherently non-normal.
• Signal processing, for adaptive filtering and noise reduction in signals with non-Gaussian noise characteristics.

Advantages and Limitations

Advantages:

• Robustness to outliers and heteroscedasticity, making it suitable for real-world data that often deviate from ideal assumptions.
• Flexibility to be applied to both linear and nonlinear models.

Limitations:

• Convergence to the global minimum is not guaranteed, especially for highly non-linear models or poor initial guesses.
• Computational complexity can be higher than OLS, especially for large datasets or complex models.

See Also

• Least squares
• Weighted least squares
• Robust statistics
• Logistic regression