Concept
Overfitting
Overview
In mathematical modeling, overfitting is the production of an analysis that corresponds too closely or exactly to a particular set of data and may therefore fail to fit to additional data or predict future observations reliably. An overfitted model is a mathematical model that contains more parameters than can be justified by the data. In the special case of a model that consists of a polynomial function, these parameters represent the degree of a polynomial. The essence of overfitting is to unknowingly extract some of the residual variation (i.e., noise) as if that variation represents the underlying model structure.
Underfitting occurs when a mathematical model cannot adequately capture the underlying structure of the data. An under-fitted model is a model that is missing some parameters or terms that would appear in a correctly specified model. Underfitting would occur, for example, when fitting a linear model to nonlinear data. Such a model will tend to have poor predictive performance.
The possibility of over-fitting exists when the criterion used for selecting the model is not the same as the criterion used to judge the suitability of a model. For example, a model might be selected by maximizing its performance on some set of training data, yet its suitability might be determined by its ability to perform well on unseen data; overfitting occurs when a model begins to "memorize" training data rather than "learning" to generalize from a trend.
As an extreme example, if the number of parameters is the same as or greater than the number of observations, then a model can perfectly predict the training data simply by memorizing the data in its entirety. (For an illustration, see Figure 2.) Such a model will typically fail severely when making predictions.
Overfitting is related to both the complexity of the chosen model and how well it is optimized during training. A function class that is too large, in a suitable sense, relative to the dataset size is likely to overfit. Even when the fitted model does not have an excessive number of parameters, it is to be expected that the fitted relationship will appear to perform less well on a new dataset than on the dataset used for fitting (a phenomenon sometimes known as shrinkage). In particular, the value of the coefficient of determination will shrink relative to the original data.
To lessen the chance or amount of overfitting, several techniques are available (e.g., model comparison, cross-validation, regularization, early stopping, pruning, Bayesian priors, or dropout). The basis of some techniques is to either (1) explicitly penalize overly complex models or (2) test the model's ability to generalize by evaluating its performance on a set of data not used for training, which is assumed to approximate the typical unseen data that a model will encounter.
Overview adapted from Wikipedia (CC BY-SA 4.0; see the article history for authors).
Possible passages
Found by search and not yet checked by a person; they may only mention the idea in passing.
The knowledge of the elementary fact enables us to put the problem in an equation. Nothing remains but to deduce from this by combination the complex fact that can be observed and verified. This is what is called _integration_, and is the business of the mathematician. It may be asked why, in physical sciences, generalization so readily takes the mathematical form. The reason is now easy to see.
Henri Poincaré, The Foundations of Science
States what Overfitting is.
In a similar manner, what is called the diurnal variation of the barometer, which is very small compared with the variations arising from the irregular changes in the state of the atmosphere, was discovered by comparing the average height of the barometer at different hours of the day. When this comparison was made, it was found that there was a small difference, which on the average was constant, however the…
John Stuart Mill, A System of Logic, Ratiocinative and Inductive
States what Overfitting is.
This postulate is called the “principle of limitation of variety”. Mr. Keynes again finds that it is needed in attempts to establish laws by statistics; if he is right, it is needed for all our scientific knowledge outside pure mathematics. Jean Nicod pointed out that it is not quite sufficiently stringent. We need, according to Mr.
Bertrand Russell, Philosophy
States what Overfitting is.
Related ideas
- Parameter
- University of Texas at Austin
- Occam's razor
- Bias
- Dependent and independent variables
- Wiley (publisher)
- Cambridge University Press
- Algorithm
Place in the tradition
Part of Philosophy of science.