In this module, an interpretation of supervised machine learning methods simply as abstract mappings from a sample of data to a predictive hypothesis is presented. As an important special case that covers a surprisingly large portion learning algorithms, we consider methods that select an optimal hypothesis based on a given measure of how well hypotheses fit to a sample of data. The measure can be just a straightforward measure of prediction performance of a hypothesis on the sample, such as classification accuracy or regression error. However, it can also be something more complicated and seemingly more distant from the learning objective, such as a function measuring the distance of Voronoi partitions from the sample points as is the case with nearest neighbor methods we consider as example methods. Furthermore, resampling and cross-validation based model selection method considered in the third module are also examples of this framework. The law of large numbers concept is revisited and the so-called bounded differences conditions under which it holds for arbitrary performance measures on a sample of data are considered.

In this module, resampling techniques for performance evaluation, such as splitting the sample into training and test set parts as well as its averaged variation known as cross-validation, are considered. Moreover, method for model selection, including selection of hyperparameter values, feature subsets or learning algorithms, based on the resampling approaches are considered. It is observed that this type of model selection is itself a learning algorithm in the same sense as the methods selecting the optimal hypothesis according to some performance measure considered in the second module, the performance measure in this case being the resampling method. Accordingly, to measure the expected prediction performance of hypotheses obtained by this type of a model selector, one has to use resampling techiques on the model selector itself resulting into nested resampling methods that include splitting the sample into training, validation and test parts as well as nested cross-validation.