Model Building for Large-Scale Machine Learning

In this post on my series on "Optimization Methods for Large-Scale Machine Learning" by Bottou, Curtis, and Nocedal, I want to focus on model building in machine learning. Section 2 of the paper describes several case studies, with the purpose of showing how "the process of machine learning leads to the selection of a prediction function through solving an optimization problem." A prediction function is a mathematical function that links the model inputs to the quantity we wish to predict. From the practitioner's point of view, a prediction function is implicitly specified by the technique the data scientist has chosen (for example, regression or neural networks) and trained model parameters (what is actually learned when the technique is applied to data). For example, the structure of a neural network amounts to a description of a family of related functions.