Learning from the Kernel and the Range Space

The learning problem in machine intelligence has been traditionally formulated as an optimization task where an error metric is minimized. In the system of linear equations, becauseit is difficulttohave anexact match between thesamplesizeand the number of model parameters, an approximation is often sought-after according to the primal solution space or the dual solution space in the least error sense. Such an optimization, particularly one that is based on minimizing the least squares error, has been a popular choice due to its simplicity and tractability in analysis and implementation. The approach is predominant in engineering applications as evident from its pervasive adoption in statistical and network learning. Attributed to the computational effectiveness of the backpropagation algorithm running on the then limited hardware (see e.g.,[1, 2, 3, 4, 5]) and the theoretical establishment of the mapping capability (see e.g., [6, 7, 8, 9]), the multilayer neural networks were once a popular tool for research and applications in the 1980s.