Linear algebra is to machine learning as flour to bakery: every machine learning model is based in linear algebra, as every cake is based in flour. It is not the only ingredient, of course. Machine learning models need vector calculus, probability, and optimization, as cakes need sugar, eggs, and butter. Applied machine learning, like bakery, is essentially about combining these mathematical ingredients in clever ways to create useful (tasty?) models. This document contains introductory level linear algebra notes for applied machine learning. It is meant as a reference rather than a comprehensive review. If you ever get confused by matrix multiplication, don't remember what was the $L_2$ norm, or the conditions for linear independence, this can serve as a quick reference. It also a good introduction for people that don't need a deep understanding of linear algebra, but still want to learn about the fundamentals to read about machine learning or to use pre-packaged machine learning ...

We present new large-scale algorithms for fitting a multivariate convex regression function to $n$ samples in $d$ dimensions---a key problem in shape constrained nonparametric regression with widespread applications in engineering and the applied sciences. The infinite-dimensional learning task can be expressed via a convex quadratic program (QP) with $O(nd)$ decision variables and $O(n^2)$ constraints. While instances with $n$ in the lower thousands can be addressed with current algorithms within reasonable runtimes, solving larger problems (e.g., $n\approx 10^4$ or $10^5$) are computationally challenging. To this end, we present an active set type algorithm on the Lagrangian dual (of a perturbation) of the primal QP. For computational scalability, we perform approximate optimization of the reduced sub-problems; and propose a variety of randomized augmentation rules for expanding the active set. Although the dual is not strongly convex, we present a novel linear convergence rate of our algorithm on the dual. We demonstrate that our framework can solve instances of the convex regression problem with $n=10^5$ and $d=10$---a QP with 10 billion variables---within minutes; and offers significant computational gains (e.g., in terms of memory and runtime) compared to current algorithms.

Linear Algebra and Optimization for Machine Learning: A Textbook (Springer), authored by Charu C. Aggarwal, May 2020. PDF Download Link (Free for computers connected to subscribing institutions only). The PDF version has links for e-readers, and is preferable in terms of equation formatting to the Kindle version. A frequent challenge faced by beginners in machine learning is the extensive background requirement in linear algebra and optimization. This makes the learning curve very steep.

These six brief videos, recorded in 2020, contain ideas and suggestions from Professor Strang about the recommended order of topics in teaching and learning linear algebra. The first topic is called A New Way to Start Linear Algebra. The key point is to start right in with the columns of a matrix A and the multiplication Ax that combines those columns.That leads to The Column Space of a Matrix and the idea of independent columns and the factorization A = CR that tells so much about A. With good numbers, every student can see dependent columns.The remaining videos outline very briefly the full course: The Big Picture of Linear Algebra; Orthogonal Vectors; Eigenvalues & Eigenvectors; and Singular Values & Singular Vectors. Singular values have become so important and they come directly from the eigenvalues of A'A.You can see this new idea developing in the first video lecture of Professor Strang’s 2019 course 18.065 Matrix Methods in Data Analysis, Signal Processing, and Machine Learning.

A 2020 Vision of Linear Algebra - The Professor Strang about the recommended order of topics in teaching and learning linear algebra. The first topic is called A New Way to Start Linear Algebra. The key point is to start right in with the columns of a matrix A and the multiplication Ax that combines those columns. The remaining videos outline very briefly the full course: The Big Picture of Linear Algebra; Orthogonal Vectors; Eigenvalues & Eigenvectors; and Singular Values & Singular Vectors. Singular values have become so important and they come directly from the eigenvalues of A'A.

Since AI is a branch of Computer Science, I think we face the same issues regarding math as other areas of CS. On the one hand, mathematical concepts are foundational to CS. On the other hand, we want to make the material accessible to as broad a range of students as possible, including those with weak math backgrounds and those who are disinclined toward mathematical thinking but might still find other aspects of computer science attractive. Probability and statistics are two topics that are seen as increasingly important for K-12 students. In AI, they form the basis of probabilistic reasoning (part of our Big Idea #2).

Riemannian optimization has drawn a lot of attention due to its wide applications in practice. Riemannian stochastic first-order algorithms have been studied in the literature to solve large-scale machine learning problems over Riemannian manifolds. However, most of the existing Riemannian stochastic algorithms require the objective function to be differentiable, and they do not apply to the case where the objective function is nonsmooth. In this paper, we present two Riemannian stochastic proximal gradient methods for minimizing nonsmooth function over the Stiefel manifold. The two methods, named R-ProxSGD and R-ProxSPB, are generalizations of proximal SGD and proximal SpiderBoost in Euclidean setting to the Riemannian setting. Analysis on the incremental first-order oracle (IFO) complexity of the proposed algorithms is provided. Specifically, the R-ProxSPB algorithm finds an $\epsilon$-stationary point with $\mathcal{O}(\epsilon^{-3})$ IFOs in the online case, and $\mathcal{O}(n+\sqrt{n}\epsilon^{-3})$ IFOs in the finite-sum case with $n$ being the number of summands in the objective. Experimental results on online sparse PCA and robust low-rank matrix completion show that our proposed methods significantly outperform the existing methods that uses Riemannian subgradient information.

In this part of lectures we will talk about simplest "ml models" -- polynoms. But alone polynoms is just some function, so we will also talk about Interpolation of them and will try to make ith with Least Squares method. Let's start from looking at magic: As we see we could interpolate our 2nd order polynom equation very precise(green line totaly cover red line). The only exception is periodic functions because polynom is not periodic. So what we can do with this thing?

This textbook develops the essential tools of linear algebra, with the goal of imparting technique alongside contextual understanding. Applications go hand-in-hand with theory, each reinforcing and explaining the other. This approach encourages students to develop not only the technical proficiency needed to go on to further study, but an appreciation for when, why, and how the tools of linear algebra can be used across modern applied mathematics. No previous knowledge of linear algebra is needed to approach this text, with single-variable calculus as the only formal prerequisite. However, the reader will need to draw upon some mathematical maturity to engage in the increasing abstraction inherent to the subject.

We need to find best rotation & translation params between two sets of points in 3D space. This type of transformation called Euclidean as it preserves sizes. There are many ways of getting things done almost in any case, but currently we will use SVD for this. You ask why? -- Because matrix is basically a transformation and with SVD can be decomposed on Rotation(U), Scaling(Σ), Rotation(V) (in special case when A is mxm matrix, but this gives us a clue. To get rotation first we need to find out center of it.