Stochastic Gradient Descent for machine learning clearly explained

Our linear regression has only two predictors (a and b), thus X is a n x 2 matrix (where n is the number of observations and 2 the number of predictors). As you can see, to solve the equation we need to calculate the matrix (X T X) then invert it. In machine learning, the number of observations is often very high as well as the number of predictors. Consequently, this operation is very expensive in terms of calculation and memory. Gradient descent algorithm is an iterative optimization algorithm that allows us to find the solution while keeping the computational complexity low.