Comparing the Moore-Penrose Pseudoinverse and Gradient Descent for Solving Linear Regression Problems: A Performance Analysis

Linear regression is a foundational algorithm in statistics and machine learning, widely employed for modeling the linear relationship between a dependent (or target) variable and one or more independent (or explanatory) variables (e.g., Montgomery et al. [15] and Weisberg [23]). Its simplicity, interpretability, and efficiency have made it an indispensable tool across diverse fields such as economics, engineering, biology, and social sciences. The core objective in linear regression is to determine the optimal set of parameters (or weights) for the independent variables that best predict the dependent variable, typically by minimizing the sum of squared differences between observed and predicted values--a criterion known as Ordinary Least Squares (OLS). Figure 1 provides a conceptual illustration of linear regression. Once the problem is formulated, the next crucial step is to solve for these optimal parameters. Two predominant strategies for achieving this are: The Moore-Penrose pseudoinverse, which provides a direct, analytical solution.