What is Gradient Descent in Machine Learning?

In every Machine Learning problem where there is an association of regression, there is one more term associated and that is called Gradient Descent. As we all know that Linear regression, Logistic regression, SVM, etc. is associated with finding the best fit line to fit in all the points where the slope of the line and bias tend to cover all the points in the dataset. This never happens as a perfect fit line leads to the condition of overfitting. So, the difference that is present between the target output and predicted output is termed as the loss function or the cost function and is given by the difference of predicted value by actual value to the power of 2. When this cost function is minimum we say that we have attained the point of least error and our model can be used as a benchmark model. In the field of statistics, there is a lot of tuning and tweaking that is done to attain the point of least error.