How to Improve my ML Algorithm? Lessons from Andrew Ng's experience -- I

One of the challenges with building machine learning systems is that there are so many things you could try, so many things you could change. Including, for example, so many hyperparameters you could tune. The art of knowing what parameter to tune to get what effect, is called orthogonalisation. In supervised learning, one needs to perform well on the following four tasks and for each of them, there should be a set of knobs which can be tuned for that task to perform well. Suppose, your algo isn't doing well on training set, you want one knob, or maybe one specific set of knobs that you can use, to make sure you can tune your algorithm to make it fit well on the training set.