How, When, and Why Should You Normalize / Standardize / Rescale Your Data?

Before diving into this topic, lets first start with some definitions. "Rescaling" a vector means to add or subtract a constant and then multiply or divide by a constant, as you would do to change the units of measurement of the data, for example, to convert a temperature from Celsius to Fahrenheit. "Normalizing" a vector most often means dividing by a norm of the vector. It also often refers to rescaling by the minimum and range of the vector, to make all the elements lie between 0 and 1 thus bringing all the values of numeric columns in the dataset to a common scale. "Standardizing" a vector most often means subtracting a measure of location and dividing by a measure of scale.