What is Bootstrap Sampling in Machine Learning and Why is it Important?

The Bootstrap Sampling Method is a very simple concept and is a building block for some of the more advanced machine learning algorithms like AdaBoost and XGBoost. However, when I started my data science journey, I couldn't quite understand the point of it. So my goals are to explain what the bootstrap method is and why it's important to know! Technically speaking, the bootstrap sampling method is a resampling method that uses random sampling with replacement. Don't worry if that sounded confusing, let me explain it with a diagram: Suppose you have an initial sample with 3 observations.