Explainable AI -- What is it? Why is it important?

"My dog accidentally knocked down the trash and found old cheesy pasta in it, and is now convinced that trash cans provide an endless supply of cheesy pasta, knocking it over every chance she gets." Sometimes, you would have seen your Machine Learning(ML) model do the same too. A notorious example is how a neural network learned to differentiate between dogs and wolves. It didn't truly learn to differentiate between dogs and wolves, instead it learnt that all the wolf pictures had snow in the background as it was their natural habitat as opposed to dogs with grass in the background. The model then differentiated the two animals by looking at whether the background was snow or grass.