What's in the box? Explaining the black-box model through an evaluation of its interpretable features