Crafting Large Language Models for Enhanced Interpretability