Enhancing User Experience in On-Device Machine Learning with Gated Compression Layers