Bridging the Granularity Gap for Acoustic Modeling