Learning a Hybrid Architecture for Sequence Regression and Annotation