A Contact-Driven Framework for Manipulating in the Blind