Prompting is not Enough: Exploring Knowledge Integration and Controllable Generation