Unlocking Pre-trained Image Backbones for Semantic Image Synthesis