Decoding Partial Differential Equations: Cross-Modal Adaptation of Decoder-only Models to PDEs