PRISM: Privacy-preserving Inference System with Homomorphic Encryption and Modular Activation