SOFARI-R: High-Dimensional Manifold-Based Inference for Latent Responses