RMoA: Optimizing Mixture-of-Agents through Diversity Maximization and Residual Compensation