From Observability Data to Diagnosis: An Evolving Multi-agent System for Incident Management in Cloud Systems

Abstract--Incident management (IM) is central to the reliability of large-scale cloud systems. Y et manual IM, where on-call engineers examine metrics, logs, and traces is labor-intensive and error-prone in the face of massive and heterogeneous observ-ability data. Existing automated IM approaches often struggle to generalize across systems, provide limited interpretability, and incur high deployment costs, which hinders adoption in practice. In this paper, we present OpsAgent, a lightweight, self-evolving multi-agent system for IM that employs a training-free data processor to convert heterogeneous observability data into structured textual descriptions, along with a multi-agent collaboration framework that makes diagnostic inference transparent and auditable. T o support continual capability growth, OpsAgent also introduces a dual self-evolution mechanism that integrates internal model updates with external experience accumulation, thereby closing the deployment loop. Comprehensive experiments on the OPENRCA [1] benchmark demonstrate state-of-the-art performance and show that OpsAgent is generalizable, interpretable, cost-efficient, and self-evolving, making it a practically deployable and sustainable solution for long-term operation in real-world cloud systems. Cloud systems have become the de facto platform for modern software services, with wide deployments across industries such as IT, government, and finance [2]. However, incidents (e.g., service disruptions and outages) [2], [3] are inevitable due to the complexity of cloud systems, often resulting in catastrophic economic and operational consequences. Google Cloud triggered a global outage that lasted nearly eight hours, disrupting more than 80 GCP services and cascading into failures across e-commerce, finance, AI applications, entertainment platforms, and transportation systems worldwide. The economic impact of this incident was substantial, as it encompassed not only Google's direct losses but also widespread hidden costs borne by countless enterprises and end users affected by the disruption [4]. Traditionally, on-call engineers (OCEs) manually inspect metrics, logs, and traces to identify the root cause when incidents occur [2].