Go back
Helping SREs detect, diagnose, and resolve incidents faster using Agentic AI
Just Visuals
Context
Modern telecom systems operate at massive scale, where millions of requests flow through interconnected services like APIs, databases, network infrastructure, and third-party systems. While this complexity is invisible during normal operation, failures can propagate quickly, turning minor issues into widespread outages.
At Cox Communications, system reliability is maintained through a combination of teams and tools. The Network Operations Center (NOC) monitors systems and detects anomalies, while Site Reliability Engineers (SREs) investigate and resolve deeper issues. Despite having access to extensive monitoring tools, the process of handling incidents remained fragmented. Engineers had to navigate multiple systems logs, metrics dashboards, infrastructure tools.
As part of a collaboration between COX and Zemoso, we explored how Agentic AI could transform this workflow. This project explored how Agentic AI could be introduced into this ecosystem not as a passive assistant, but as an active collaborator capable of investigating, reasoning, and supporting decision-making.
Stakeholders
Category
Design & Improvement
My Role
Senior Product Designer
Team
1 PD, 2 PM, 16+ Engineer
Timeline
12 Weeks
What did I do?
I led the design from 0 → 1, working closely with the PM to:
Understanding telecom and SRE workflows from first principles
Defining the problem space in an ambiguous environment
Shaping the product direction and system behavior
Design the end-to-end system
I was involved from the initial client discussions to final delivery, contributing not only to the design but also to product thinking and solution strategy.
Understanding the Problem
When a system issue occurs, engineers are not given answers they are given signals. These signals appear as:
Alerts from monitoring tools
Spikes in latency or error rates
Logs from services
Infrastructure anomalies
Each signal indicates that something is wrong, but none of them explain why.
Alerts signal that something is wrong — but not why it’s happening
The responsibility of the SRE is to interpret these signals and identify the underlying cause a process known as Root Cause Analysis (RCA). In practice, this process is:
Highly manual
Fragmented across tools
Dependent on experience and intuition
Engineers constantly switch between systems, correlate timelines, and build a mental model of the problem. Even for experienced engineers, this can take significant time especially in complex or subtle failures. This clearly tells…
The system wasn’t lacking data it was lacking structure, context, and orchestration.
The job map shows effort lies in understanding, not resolving issues
Engineers constantly switch between systems, correlate timelines, and build a mental model of the problem. Even for experienced engineers, this can take significant time especially in complex or subtle failures. This clearly tells…
" The system wasn’t lacking data it was lacking structure, context, and orchestration.
The Existing Workflow
To better understand this, I mapped the lifecycle of an incident.
An anomaly is first detected by the NOC, which logs the incident and performs initial triage. If the issue is complex, it is escalated to the SRE team. The SRE then investigates by analyzing logs, metrics, and dependencies across services. Once the root cause is identified, they implement a fix, verify stability, and document the incident through an RCA report.
NOC detects issues; SRE investigates
While this flow appears structured, it is fragmented in practice. Each stage is supported by different tools, and transitions between them require manual effort. Engineers spend as much time navigating systems as they do solving problems.
Defining the Core Problem
The challenge was not detecting issues it was understanding them quickly and confidently. Specifically:
Signals were disconnected across systems
Root cause analysis required manual correlation
Engineers faced high cognitive load
Learnings from past incidents were not effectively reused
This revealed a deeper opportunity, not to improve individual tools, but to design a system that supports decision-making under uncertainty.
This shifted the direction of the project. Instead of optimizing monitoring or debugging tools, the goal became:
Design a system that helps engineers move from signals → understanding → action as quickly as possible
This meant transitioning from a reactive workflow to a proactive, AI-assisted system where insights are surfaced early and decisions are supported intelligently.
Solution Approach
The problem wasn’t just about improving visibility or adding another tool it was about fundamentally rethinking how engineers move from signals to understanding to action. This required a shift in approach.
Instead of building a system that passively presents data, we explored how the system itself could actively participate in the investigation process helping engineers make sense of complexity as it unfolds.
This led to the concept of an Agentic AI system.
Unlike traditional tools that rely on engineers to manually correlate information, this system continuously works in the background — analyzing signals across systems, forming hypotheses, investigating patterns, and surfacing insights in a structured way. Rather than waiting for input, the system behaves more like a collaborative investigator — one that assists engineers in navigating ambiguity, rather than leaving them to piece everything together on their own.
To make this behavior tangible and scalable, the system was structured into four interconnected layers:
System structured into four layers for end-to-end workflow
Together, these layers transform a fragmented workflow into a continuous loop — connecting detection, diagnosis, and resolution into a single, coherent system.
Importantly, the goal was not to replace engineers.
It was to augment their ability to reason under pressure providing clarity early, supporting decisions in real time, and reducing the effort required to move from uncertainty to action.
Golden path defines the ideal flow from detection to resolution
How the System Works
When an SRE logs into the system, they are presented with incidents that are AI-prioritized based on SLA impact and severity, reducing the need to manually scan alerts.
Dashboard highlights prioritized incidents, helping engineers focus immediately
Opening an incident provides a structured understanding of the problem including a summary, urgency, and an AI-generated root cause hypothesis. This allows engineers to begin with context rather than starting from raw data.
Clear incident summary provides immediate context and urgency
The system aggregates evidence from multiple tools logs, metrics, and monitoring systems into a unified view. A service map visualizes dependencies across components, helping identify where failures originate.
AI suggests root cause hypotheses to guide investigation early
Service map visualizes dependencies to trace where failures originate
For deeper analysis, engineers can initiate an AI-assisted investigation. The AI actively queries systems, analyzes patterns, and builds correlations across data sources.
AI chat offers suggested queries to accelerate investigation flow
A key design decision was to make this process transparent. The system exposes how the AI operates — what queries it runs, what data it analyzes, and how it reaches conclusions. This transforms AI from a black box into a visible and auditable collaborator, which is critical for trust in high-stakes environments.
Engineers orchestrate actions with AI across chat and ticketing workflows
As the investigation progresses, the system refines its hypothesis and presents a likely root cause, along with a timeline showing how events evolved and connected. From there, the system transitions into action generating recommendations and drafting structured tickets that include root cause, evidence, and next steps. Engineers review and approve these actions before execution.
AI generates structured incident tickets
What previously required navigating multiple tools and assembling context manually is now handled within a single, unified workflow.
Designing for Real-world Constraints
During early iterations, a critical challenge emerged.
While the AI system could generate deep root cause analysis, it required 4–5 minutes for complex investigations. Although significantly faster than manual analysis, this introduced a new problem waiting time.
Experienced engineers do not simply wait for results. Given partial context, they begin forming hypotheses and exploring the system independently. The initial design assumed that AI would lead the investigation. In reality, this created a mismatch between system behavior and user behavior.
Exploring workflows and constraints to align design with real-world conditions
Solution: Human + AI Collaboration
To address this, I introduced a terminal-based workflow integrated into the system. While the AI investigates in the background, engineers can:
Run commands
Validate hypotheses
Explore the system independently
Meanwhile, the AI continued deeper analysis in the background — correlating logs, metrics, and timelines.
This created a parallel workflow, where:
AI handles large-scale correlation and pattern detection
Engineers apply intuition, experience, and fast validation
Instead of replacing human effort, the system now supports a more natural way of working.
Engineers act in terminal while AI continues investigation in parallel
Terminal expands for focused work while AI continues in background
" The goal was not to eliminate human intuition, but to amplify it allowing engineers to move faster without losing control.
Impact
The system demonstrated strong potential in transforming how incidents are handled shifting engineers from reactive investigation to faster, more structured decision-making.
From an operational standpoint, engineers were able to move from raw signals to actionable insights much faster, reducing the effort required to understand and resolve incidents. The introduction of AI-assisted investigation meant that patterns across logs, metrics, and infrastructure could be surfaced early allowing teams to focus on validation and decision-making rather than data gathering.
Earlier anomaly detection (~80–90%), helping teams act before escalation
~40–50% reduction in RCA time through early hypothesis generation and reduced manual correlation
Lower cognitive load by unifying fragmented tools into a single workflow
A few months after the initial release, as the product started gaining traction, my manager shared this feedback:
Copyright © 2025 Vaibhav Agrawal
All rights reserved.
Protected Access
This case study contains confidential product work and is available only upon request. Please enter the password provided to you.
Need access to this case study?
vaibhavagrawal2907@gmail.com