ORCA-Bench: How Ready Are Language Model Agents for Oncall?
About This Publication
The Traversal AI Research team, in collaboration with Columbia University and Cornell Tech, recently introduced ORCA-bench, a new public benchmark for evaluating AI agents on production root cause analysis (RCA). Unlike traditional coding benchmarks, ORCA-bench evaluates agents in production-fidelity on-call environments that reflect how AI SREs investigate real production incidents.
This page summarizes the key findings from the full 33-page paper. The complete methodology, benchmark, public dataset, leaderboard, and evaluation framework are available at the bottom of this page.
Authors: Albert Gong, Kyuseong Choi, Abhineet Agarwal, Jason Schechner, Ryan Huang, Raj Agrawal, Anish Agarwal, and Raaz Dwivedi.
Why Production RCA Needs Its Own Benchmark
Root cause analysis is one of the most demanding tasks in modern software operations. When a production incident occurs, an on-call engineer rarely starts with a clear diagnosis. Instead, they begin with an ambiguous report such as "users are reporting site issues" and work backwards through thousands of metrics, logs, traces, recent deployments, and source code to determine the single upstream condition that triggered the incident.
Unlike software engineering tasks, there is no failing unit test pointing toward the answer and no clearly defined specification describing what success looks like. The evidence is noisy, distributed, incomplete, and constantly changing. The challenge is determining which explanation is actually consistent with how the entire system behaves.
As AI systems become increasingly capable of writing software, evaluating whether they can operate software has become equally important. Public benchmarks such as SWE-bench accelerated progress in coding agents by giving the research community a common way to measure improvement. Production root cause analysis requires the same foundation.
Why Existing Benchmarks Fall Short
Although several benchmarks have been proposed for AI SRE and operations workflows, none capture the conditions that define real production on-call.
Most existing evaluations are built around static repositories, offline datasets, or highly-specified tasks. Agents receive precise instructions, operate over frozen environments, and are judged against deterministic answers. While useful for measuring certain capabilities, these settings remove much of what makes production troubleshooting difficult.
Real incidents look very different.
Engineers begin with incomplete user reports rather than precise task descriptions. They investigate live systems instead of static repositories. They move between metrics, logs, traces, dashboards, and source code while continuously forming and testing hypotheses. Their goal is to determine the causal chain that produced the incident, not just identify an error.
Prior benchmarks typically omit at least one of the three primary information sources used during production investigations:
- A live observability interface over metrics, logs, and traces.
- The telemetry generated by a running distributed system.
- The underlying source code.
Removing any of these artifacts fundamentally changes the task. As a result, existing benchmarks measure only portions of production root cause analysis rather than the complete on-call workflow.
How ORCA-bench Brings Production Fidelity to AI Evaluation
ORCA-bench was designed to evaluate AI agents under conditions that more closely resemble real production environments.
The benchmark is built around a live OpenTelemetry-instrumented e-commerce application consisting of 19 microservices written in 13 programming languages. Over six days of execution, the system generates more than 50GB of production telemetry while exposing agents to the same observability interfaces used by engineers, including Prometheus, Jaeger, OpenSearch, and Grafana.
The benchmark contains 1,079 production RCA tasks spanning a wide range of operational scenarios.
Rather than presenting identical incidents repeatedly, ORCA-bench systematically varies several dimensions that make production investigations difficult:
- Report specificity, ranging from precise service failures to vague reports such as "users are reporting site issues."
- Detection delay, from incidents identified within minutes to failures discovered many hours later.
- Concurrent failures, including isolated issues, cascading failures, and scenarios with multiple overlapping faults.
- Control tasks with no injected failure, testing whether agents correctly determine that no incident exists.
Every task provides access to both live telemetry and full source code, mirroring the information available to a human on-call engineer.
To ensure reliable evaluation, every ground-truth answer was curated and validated by expert SREs. The evaluation methodology was further validated through independent human annotation, achieving near-perfect agreement between human judges and the benchmark's automated scoring.
Together, these design choices make ORCA-bench one of the most production-faithful public evaluations developed for AI SRE.
What ORCA-bench Found
ORCA-bench evaluated five frontier models across 1,079 production RCA tasks. The results consistently show that today's strongest AI agents remain far from handling production on-call reliably.
1. Overall RCA accuracy remains low.
Even under the benchmark's medium-difficulty setting, the strongest model correctly identified every root cause in only 25.3% of incidents.
These results suggest that production RCA remains a substantially more difficult reasoning problem than many existing software engineering benchmarks.
2. Ambiguity dramatically increases difficulty.
The less information agents receive at the start of an investigation, the larger the space of plausible explanations becomes.
On the benchmark's hardest tasks, where agents began with only vague reports such as "users are reporting site issues," the best-performing model achieved just 10% accuracy.
As report specificity decreases, agents must independently determine which services deserve investigation, distinguish symptoms from causes, and eliminate competing hypotheses using telemetry and source code. Performance deteriorated sharply under these more realistic conditions.
3. Source code is essential for accurate diagnosis.
Removing source code access reduced RCA accuracy across every frontier model evaluated.
Performance dropped between 9 and 16 percentage points, while hallucination rates generally increased.
This finding suggests that observability data alone is often insufficient for accurate causal reasoning. Understanding implementation details remains critical for identifying why an incident occurred rather than simply describing what happened.
4. Models frequently identify symptoms instead of root causes.
Across many investigations, agents gravitated toward the most visible downstream symptom rather than tracing failures back to their upstream origin.
This failure mode is especially important for multi-hop incidents, where the service exhibiting failures is often several dependencies removed from the component that actually introduced the issue.
5. Hallucinations remain a significant challenge.
The weakest model proposed an implausible root cause in roughly 40% of incident investigations.
In practice, incorrect diagnoses can misdirect investigations, delay identification of the underlying issue, and reduce confidence in automated systems.
Why Real-World Performance Is Likely Even Lower
Despite its production fidelity, ORCA-bench intentionally simplifies several aspects of real-world on-call work to make evaluation tractable.
The benchmark uses a public codebase rather than proprietary production systems. Agents investigate one incident at a time instead of balancing multiple simultaneous operational priorities. They begin each task without accumulated organizational context, historical knowledge, or familiarity with the environment.
Real production systems are typically larger, evolve continuously, contain significantly more telemetry, and rely on undocumented operational knowledge that engineers accumulate over months or years.
Taken together, these factors mean the benchmark likely overestimates how today's frontier models would perform in enterprise production environments. If the best models struggle in an environment that is intentionally simpler than most real-world systems, their performance in production is likely to be even lower. The gap between today's AI agents and the level of reliability required for production root cause analysis is therefore probably larger than these benchmark scores alone suggest.
The benchmark also reinforces an important point: production reasoning is not simply a frontier-model problem. Closing this gap will require systems that can efficiently access production-scale telemetry, reason across complex service dependencies, and evaluate competing causal hypotheses. Those capabilities depend as much on purpose-built data infrastructure and reasoning systems as they do on advances in foundation models.
What ORCA-bench Means for AI SRE
ORCA-bench provides more than a new set of benchmark scores. It establishes a common evaluation standard for production root cause analysis.
A shared benchmark allows researchers to compare architectures, reasoning strategies, and models using the same production-fidelity tasks. It creates a reproducible foundation for measuring progress, identifying failure modes, and advancing the state of AI SRE in the same way that public coding benchmarks accelerated research on software engineering agents.
For Traversal, the benchmark is one component of a broader evaluation infrastructure.
The benchmark provides the tasks and ground truth. Combined with an automated evaluation harness, it enables continuous regression testing, architectural evaluation, model comparisons, and post-training experiments. Rather than relying on anecdotal improvements, every change to the system can be measured against a consistent production workload.
This tighter evaluation loop allows research improvements to translate directly into product improvements while providing the broader AI community with a rigorous, production-oriented benchmark for measuring progress.
.png)