ORCA-Bench: How Ready Are Language Model Agents for Oncall?
About This Publication
The Traversal AI Research team, in collaboration with Columbia University and Cornell Tech, recently introduced ORCA-bench, a new benchmark for evaluating AI agents on production root cause analysis. Unlike traditional coding benchmarks, ORCA-bench evaluates agents in production-fidelity on-call environments, reflecting how AI SREs investigate real production incidents.This page provides a summary of the paper. The full publication, benchmark, public dataset, leaderboard, and evaluation methodology are available at the bottom of this page.
Authors: Albert Gong, Kyuseong Choi, Abhineet Agarwal, Jason Schechner, Ryan Huang, Raj Agrawal, Anish Agarwal, and Raaz Dwivedi.
What Is ORCA-Bench and Why Does It Matter?
Root cause analysis (RCA) is one of the most demanding tasks in modern software operations. When something breaks in a production system, an on-call engineer must sift through thousands of metrics, logs, and traces, often starting from nothing more than a vague user complaint like "the site is down," and work backward through a web of interconnected services to find the single upstream condition that started the cascade. It requires deep system knowledge, pattern recognition under pressure, and the ability to reason causally across data that is noisy, incomplete, and constantly changing. Unlike writing code or fixing a bug in a static repository, production RCA has no failing test to point you in the right direction, no clear task specification, and no binary pass or fail verdict. The evidence is live, distributed, and ambiguous, and the clock is running the entire time. ORCA-bench is the first benchmark ever built to test whether AI agents can actually do this work at production scale, under the same conditions a real on-call engineer faces at 2am when a critical service goes down and users are already impacted.
How ORCA-Bench Works — The Benchmark Explained
ORCA-bench pairs a live, OpenTelemetry-instrumented microservice e-commerce system with 19 services written in 13 programming languages, generating over 50GB of real telemetry across six days, with 1,079 carefully designed incident tasks. Each task gives an AI agent the same information a real on-call engineer would have: a vague user-facing complaint, a detection time, access to live telemetry through Prometheus, Jaeger, and OpenSearch via Grafana, and full source code access via a terminal. The tasks systematically vary how specific the user report is, ranging from precise error messages like "the product page for the National Park Foundation Explorascope is returning HTTP 500" all the way down to "users are reporting site issues," which is the hardest and most realistic setting. They also vary how long ago the incident started, with time-to-detection windows ranging from 15 minutes to 24 hours, and how many overlapping faults are active at the same time, including isolated failures, cascading failures, and scenarios where six concurrent feature flags are all active simultaneously on what the researchers called a "FAFO Friday" chaos peak. Every ground-truth answer was curated and validated by expert SREs, not generated automatically, and a subset of 40 tasks was independently re-scored by human annotators to confirm the benchmark's reliability, achieving a Cohen's kappa of 0.90, which indicates near-perfect agreement between the human judges and the LLM judge. This level of methodological rigor makes ORCA-bench the most realistic and trustworthy AI SRE evaluation ever published.
The Gap Between AI Benchmarks and Real Production Oncall
When a production incident hits, an on-call engineer is handed a vague complaint like "checkout is broken" or "users can't pay," often hours after it started. They open dashboards, cross-reference logs, check recent deployments, search Slack for context from the last shift, and chase hypotheses across dozens of services while the business impact is compounding in real time. Most existing AI benchmarks evaluate agents on frozen codebases with precise task specifications and binary test verdicts, which looks superficially similar to RCA but differs in every dimension that actually matters. The input in real oncall is an ambiguous natural-language complaint, not a failing test. The evidence is a running distributed system, not a static repository. And the success criterion is a defensible causal diagnosis, not a green checkmark. Every prior SRE benchmark strips out at least one of the three artifacts that real on-call engineers rely on: a telemetry interface over metrics, logs, and traces; the raw telemetry generated by live services under real load; and the source code of those services. ORCA-bench is the first benchmark to expose all three simultaneously, under the same ambiguity and time pressure that defines real production oncall work. The gap it reveals between what AI agents can do today and what production reliability actually requires is significant, and it is almost certainly an underestimate of the true gap.
What ORCA-Bench Found: Frontier AI Agents Are Not Ready for Oncall
Built by researchers from Cornell Tech, Traversal, and Columbia University, ORCA-bench is the first benchmark to put frontier AI agents in true production-fidelity oncall conditions, with a live OpenTelemetry-instrumented microservice system, 50GB of real telemetry across six days, full source code access, and 1,079 incident tasks that vary report specificity, time-to-detection, and co-occurring fault scenarios exactly as they appear in production. Every ground-truth answer was curated and signed off by expert SREs. Across five frontier models including Claude Opus 4.7, Sonnet 4.6, GPT-5.5, GLM-5, and DeepSeek-V4-Pro, the best RCA accuracy on realistic Medium-difficulty tasks was just 25.3%, meaning that even the strongest model correctly identified all root causes in only one in four incidents. On Hard tasks, where agents only see "users are reporting site issues" and must reason over an average of 4.4 plausible root causes simultaneously, accuracy collapsed to 10%. The weakest model hallucinated an entirely implausible root cause in 40% of incident reports, which in a live production environment would mean paging the wrong team, triggering an unnecessary rollback, or missing the real problem entirely while the outage continues. When source code access was removed, RCA accuracy dropped by 9 to 16 percentage points across every model tested, and hallucination rates spiked significantly, confirming that agents depend on source code not just for context but for the causal reasoning that separates a correct diagnosis from a confident guess. Perhaps most tellingly, agents consistently locked onto the loudest symptom rather than the upstream origin, a pattern that mirrors the exact failure mode that makes multi-hop incidents so expensive and time-consuming for human engineers.
Why This Gap Is a Lower Bound — And What Comes Next
Built by researchers from Cornell Tech, Traversal, and Columbia University, ORCA-bench is the first benchmark to put frontier AI agents in true production-fidelity oncall conditions, with a live OpenTelemetry-instrumented microservice system, 50GB of real telemetry across six days, full source code access, and 1,079 incident tasks that vary report specificity, time-to-detection, and co-occurring fault scenarios exactly as they appear in production. Every ground-truth answer was curated and signed off by expert SREs. Across five frontier models including Claude Opus 4.7, Sonnet 4.6, GPT-5.5, GLM-5, and DeepSeek-V4-Pro, the best RCA accuracy on realistic Medium-difficulty tasks was just 25.3%, meaning that even the strongest model correctly identified all root causes in only one in four incidents. On Hard tasks, where agents only see "users are reporting site issues" and must reason over an average of 4.4 plausible root causes simultaneously, accuracy collapsed to 10%. The weakest model hallucinated an entirely implausible root cause in 40% of incident reports, which in a live production environment would mean paging the wrong team, triggering an unnecessary rollback, or missing the real problem entirely while the outage continues. When source code access was removed, RCA accuracy dropped by 9 to 16 percentage points across every model tested, and hallucination rates spiked significantly, confirming that agents depend on source code not just for context but for the causal reasoning that separates a correct diagnosis from a confident guess. Perhaps most tellingly, agents consistently locked onto the loudest symptom rather than the upstream origin, a pattern that mirrors the exact failure mode that makes multi-hop incidents so expensive and time-consuming for human engineers.
.png)
.png)
