ORCA-Bench: How Ready Are Language Model Agents for Oncall?
Researchers from Cornell Tech, Traversal, and Columbia University built ORCA-bench — the first benchmark to test AI agents on real production oncall work. The verdict: even the best frontier models correctly identify root causes only 25% of the time, and hallucinate an implausible cause in up to 40% of incidents.
- RCA is the hardest job in oncall — AI agents must reason causally across live, noisy, distributed systems with no clear starting point and no time to spare.
- ORCA-bench is the most realistic AI SRE benchmark ever built — 50GB of live telemetry, 19 microservices, 1,079 expert-validated incidents, and the full evidence stack a real engineer would use.
- Existing AI benchmarks don't test real oncall — they use frozen codebases, precise inputs, and binary verdicts. Production RCA has none of those.
- The best frontier model gets RCA right only 25% of the time — and hallucinates a wrong root cause in up to 40% of incidents.
- The gap is bigger in production than the numbers show — ORCA-bench is easier than real oncall in every dimension. The real readiness gap is larger.
