The Model Isn't the Bottleneck: Findings From Benchmarking on High-Severity Incidents in Enterprise
.jpg)
Every engineering leader evaluating an AI SRE has the same thought, usually in the first meeting: couldn't we build this ourselves?
It's a fair question. Frontier models are remarkable at debugging. Your observability vendors expose APIs. MCP servers turn those APIs into tools. Wire it together, point it at an incident, and you have an AI investigator. No vendor required.
So we tested exactly that. Not on synthetic tasks or tidy alerts, but on 25 real production incidents at severity 1 through 3. The tier that pages humans, opens a bridge, and pulls in cross-team responders, up to full customer-facing outages, at a large financial-services enterprise. All judged blind, head-to-head against Traversal.
Three key findings:
- ~21 months ahead: Traversal delivered the better RCA 83% of the time against the strongest direct-access setup, a 255 Elo gap roughly equivalent to 21 months of frontier model progress
- 2–3× faster: Traversal completed investigations in 9.5 minutes on average, versus 19.7 and 33.5 minutes for the top two direct-access baselines
- 3× more lift from better models: Over the last three months, Traversal gained 153 Elo, versus 41 from upgrading the frontier model with direct tool access
The DIY setup didn't lose by a model release. It lost by an era.
The Benchmark: Real Incidents, No Answer Key
Most AI benchmarks hand the agent a clean problem. Production doesn't.
Each agent in this benchmark got exactly what a human on-call gets at t=0: the incident timestamp, the original ticket description, and access to the customer's observability systems. No answer key. No pre-staged telemetry. No postmortems or chat history that give away the ending. Tools that could leak the answer were blocked, and change searches were restricted to what was knowable at the incident timestamp.
And the environment is the reason "just search the logs" isn't a strategy:
- 4,000+ services across cloud-native, service mesh, legacy data stores, identity, and credentialing infrastructure
- 500 TB+/day MELT data, 100K+ code repos across 10+ observability and coding systems
- 150+ people pulled into a major incident for 2–4 hours
In this environment, root causes sit several hops from the symptom. One unhealthy pod behind an instant-decision service produced 631 of 634 failures, invisible unless you compare it against its healthy peers. An invalid credential caused an identity system to revoke a service ID and cascade into dependent servicing functions. A disaster-recovery switch quietly added latency to a downstream data path; the median looked fine, and only the tail gave it away.
There's no runbook for any of that. Getting from "users can't log in" to the specific causal state is a 10-to-30-hop investigation across millions of tokens.
The Contenders
We compared three setups:
- Traversal
- Frontier models + access to systems via MCP
- Frontier models + access to systems via API
The access covered MELT, system of record, code, change/deployment management. Setups 2 and 3 are direct tool access: a frontier model (we ran Fable 5, Sol 5.6, and Opus 4.8) pointed straight at the observability stack. This is the floor of building it yourself. Any serious in-house system needs far more, but this is where every DIY journey starts.
To keep it sporting, the direct-access baselines got a 30-minute budget. Traversal got 12.
Quality was scored by blinded pairwise judgment, the same Elo system that ranks chess players, because two RCAs can be partially right in different ways and pass/fail can't capture that. A blind judge saw the incident, the raw ground-truth records, and two answers in randomized order, and picked the more accurate root cause and mitigation. Five votes per pair, order shuffled, majority wins. Ground truth came from the raw human record of each incident; system-generated analyses were excluded, so no agent was ever graded against another agent's summary.
Reading Elo is simple: 100 points higher means winning about 64% of matchups. 200 points higher, about 76%.
Finding #1: The Gap is an Era of Model Progress, Not a Model Release
Traversal reached 1696 Elo. The best direct-access baseline, Fable 5 + MCP, reached 1441.
That's a 255-point gap. In head-to-head terms: Traversal was judged the better RCA 83% of the time against the strongest DIY setup, and up to 90% against the weakest.
Here's what 255 Elo means in units you already track. On the Arena math leaderboard, the distance from Claude 3.5 Haiku to Fable 5 is roughly the same size: about 21 months of frontier model progress.
A frontier model wired to your observability tools isn't one release behind. On hard production RCA, it's roughly an era behind. And it doesn't catch up by waiting for the next model. More on that in finding #3.
What the gap looks like on a single incident
Take the unhealthy pod. Fourteen minutes of HTTP 500s and 75-second timeouts across an instant-decision service; one pod out of a large fleet producing nearly all the failures.
Traversal compared the suspect against its healthy peers, isolated the single bad pod, and recommended draining it so traffic moved to healthy replicas.
The direct-access baselines returned plausible, confident, wrong answers: a shared infrastructure event. A downstream provider issue. A mainframe database hang. A recent config deployment.
Every baseline found a reasonable neighborhood. Traversal found the house.
Finding #2: Better Answers, 2–3× Faster
Speed usually trades against quality. Here it didn't.
Against the top two direct-access baselines by Elo, Traversal finished in 9.5 minutes, versus 19.7 minutes for Fable 5 + MCP and 33.5 minutes for Sol 5.6 + raw API. The better answers arrived first, on a time budget less than half the size.
During an incident, those minutes aren't abstract. Every one of them extends the outage, keeps 150 people in the war room, and delays mitigation your customers are waiting on.
Finding #3: The Gap is Widening
This is the finding that should reframe the build-vs-buy math entirely.
Over the last three months, Traversal upgrades gained 153 Elo on this benchmark. Over the same period, swapping Opus 4.8 for Fable 5 in a direct-access setup gained 41 Elo.
The same underlying force, better models, delivers roughly 3× more lift inside Traversal. Because Traversal doesn't just consume model progress; it compounds it:
- Model pooling. Traversal routes each subtask to the strongest model available for it, across every frontier lab. When a lab ships a better model, it enters the pool and lifts every investigation, with no re-architecture on your side.
- Production World Model. A proprietary model of your environment, continuously optimized for how agents actually use it, so intelligence goes into causal reasoning instead of rediscovering topology every run.
- Causal Search Engine. An agentic harness that folds each frontier advance into faster hypothesis generation, sharper evidence selection, and verified answers: causal reasoning, not correlated guessing.
A stronger model handed the wrong hundred log lines is still reading the wrong hundred log lines.
Why Direct Tool Access Falls Short, and Why This is the Floor
Direct tool access sounds like a shortcut: give the model the same tools a human uses, let it investigate. In practice, a bare model has to solve everything at once, mid-incident: learn your topology live, decide which signals matter, generate competing hypotheses, retrieve evidence across 15 systems, compare suspects against healthy peers, dodge plausible-but-non-causal explanations, and verify before stopping.
Traversal separates and reinforces those steps. The Production World Model narrows the search space before the incident fires, the Casual Search Engine tests hypotheses against real dependency structure, and the model pool puts the right model on the right subtask. The result isn't more tool calls. It's a better investigation loop.
And remember: these baselines are the floor of DIY, not the ceiling. Closing a 255-Elo gap in-house means rebuilding that machinery yourself, while the gap grows 3× faster on the other side.
Full Results
Method notes. 25 real production incidents at a large financial-services enterprise. Ground truth from the raw human incident record; system-generated analyses excluded. Blinded pairwise judging, five votes per pair with answer order shuffled; strict majority wins, ties otherwise; Elo fit to all win/tie/loss outcomes, centered at 1500. Direct-access baselines ran with a 30-minute budget; Traversal with 12 minutes.
The Bottom Line
On hard production incidents in a real enterprise environment, direct tool access isn't a near substitute. It finished 255 Elo and an era of model progress behind, at 2–3× the latency, and it loses more ground with every model release.
If you're weighing build versus buy, the benchmark's answer is the same from every angle: Traversal sets a higher ceiling today, and turns tomorrow's model progress into a bigger gain than one can directly capture by swapping models alone.
See how Traversal's AI SRE diagnoses real incidents in your production environment. Book a demo.
