How to Reduce Engineering Toil During Incidents
The standard advice for engineering toil is to automate the repetitive work away. That advice is right about restarts, scaling actions, and deploy scripts. It is wrong about incidents.
Reducing engineering toil during incidents is a different problem, because most of that toil is investigative, not repetitive. During an incident, engineers causally connect symptoms across services, reconstruct the dependency chain, and decide which fix is safe. None of that yields to a runbook, and runbooks can't hold up for novel incidents.
The reason is structural, and it is why diagnosis consumes most incident time: the incident clock is spent on diagnosis, not the fix. So the toil concentrates exactly where scripts cannot reach. Here is the key distinction: there is toil you can automate away, and there is toil you can only reason away.
The good news for reliability leaders is that the investigative toil is removable, not permanent. See how Traversal reduces engineering toil.
What incident toil actually is
Toil is defined as manual, repetitive work, tied to running a service, that scales as the service grows and produces no lasting value. The classic examples are scriptable: restarts, capacity bumps, certificate rotations.
The concept is not new. Google's SRE Book set a goal of keeping toil below 50% of each SRE's time. It warned that unchecked toil expands to consume an engineer's whole week and drives burnout. During an incident, however, toil does not disappear; it changes form.
Engineers triage alert storms, correlate logs and metrics across tools, page and re-page teams, and rebuild the postmortem from memory. Every escalation handoff resets part of the investigation.
The modern surface is larger still: maintaining runbook libraries and per-service alert thresholds that drift from reality is itself toil. Teams describe the result as death by a thousand cuts.
Reliability leaders can measure incident toil before it compounds, using a few concrete signals:
- Investigate vs. fix ratio: the share of on-call time spent diagnosing rather than applying the fix.
- Repeat investigations: how often the same failure class gets re-diagnosed from scratch.
- Interrupt hours per incident: engineer-hours pulled into correlation and handoffs during each incident.
That is the part the classic playbook misses. Scriptable toil is bounded and predictable, so automation retires it cleanly.
Investigative toil is open-ended, because every serious incident is a little different from the last one. You cannot pre-write the reasoning for a failure you have not seen yet. This is the toil that survives every automation project and quietly consumes your most experienced engineers.
Why this is hard (and getting harder)
Incident time is expensive, which raises the stakes on every hour of toil. In ITIC's 2024 downtime survey, the average single hour of downtime exceeded $300,000 for over 90% of mid-size and large enterprises. Those were self-reported estimates that excluded litigation and regulatory penalties.
The structural obstacles compound that cost:
- Failures travel. Distributed systems fail across service boundaries, so the component that pages is rarely the one that broke.
- Tools show correlation, not causation. Observability platforms were built to display signals, so a human still supplies the causal reasoning by hand under pressure.
- Noise buries signal. Alerting was designed to fire on thresholds, not to explain them, so alert fatigue grows with every new service.
- Knowledge concentrates. The context that cracks a hard incident lives in a few senior heads, and it leaves when they do.
- AI is raising the volume. DORA's 2024 State of DevOps research found AI adoption boosts productivity yet degrades delivery stability and throughput.
More change reaches production, from more contributors, against more dependencies. The investigative surface expands with it.
Notice what these obstacles have in common. None of them is a bug in a specific tool. Observability, alerting, and incident management each do exactly what they were designed to do.
The gap sits between those tools, in the causal work that no product on the incident stack was built to perform. That is why bolting on one more integration rarely moves the toil number.
Why 'more automation' doesn't fix incident toil
The obvious response is to automate more. Write more runbooks, add more scripts, tune more alerts. That reduces some scriptable toil, it adds maintenance toil, and it never touches the investigation.
A useful way to judge any toil-reduction agent is by how much autonomy it actually has. Traversal describes three phases of agent autonomy:
- Invoked. Copilots and chatbots act only when a human asks, so the human still drives the investigation.
- Background. Cron jobs, rules, and alert routing that run on a fixed schedule or trigger, useful but blind to novel failures.
- Proactive. Agents that decide for themselves when to engage, then run the investigation without being summoned.
Most tooling marketed as toil reduction stops at Invoked or Background. A human still owns the diagnosis, so the investigative toil stays put. Durable reduction of that toil requires Phase 3, where the agent engages on its own and does the reasoning before anyone is paged.
The phases are cumulative, not competing. Background automation still handles the routine actions it is good at, and a human can still invoke help on demand.
What Phase 3 adds is the one thing the first two phases structurally cannot deliver. It notices an incident, decides to engage, and completes the investigation without waiting for a prompt. That is the difference between a tool that helps you do the toil and a system that does it for you.
The proof: hours given back, not just minutes shaved
Leaders do not buy minutes shaved off a chart. They buy back senior engineering capacity, and that is where the evidence lands.
A global crypto exchange reclaimed roughly 2,000 senior engineering hours per month after handing the investigation to Traversal. The same team projects a 40%+ reduction in mean time to recovery.
DigitalOcean's reliability team reclaimed thousands of engineering hours per year and saw fewer incidents overall. Those are hours returned to product work, not consumed by correlation.
The pattern repeats at scale. A Fortune 100 financial services company identifies true root cause in minutes with high accuracy. In each case, the win is the same: the investigative toil is removed, not reshuffled.
This is the number a board understands. Reclaimed senior-engineer hours convert directly into roadmap capacity, retention, and fewer war rooms.
A percentage point off a recovery-time chart is real, but it does not tell a reliability leader how many people got their week back. Hours given back does.
What reducing engineering toil during incidents actually requires
Reducing engineering toil during incidents durably takes more than another dashboard. It takes a system that reasons causally about production and externalizes the operational knowledge that otherwise lives in people's heads.
Traversal is built as one causal pipeline. Agentless Data Capture™ captures telemetry, code, and production data with no agents or sidecars.
The Causal Indexer™ distills raw production data by orders of magnitude, without sampling. That makes causal reasoning economical at petabyte scale.
The Production World Model™ is a living, auto-maintained map of how the environment behaves. The Knowledge Bank™ holds operational knowledge that is mostly auto-discovered after each investigation, with human input as last-mile refinement.
The Causal Search Engine™ returns one causally consistent root cause, with evidence and a remediation path, using multi-hop reasoning across the dependency graph.
The differentiator is the causal depth and the auto-discovered knowledge, which removes maintenance toil instead of adding it. Each investigation makes the next one cheaper, because the knowledge that used to live in one senior engineer's memory is captured automatically and reused.
Over time the system gets more accurate on your environment specifically, rather than staying generic. That compounding is what turns a one-time efficiency gain into a durable reduction in investigative toil.
See Traversal cut engineering toil during incidents in your own environment.
FAQ
Incident toil is mainly investigative rather than repetitive, so engineers must correlate symptoms across services, reconstruct dependency chains, and decide which fix is safe. This reasoning cannot be pre-written into runbooks like scriptable work such as restarts or capacity bumps.
They can track the investigate versus fix ratio, how often the same failure class is re-diagnosed from scratch, and interrupt hours per incident due to correlation and handoffs. These signals capture the investigative work that typical playbooks miss.
More automation can reduce scriptable toil, but it adds maintenance toil and does not address the investigation. Tooling that only supports invoked or background help still leaves diagnosis to a human, so investigative toil remains.
Durable reduction requires a system that reasons causally about production and externalizes operational knowledge that otherwise stays in senior engineers’ heads. The article describes an agent-led approach with a causal pipeline that proactively performs investigation and uses continually updated knowledge.
Traversal is described as having an Agentless Data Capture pipeline to ingest telemetry, code, and production data, a Causal Indexer to distill raw production data, and a Production World Model that maintains a living map of environment behavior. It also includes a Knowledge Bank for auto-discovered operational knowledge and a Causal Search Engine that returns causally consistent root cause with evidence and a remediation path.
