INCIDENT RCA
The hardest problem in production, solved first.
Traversal isolates root cause across services, dependencies, and changes, even when it's multiple hops from the symptom, in minutes.
>85%
MTTR reduction
>82%
Accurate remediation
<5 min
Multi-hop root cause
Traverses your entire production environment.
Not just the failing service; every signal that could hold the answer.
Checkout unable to place new orders
FeedbackCopy link
Checkout unable to place new orders
Gathering incident details from the Jira ticket, recent alerts, and runbooks for checkout write failures
orders-api, checkout-svc, and cart-svc are failing writes with
relation "orders.line_items" does not exist since 14:31 UTC LogsHypotheses to test: a recent deploy, connection pool exhaustion, the 14:12 credential rotation, or a database event
Testing whether a recent deploy introduced the regression
No deploys to orders-api or checkout-svc in the prior 12 hours, so a deploy is ruled out Deploys
Checking orders-db connection pool and credential activity
Pool at 34% of max with flat latency, and the 14:12 key rotation only touched the analytics cluster. Both ruled out Metric
Searching database failover and DDL events on orders-db
Using Production World Model™
HypothesisAutomated failover promoted orders-db-02 to primary at 14:30:52 UTC, 12 seconds before the first error. This is the leading path Logs
Comparing schemas on orders-db-01 and orders-db-02
Traversing in Deep Mode...
Ask Traversal something...Fast
AI-generated content may be inaccurate, sensitive, or executable. Review before downloading, running, or sharing.
Isolates the true cause in minutes.
No matter how many hops from the symptom.
Checkout unable to place new orders
FeedbackCopy link
Checkout unable to place new orders
Traversed in Deep Mode for 3 minutes
Checkout Outage Due to Missing
orders Schema on Promoted ReplicaSymptoms Confirmed
- Checkout success rate fell from 99.8% to 81% across all regions by 14:35 UTC Metric
- Write errors on orders-db jumped from a 0.02/s baseline to ~900/s starting 14:31 UTC Metric
- Customers could add to cart but every order submission failed Logs
Root Cause:
orders Schema Missing from the New PrimaryAn automated failover at 14:30:52 UTC promoted
orders-db-02 to primary. That replica was rebuilt last week from a snapshot taken before the orders schema existed, so every write after the failover failed:ERROR: relation "orders.line_items" does not exist (SQLSTATE 42P01)
The database host stayed healthy and PgBouncer kept routing traffic, which is why no infrastructure alert fired. Four hops from the checkout symptom to the cause. Logs
Ask Traversal something...Fast
AI-generated content may be inaccurate, sensitive, or executable. Review before downloading, running, or sharing.
Reconstructs the full timeline and blast radius.
Every event, every affected service, in order.
Checkout unable to place new orders
FeedbackCopy link
Customer-Facing Propagation
Within a second of the first write error,
checkout-svc began returning HTTP 500 on /v2/orders, surfacing the outage on the storefront. The full cascade, Postgres error 42P01 → orders-api failures → checkout 500s → abandoned carts, is captured in the request trace Trace.Hypotheses Investigated and Ruled Out
| Hypothesis | Result |
|---|---|
Recent deployment of orders-api or checkout-svc introduced the regression | No deployments to either service in the prior 12 hours |
| Connection pool exhaustion on orders-db | Pool at 34% of max; latency flat throughout |
| Credential rotation at 14:12 UTC affected the orders backend | Rotated keys belong to the analytics cluster, not orders |
Blast Radius
- All four serving regions affected: us-east-1, us-west-2, eu-west-1, ap-south-1
- Every service writing to orders-db failed:
orders-api,checkout-svc,cart-svc,fulfillment-worker,billing-events - Customer-facing impact: order submission blocked and HTTP 500s on
/v2/orders(first 500 at 14:31:04 UTC) - Sustained write errors (~900/s) and checkout success near 81% were still in progress at escalation Logs Metric Metric
Checkout Outage — orders Schema Missing on Promoted Replica
What happened?The
orders schema was missing from the newly promoted primary at 14:30 UTC, driving error 42P01 across every service that writes orders while customers could not check out…Ask Traversal something...Fast
AI-generated content may be inaccurate, sensitive, or executable. Review before downloading, running, or sharing.
Fixes it, or tells you exactly how.
Traversal proposes the fix, or applies it. You choose.
Full investigation report
Checkout Outage —
orders Schema Missing on Promoted ReplicaHigh confidenceWhat happened
Timeline
Request analysis
Mitigation strategy
Associated evidence
checkout-svc began returning HTTP 500 on /v2/orders at 14:31:05…Oct 6, 2026 2:31:05 PM
orders-api logged relation "orders.line_items" does not exist…Oct 6, 2026 2:31:04 PM
Page Size: 10 ⌄1 to 3 of 3Page 1 of 1
Mitigation Strategy
- Verify the
ordersschema is missing onorders-db-02and present onorders-db-01; capture the\dnoutput before any remediation. - Audit the 14:30 failover and recent snapshot restores on orders-db to confirm how
orders-db-02was rebuilt without the schema. - Point PgBouncer's
orderspool back toorders-db-01, or restore the schema toorders-db-02from the latest backup. - Confirm recovery by watching checkout success return above 99.5% and orders-db write errors return to their ~0.02/s baseline.
Associated evidence
Filter by keywordAll evidence (7)
Write errors on orders-db
Checkout success rate by region
Background task completePull request ready
PR #3141 · Point PgBouncer's orders pool back to orders-db-01
