How to Instrument Systems and Filter MELT Data for AI SRE

Teams instrument everything, then sample aggressively to control the bill, and still cannot answer the 2:47 AM question: why is this incident happening right now? The instinct to collect more telemetry is understandable. Production complexity is exploding, and engineers want visibility. But more telemetry is not the same as better understanding. The problem is not how much data you collect. It is whether that data is structured well enough for something to reason over it. This piece explains how to instrument and filter MELT (metrics, events, logs, and traces) so an AI SRE can reason causally, instead of correlating data.

What It Means to Properly Instrument a System

Instrumentation is the code that makes an application observable by emitting telemetry. That telemetry includes MELT, the data your tools consume to answer questions about system behavior.

OpenTelemetry frames the goal as a quality bar, not a volume target. An application is properly instrumented when developers already have the information they need to troubleshoot an issue without adding more instrumentation. That bar matters because collecting signal and being able to explain an incident are two different things.

Telemetry gives you raw data. Observability is the ability to understand the internal state of a system by examining that data. You can have terabytes of metrics and logs and still not be able to answer why a failure happened. Instrumentation creates the foundation, but only if it captures the right signals in the right structure.

The distinction is critical for enterprise environments. A system might emit millions of log lines per minute and still lack the correlation IDs and span context needed to follow a single request across services. Volume is not the same as completeness. Completeness means capturing the relationships between signals, not just the signals themselves.

Ready to see causal reasoning in action? Book a Demo.

MELT Explained: Metrics, Events, Logs, and Traces

MELT stands for the four telemetry signal types: metrics, events, logs, and distributed traces. Each answers a different question about system behavior, and an investigation usually needs more than one. Understanding their differences helps you decide what to capture and how to filter.

Metrics

Metrics are pre-aggregated numerical measurements collected at regular intervals. They answer the question "is something wrong?" by showing symptoms and trends. Common examples include request rate, error rate, latency percentiles, CPU utilization, and memory usage.

Metrics are cheap to store and query because they summarize data (counts, rates, percentages) rather than preserve individual records. They are ideal for dashboards, alerts, and capacity planning. The tradeoff is that metrics drop individual detail. You can see that error rate spiked to 5%, but you cannot see which specific requests failed or why.

High-cardinality metrics (those with many unique label combinations) can quickly become expensive. A metric tagged by user ID, endpoint, region, and status code generates a new time series for every unique combination. Managing cardinality is one of the most common cost challenges in observability.

Events

Events are discrete records of change. They capture deploys, configuration changes, scaling actions, feature flag toggles, and other deliberate modifications to the environment. Events answer the question "what changed?"

During an incident, events provide context. If latency spiked right after a deploy, the event log shows exactly when that deploy occurred and what it contained. Without event data, you are left correlating timestamps manually and guessing which change might be responsible.

Events are underutilized in many observability setups. Teams capture metrics and logs but fail to surface deployment events, infrastructure scaling decisions, or configuration changes alongside their telemetry. This gap makes causal analysis harder because the "what changed" context is missing from the investigation.

Logs

Logs are semi-structured or unstructured text records emitted by applications and infrastructure. They provide the richest per-record evidence and answer the question "what specifically happened?"

Logs excel at capturing error messages, stack traces, request payloads, and detailed runtime state. They are often the final source of truth during root cause analysis because they contain the specific details that metrics and distributed traces summarize away.

However, logs also generate the highest volume of any signal type. High-cardinality log data (unique transaction IDs, user IDs, session tokens) can drive storage costs quickly. Many teams log too much in production (verbose debug output, routine success messages) and too little of what matters (structured error context, request identifiers).

Traces

Traces represent the path of a single request as it flows across service boundaries. They answer the question "where in the system did it break?"

A trace consists of spans, each recording work done by a single service. Spans link together to form a tree that shows which service called which, in what order, and how long each step took. Distributed traces are the most powerful signal for understanding failures in microservices architectures because they preserve the request path across network boundaries.

The challenge with distributed traces is that they require coordination. Every service in the request path must propagate context (a trace ID and span ID) so that spans can be stitched together. If one service drops the context, the trace breaks. This coordination overhead is one reason distributed traces are harder to implement than metrics or logs.

According to the Grafana 2025 Observability Survey, 95% of teams use metrics and 87% use logs, but only 57% use distributed traces. The signal most useful for understanding cross-service failures is the least adopted. For AI SRE, this gap is significant: distributed traces provide the causal chain that lets an AI follow a failure from symptom to root cause.

‍

Why More Telemetry is Not Better Telemetry

The common pattern looks like this: teams instrument everything and ship it all to their observability platform. Then the bill arrives, so they reach for aggressive sampling to control cost. Then the intermittent 1% failure disappears from monitoring entirely. Sampling to save money is like turning off smoke detectors to save on batteries.

According to the Grafana 2026 Observability Survey, the top three observability concerns are complexity and overhead (38%), signal-to-noise challenges (34%), and cost (31%). These three problems are connected. More telemetry creates more complexity and noise, which drives cost, which triggers sampling, which creates blind spots.

Dashboards correlate what moved together, but they do not prove cause. You can see that latency spiked at the same time as CPU usage, database connections, and three different deploys. Which one caused the failure? Correlation shows coincidence. Causation proves mechanism.

Key point: Filtering telemetry is too often used as a financial control rather than an engineering decision about relevance. The observability stack was built to surface signals to humans, not to let software reason causally over production. That structural gap is why more data does not equal better answers.

The volume-versus-usefulness tension is a symptom of a deeper problem. Most observability tools are designed to help humans find patterns. They assume a human will look at a dashboard, notice anomalies, correlate signals, and form a hypothesis. But AI SRE changes the equation. An AI agent can process vastly more data than a human, but it needs that data to be complete enough and structured enough to support causal reasoning.

Want AI that reasons causally, not just correlates? Book a Demo.

How to Filter MELT Data to Keep What is Useful

Filtering well means keeping the signal that supports an explanation and dropping the rest, deliberately, not just to cut the bill. The goal is causal usefulness: does this signal help explain why something failed?

Sample by Signal, Not by Budget

Different signal types require different sampling strategies. Logs can be sampled statistically because you still retain a representative view of behavior. A 10% sample of error logs still shows you error patterns.

Distributed traces are different. A trace must be kept whole or dropped whole. If you lose spans in the middle of a trace, the entire trace becomes useless for understanding the request path. You cannot reconstruct the causal chain when pieces are missing.

Head-based sampling (decide at the start of a request whether to keep the trace) is simpler to implement but misses failures. If you sample 1% of requests at the start, you will likely miss the one request that failed. Tail-based sampling (decide after the request completes) lets you keep all failed or slow requests while sampling successful ones. This approach preserves the traces that matter most for incident investigation.

Key point: Sample by the value of the signal, not by how much you are willing to spend. Keep 100% of errors and slow requests. Sample routine success cases.

Aggregate and Tier Retention

You do not need 15-second granularity on every metric forever. According to Google Cloud, on its Managed Service for Prometheus, changing the metric scraping period from 15 seconds to 60 seconds can yield approximately 75% cost savings without sacrificing cardinality. For high-traffic systems, Google notes that most customers can sample distributed traces at 1-in-1,000 (or even 1-in-10,000) and still have enough data for performance analysis.

Retention tiers also help. Keep high-resolution data for 7 days, roll up to hourly aggregates for 30 days, and archive daily aggregates for a year. This lets you investigate recent incidents at full fidelity while retaining enough history for trend analysis.

Structure for Correlation

Filtering cannot fix poorly structured data. Structured logs (JSON with consistent field names), consistent metadata across services, and correlation IDs that link logs to spans to metrics let you stitch signals into one story.

Without correlation IDs, you are left searching logs by timestamp and hoping you find the right records. With them, you can pull every signal related to a single request in seconds. A correlation ID is typically a unique identifier (like a UUID) generated at the start of a request and propagated through every service that handles that request.

Consistent metadata is equally important. If one service logs user IDs as "user_id" and another logs them as "userId" and a third logs them as "customer," joining those signals becomes difficult. Standardizing field names, tag formats, and labeling conventions across your stack makes filtering and correlation far more effective.

KeepAggregateSampleDropAll errors and exceptionsHigh-frequency metrics (15s → 60s)Successful request traces (1-in-100 or 1-in-1,000)Debug-level logs in productionAll slow requests (above P95 threshold)Verbose infrastructure metricsHealth check requestsDuplicate/redundant signalsDeployment and config change eventsApplication-level success logsStatic asset requestsHigh-cardinality labels with no diagnostic valueTraces containing errors or timeouts

Correlation vs. Causation: What an AI SRE Actually Needs From Your Data

Correlation shows what changed at the same time. Causation proves which change caused the failure. This is the distinction that matters most for AI SRE.

An AI reasoning over noisy or incomplete MELT will confidently surface a correlation, not a cause. If CPU spiked alongside memory pressure, a slow database query, and a code deploy, the AI can identify that all four moved together. But unless it can follow the failure path across service boundaries and map dependencies, it cannot determine which one caused the others.

Key point: This is the structural reason behind hallucinated root causes. The AI lacks the trace and event context to prove mechanism, so it falls back to statistical coincidence.

What Enterprise-Grade AI SRE Requires From Your MELT

The requirements above create a paradox for enterprise teams. You need complete telemetry to reason causally, but complete telemetry at enterprise scale is prohibitively expensive. The solution is architecture that captures everything and compresses intelligently, rather than forcing you to choose between coverage and cost.

Traversal was built to address exactly this paradox. Agentless Data Capture™ provides complete telemetry capture without the instrumentation burden of deploying agents or sidecars across every service. The Causal Indexer™ distills production data up to 1,000:1 without causal signal loss, making it possible to retain the full picture at scale.

The Production World Model™ is a continuously updated, causal model of your environment. It represents how services depend on each other, how changes propagate, and how failures cascade. The Knowledge Bank™ makes runbooks, documentation, and past incidents usable in real time, so the system learns from historical context. The Causal Search Engine™ reasons across 10+ hops (from applications to services to infrastructure to networking) to find root cause, not just correlation.

At American Express, Traversal ingests 250 billion+ logs per day and delivers 85%+ improvement in MTTR (mean time to resolve) and MTTD (mean time to detect). That result is only possible because the architecture captures complete MELT and reasons over it causally. Evidence-backed root cause analysis replaces correlation-based guessing.

The deployment model matters as well. Traversal is agentless, sidecarless, read-only, and runs entirely in the customer's environment. There is no extensive tuning required and no forward-deployed engineering army needed to get value. Customers typically see results in under two weeks. Across customers operating at petabyte scale, Traversal delivers 82%+ accurate root causes in under 5 minutes on average.

You should not have to choose between coverage and instrumentation burden. Completeness plus compression is what makes causal reasoning possible at enterprise scale.

What Traversal Is

Traversal is the first and only AI SRE validated within the Fortune 100. Our founders spent over a decade on causal machine learning and AI, and the company was founded by AI researchers and professors from MIT, Columbia, Berkeley, and Cornell. We run a team of superhuman AI SRE agents that find, fix, and prevent incidents in minutes by reasoning causally over production.

Reliability is not fundamentally an observability problem. It is a causality problem.

Ready to see what causal reasoning looks like at enterprise scale? Book a Demo.