← Blog · Engineering Management · June 21, 2026 · 8 min read · By Operate Editorial

Your Observability Bill Is a Hedge Against Bad Investigations

Every cost guide says collect less telemetry. The real lever is why you store it: an investigation model that can only query pre-collected data. Fix that instead.

Your Observability Bill Is a Hedge Against Bad Investigations

Your Observability Bill Is a Hedge Against Bad Investigations

TL;DR: High observability costs are primarily a "hedge" against the inability to query live systems during an incident. By shifting from a model that requires pre-collecting all possible answers to one that queries infrastructure directly at incident time, teams can decouple investigation depth from telemetry storage volume.

The bill nobody can defend line-by-line

In engineering departments across the globe, a specific ritual occurs every twelve to twenty-four months: the observability renewal meeting. High-ranking engineering leaders sit across from procurement specialists, staring at a line item that has quietly ballooned to rival their primary compute bill. The conversation usually follows a predictable pattern. Finance asks why the cloud bill grew by 20% while the observability bill grew by 80%. Platform teams respond with vague justifications about "visibility," "velocity," and the existential risk of a "dark" production environment.

The reality is that almost no one can defend their observability costs line-by-line. We track custom metrics that no dashboard reads. We ingest terabytes of DEBUG logs that haven't been queried since the service was in staging. We pay for high-cardinality indexing on tags that are never used in a GROUP BY clause. When pushed, the defense isn't technical—it’s emotional. It’s an insurance policy. We pay for the data because we are afraid of the one 3:00 AM incident where we might need it and won’t have it.

This fear has turned observability into a tax on ignorance. Because we don't know what questions we will need to ask when the system breaks, we pay to store every possible answer in advance. It is a storage-heavy model for a logic-heavy problem, and as systems grow more complex, the economics of this model are beginning to fracture.

The 12-tactics industry: what supply-side optimization actually is

There is an entire industry dedicated to observability cost optimization. If you search for ways to lower your bill, you will find a standardized list of "supply-side" levers. According to [OpenObserve], these tactics include ingest filtering, tail-based sampling, deduplication, and tiered retention. These are the classic moves of a cost-conscious SRE:

  1. Drop DEBUG logs at the edge: Don't pay for what you don't need.
  2. Tail Sampling: Only keep the traces that look "interesting" (errors or high latency).
  3. Aggregation: Turn high-volume logs into low-volume metrics before they hit the wire.
  4. Retention Tiers: Move old data to S3 or "cold storage" where it's cheaper to store but slower to query.

According to [Mezmo] and other vendors, these tactics can lead to 30% to 70% savings. These are not bad ideas; they are necessary hygiene for any mature platform. But they are also a form of "starving the beast." They assume that the only way to save money is to collect less data. They never question why the data is being collected in the first place. This is supply-side thinking. It focuses on the pipe and the bucket, rather than the thirsty person standing at the end of it.

The assumption under all of it: investigation = querying yesterday's collected data

Every cost-reduction guide on the market shares a single, unspoken premise: Investigation requires telemetry that was already collected and stored.

Under this paradigm, the "observability" of a system is defined as the volume of telemetry it emits. If you want to know why a database is slow, you must have already configured the database to export its slow-query logs to a central provider. If you want to know why a pod is crashing, you must have already been shipping its stdout to a log aggregator.

This creates a linear relationship between the complexity of your system and the cost of your bill. As you add more microservices, more sidecars, and more abstractions, you must ship more telemetry to maintain the same level of "investigatability." You are essentially trying to build a perfect historical record of every state change in your system, just in case one of those states leads to a failure.

Key Takeaway: Most observability spend is an insurance premium paid to guarantee that yesterday's data will be available to answer tomorrow's unpredictable questions.

Your telemetry hedge: how much stored data is never read

If you view your observability costs as an insurance premium, the "claims" are the queries you actually run during an investigation. For many organizations, the ratio of data stored to data queried is staggering.

Consider a standard log management setup. A team might ingest 5 TB of logs per day. During a major incident, a developer might query 50 GB of those logs to find a specific error pattern. Over the course of a month, perhaps less than 1% of the total ingested data is ever actually touched by a human or an automated alert.

The other 99% is the "hedge." It is data that exists solely to provide comfort. We are paying the vendor for the possibility of a query, not the query itself. This is why reduce observability costs initiatives often fail to make a permanent dent: as soon as an incident occurs where the "dropped" 99% contained the root cause, the team panics and dials the ingestion back up to 100%.

Why the hedge is growing again: LLM observability and AI agent traces

Just as teams were getting a handle on traditional microservice telemetry, a new driver of spend emerged. According to [r/sre], many organizations are seeing a massive spike in observability costs driven by LLM-based features.

The rise of LLM observability cost is due to the conversational and non-deterministic nature of AI. Debugging a RAG (Retrieval-Augmented Generation) pipeline requires tracing not just the API call, but the retrieved chunks, the prompt templates, the token usage, and the model's intermediate reasoning.

Standard distributed tracing wasn't built for this. Organizations are now adding specialized "AI traces" on top of their existing APM. Because AI behavior is unpredictable, teams are even more hesitant to sample this data. They want every single interaction stored. According to [Datadog LLM observability cost] discussions in industry forums, these specialized traces can quickly become one of the most expensive lines on the bill because they are high-cardinality and high-volume by design. We are once again hedging against the unknown, but this time, the "unknown" is the output of a black-box model.

The demand side: what if the investigation could ask the system directly?

To truly reduce observability costs, we have to move beyond supply-side filtering and address the "demand side"—the way we investigate.

The radical alternative to store-everything hedging is "query-time investigation." This model acknowledges that the system itself is the best source of truth, and it is usually still there when we are investigating. Instead of pre-shipping every possible metric to a third-party vendor, an investigator (human or machine) can query the live infrastructure for the specific data needed to solve a specific problem.

Examples of high-value, low-cost investigation include:

By querying these sources directly, you shift the cost from "storage" to "compute." You only pay for the investigation you actually perform, rather than the hedge you hope you won't need.

What still deserves hot retention?

Moving to a demand-side model doesn't mean deleting your Datadog or Grafana account. There are specific types of telemetry that absolutely belong in high-availability, hot-retention storage. An honest cost model splits telemetry into three buckets:

  1. Detection (Keep): High-level SLIs (errors, latency, throughput). You need this to know something is broken. It is low volume and high value.
  2. Transient States (Keep): Data that disappears as soon as a pod restarts or a system recovers. If you don't capture this immediately, it’s gone forever.
  3. Investigation (Move to Demand-Side): The deep, granular data used to find why something broke. This is where the 90% of waste lives.

According to [Charity Majors], the economics of observability are being remade by columnar storage engines like ClickHouse, which allow for much cheaper "wide" telemetry. However, even with better storage, the most efficient way to investigate is to only fetch the data you need when you need it.

A decision framework: detection, investigation, or fear?

When evaluating your next renewal or architectural change, put every telemetry stream through this checklist:

The economics rerun: a new model for SRE

The "13th tactic" for observability cost optimization is not about better compression or more aggressive sampling. It is about changing the investigation model.

When you decouple detection from investigation, the economics of your platform change. You no longer have to fear the next scale-up of your LLM agents or the next 100 microservices being deployed. Your "detection bill" scales linearly with your service count, but your "investigation bill" only scales with the number of incidents you actually have.

This is where Operate changes the game. Unlike traditional observability tools that require you to ship every byte of data to their cloud, Operate investigates by querying your systems directly at incident time. It talks to your database catalogs, inspects your infrastructure state, reads logs in place, and analyzes your git history—all with read-only access within your own VPC. By shifting the work of investigation from your storage bill to our automated reasoning engine, Operate allows you to maintain deep "investigatability" without the crushing premium of the telemetry hedge.

Sources & further reading:

#observability#cloud costs#sre#platform engineering