VesselAPI found requests that returned nothing used 71% of their database time. The miss is the expensive path, and dashboards averaged by raw path cannot show it.

Seventy-One Percent of the Database Time Went to Answers That Did Not Exist

TL;DR: The most expensive queries in your production environment are often the ones that return zero results, as a "miss" forces the database to exhaust the entire search space while a "hit" can stop early. VesselAPI recently discovered that requests returning "not found" consumed 71% of their database time on a critical endpoint because common observability defaults, such as averaging latency by raw path, structurally hide these systemic failures.

Twenty-eight milliseconds, or two and a half seconds, for the same query

On September 9, 2026, the engineering team at VesselAPI discovered a startling asymmetry in their production database performance. According to a detailed report from VesselAPI, their API could return a vessel’s most recent port event in just 28 ms when the data existed. However, if the vessel had no recorded events, the exact same query took between 2.0 and 2.5 seconds.

The gap was even more pronounced on their ETA lookups. When an IMO (International Maritime Organization) number could not be mapped to an MMSI (Maritime Mobile Service Identity), the request took 12 seconds on a quiet database and upwards of 30 seconds under load, often ending only when the statement timeout killed the process. The most significant finding was the aggregate impact: over a single night, requests that ended in a "not found" result consumed 71% of all database time spent on the ETA endpoint.

Why a miss costs more than a hit, in every store you run

This is not a quirk of a specific database; it is a fundamental property of search. In almost every data store, a "hit" has the luxury of being $O(stop\ early)$. If you are looking for the latest record with a LIMIT 1 and an ORDER BY, the database can stop the moment it satisfies the predicate.

A "miss," however, is $O(everything)$. To return a zero-result set with absolute certainty, the database must prove the absence of the data. This requires exhausting the entire search space: visiting every partition, every chunk, and every index leaf that hasn't been explicitly pruned.

Key Takeaway: Whenever your fast path relies on "stopping early," your slow path will be "visiting everything"—and because the slow path returns no rows, it leaves almost no trace in your application logs.

This failure class manifests in several common ways:

Partition Mismatches: When a pruning predicate fails to filter, forcing a scan of all partitions.
Filter Failures: Bloom or min-max filters that are improperly configured and cannot reject a lookup.
Negative Caching Gaps: Systems that cache successful hits but leave the "not found" path to hit the disk every time.

The dashboard could not have shown them this

The reason this behavior persisted is that traditional observability is often structurally blind to it. VesselAPI’s latency dashboard averaged performance by raw URL path. This meant that a single outlier taking 40 seconds would sit at the top of a "slowest paths" panel, appearing as a transient blip or a "noisy neighbor" problem.

It was only when they regrouped their metrics by route template that the systemic nature of the problem emerged: the endpoint wasn't just slow for a few people; it was slow in the median, every single time a request yielded an empty answer.

Any aggregation that mixes high-cardinality identifiers (like a specific UUID or vessel ID) into the group key converts a systematic failure into background noise. Furthermore, because these requests return a valid (though empty) response, they emit no errors and rarely trigger customer complaints. A user waiting four seconds for a "No results found" screen often assumes the network is slow, not that the database is doing a full table scan.

The index that was still defined and held nothing

The specific mechanism involved TimescaleDB compression. The port_events table had 764 one-hour chunks, 760 of which were compressed and segmented by mmsi. According to the VesselAPI investigation, a B-tree index on (vessel_imo, timestamp) still existed as a definition on those compressed chunks, but it held no rows.

Consequently, a lookup by imo had no usable index on 760 out of 764 chunks. A "hit" might find what it needed in the four uncompressed head chunks and stop. A "miss" was forced to run a sequential scan on every single compressed chunk. Interestingly, two specific data conditions disabled the built-in safety nets:

The NULL Bloom Filter: A batch where imo values were all NULL carries a NULL bloom filter. In this state, the filter passes every lookup, effectively silencing the optimization.
Subquery Pushdown: A COALESCE subquery in the WHERE clause was not pushed down to the bloom filter at all, resulting in a 6.05s execution time compared to 0.84s for a plain bound parameter.

This reinforces a critical lesson: a query optimization you have not verified against your actual data distribution is merely a hypothesis, not a guarantee.

Compression saved two megabytes and cost seven hundred and sixty indexes

The irony of the compression strategy was the return on investment. The hypertable_compression_stats showed the table occupied 1,689 MB before compression and 1,687 MB after. By segmenting by a high-cardinality key (mmsi) within a narrow one-hour window, most segments contained only a single row. There was nothing to compress.

For the sake of saving 2 MB of disk space, the system sacrificed the utility of every index (except the segment column) across 760 chunks. This wasn't a failure of the tool, but a failure of configuration drift—copying settings from a high-volume table where they worked to a low-volume table where they didn't.

The check compared four thousand one hundred and five responses and tested nothing

Before shipping their fix, the team ran a verification harness. They compared 4,105 API responses between the old and new read paths. The results were byte-identical. The migration was green-lit.

However, the check was worthless. A server from an earlier test was still holding the port; the new servers failed to bind with an "address already in use" error in a log file no one was watching. Every single one of those 4,105 requests had been served by the same old process. They were comparing a server to itself.

Even if the servers had started correctly, a second "dead check" was discovered later: the test script was requesting a page of 200 items against a hard-coded API limit of 50. Every request returned a 400 Bad Request. The harness was successfully verifying that two identical error messages are, indeed, identical.

What they changed, and the one change that generalises

VesselAPI moved to daily chunks without compression for port_events and introduced a plain "latest-row" table for ETAs. The results were dramatic: p99 latency dropped from seconds to 32 ms.

But the most important change was to their verification philosophy. Their checks now fail loudly if they cannot confirm independence. The diff script exits if its port is already taken. Probes now scan for a known real row and fail if the commit is broken.

At Operate, we see this pattern frequently. It is why our platform is designed for proactive investigation rather than reactive paging. Operate’s engine watches production for these "silent" database costs—like the expensive miss path—before they become 71% of your budget. Crucially, when Operate proposes a fix, the verification is performed by an agent running on a different model than the one that produced the diagnosis. Agreement is only evidence if you have independently established that you are checking two different things.

Four questions to take to your own system this week

Audit your dashboards: Group one latency panel by route template instead of raw path. Do the medians still look healthy?
Measure the "Empty" case: For your three highest-traffic lookups, explicitly measure the latency of zero-result queries versus successful hits.
Test your optimizations: For every index or compression policy you rely on, identify the data condition that would silently disable it. Does that condition exist in your production set?
Break your verification: For every test harness you trust, describe a failure mode where it could pass while testing nothing. If that happens, does the harness stay green?

Sources & further reading

According to VesselAPI, the "miss" path was the primary driver of endpoint exhaustion.
As noted by Ujwalendu on LinkedIn, a slow query is an observation, not a diagnosis.
Hussein Nasser argues that caching is often used to mask underlying architectural inefficiencies rather than solve them.