Pod healthy, service healthy, endpoint healthy - and the request times out. Four real incidents where every dashboard was green, and a cross-layer protocol for the failure class between components.

Every Component Is Green and the System Is Down

TL;DR: When every component reports healthy but the system is failing, you are likely facing an "edge failure" rather than a "node failure." Modern monitoring excels at tracking individual service health but often misses the silent failures in network policies, implicit type conversions, and protocol timers that exist between services.

The most frustrating moment for any SRE is looking at a sea of green lights while the Slack channel is screaming that the application is offline. This is the dashboards green but system down phenomenon. It is a specific failure class where individual components are factually healthy, yet the system-level outcome is binary failure. We have been taught that if every microservice, database, and load balancer passes its health check, the system is functional. This is a dangerous myth.

The myth: green means healthy

Per-component health checks are existence proofs, not path proofs. A Kubernetes Pod reporting Ready only means its process is running and its readinessProbe returned 200 OK. It does not guarantee that the service is reachable, that the database is accepting its specific dialect of traffic, or that the network fabric between them hasn't been severed by a config change.

Monitoring traditionally watches nodes (the components). However, the most insidious failures in distributed systems live on the edges—the connections, policies, and translations that bind those nodes together. When the edges fail, the nodes stay green because they have no internal awareness of the vacuum surrounding them.

Four incidents where every check passed (July 2026)

In the last month alone, four distinct high-profile incidents reminded the community that silent failures in production are the new baseline.

The Default-Deny NetworkPolicy: In an incident described by SRE practitioner Mike, a Kubernetes service was completely unreachable. The Pods, Endpoints, and Services all showed green in the dashboard. According to [sreThoughts], the culprit was a forgotten NetworkPolicy that defaulted to deny, silently dropping traffic after an infrastructure update. The monitoring saw the service exist, but not the path's exhaustion.
The Stuck EKS Runners: GitHub Actions runners on EKS became unresponsive during a deployment. Dashboards reported the nodes as healthy and the runners as "Idle," yet no jobs were processed. The failure lived in the scaling logic—an edge between the runner manager and the EKS API—where the "all-clear" signal masked a deadlock in job assignment.
The VARCHAR to NVARCHAR Conversion: A database performance degradation occurred where query logs showed no errors, and CPU/Memory were stable. According to [dilshanw], the issue was an implicit data type conversion during a schema change. The database stayed green, but the logic "edges" between the application and the storage layer became 10x slower, effectively taking the system down under load.
The 9-Minute SIP Timer: A Voice AI company faced a bizarre issue where every call died at exactly nine minutes. According to [KK Upadhayay on LinkedIn], all VoIP infrastructure reported 100% uptime. The failure was a SIP session timer mismatch—a protocol edge case that no standard health check was designed to probe.

The pattern: nodes vs edges

Health lives in components; failure lives in paths, policies, conversions, and timers between them. When you search for why dashboards are green but the system is down, you are usually looking at a "Silent Crash." According to [Bervice on Medium], these crashes occur without leaving a trace because the monitoring system's expectations do not match the system's actual interaction patterns.

Key Takeaway: Most modern outages are not caused by component failure, but by "edge failures" where the interaction between two healthy components is broken by policy, protocol, or configuration.

A cross-layer evidence protocol

When your dashboards lie to you, stop looking at them. Instead, follow a cross-layer evidence protocol to find the "blind spot."

Trace the request path end-to-end first. Do not trust a service's internal health check. Use curl or a diagnostic sidecar to jump into the network namespace of the caller and attempt to reach the destination.
Diff declared vs. effective configuration. Check the policies that govern the connection. Is there a NetworkPolicy, an IAM role, or a Security Group that was applied "at the edge" by an automated CI/CD process?
Test the path, not the port. A port might be open (SYN/ACK), but the protocol might be failing (TLS handshake, SIP timer, etc.). Use protocol-specific probes.
Suspect the layer you don't own. If your app and DB are green, the problem is likely the VPC, the Service Mesh, or the Cloud Provider's routing table.

Why 'add more monitoring' is the wrong first answer

The knee-jerk reaction to a false all-clear in monitoring is to add more dashboards. However, as [CubeAPM] notes, Kubernetes has inherent blind spots that standard metrics won't fix. You cannot pre-instrument every possible edge failure because edges are often ephemeral or emergent properties of the system.

Instead of more data, humans need better correlation. Adding a 101st dashboard just increases the cognitive load during an incident. The goal is to see how the existing layers (Logs, Metrics, Config) contradict one another.

Checklist for on-call runbooks

Validate the Symptom: Is the "down" status coming from real users or a synthetic probe?
Identify the Gap: Which two "green" components are trying to talk to each other?
Check the "Invisible" Layer: Review recent changes to NetworkPolicies, ConfigMaps, or IAM permissions.
Test with Payload: Send a real request through the path, not just a ping.

Why this matters

With the rise of "Infrastructure as Code," we are shifting failure from hardware to configuration. This makes monitoring blind spots more common. When the "code" is a YAML file governing a network, the failure doesn't show up as a 500 error; it shows up as a timeout that the dashboard interprets as "no traffic (and therefore no errors)."

Sources & further reading

According to [sreThoughts], K8s service reachability issues often hide behind successful internal probes.
According to [dilshanw], monitoring often fails to capture the "silent" cost of implicit data conversions.
According to [KK Upadhayay], protocol-level timers (like SIP) are a prime source of "everything is green" outages.
According to [ServiceNow], "dependency-driven false all-clears" are a major hurdle for modern ITSM.

Finding these "edge failures" is precisely where cross-layer correlation beats human tab-switching. Operate's Context agent pulls code, configuration, database state, and logs into a single view, which is how green-component/broken-path root causes are discovered without manual hunting.

If your dashboards are green but your users are unhappy, stop looking at the nodes. Start looking at the space between them.