ReadWriteOnce Does Not Mean One Pod: The Misconfig That Waits for a Node Drain

TL;DR: The ReadWriteOnce (RWO) access mode in Kubernetes restricts a volume to being mounted by a single node, not a single pod. This means two pods on the same node can simultaneously write to the same RWO volume, leading to silent data corruption or deployment deadlocks that only trigger during node drains or scaling events.

If you are running stateful workloads in Kubernetes, you likely believe that ReadWriteOnce (RWO) is a safety guardrail. The common assumption among DevOps and SRE teams is that RWO ensures exactly one pod can access a persistent volume (PV) at any given time.

However, this assumption is technically incorrect and operationally dangerous. The "Once" in ReadWriteOnce does not refer to the Pod; it refers to the Node. This subtle distinction between readwriteonce vs readwritemany creates a class of configuration debt that remains invisible during development and staging, only to manifest as a production outage during a routine node drain or cluster upgrade.

The experiment: two pods, one RWO volume, zero errors

To see this behavior in action, we can recreate an experiment popularized by engineering research into Kubernetes storage semantics. Imagine a standard deployment where you have a PersistentVolumeClaim (PVC) using ReadWriteOnce. You might expect that if you try to attach two different pods to this same PVC, the second pod will fail with a mounting error.

In a test cluster—which is often small, perhaps even a single-node Minikube or KIND instance—the following happens:

1. You define a PVC with accessModes: [ReadWriteOnce].
2. You define two separate Pods (Pod A and Pod B) that both reference that same PVC.
3. You apply the manifests.
4. Both Pods enter the Running state. Both Pods successfully mount the volume. Both Pods can write to the disk simultaneously.

There are no errors in the events log. No FailedMount warnings appear. According to Jimmy Ayoade, who documented this behavior, Kubernetes allows this because both pods were scheduled on the same worker node. Since the volume is attached to the node at the hypervisor or storage fabric level, the Linux kernel on that node sees a single block device and allows multiple processes (containers) to mount it.

What ReadWriteOnce actually scopes (node, not pod)

To understand why this happens, we have to look at the four primary access modes defined in the Kubernetes specification. The scoping is often misunderstood because the naming convention focuses on the "how many" (Once vs. Many) rather than the "what" (Node vs. Pod).

As seen in the table above, the standard ReadWriteOnce is a node-level lock. This is primarily because of how underlying cloud providers handle block storage. According to official Kubernetes documentation, block storage services like AWS EBS, Azure Disk, and Google Compute Engine PD are physically attached to a virtual machine (the Kubernetes Node). The storage provider has no concept of a "Pod"; it only knows that the "Disk" is attached to "Instance-X."

Key Takeaway: ReadWriteOnce is a node-level constraint that allows multiple pods to share a volume if they are co-scheduled on the same host, which frequently leads to architectural assumptions that fail during infrastructure shifts.

Why your test cluster lies to you: co-scheduling hides the constraint

The reason this misconfiguration is so prevalent is that our testing environments are designed to hide it. In a small developer cluster or a staging environment with limited nodes, the Kubernetes scheduler is highly likely to place "related" pods (those using the same PVC) on the same node to satisfy data locality or simply because there are no other nodes available.

When pods are co-scheduled, the ReadWriteOnce constraint is technically satisfied. The volume is on one node. The storage driver is happy. The developer sees "Green" across the dashboard.

This creates a false sense of security. Because the code works in staging, the team assumes the RWO access mode is providing pod-level exclusivity. The configuration is a ticking time bomb, waiting for the topology of the cluster to change in a way that the scheduler can no longer place both pods on the same machine.

The two production failure modes

When the workload moves to a production environment with high availability (HA) requirements, the "Node vs. Pod" distinction causes two types of catastrophic failures.

1. Concurrent writers and silent data corruption

If you have a legacy application or a database that is not designed for multi-writer access, and you accidentally run two replicas on the same node (because of the RWO loophole), both will write to the same filesystem.

Since traditional filesystems like Ext4 or XFS are not "cluster-aware," they do not have a distributed lock manager. One pod may overwrite the metadata of another, or two pods may attempt to write to the same blocks. The result is silent data corruption. You won't know the volume is corrupted until the application crashes or you try to perform a backup and realize the data is unrecoverable.

2. The node-move failure: drain, scale event, ContainerCreating purgatory

This is the most common "incident" caused by the RWO misunderstanding. It typically unfolds like this:

1. You have a Pod running on Node A, mounting an RWO volume.
2. You perform a kubectl drain node-a for maintenance.
3. The scheduler tries to move the Pod to Node B.
4. The cloud provider attempts to attach the volume to Node B.
5. The attachment fails because the volume is still technically "in use" or attached to Node A (which is still shutting down or hasn't fully released the volume).
6. Or, even worse: you have a Deployment with 2 replicas and a single RWO PVC. On a 10-node cluster, the scheduler puts Replica 1 on Node A and Replica 2 on Node B. Replica 1 starts fine. Replica 2 stays in ContainerCreating forever with a Multi-Attach error.

The failure signature: events and states to alert on

When this misconfiguration bites, it leaves a specific trail of evidence. If you see pods stuck in ContainerCreating or Pending for more than 5 minutes, you should immediate check the kubectl describe pod output for these specific Event patterns:

• FailedAttachVolume: "Multi-Attach error for volume... Volume is already exclusively attached to one node and can't be attached to another."
• FailedMount: "MountVolume.SetUp failed for volume... volume is already used by another pod." (This occurs when the CSI driver is more restrictive than the default behavior).

This is a "topology-dependent" failure. The manifests are valid YAML; the storage class is functional. The failure only exists in the relationship between the workload's replica count and the node's availability.

Choosing correctly: RWOP for exclusivity, RWX for sharing

To fix this, you must match your access mode to your actual intent.

If you truly need exactly one pod to access a volume for safety (e.g., a Prometheus TSDB or a Postgres data dir), you should use ReadWriteOncePod (RWOP). Introduced as a stable feature in Kubernetes 1.27, RWOP closes the "same-node" loophole. If you use RWOP, Kubernetes will prevent a second pod from starting even if it is on the same node as the first pod.

If you actually need multiple pods to share data (e.g., a shared asset folder for a web server), you must move away from block storage and use a real ReadWriteMany (RWX) backend. This usually involves:

• AWS: Using EFS (Elastic File System) instead of EBS.
• Azure: Using Azure Files instead of Azure Disk.
• On-prem: Using CephFS or a dedicated NFS server.

Auditing your cluster for this today

You don't need to wait for a node drain to find these problems. You can proactively audit your cluster for any PersistentVolumeClaims that are labeled as ReadWriteOnce but are being consumed by multiple pods or high-replica controllers.

Here is a simple kubectl recipe to identify potential RWO risks:

# Find all RWO PVCs that are currently mounted by more than one Pod
kubectl get pods -A -o json | jq -r '
  .items | 
  map({pod: .metadata.name, volumes: [.spec.volumes[].persistentVolumeClaim.claimName] | select(. != [null])}) | 
  group_by(.volumes[]) | 
  .[] | 
  select(length > 1) | 
  "Warning: Multiple pods \([.[].pod]) are using the same RWO volume: \(.[0].volumes[0])"'

Additionally, check your Deployments. If a Deployment has replicas: 2 (or greater) and uses an RWO volume without a podAntiAffinity rule to force them onto separate nodes, you are either courting data corruption (if they land on the same node) or a Multi-Attach error (if they land on different nodes).

Why this matters

This issue illustrates a broader truth about Kubernetes: Infrastructure configuration is only as good as the topology it is tested against.

A "working" system in a single-node staging environment is often an "unreliable" system in a multi-availability-zone production cluster. As teams move toward platform engineering and self-service infra, the gap between "it runs" and "it is resilient" grows wider. We rely on access modes to enforce our architectural constraints, but when those modes are scoped to the node rather than the workload (the Pod), the safety mechanism itself becomes a source of drift.

Sources & further reading

• According to Jimmy Ayoade, testing on local clusters frequently masks the node-level scoping of RWO because pods are co-scheduled by default.
• According to the official Kubernetes documentation, the access mode is defined by the storage provider's capabilities, not by Kubernetes software logic alone.
• The introduction of ReadWriteOncePod was a direct response to the community's need for a workload-scoped locking mechanism that block storage couldn't provide at the node level.

The bigger lesson: config assumptions that only production topology can falsify

Reliability in Kubernetes isn't just about avoiding "CrashLoopBackOff." It's about ensuring that the state of the cluster remains consistent when the underlying nodes are in flux. The ReadWriteOnce dilemma is a perfect example of a configuration that passes every "lint" test and unit test, yet fails spectacularly the moment a node needs a security patch.

These "shadow failures" are exactly why localized monitoring isn't enough. You need visibility that correlates the storage layer, the scheduler events, and the controller state. Most teams only discover they have an RWO/RWX mismatch when the incident is already underway and the volume is stuck in a "waiting to detach" loop during a high-stakes maintenance window.

This is where Operate changes the math. Because Operate runs as a self-hosted SRE platform, it continuously audits your infrastructure configuration against known failure patterns like the RWO-multi-pod collision. It doesn't just wait for an alert; it looks at your PVC definitions, compares them to your ReplicaSet counts, and identifies the "node-drain debt" before it becomes an outage. When a FailedAttachVolume event does occur, Operate investigates the root cause, finds the co-scheduled pods, and drafts the necessary podAntiAffinity or accessMode PR to fix the architecture permanently—all within your own environment, keeping your storage topology and audit logs private.

Stop letting your node drains be your only form of configuration auditing. Fix the RWO assumption before the next scale event does it for you.