No Space Left on Device Errors That Are Not a Full Disk

No space left on device is the text of ENOSPC, error number 28, which the Linux kernel returns when a write or an allocation cannot get the resource it needs. We treat it as a category rather than a diagnosis, because the resource that ran out is often not disk bytes at all.

Most guides answer the error with one instruction, delete files. On a modern stack with containers, Kubernetes, Postgres and CI runners, that advice is often wrong, and deleting the wrong files can make the incident worse.

In this post we cover why one error code hides many failures, how to start from the failing path, eight checks in order, where ENOSPC shows up beyond a plain host, what it breaks after the write fails, and when the procedure points the wrong way.

One Error Code Covers Many Different Failures

The Linux manual pages show the ambiguity directly. For write, ENOSPC means "the device containing the file referred to by fd has no room for the data", which is the meaning most people assume.

The same code appears elsewhere with a different meaning. The manual page for inotify_add_watch lists ENOSPC for "the user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource." That failure has nothing to do with storage.

Between those two sit inode exhaustion, space held by deleted files, reserved blocks, quotas and filesystem-specific allocation limits. We work through them in a fixed order so each check rules something out.

Start From the Path That Failed, Not the Root Filesystem

The most common wasted step we see is checking the wrong filesystem. The root volume may have hundreds of gigabytes free while the process is writing to a small tmpfs, a container volume or a separate data mount that is full.

We take the exact path from the error message and ask which mount it lives on, then check blocks and inodes for that mount only.

findmnt -T /var/lib/app/data
df -hT /var/lib/app/data
df -ih /var/lib/app/data

If both block and inode usage are low on the mount that findmnt returns, simple exhaustion is ruled out and the remaining checks matter.

How to Diagnose No Space Left on Device in Eight Checks

We run these in order, cheapest and most common first. Each one either finds the cause or removes a branch.

1. Blocks Are Exhausted

If df -hT shows the mount at or near full use, the disk really is full. We then find what grew, usually logs, temporary files or a runaway cache, before deleting anything, so the cause does not simply refill the volume.

2. Inodes Are Exhausted

If df -ih shows inode use at 100% while bytes are free, the filesystem has run out of file entries. New files cannot be created even when they are empty, which is typical of directories holding millions of small session, cache or queue files. Deleting a few large files does not help here, because the fix is removing many small ones.

3. Deleted Files Are Still Held Open

When du and df disagree sharply, a process is usually holding deleted files open, and their space is not freed until the descriptor closes. The lsof manual explains that the +L1 option "will select open files that have been unlinked", so lsof -nP +L1 lists them.

Restarting the holding process, or truncating the file instead of deleting it, releases the space.

4. Reserved Blocks Are Hiding the Last Few Percent

On ext4, part of the filesystem is reserved for the superuser. The mke2fs manual says the reserve lets root-owned daemons keep working after non-privileged processes are prevented from writing, and "the default percentage is 5%." A service running as an ordinary user hits ENOSPC while df still shows a little space.

5. A Quota Applies to This User

If writes fail for one user or project but succeed for another on the same mount, we check quotas with quota -s or the filesystem's own quota tools. No quota configured means this branch is closed.

6. Container or Node Storage Is Full

Containers write to layers and volumes that live on the node's disk, so a full node can fail writes in every container on it. We check the runtime with docker system df and the node with kubectl describe node, looking for the DiskPressure condition.

Per-container limits behave differently from a full disk. The Kubernetes documentation on local ephemeral storage says that if a container's writable layer and log usage exceeds its storage limit, the kubelet marks the Pod for eviction, so the symptom there is an evicted pod rather than an ENOSPC write error.

7. The Filesystem Cannot Allocate Despite Free Space

Some filesystems can report free space and still fail allocations. Btrfs keeps data and metadata in separate chunks, so metadata can fill while data space remains, and thin-provisioned LVM pools can run out of real backing space while the volumes on top look empty. We check with btrfs filesystem usage or lvs.

8. The Failing Call Was an Inotify Watch

If blocks, inodes and open deleted files are all clean, we look at which call failed. When it was inotify_add_watch, the limit reached is fs.inotify.max_user_watches, not storage, and file watchers in development servers, log shippers or IDEs are the usual consumers.

Where ENOSPC Shows Up Beyond a Plain Linux Host

In production the error often surfaces through another layer, worded differently. We keep a short map of where to look first.

Layer	What we usually see	First check
PostgreSQL	SQLSTATE 53100, disk full, often from WAL growth or large temporary files	The data and WAL mounts, and temporary file usage
CI runners	Builds failing after several container image builds	Image and build cache size on the runner
Kubernetes nodes	DiskPressure, evicted pods, image pulls failing	Node filesystem and image storage usage
Log pipelines	Agents dropping data when local buffers fill	Buffer directory mount and agent logs

The PostgreSQL error codes appendix lists 53100 as disk_full in the insufficient resources class. Large temporary files from exploratory queries are a common cause, and we explain how they get there in read-only production access as a capacity change.

Across all these layers the investigation means correlating a mount, a process and a time window from several sources at once. The practical next step for teams that do this often is to connect an AI agent read-only to your logs and database, so the evidence is collected before anyone starts deleting files.

What a Full Disk Breaks After the Write Fails

The failed write is rarely the whole damage. A database that cannot write its WAL may shut down, and a service that half-writes a configuration or state file can fail on every restart afterwards, which often surfaces as a CrashLoopBackOff.

Logging is the other casualty. When the log volume fills, the evidence of what filled it is often the first thing lost, so we capture the output of the checks above before cleaning up.

Three habits prevent repeat incidents, and they are the actions we would expect a root cause analysis of this error to end with.

• Alert on inode use as well as byte use, since many setups watch only bytes.
• Truncate active log files instead of deleting them, so the space is actually released.
• Capture the check outputs before cleaning up, so the cause survives the fix.

When the Procedure Points the Wrong Way

The order above suits local Linux filesystems, and we adjust it in a few cases.

Network filesystems can report space and quotas differently from the server's view, so a clean local df on an NFS mount does not settle anything. We check on the server side as well.

Transient fills are another trap. A large query or build can fill a volume, fail, and clean up its temporary files within minutes, so every check comes back clean by the time anyone looks. In those cases the evidence is in logs and metrics from the failure window, not in the current state of the disk.

Finally, the descriptor limits behind "too many open files" are a different resource with a similar feel, and we cover that error separately in our too many open files field guide.

Read the Failing Path Before Deleting Anything

We would make one rule standard for on-call engineers. Take the path from the error, find its mount, check blocks, inodes and open deleted files on that mount, and only then decide what to remove or which limit to raise.

That order turns no space left on device from a vague alarm into a specific resource with a specific fix. Finding that resource from evidence is the kind of investigation Operate runs, self-hosted and read-only, with every finding citing the evidence behind it.