No Space Left on Device Is Not a Disk Problem Eight Times Out of Ten: A Production Decision Procedure for ENOSPC

TL;DR: The Linux kernel returns ENOSPC (Errno 28) for at least eight structurally unrelated conditions, many of which have nothing to do with available disk bytes. To diagnose a failure when df looks healthy, you must systematically check for inode exhaustion, deleted-but-open file handles, and inotify watch limits.

Errno 28 is one number for eight different problems

In production environments, receiving a no space left on device error is rarely as simple as a full hard drive. The kernel returns ENOSPC (Error No Space) for a variety of resource exhaustion scenarios. If you have ever run df -h, seen 40% availability, and still watched a write operation fail, you are experiencing the structural ambiguity of Errno 28.

The problem with most troubleshooting guides is that they treat no space left on device as a single issue with one fix: delete files. In a complex stack involving Docker, Kubernetes, and Postgres, that advice is often irrelevant. You need a decision procedure that rules out branches of possibility until the root cause is isolated.

Check the path that failed, not the root filesystem

The most common wasted step in an incident is checking the wrong filesystem. A system might have 500GB free on /, but the process is trying to write to a 64MB tmpfs partition or a specific container volume that is at its limit.

The Rule: Run findmnt -T on the exact failing path provided in your logs.

# Identify which mount the failing path belongs to
findmnt -T /var/lib/data/my-app/logs

# Check both block and inode usage for THAT specific mount
df -hT /var/lib/data/my-app/logs
df -ih /var/lib/data/my-app/logs

What this rules out: The "Wrong Filesystem" error. If the usage is low on the specific mount returned by findmnt, you have ruled out simple block exhaustion and must look deeper.

The eight conditions, each with its tell and what it rules out

Follow this order. Each step is designed to eliminate a specific architectural cause of linux no space left on device errors.

1. Blocks exhausted

• The Tell: Use% at 100% in df -hT.
• Elimination: If this is clean (under 95%), the problem is not volume capacity. Move to inodes.

2. Inodes exhausted

• The Tell: IUse% at 100% while df -h shows plenty of bytes free.
• Check: df -ih
• The Consequence: You cannot create new files, even if they are empty. This is common in directories with millions of small session files, mail queues, or node_modules folders. Deleting large files will not help; you must delete many files.

3. Deleted but still open

• The Tell: du (disk usage) and df (disk free) disagree significantly.
• Check: lsof -nP +L1
• The Cause: When you delete a file that a process (like Postgres or Nginx) still has open, the space is not reclaimed until the process closes the descriptor.
• Elimination: If lsof shows no large (deleted) files, you don't have a "ghost file" problem.

4. Reserved blocks

• The Tell: df shows 5% free, but non-root users get no space left on device.
• Check: tune2fs -l /dev/sda1 | grep "Reserved block count"
• The Cause: By default, ext4 reserves 5% of blocks for the root user to prevent a total system lockup. If a service runs as a non-root user, it hits the wall early.

5. Quotas

• The Tell: Write fails for one user but works for another on the same mount.
• Check: quota -s
• Elimination: If no quotas are defined, this is ruled out.

6. Container and orchestrator limits

• The Tell: The host is fine, but the no space left on device docker error persists.
• Check: docker system df or kubectl describe node | grep DiskPressure.
• The Cause: In Kubernetes, ephemeral-storage limits on a pod or a full Docker overlay2 storage driver can trigger ENOSPC even if the underlying EBS volume is empty.

7. Filesystem-specific allocation (Btrfs/XFS/LVM)

• The Tell: df reports free space, but the filesystem cannot allocate new chunks.
• Check: btrfs filesystem usage /mnt or lvs.
• The Cause: Btrfs metadata chunks can be full while data chunks are empty. Thin-provisioned LVM pools can also run out of "real" backing space while the virtual volume looks empty.

8. Not about disk at all: inotify watch exhaustion

• The Tell: The failing syscall is inotify_add_watch returning -1 ENOSPC.
• Check: sysctl fs.inotify.max_user_watches
• The Cause: This is the ultimate "false" ENOSPC. The kernel uses the same error code when you exceed the maximum number of file watchers. It has zero to do with storage.

Key Takeaway: If df, df -i, and lsof +L1 all come back clean, stop looking at storage and check whether the failing syscall was inotify_add_watch.

The fast triage block

Paste this into your terminal to capture the state of a failing mount immediately:

TARGET_PATH="/path/to/failure"
echo "--- Mount Info ---"
findmnt -T $TARGET_PATH
echo "--- Block & Inode Usage ---"
df -hT $TARGET_PATH && df -ih $TARGET_PATH
echo "--- Deleted but Open Files ---"
lsof -nP +L1 | head -n 20
echo "--- Inotify Limits ---"
sysctl fs.inotify.max_user_watches

Where this shows up outside a plain Linux host

In modern production, ENOSPC often originates in the managed service layer:

• Postgres: Returns SQLSTATE 53100. This often happens during pg_wal (Write Ahead Log) spikes or when a bad query plan spills massive temporary files to disk. On AWS RDS or Aurora, this might trigger an storage auto-scaling event, but there is a lag where writes will fail.
• CI Runners: A github action no space left on device error is usually caused by Docker layer accumulation. Ephemeral runners often have small disks (14-20GB) that fill up after multiple docker build steps if the cache isn't pruned.
• Log Pipelines: Monitoring agents like Fluentd or Vector may silently drop data when their local buffers hit ENOSPC, meaning the evidence of the disk failure is the first thing to be deleted.

Blast radius: what ENOSPC breaks that you will not notice

The danger of a no space left on device error isn't just the failed write; it's the state inconsistency it leaves behind. A database that cannot write its WAL may shut down. A service might half-write a configuration file, leading to a CrashLoopBackOff on restart.

Because these investigations require checking multiple layers—from the kernel syscall to the Kubernetes event log to the Postgres internal state—they are difficult to perform manually during an active incident. The reason this procedure is worth writing down is that a human doing it at 3am has to hold eight branches in their head while the incident is still running. This is exactly the kind of investigation Operate automates. Operate watches production, finds the root cause with evidence across the stack (like identifying an inotify exhaustion while others are looking at df), and drafts the fix as a PR.

Common pitfalls

• Deleting logs without restarting: If you rm a log file but the app is still writing to it, the space isn't freed. Always truncate (> file.log) instead of deleting, or restart the service.
• Ignoring Inodes: Many monitoring setups only alert on byte percentage. You must alert on inode usage separately to catch high-file-count failures.
• Checking / instead of the mount: Always use the failing path as the argument to df.

Sources & further reading

• According to OneUptime, the disagreement between du and df is the primary indicator of deleted-but-open file handles.
• According to DataCamp, Docker's overlay2 driver is a frequent source of hidden disk exhaustion via dangling images and build cache.
• According to the Gentoo Wiki, ENOSPC can occur during cache updates even when df shows 50% availability due to inode limits.
• The errno 28 manual page specifies that this error is returned for both physical space and "exhaustion of other resources" like inotify watches.

Related: Too Many Open Files Is Almost Never a ulimit Problem