EMFILE vs ENFILE vs Inotify for Too Many Open Files

EMFILE vs ENFILE is the first fork in any too many open files incident, and the inotify limits add a third branch that looks like a descriptor problem and is not. We think the fastest way through is to stop reading the message and read the errno instead.

The message is a generic string. The number behind it names the ceiling you hit, and each ceiling lives in a different place and has a different fix.

This post compares the three ceilings, shows how to read each limit on a live system, explains why inotify surprises people and says when raising a limit is the wrong move. For the leak-versus-load question and the full fix, we keep that in our field guide to file descriptor exhaustion.

One Sentence Comes From Three Different Ceilings

The three ceilings differ in scope. One belongs to a process, one belongs to the machine and one belongs to a user. A single table keeps them apart, and we would pin it next to the runbook.

Ceiling	Errno	Scope	Where to read it
Per-process descriptor limit	EMFILE, 24	One process	The Max open files line in the process limits file
System-wide file table	ENFILE, 23	The whole kernel	The file-nr and file-max sysctls
Inotify instances	EMFILE, 24	One user	The max_user_instances sysctl
Inotify watches	ENOSPC, 28	One user	The max_user_watches sysctl

The fourth row is the one that misleads. It prints a message about disk space, so it looks like a full filesystem when no disk is involved, and responders lose time checking df.

EMFILE vs ENFILE vs Inotify Is Settled by the Errno

We start every one of these incidents the same way. If the application logs do not show the errno, trace the failing process for a few seconds. The calls that matter are open, socket, accept and the inotify calls, and the return value names the ceiling directly.

strace -f -p PID -e trace=openat,socket,accept4,inotify_init1,inotify_add_watch

1. EMFILE From a Process Limit

The Linux manual page for open(2) describes EMFILE as "The per-process limit on the number of open file descriptors has been reached." The limit that counts is the effective limit of the running process, not what your terminal prints.

grep "Max open files" /proc/PID/limits
ls /proc/PID/fd | wc -l

If the soft limit is 1024 and the process holds 1024 descriptors, you have hit EMFILE. A service started by systemd inherits its limit from the unit file, so a command run in your own shell tells you nothing about it.

2. ENFILE From the System Table

The same page describes ENFILE as "The system-wide limit on the total number of open files has been reached." This is the rare branch on modern hosts and the more likely one on a crowded shared node.

cat /proc/sys/fs/file-nr

The kernel's sysctl documentation says the three values in file-nr denote the number of allocated file handles, the number of allocated but unused file handles, and the maximum number of file handles. When the first value reaches the third, the table is full for every process on the machine.

3. Inotify Instances and Watches

Inotify has two user-wide ceilings. Creating an instance can fail with EMFILE because, in the words of the inotify_init(2) manual, "The user limit on the total number of inotify instances has been reached." Adding a watch can fail with ENOSPC because, per inotify_add_watch(2), "The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource."

sysctl fs.inotify.max_user_instances fs.inotify.max_user_watches
lsof -nP | grep inotify | awk '{print $1, $2}' | sort | uniq -c | sort -nr | head

The second command lists the processes holding the most inotify handles, which usually names the offender in one screen. Run it as a privileged user, because other users' handles are hidden from an ordinary account, and we would expect an empty result from a normal shell to mean nothing.

The table below condenses the three checks into one view, so a responder can run them in order without rereading the section.

Check	Command target	A full ceiling looks like
Process limit	The limits file and fd directory of the process	Descriptor count equal to the soft limit
System table	The file-nr sysctl	Allocated handles equal to the maximum
Inotify	The inotify sysctls and a per-process handle count	One user holding instances up to the limit

Why Inotify Looks Like a Descriptor Problem and Is Not

We think the inotify case is the one that costs the most time, because the usual evidence says everything is fine. The process holds a few dozen descriptors, the system table is nearly empty and the soft limit is generous. Yet the process cannot start a file watcher.

The reason is scope. The inotify limits apply per user, so one noisy container or tool that leaks instances can block every other process running under the same user ID. The victim and the culprit are different processes, and process-level evidence points at the wrong one.

Anything that watches files can hit this, including development servers, log shippers and configuration reloaders. When a tool reports that it could not create a watcher, we treat it as an inotify question first and a descriptor question second.

Raising the Wrong Limit Changes Nothing

Most advice stops at raising a limit, and the right limit depends on the errno. The table below pairs each symptom with the fix that works and the common fix that does not.

Errno	Fix that works	Fix that does nothing
EMFILE from a process	Raise the limit in the service definition, such as LimitNOFILE in a systemd unit	Running ulimit in an interactive shell
ENFILE	Find which process holds the most handles, then raise file-max if load is legitimate	Raising the per-process limit further
Inotify instances	Raise max_user_instances or stop the leaking watcher	Raising the per-process limit
Inotify watches	Raise max_user_watches or watch fewer paths	Freeing disk space

For containers, the same idea applies at a different layer. Docker accepts a nofile value through its ulimit option when a container is started, and a web server such as nginx has its own worker setting for the same purpose. In both cases the number must reach the process that is failing.

Go programs deserve a note. The Go 1.19 release notes say Go programs that import the os package now automatically raise the open file limit to the hard limit on Unix systems. A Go service that still hits EMFILE is therefore working against its hard limit, and that is where we would look.

When Raising the Limit Is Not the Answer

A raised limit is the right answer only when the count follows legitimate load and falls back when traffic does. If the number climbs and never recedes, the process is leaking descriptors, and a bigger ceiling only postpones the next crash by a few hours.

We would not try to settle that question from the errno alone. The errno names the ceiling, while the shape of the descriptor count over time names the cause. Our guide to the leak test and the composition of the descriptor table covers it, and we link it above for that reason.

Evidence for these incidents is scattered across process files, service units and container configuration. If you want an agent to gather it, we would start with one log source and one repository and judge it on an incident you already understand, with a person keeping the decision on the fix.

A Sixty Second Triage Order

This is the order we would follow when the page says too many open files and nobody knows which ceiling it is.

1. Capture the errno from the logs, or from a short trace of the failing process.
2. If it is 24, read the effective limit and the live descriptor count of that process.
3. If it is 23, read the system file table and find the process holding the most handles.
4. If the process looks healthy but cannot start a watcher, read the inotify limits and list handles by process.
5. Decide between raising the limit and fixing a leak only after you know which ceiling you are on.

Our notes on the other ENOSPC cases, where the disk is not full cover the sibling confusion for the inotify watch limit. Error strings with several unrelated causes are common, and our guide to attributing a connection reset to the component that sent it uses the same read-the-detail-first habit.

Name the Ceiling Before You Raise Anything

EMFILE vs ENFILE is a question about scope, and the inotify limits add a user-wide scope that no process-level check will reveal. Read the errno, read the matching limit and change only that one.

We would put the table in the runbook this week, so the next responder spends the first minute on the right ceiling. When you want agents to read the effective limits and descriptor growth while your engineers decide, Operate runs dedicated AI agents that investigate production problems with cited evidence and propose fixes for your team to review.