Too Many Open Files: Read the Errno Before You Raise the Limit

The error message "too many open files" is a generic string printed by three unrelated Linux exhaustion conditions: EMFILE (per-process limit), ENFILE (system-wide limit), and inotify ceiling exhaustion. To fix it, you must first identify the specific error code; raising the ulimit is the correct fix for EMFILE but will fail to resolve incidents caused by system-wide table saturation or filesystem watcher limits.

In a production incident, the goal is to determine which ceiling you hit in under 60 seconds. Most guides suggest blanket configuration changes, but if you are dealing with a descriptor leak in code, raising the limit only delays the next crash.

The thirty-second triage

The string "too many open files" is ambiguous, but the errno returned by the kernel is not. Your first step is to identify the error code.

• Branch 1: EMFILE (errno 24). This means "Process reached its own limit." You are hitting a per-process RLIMIT_NOFILE.
• Branch 2: ENFILE (errno 23). This means "Too many open files in system." The kernel’s global file table is full.
• Branch 3: inotify (errno 24 from inotify_init). This is a "stealth" EMFILE. You have plenty of file descriptors available, but you’ve exhausted fs.inotify.max_user_instances.

If you cannot see the errno in your application logs, use strace on the failing process: strace -p <PID> -e trace=open,openat,socket,accept4,inotify_init

Branch one: EMFILE, errno 24, the per-process limit

When you see too many open files linux errors, this is the most common culprit. However, the limit that matters is not what ulimit -n says in your terminal; it is the effective limit of the running process.

According to the proc(5) man page, every process has its own limits defined at boot or exec time. A process started by systemd inherits limits from the unit file, not your shell profile.

Verify the effective limit:

grep "Max open files" /proc/<PID>/limits

If the "Soft Limit" is 1024 and your process has 1024 files open (check with ls /proc/<PID>/fd | wc -l), you have hit EMFILE.

Key Takeaway: The effective file descriptor limit for a process is often inherited from its parent (like systemd or a container runtime) and can differ significantly from the global shell ulimit setting.

Branch two: ENFILE, errno 23, the system-wide table

If the error code is 23, the too many open files in system error indicates the kernel itself cannot allocate more file structures. This is rare on modern kernels but common in heavily multi-tenant environments or legacy systems.

Check the system-wide usage:

cat /proc/sys/fs/file-nr
# Output: 1216  0  1000000

The first number is the current number of allocated file handles. The third number is the maximum (fs.file-max). If the first equals the third, you have a system-wide exhaustion. According to the Linux kernel documentation (fs.rst), fs.file-max now defaults to a very high value (often 10% of RAM) on modern kernels, making this branch less likely unless you are on a shared node.

Branch three: The inotify ceiling that is not a descriptor problem

You might see errno 24 while your process only has 50 files open. This usually happens when Go or Kubernetes tooling tries to create a filesystem watcher.

• inotify_init() returns EMFILE when fs.inotify.max_user_instances is hit.
• inotify_add_watch() returns ENOSPC ("No space left on device") when fs.inotify.max_user_watches is hit.

This is a per-user limit. One noisy container leaking watches can block every other process running under the same UID from opening new instances. Use find /proc/*/fd -lname "anon_inode:inotify" | cut -d/ -f3 | uniq -c to find the offender.

Leak or legitimate concurrency: The question the flowcharts skip

Once you know you hit a limit, you must decide: Do I raise the limit or fix the code? The absolute number of open files doesn't tell you the answer. The shape of the data does.

1. The Shape Test: Plot process_open_fds against request rate. If the count tracks the load and drops to a baseline when traffic stops, you have a concurrency problem—raise the limit. If the count steps up and never recedes, you have a resource leak.
2. The Composition Test: Run ls -l /proc/<PID>/fd. If you see thousands of anon_inode:[eventpoll] entries, your async runtime is leaking. If you see thousands of socket entries, your network code is failing to close connections.
3. The Socket State Test: Run ss -tanp -p <PID> | awk '{print $1}' | sort | uniq -c. A growing number of sockets in CLOSE_WAIT means the remote peer closed the connection, but your application did not—a classic bug.

When raising the limit is the right answer

As Brian Brazil notes at Robust Perception, file descriptors are not inherently scarce. For a high-performance reverse proxy (like Nginx) handling 20,000 concurrent users, each user might require two descriptors (one for the client, one for the upstream). In this case, setting LimitNOFILE=65536 is the correct operational move, not a workaround.

Per-environment quick reference

Common Pitfalls

• The Shell Trap: Running ulimit -n 65535 in your terminal only affects that shell and its future children. It does not affect already running services.
• The Container Override: Kubernetes pods do not respect /etc/security/limits.conf. They inherit limits from the Docker/containerd daemon.
• The Java JVM: Java will throw java.net.SocketException: Too many open files. Note that the JVM reads the limit at startup; you cannot change it without a restart.

FAQ

What does "too many open files" mean?

It means the operating system has blocked a process from opening a new file, socket, or pipe because a resource ceiling (per-process, system-wide, or inotify) has been reached.

Why do I still get the error after raising ulimit to 65535?

You likely changed the limit for your current user session, but the service is running under a different context (like systemd or a container) that hasn't received the new configuration.

What is the difference between error 23 and error 24?

Error 24 (EMFILE) is a per-process limit. Error 23 (ENFILE) is a system-wide exhaustion of the kernel's file table.

How do I check the real limit of a running process?

Check the kernel's process status file: cat /proc/<PID>/limits | grep "Max open files".

What does "failed to create fsnotify watcher" mean?

This usually means you've hit the fs.inotify.max_user_instances limit, which surfaces as a "too many open files" error even if your descriptor count is low.

Sources & further reading

According to the inotify(7) man page, reaching instance limits returns EMFILE. According to proc(5), individual process limits are tracked independently of the global file-max. Brian Brazil at Robust Perception argues that for modern services, file descriptors should be treated as abundant, provided monitoring is in place to catch leaks.

Investigating these errors manually is time-consuming because the evidence is scattered across /proc, systemd units, and container configs. Operate automates this by opening a case when these errors occur, immediately reading the effective limits of the process, and correlating descriptor growth with request rates. It identifies whether the root cause is a configuration mismatch or a code-level leak, providing the evidence needed to fix the issue without the manual strace and lsof overhead.