Too Many Open Files Is Almost Never a ulimit Problem: A Production Field Guide to File Descriptor Exhaustion

TL;DR: The "too many open files" error occurs when a process or system hits its allocated limit for file descriptors, which include sockets, pipes, and files. Fixing it requires distinguishing between a resource leak, a traffic-driven sizing issue, or a latency-induced spike before blindly raising limits that may not even apply to your specific service runtime.

In production environments, encountering too many open files is a rite of passage for on-call engineers. Most guides suggest a quick ulimit -n adjustment, but for a distributed system, this rarely addresses the root cause. This field guide moves beyond the snippet to help you diagnose why your descriptors are exhausted and why your configuration changes might be failing.

The error, precisely

The error usually manifests as errno 24 too many open files (EMFILE) or too many open files in system (ENFILE). While they look similar, they indicate different exhaustion points. EMFILE means a single process has hit its per-process limit. ENFILE means the entire operating system has exhausted its total allocation of descriptors—a much more severe state that can freeze the host.

A "file" in Linux is a broad abstraction. When you see too many open files linux errors, you aren't just looking at text files on a disk. File descriptors (FDs) represent:

• Network sockets (inbound and outbound)
• Pipes and FIFOs
• Event-monitoring instances like epoll and inotify
• Memory maps
• Deleted files that are still being held open by a process

The error message is identical whether you have a genuine leak of 1,000 handles or you simply sized a high-traffic gateway too small.

The three shapes of descriptor growth

To fix the issue, you must identify the "shape" of the exhaustion.

Key Takeaway: Before you touch a limit, sample the descriptor count over time against traffic: if it rises under flat traffic you have a leak, if it tracks traffic you have a sizing decision, if it spikes when a dependency slows you have a latency problem.

The limit you set is probably not the limit the process got

One of the most common frustrations is seeing too many open files ubuntu or CentOS errors persist after running ulimit -n. This happens because ulimit only affects the current shell session and its children.

• Authority: The only authoritative source of truth is cat /proc/PID/limits. Look for the "Max open files" line.
• Systemd: Modern Linux services ignore limits.conf. You must set LimitNOFILE=65536 within the [Service] section of your systemd unit file.
• Containers: A docker too many open files error usually stems from the container inheriting the daemon's default (often 1024). You must pass --ulimit nofile=65536:65536 or set it in the container engine's configuration.
• Kubernetes: In a kubernetes error too many open files scenario, remember that pods often inherit limits from the node's container runtime. Furthermore, certain limits like fs.inotify.max_user_instances are namespaced to the node, meaning one "leaky" pod can exhaust the watch limit for every other pod on that host.

Diagnose in five commands

If you are mid-incident, run these five commands to locate the pressure point:

1. Check the count: ls /proc/PID/fd | wc -l Sample this every 30 seconds. A positive slope under flat traffic confirms a leak.
2. Identify the type: lsof -p PID | awk '{print $5}' | sort | uniq -c This shows if your descriptors are REG (files), IPv4/v6 (sockets), or unix (local pipes).
3. Check socket states: ss -tan | grep PID A high volume of CLOSE_WAIT indicates an application bug where the code is not calling close(). TIME_WAIT usually indicates high connection churn.
4. Find "ghost" files: lsof -p PID | grep '(deleted)' This identifies files that were deleted from disk (like a rotated log) but are still held open, consuming space and descriptors.
5. Inspect the targets: ls -l /proc/PID/fd This provides the actual paths or socket inodes for the descriptors, showing exactly what the app is "holding."

The same ceiling at every layer

The too many open files error propagates differently depending on your stack.

• Nginx: You must balance worker_connections with worker_rlimit_nofile. If the latter is lower than the former, Nginx will fail to accept new connections under load.
• Postgres: High connection churn can trigger too many open files in the database before the application sees it. Setting max_files_per_process is critical here.
• Runtimes: In too many open files java or python too many open files scenarios, the culprit is almost always a failure to close an HTTP response body or a database cursor in an error-handling catch or except block.
• Sidecars: If you use a service mesh (Istio, Linkerd), every request consumes descriptors twice—once for the inbound proxy and once for the outbound.

Little's Law is the model you are missing

Many engineers treat descriptors as a static resource, but they are a function of concurrency. According to Little's Law, L = λW (where L is concurrency, λ is arrival rate, and W is duration).

If your service handles 1,000 requests per second at a 50ms latency, it uses ~50 descriptors. If a downstream database slows down and latency jumps to 10 seconds, that same 1,000 rps now requires 10,000 descriptors. This is why socket too many open files errors often signal a latency crisis, not a configuration error.

Common pitfalls

• Restarting too early: A restart "fixes" the problem by releasing descriptors, but it destroys the evidence in /proc/PID/fd. Capture your lsof and ss output before you kill the process.
• Blindly setting 1,000,000: While high limits are generally safe, they can mask severe leaks that eventually exhaust system-wide kernel memory or fs.file-max.
• Ignoring inotify: On Kubernetes, you may hit fs.inotify.max_user_watches long before you hit a file descriptor limit. This often breaks build tools or file-syncing sidecars.

Sources & further reading

• According to [OneUptime], systemd services require explicit LimitNOFILE settings as they bypass limits.conf.
• According to [Baeldung], lsof is the primary tool for mapping file descriptors to physical resources and network states.
• According to [Robust Perception], Prometheus and other high-ingestion tools require specific sizing of descriptors to handle massive scrape targets.

Why this matters now

In an era of sidecars, microservices, and high-concurrency AI workloads, the surface area for descriptor exhaustion has expanded. When an incident occurs, the evidence—socket distributions, file handles, and slope data—is ephemeral.

This is where Operate changes the recovery flow. Descriptor exhaustion is a perfect example of a case where the evidence is on the box, spread across /proc, lsof output, socket states, and the traffic graph, and it is gone the moment someone restarts the service. Operate watches your production environment and, upon detecting an incident, automatically investigates by reading across code, database logs, and infrastructure metrics. It assembles the trail and produces a root cause with the evidence attached, verified by a second model. Instead of just suggesting a ulimit change, Operate can find the specific unclosed HTTP body in your PR history and draft a fix. Because it is read-only and self-hosted, your data stays private while you get the auditability needed to solve complex kernel-level failures.

First ten minutes: a checklist

1. Check cat /proc/PID/limits to see the actual limit.
2. Run ls /proc/PID/fd | wc -l three times to find the growth slope.
3. Check ss -tan for a CLOSE_WAIT pile-up (indicates a code leak).
4. Check downstream latency; is the "leak" just a pile-up of slow requests?
5. If a restart is required, run lsof -p PID > fd_dump.txt first.