Kubernetes Requests and Limits Can Be Cut Without the Blame
Padded Kubernetes requests and limits survive because cutting them is a personal risk. What a request controls, four real risks and a blame-safe rollout.
Quick Answer
Padded Kubernetes requests and limits survive because cutting them puts one engineer's name on the next latency blip, while the saving belongs to nobody. Requests drive scheduling, so CPU requests can usually come down toward measured usage without touching limits. Memory requests need more care. A namespace-by-namespace rollout with rollback criteria agreed in advance turns the cut into a procedure instead of a personal bet.
Kubernetes requests and limits are the two numbers each container declares for CPU and memory. The request is what the scheduler reserves when it places a pod, and the limit is the ceiling the kernel enforces at run time. When requests sit far above real usage, nodes look full while doing little work, and the cluster bill reflects the reservation, not the load.
We keep meeting the same cluster. It is nearly fully allocated, it runs at a fraction of that, everyone has seen the dashboard, and nobody cuts anything. The reason is rarely technical.
In this post we cover why the graphs do not move anyone, what a request actually controls, the four ways a cut really does bite, a rollout that makes rollback a rule rather than a judgement, how to move the decision to service owners, and when low utilization is the right answer.
Why the Utilization Graphs Never Move Anyone
Most rightsizing advice starts with measuring usage. Most teams already have that measurement. They have a dashboard of requested versus used CPU, and often a Vertical Pod Autoscaler running in recommendation-only mode with numbers nobody has applied.
What stops the change is asymmetry. The saving shows up in a cloud bill months later and is credited to nobody in particular, while a latency regression three weeks after a cut is credited to the person whose commit changed the request. Given those two outcomes, leaving the padding alone is the rational choice for any individual engineer.
We think that is the real problem to solve. The calculation is a query, and the hard part is a process that lets someone make the change without owning every future incident on that service.
A Request Controls Placement, Not Speed
The Kubernetes resource management documentation separates the two numbers cleanly. When you specify a request, "the kube-scheduler uses this information to decide which node to place the Pod on." When you specify a limit, the kubelet enforces it so the container cannot use more than that amount.
That distinction removes most of the fear around CPU. A container is not slowed down by a smaller CPU request while the node has spare cycles. The same page says the CPU request "typically defines a weighting", so requests only decide shares when several containers compete for a contended node.
Limits are where speed is decided. CPU limits are enforced by CPU throttling, and memory limits by out-of-memory kills. That is why we cut padded CPU requests toward measured usage and leave limits where they are on the first pass, so a pod that needs to burst still can.
Four Ways a Request Cut Really Bites
Separating requests from limits makes cuts safer, not free. We check four specific risks before changing anything, because each has caused a real regression on a cluster that looked overprovisioned.
1. Memory Requests Change Who Gets Evicted
Memory is not compressible the way CPU is. The pod QoS documentation says that when a node evicts pods for resource pressure, "only Pods exceeding resource requests are candidates for eviction." Cutting a memory request below real usage moves that pod to the front of the queue.
So we cut memory requests more conservatively than CPU, and only against a window that includes the service's own peaks. An OOM kill shows up as exit code 137, and our guide to attributing exit code 137 helps tell an eviction from a limit kill afterwards.
2. Nothing Saves Money Until Nodes Are Removed
Smaller requests free space on nodes, but the bill only drops when the cluster autoscaler removes a node. Its FAQ lists what prevents that, including pods with a restrictive PodDisruptionBudget, pods not backed by a controller, pods with local storage, and pods annotated safe-to-evict false.
We audit those blockers before the cut, not after. Otherwise the requests go down, the nodes stay, and the team concludes that rightsizing did not work.
3. One Pod Can Pin a Whole Node
The same FAQ notes that pods which cannot schedule anywhere else keep their node alive, for example because of a node selector, a hostPort or a one-pod-per-node rule. A single such pod can hold an expensive node in the cluster after every other workload has moved off it.
4. Contention Returns When the Node Fills Up
Tighter packing means more pods share each node. When they all peak together, CPU shares follow the requests, so a service whose request was cut hardest gets the smallest share. We look at co-peaking services and at p99 latency, not the average, because the regression appears in the tail first, as our p99 latency field guide explains.
A Rollout That Makes Rollback a Rule
The fix for blame asymmetry is a procedure that everyone agreed to before the change. When the rule says what failure looks like and what happens next, a revert is the process working, not a person failing. We run it in five steps.
- One namespace at a time. Start with an internal, low-criticality service and use it to show the procedure works before touching anything customer-facing.
- A window that includes real peaks. Base new requests on usage over a window long enough to include the service's weekly and monthly peaks, plus a margin the team agrees in advance.
- Evidence in the change itself. Put the old values, the new values and the usage graph in the change description, so the revert is one line and the reasoning survives the person who wrote it.
- Rollback criteria signed before merge. The service owner agrees in writing which latency and error thresholds trigger an automatic revert, and for how long after the change they apply.
- Watch, then widen. Hold the change through at least one peak, then move to the next namespace with the same rules. A namespace that needed a revert goes back into the queue with a smaller cut, not out of the programme.
The evidence step is where most teams stall, because it means pulling usage, events and recent incidents for each service. It helps to connect an investigator read-only to your logs and metrics, so the before-and-after comparison for each namespace does not depend on one person's queries.
Move the Decision to the People Who Own the Service
The durable fix is to make padding visible to the team that pays for it. Showback, a report that ranks teams by requested versus used capacity, changes the default question from "who will cut this" to "why is this padded". Chargeback goes further by putting the cost in the team's own budget.
Automation can carry the rest. Run the Vertical Pod Autoscaler with updateMode set to Off, which its quick start describes as calculating recommendations without changing pods. Then publish a policy that recommendations are applied after a review period unless the owner overrides them with a reason.
Restarts used to make every change feel bigger than it was. In-place resize of container CPU and memory has been stable since Kubernetes v1.35, after first appearing in v1.27, so on current clusters a resize no longer has to mean a rollout. We would still test it per workload before relying on it.
When Low Utilization Is the Correct Answer
Low average utilization is not always waste. A batch job that is idle most of the day and saturates its request for an hour looks terrible on an average and is sized correctly for its peak. Headroom for a failed node or a zone loss also looks like idle capacity until the day it is needed.
The effort can also cost more than it saves. On a small cluster, a series of meetings and production changes to save a modest monthly amount is a bad trade. We would rightsize where the spend is large and the services are well understood, and leave the rest.
Finally, rightsizing changes who carries risk. Our post on who owns the reliability bill after a cost saving covers that side, and the short version is that a saving with no owner for its risk is not finished.
Cut Kubernetes Requests and Limits by Procedure, Not Courage
The technical part of rightsizing is small. Cut padded CPU requests first, keep limits on the first pass, treat memory carefully, and make sure nodes can actually be removed. We would spend most of the effort on the procedure, because Kubernetes requests and limits only come down when changing them stops being a personal risk.
Start with one namespace, one signed rollback rule and one week of watching. Operate investigates production issues through read-only adapters to logs, databases and repositories, cites the evidence for every finding, and proposes any code fix as a patch file an engineer reviews and applies.
Frequently Asked Questions
A request is the amount of CPU or memory the scheduler reserves for a container when it picks a node. A limit is the maximum the container may use, enforced by throttling for CPU and by OOM kills for memory. We treat requests as a placement decision and limits as a safety ceiling.
Setting them equal for CPU and memory in every container gives a pod the Guaranteed QoS class, which makes it the least likely to be evicted. We do that for critical, steady workloads. For bursty services we keep limits above requests, so they can use spare node capacity without reserving it all day.
The container is scheduled using its request and can use any spare capacity on the node. For CPU we often accept that, since contention is shared by request weight. For memory we are more careful, because a container with no limit can grow until the node itself is under pressure.
We use kubectl with a custom output that prints each container's resources, or kubectl describe node to see allocated requests and limits per node. For trends over time we query the requested and used metrics in our monitoring stack, since a single snapshot hides weekly and monthly peaks.
On current Kubernetes, yes. In-place resizing of CPU and memory is stable since v1.35, so a container's resources can change without recreating the pod. We still check each container's resize policy, because memory changes often use RestartContainer, since many runtimes cannot adjust memory while running.
About the author
Operate Team
The team behind Operate
Operate Team builds Operate, enterprise AI agents for production operations. Operate answers production questions from Slack and Microsoft Teams, investigates alerts, and reviews performance, security and cost. It finds the root cause with evidence and proposes a fix for an engineer to review, with controlled access to your systems.
