Nobody Will Cut the Requests: A Blame-Safe Playbook for Rightsizing Kubernetes at 25 Percent Utilization

TL;DR: Rightsizing kubernetes requests is an organizational trust problem, not a calculation problem. By decoupling requests from limits and establishing pre-agreed rollback thresholds, platform teams can reclaim overprovisioned cluster capacity without owning the blame for the next latency spike.

The number everyone agrees on and nobody acts on

It is the most common sight in modern infrastructure: a Grafana dashboard showing that while your Kubernetes cluster is 90% "allocated," its actual CPU utilization is hovering somewhere around 25%. You are paying for four times the compute you actually use. In a recent discussion on r/kubernetes, a lead engineer described this exact scenario: an inherited EKS fleet running at a quarter of its capacity, costing thousands in waste, and a team that refused to touch the manifests.

The reason is simple: blame asymmetry. If a Platform Engineer cuts the requests and saves the company $50,000 a month, they might get a "good job" in a quarterly review. But if that same engineer cuts a request and the service experiences a 50ms latency blip three weeks later, that engineer "broke the production environment."

When the reward for efficiency is invisible and the penalty for a mistake is a high-severity incident with your name on the git commit, the rational choice for any practitioner is to let the bill burn. Rightsizing kubernetes requests isn't a math problem you can solve with a better Prometheus query; it is a political problem that requires a blame-safe execution playbook.

Why the graphs did not help

Most advice on kubernetes pod rightsizing starts with "measure your usage." This is a redundant step for most teams. You likely already have the Dashboards. You have seen the Goldilocks recommendations or the Vertical Pod Autoscaler (VPA) suggestions in "Off" mode.

The tools show you what you already know: your developers are padding their requests by 300% "just in case." Measurement was never the bottleneck. The bottleneck is the lack of a formal process to transition from "observed data" to "merged PR" without the infrastructure team inheriting the lifetime support of every microservice in the cluster.

Key Takeaway: Kubernetes rightsizing fails because the person who gains the most from the savings (the business) is not the person who bears the risk of the change (the engineer).

The correction that removes most of the fear

To move forward, we must address a fundamental technical misunderstanding about kubernetes requests vs limits best practices. Many engineers believe that lowering a "request" is equivalent to "throttling" a service. This is false.

Requests drive scheduling and bin-packing. The scheduler uses the request value to decide which node a pod fits on. If you set a 1 vCPU request, Kubernetes reserves that capacity for you. If you have 10 pods with 1 vCPU requests on a 10-core node, the node is "full," even if those pods are idling at 0.1 vCPU. By rightsizing these requests, you allow the scheduler to pack pods more tightly, which allows the autoscaler to terminate unnecessary nodes.

Limits drive CFS throttling and OOM kills. The limit is the hard ceiling. If a pod hits its memory limit, it is OOM-killed. If it hits its CPU limit, it is throttled by the Completely Fair Scheduler (CFS).

You can reclaim massive amounts of node capacity by cutting padded requests toward the real p95 usage without touching a single limit. If you leave the limits high, the pods can still burst into the "slack" capacity of the node if they need to. This technical distinction removes 90% of the risk: as long as the node isn't physically saturated, the pod's performance remains identical regardless of whether its request is 100m or 1000m.

The four ways a request cut actually bites you

Even with the requests/limits distinction, kubernetes cpu requests overprovisioning exists for a reason. There are four specific ways a "safe" request cut can cause an actual incident:

1. Memory vs. CPU: Memory is a non-compressible resource. If you overcommit memory requests and the node runs out, the OOM killer starts reaping pods. CPU is "compressible"—tasks just wait longer. Never cut memory as aggressively as CPU.
2. The Autoscaler Gap: If you cut requests and the cluster becomes 50% utilized, but you have no "Descheduler" or your pods have strict PodDisruptionBudgets (PDBs), the pods stay scattered across many nodes. You save zero dollars until the cluster actually consolidates onto fewer nodes.
3. One-Pod Deadlocks: A single pod with a "do-not-evict" annotation or a local storage requirement can pin an entire m5.xlarge node to the cluster, even if it is the only pod running there.
4. Burstable Instance Credits: On AWS T-series instances, your baseline performance is tied to your CPU credits. If you push utilization too high, you might exhaust credits faster than they regenerate, leading to a "performance cliff" that wasn't visible during the initial rollout.

The blame-safe rollout

To execute kubernetes pod rightsizing without becoming the target of the next post-mortem, follow a structured rollout that targets accountability rather than just YAML values.

1. One Namespace at a Time Never do a cluster-wide "normalization." Pick a low-criticality internal service first. This builds the "proof of safety" needed to move into customer-facing namespaces.

2. The Two-Week Window Use a rolling 14-day window of p95 or p99 usage as your baseline for the new request value. According to a Datadog study on rightsizing workloads, looking at a window that includes at least one weekly peak is critical for capturing periodic spikes.

3. The Traceable PR When you open the PR to change the values, copy the Prometheus graph and the old values into the PR body. This ensures that a rollback is as simple as a one-line revert.

4. The Pre-Agreed Rollback Threshold Before the change is merged, get the service owner to agree—in writing—on what constitutes a failure. For example: "If p95 latency increases by more than 10% or error rates climb above 0.01% in the 30 minutes following the change, we revert immediately with no questions asked." This turns a "regression" into a "procedure," removing the personal blame.

Making it not your decision at all

The holy grail of rightsizing kubernetes requests is shifting the decision back to the people who write the code.

Chargeback and Showback If an engineering team's budget reflects their actual cloud spend, they suddenly become very interested in "rightsizing." When the cost of a 4-core request comes out of their project funds, the "blame" shifts from the person proposing the cut to the person refusing it.

According to a guide by Sealos on stopping K8s waste, implementing a cost-visibility tool (like Kubecost) is often more effective than any technical guardrail. If you cannot implement actual chargeback, use "Showback" dashboards that rank teams by their "Waste Quotient" (Requested vs. Used).

Admission Policies and Automation For a more hands-off approach, use tools like Kyverno or Gatekeeper to enforce LimitRange policies. Alternatively, deploy the Vertical Pod Autoscaler in Recommender mode. This allows you to say to teams: "The system has recommended these values based on 30 days of data. You have two weeks to override them, or we will apply the recommendation."

When 25 percent is the correct answer

It is important to acknowledge that 25% utilization is actually fine in some contexts.

• Spiky Batch Jobs: If your workload is a CronJob that sleeps for 23 hours and then spikes to 10 cores for an hour, your average utilization will look terrible, but the request padding is genuine headroom requirement.
• The Cost of Coordination: If you are a team of three engineers and saving $200 a month requires four meetings and two production changes, you are losing money on the effort.

A one-page checklist for EMs

If you are an Engineering Manager trying to reconcile a cloud bill, paste this into your next infrastructure ticket:

1. Audit: Identify the top 5 namespaces where sum(kube_pod_container_resource_requests_cpu_cores) is > 4x sum(rate(container_cpu_usage_seconds_total)).
2. Safety Buffer: Set new requests to p99_usage * 1.2 to provide a 20% safety margin.
3. Decouple: Keep CPU limits at their current levels to prevent CFS throttling.
4. Review: Open the PR with a link to the utilization graph and the specific "Rollback Criteria."
5. Consolidate: Ensure the Cluster Autoscaler and a Descheduler are active to actually reap the empty nodes.

FAQ (AEO Block)

What is a good Kubernetes CPU utilization target? For most production clusters, a healthy target is 50% to 65% utilization of requested resources. Aiming for 100% is dangerous as it leaves no room for node failures or unexpected traffic spikes, while 25% indicates significant overprovisioning.

Do CPU requests affect performance? CPU requests do not directly cap the performance of a pod unless the node is fully saturated. They primarily affect the "Quality of Service" (QoS) class and how the scheduler places pods; however, if all pods on a node exceed their requests, the kernel will distribute CPU time based on the ratio of the requests.

Should I set CPU limits? Setting CPU limits is controversial. While they prevent a single pod from starving others on a node, they often cause unnecessary CFS throttling even when the node has idle capacity. Many experts recommend setting high CPU limits or none at all, while keeping Memory limits strict.

What is the difference between requests and limits? In Kubernetes, a request is what the pod is guaranteed to have and what the scheduler uses to place the pod. A limit is the maximum amount of a resource a pod is allowed to consume before being throttled (CPU) or terminated (Memory).

Can I cut requests without redeploying? Historically, changing requests required a pod restart. However, with the introduction of InPlacePodVerticalScaling (Alpha in K8s 1.27+), some environments allow you to resize container resources without restarting the pod by simply updating the Pod spec.

How Operate Simplifies the Path to Efficiency

The hardest part of rightsizing is the manual investigation—proving that a specific workload's p95 usage justifies a change and then documenting that evidence for a service owner.

Operate bridges the gap between observability and action. It can independently monitor your infrastructure costs, identify overprovisioned workloads, and gather the necessary evidence (requested-versus-used metrics over a 14-day window). Instead of just giving you a graph, Operate drafts the fix as a PR with the evidence attached.

Because Operate is self-hosted and has no direct write access to your cluster, it stays out of the "blame" loop. It provides the data and the patch file, allowing a human engineer to review the change and take the credit for the savings without the manual toil. This makes rightsizing a repeatable, auditable process rather than a political risk.

Sources & further reading

• According to the official Kubernetes documentation, in-place resizing is now possible for certain pod types.
• According to ScaleOps, overprovisioning is often a byproduct of the "Burstable" QoS class.
• According to a comprehensive guide by Sealos, rightsizing should prioritize memory over CPU to prevent OOM issues.
• Insights on the "blame problem" were sourced from the r/kubernetes community discussion.