Flink Autoscaler Migration at Netflix and the Wrong Unit
Netflix is moving more than 30,000 Flink jobs to the open-source autoscaler. The old one worked, and the new one changes the unit of the decision.
Quick Answer
More than 30,000 Netflix Flink jobs are moving from the company's own cluster-level autoscaler to the open-source Flink Autoscaler. The old system worked for years, but it reasoned about whole clusters while modern jobs need per-operator decisions. The lesson for any automated operations tool is to check the unit it acts on, the signal it reads and the cost of its own action.
Netflix built its first Flink autoscaler around 2019. According to InfoQ's report, it ran on Mantis and consumed cluster-level telemetry from Atlas, including CPU, network utilization and Kafka lag. It adjusted the total number of TaskManagers for a job, and it reduced resource usage by 25 to 45 percent across thousands of pipelines.
We think this is useful precisely because it is not a story about broken automation. By ordinary SRE measures the system worked, and the question is what makes a working automation reach its limit.
The Automation That Worked for Years
The original autoscaler retired manual toil and saved real compute. Its design was sensible for the pipelines of 2019, which were mostly simple. We read the history as a reminder that automation is judged against the workload it was built for, and workloads move.
As jobs grew into directed acyclic graphs with branches, joins and large state, the same logic began to reach its structural limits. The signals it read were still true, and they described the cluster more than the work.
What Changed Underneath the Original Autoscaler
We locate the failure mode in granularity. InfoQ quotes the explanation directly. "Because the original autoscaler reasoned about the cluster rather than individual operators, all operators in a job effectively shared the same scaling decision."
A modern streaming job often has skewed needs, where one windowed join is the bottleneck while upstream operators sit idle. When the scaling unit is the cluster, every operator shares one decision. No amount of tuning that single number can fix a mismatch in resolution.
Key Takeaway. The failure mode of a mature operational automation is usually a lack of resolution, reasoning at the wrong granularity for the system it has become, rather than a lack of correctness.
How the Flink Autoscaler Reads the Job Instead of the Cluster
The old autoscaler looked at the system from the outside in. Is the CPU high, is Kafka lag rising? Those are symptoms and not causes, and we have written about how consumer lag shapes can tell you not to scale out.
The open-source approach, described in FLIP-271, reads metrics from the running job. As InfoQ puts it, it "uses metrics exposed by the running job to estimate each operator's true processing rate from throughput and busy time." It then walks the job graph and computes parallelism for each vertex.
The idea descends from DS2, presented at OSDI 2018. The paper's abstract says DS2 "combines a performance model with lightweight instrumentation to estimate the true processing and output rates of individual dataflow operators." The replacement is not a smarter model. It is a model that reads the causal structure of the job instead of its shadow.
| Question | Original cluster-level autoscaler | Open-source Flink Autoscaler |
|---|---|---|
| Unit of decision | The whole job, through its TaskManager count | Each operator, through its own parallelism |
| Signals read | CPU, network, Kafka lag and rates from Atlas | Throughput and busy time exposed by the running job |
| Weak spot | Cannot tell which operator is the bottleneck | Still has open issues around backpressure |
We find the third row the most useful for planning. Neither design is complete, and the useful question is which gap you can live with at your scale.
Automation That Prices Its Own Action
Rescaling a stateful Flink job is not free, since it needs a restart and incurs catch-up lag. InfoQ reports that Netflix runs a utilization target of 0.45, against a community default of 0.7. We read that buffer as a deliberate choice, because a lower target means fewer rescales and fewer costly restarts for large stateful jobs.
This is a general design question for any automated actor. If the action is expensive, the automation should know it and act less often, or someone should work to make the action cheaper.
Buying the Decision and Keeping the Integration
According to InfoQ, Netflix runs the open-source decision engine inside its existing control plane, which the article describes as a Spring Boot service that uses Temporal workflows, and the setup supports jobs with up to 3,000 subtasks.
We take a build-versus-buy lesson from that. The commodity was the decision logic, and the value stayed in the integration with its own platform. Our write-up on build versus buy for operations copilots covers the same split.
The Limits of the Analogy and the Open Edges
Granularity matters because of Netflix's scale, and we would not copy it blindly. For small fleets, a cluster-wide proxy signal may remain the cheapest way to automate. InfoQ reports that one Netflix team cut annualized Flink compute spend by 58 percent, roughly $1.1 million, but that figure comes from a fleet where per-operator decisions are worth the engineering.
The migration is also unfinished. InfoQ names an open issue, FLINK-38538, tracking how backpressure interacts with scaling decisions. As the migration continues, the remaining work is in how the automation perceives backpressure and sink bottlenecks.
Ask Three Questions of Every Automated Actor
Even without running Flink, the migration gives a framework for evaluating any automated operational tool. We would ask these three questions.
- What unit does it act on? Does it scale a whole cluster when it should adjust a specific service or thread pool?
- What signal does it read? Does it read symptoms such as CPU and memory, or the internal work queue and processing rate?
- Does it know what its action costs? Does it account for a restart or a configuration change on the rest of the system?
The same test applies to AI tools for operations. We think their usefulness depends less on the model than on access to the causal structure of the software, such as dedicated AI agents that investigate with evidence from read-only adapters across code, logs and databases. Our analysis of cutting Kubernetes requests and limits without the blame applies the same reasoning to resource settings.
Check the Resolution of Your Own Automation
The Flink Autoscaler story is a story about resolution, not about a bad system. An automation that worked for years can still be reasoning at the wrong resolution. List the automated actors in your stack, note the unit, signal and action cost of each, and look first at the ones whose unit is coarser than the problem.
We would start with autoscaling and remediation scripts, because their decisions are cheap to audit. If you want agents that investigate with evidence while engineers keep the decision, Operate runs dedicated AI agents that investigate production problems with cited evidence and propose fixes for your team to review.
Frequently Asked Questions
We describe it as a component that uses metrics from a running Flink job to estimate each operator's true processing rate, then computes the parallelism each vertex needs. That per-operator view is what separates it from autoscalers that only scale a whole cluster.
We read it as the rate at which an operator could process records if it were busy all the time, estimated from observed throughput and busy time. It lets a scaler size each operator for the work it faces rather than for overall resource use.
According to InfoQ, the original autoscaler reasoned about the cluster, so all operators in a job shared one scaling decision. As jobs became more complex, we think that coarse unit stopped matching the work, and Netflix moved toward the per-operator approach.
We do not think there is a universal number. Netflix runs a target of 0.45 where the community default is 0.7, because rescaling large stateful jobs is costly. The right target depends on how expensive each scaling action is for your workload.
We use it for a measurement that correlates with the real problem without describing it, such as CPU use or consumer lag standing in for a bottlenecked operator. Proxy signals are cheap to collect, and they can drive the wrong decision when the cause lies elsewhere.
About the author
Operate Team
The team behind Operate
Operate Team builds Operate, enterprise AI agents for production operations. Operate answers production questions from Slack and Microsoft Teams, investigates alerts, and reviews performance, security and cost. It finds the root cause with evidence and proposes a fix for an engineer to review, with controlled access to your systems.
