Seven Years, Right Answer, Wrong Unit: What Netflix's Flink Autoscaler Migration Says About Automated Operational Decisions

TL;DR: Netflix is migrating its fleet of over 30,000 Flink jobs from a custom cluster-level autoscaler to the open-source Apache Flink Autoscaler. The transition highlights a fundamental shift in automated operational decisions: moving from outside-in proxy signals like CPU and Kafka lag to granular, internal job-graph analysis that accounts for the true processing rate of individual operators.

The automation that worked

Netflix has been a power user of Apache Flink since 2017. By 2019, the team had built its own internal autoscaler to manage the massive scale of its streaming pipelines. This system was not a failure; in fact, it was a resounding success.

Running on Mantis and consuming cluster-level telemetry from Atlas—including CPU utilization, network throughput, and Kafka consumer lag—the original autoscaler adjusted the total number of TaskManagers for a given job. According to the Netflix Technology Blog, this automation cut resource usage by 25 to 45 percent across thousands of pipelines for over half a decade.

It is important to establish this clearly: this is not a story about a broken automation. By every traditional SRE metric, the system worked. It retired manual toil and saved millions in compute costs. However, as the underlying workloads evolved, the logic that governed those savings began to reach its structural limits.

What changed underneath it

In the seven years since the first autoscaler was deployed, Netflix’s streaming jobs became significantly more complex. Simple pipelines were replaced by sophisticated directed acyclic graphs (DAGs) featuring multiple branches, joins, and terabytes of state.

The failure mode that emerged was one of granularity. The original autoscaler treated the Flink job as a monolithic unit. If the cluster needed more resources, it scaled everything. But a modern streaming job often has "skewed" requirements; a single operator (like a windowed join) might be bottlenecked while five other upstream operators are idle.

Key Takeaway: The failure mode of a mature operational automation is usually a lack of resolution—reasoning at the wrong granularity for the system it has become—rather than a lack of correctness.

When your scaling unit is the cluster, every operator in a job shares one decision. A job whose slowest and fastest operators need different parallelism cannot be served by a single global number. No amount of fine-tuning that global number can fix a structural mismatch in granularity.

Proxy signals versus the system's own account of itself

The shift Netflix is making involves moving away from "proxy signals." The old autoscaler looked at the system from the outside-in: Is the CPU high? Is the Kafka lag increasing? While useful, these are symptoms, not causes.

The new approach, powered by the open-source flink autoscaler (specifically FLIP-271), reads the system's own account of itself. It estimates each operator’s "True Processing Rate" by looking at throughput and busy time. By walking the job graph, it computes the required parallelism for every vertex individually.

This technique descends from the DS2 research project and the 2018 OSDI paper by Kalavri et al. Interestingly, Vasiliki Kalavri has noted that the project initially explored complex critical path analysis before settling on True Processing Rate because the simpler baseline worked better in practice. The replacement isn't necessarily a "smarter" AI model; it is simply a model that reads the causal structure of the job instead of its shadow.

Automation that prices its own action

One of the most sophisticated aspects of the Netflix migration is how they handle the cost of change. In Flink, rescaling a stateful job is not free; it requires a savepoint or checkpoint and a restart, which incurs downtime and "catch-up" lag.

To manage this, Netflix runs a 0.45 utilization target, which is significantly lower than the Flink community default of 0.7. According to Netflix engineering, this buffer is intentional. They have priced the cost of the "action" (rescaling) and determined that being slightly less aggressive helps avoid frequent, expensive restarts for large stateful jobs.

They are also investigating Flink 2’s disaggregated state architecture. This represents the other way to solve the trade-off: if you can't make the automation smarter, make the action cheaper. By reducing the time it takes to recover state, they can eventually afford to be more aggressive with their scaling targets.

Buying the decision and keeping the integration

Netflix’s migration strategy offers a lesson in "Build vs. Buy." They did not adopt the Flink Kubernetes Operator wholesale. Instead, they extracted the open-source decision engine and wired it into their existing control plane—a Spring Boot service using Temporal workflows to isolate per-job decisions.

They still had to write significant custom code to make it work at their scale, including:

• JobManager metric collection for jobs with up to 3,000 subtasks.
• Server-side metric filtering to reduce telemetry overhead.
• Logic to preserve forward-connected subgraphs during scaling.

The commodity here was the decision logic (the algorithm), but the value remained in the integration (the control plane). They retired seven years of in-house algorithmic work but retained their proprietary operational wrapper.

What to take from this if you do not run Flink

Even if you aren't managing 30,000 streaming jobs, the Netflix migration provides a framework for evaluating any automated operational actor:

1. What unit does it act on? Is it scaling a whole cluster when it should be adjusting a specific microservice or thread pool?
2. What signal does it read? Is it looking at symptoms (CPU/RAM) or the internal work queue and processing rate?
3. Does it know what its action costs? Does the automation understand the impact of a restart or a configuration change on the rest of the system?

It is worth noting the limits of the analogy: Netflix's scale is what makes this level of granularity necessary. For smaller fleets, a cluster-wide proxy signal may still be the most cost-effective way to automate. However, the reported results are hard to ignore: one Netflix team reduced their annualized Flink compute spend by 58 percent—roughly $1.1 million a year—simply by switching to this more granular approach.

The open edges

The transition isn't entirely solved. Netflix engineers noted that FLINK-38538 remains an open issue where busy operators can be adversely affected by output-ratio-based scaling decisions. As they continue to migrate the remaining internal use cases, the focus remains on refining how the automation perceives backpressure and handles sink bottlenecks.

Why this matters now

Engineering teams are currently facing a deluge of "agentic" operational tools. This case study serves as a reminder that the efficacy of an AI or automated SRE tool isn't just about the LLM or the logic; it’s about whether the tool has access to the internal causal structure of the software it is trying to fix.

At Operate, we apply these same three questions to our own platform. Operate doesn't reason about a "service" or an "alert" in isolation; its unit of analysis is the case, encompassing the entire incident context. Its signals are not just dashboard proxies, but the system’s own artifacts—code, database schemas, and logs. Finally, the cost of action is bounded: Operate is self-hosted and read-only by design, proposing fixes as PRs so that the most expensive action—deployment—always remains under human control.

Sources & further reading

• According to the Netflix Technology Blog, the original autoscaler was built on Mantis and utilized Atlas telemetry.
• InfoQ reports that the new system manages over 30,000 jobs and has saved individual teams over $1 million annually.
• The technical foundation for the new scaler is detailed in FLIP-271: Autoscaling for heterogeneous streaming jobs.
• The concept of True Processing Rate was established by Kalavri et al. in their 2018 OSDI paper, DS2: Deterministic Scalable System for Stream Processing.
• Utilization defaults and scaling logic are documented in the Apache Flink Autoscaler official documentation.