Netflix is shifting to the open-source Apache Flink Autoscaler for more than 30,000 streaming jobs running across multiple AWS regions, with one team cutting annualized Flink compute spending by 58% and saving approximately $1.1 million per year. The move comes after the streaming giant determined its original cluster-level autoscaling system was less effective for complex, stateful pipelines where different operators have varying processing needs, according to a report published by InfoQ.
Netflix has operated Apache Flink since 2017 and created its first autoscaling system around 2019. That initial system ran on Mantis and pulled cluster-level telemetry from Atlas, tracking CPU usage, network utilization, Kafka lag, input rate, and consume rate. It modified the total count of TaskManagers and lowered resource consumption by 25% to 45% across thousands of pipelines. The constraint was the scaling unit itself: because the original autoscaler evaluated the cluster rather than individual operators, all operators in a job effectively received the same scaling decision, which proved unsuitable for stateful pipelines containing branches, joins, and terabytes of state where different sections of the dataflow have distinct processing demands.
The report explains that the Apache Flink Autoscaler instead relies on metrics exposed by the running job to estimate each operator's actual processing rate from throughput and busy time, then walks the job graph and calculates required parallelism for individual vertices. The technique is outlined in FLIP-271, which tackles autoscaling for heterogeneous streaming jobs and the expense of rescaling stateful applications, and builds on research from the DS2 project. Vasiliki Kalavri, a systems researcher involved in the work, said the project initially explored critical path analysis before adopting True Processing Rate as a simpler baseline, noting "this very simple idea worked really well."
Netflix integrated the autoscaler with its internal control plane instead of deploying it directly through the Flink Kubernetes Operator. A Spring Boot service uses Temporal workflows to isolate autoscaling decisions for individual jobs. The company also modified JobManager metric collection to support jobs with up to 3,000 subtasks, added server-side metric filtering, preserved forward connected subgraphs during scaling, and added handling for sink backpressure. Unlike generic event-driven autoscalers such as KEDA, which scale workloads from external metrics or events, Flink's autoscaler evaluates the internal dataflow graph and operator capacity.
Netflix currently uses a 0.45 utilization target, below Flink's 0.7 community default, to reduce aggressive rescaling of large stateful jobs. The company plans to migrate its remaining internal autoscaling use cases to the open-source implementation, while investigating Flink 2's disaggregated state architecture to address the cost of state recovery during rescaling. Operator-aware autoscaling may become table stakes for any organization running large-scale stateful stream processing, particularly where infrastructure efficiency directly impacts bottom-line margins. The economics of cloud compute will likely push more teams toward fine-grained resource optimization rather than cluster-wide approximations.