Netflix has shifted the majority of its batch workloads to Kueue, an open-source cloud-native job execution platform that surpassed the capabilities of its internally built solution, according to a blog post published by the company. The streaming giant is now running millions of batch workloads in production using Kueue, though the transition remains in progress. Engineers achieved the migration by maintaining API compatibility with the legacy system, which allowed for a gradual, risk-reduced rollout and easy reversal if problems emerged.
Netflix originally developed Compute Managed Batch (CMB) in 2018 to handle and execute batch jobs on its Titus container platform, where it controlled capacity through tenant hierarchies and distributed workloads across multiple Kubernetes clusters. Since CMB's creation, many of its core capabilities have been absorbed into open-source projects that matured within the Kubernetes ecosystem. Building new functionality into CMB became increasingly challenging because the solution lacked tight integration with Kubernetes, prompting engineers to select Kueue as a replacement for its broad feature set, flexibility, widespread adoption, and rapid innovation pace.
The migration team made the transition completely invisible to existing CMB users and ensured the new platform could handle the necessary container launch rate and peak throughput from day one, the authors wrote. Engineers mapped internal tenants in CMB to Cohorts in Kueue and leaf tenants to paired ClusterQueue and LocalQueue resources, while resource flavors and nominal quotas configured capacity requirements imported from the legacy system. The process was tenant-specific and allowed for straightforward rollback if issues surfaced. The team shared a key lesson learned: "Don't wait until the end to migrate the most complex use case," noting they moved their largest and most intricate customer first, which built confidence for subsequent migrations and compressed the production rollout to just four weeks.
Netflix chose Kueue because the evolving open-source ecosystem had caught up to what the company built internally, making continued investment in a homegrown solution increasingly expensive. Developing new features in CMB had grown more difficult as it wasn't as closely integrated with Kubernetes, while Kueue offered job queuing based on priorities with various strategies, sophisticated resource management, multi-cluster and topology-aware scheduling, numerous integrations, and comprehensive observability. More recently, engineers significantly boosted average resource utilization by using preemption-based fair sharing to preserve reservation semantics while lending unused capacity to other tenants. The team emphasized that preserving API compatibility while swapping underlying components substantially lowered project risk and delivered a seamless experience for end users, while load testing in non-production environments helped them tune performance settings to meet the platform's throughput demands.
The migration confirms that even technology giants with deep engineering talent are consolidating around open-source standards when internal tools no longer justify their maintenance costs. Organizations betting on proprietary batch systems should weigh whether their unique requirements truly warrant custom development or if adopting proven community-backed platforms offers better long-term velocity and talent retention.