The upcoming release of Polars 2.0 will make the streaming engine the automatic default for all LazyFrame queries, delivering what the company describes as "easily 5x faster" performance in aggregate, according to an announcement published last week about the first release candidate. Polars, an open-source library used by developers and data analysts to clean, combine, and analyze tables of data, aims to solve two persistent challenges with large datasets: sluggish queries and out-of-memory errors. But the performance leap comes with a significant caveat that could silently break existing code.
The streaming engine processes lazy queries in batches instead of handling all data simultaneously, allowing users to work with datasets larger than available memory, the company explains. Users can anticipate "massive memory and performance improvements on most queries" from this change. However, the streaming engine can alter the sequence in which rows appear, particularly for operations like join, group_by, and unpivot. The company highlights this migration risk in a red "danger" box in its Version 2.0-rc user guide, acknowledging the change "may silently impact the results of your pipelines."
According to the announcement, the "streaming engine doesn't guarantee row-order by default for certain operations." For users whose code expects rows to appear in a certain order, that could create more problems for downstream processes. Polars offers two main mitigation strategies: sort explicitly or set maintain_order=True where applicable. Alternatively, users can keep the in-memory engine as default by setting the engine affinity, though this forgoes the performance benefits. The pre-release also removes many ambiguous casts, directing users to employ .str.to_date()/.str.to_datetime() to parse strings to temporal data types, giving users "one obvious way to parse data."
The shift to streaming by default addresses a fundamental trade-off in data processing: speed and memory efficiency versus predictability. By executing queries in smaller chunks rather than loading entire datasets, the streaming engine prevents memory overflow that can crash programs handling massive files. The row-order issue stems from parallel processing, where different batches can complete at different speeds. Polars ships pre-releases rather than holding features back because the company "[doesn't] gate new features" and prefers to release them as soon as they're ready. Looking ahead to version 2.x, the company hints at a new IO-plugin design, a faster S3 reader, a cost-based planner, join reordering, and substantial SQL coverage improvements. For developers exploring the release candidate now, the takeaway is clear: better memory and performance are worth getting excited about, but don't forget to watch that row order. The decision to flip this default reveals a maturation point where performance optimization can no longer coexist invisibly with legacy behavior assumptions. Organizations that treat row order as incidental rather than contractual may discover how many implicit dependencies live in production systems.