Spotify has unveiled a storage architecture called Random Access Parquet (RAP) that lets online services and AI applications retrieve individual records directly from its data lake without copying datasets into separate operational databases. The company announced the system in an August 2026 blog post, explaining that RAP adds an external indexing layer over Apache Parquet files to enable interactive lookups while continuing to use the same datasets for analytics, machine learning, and online serving. The streaming giant currently maintains petabytes of online data in Bigtable while exabytes sit in its Google Cloud Storage-based data lake, making large-scale replication into serving databases increasingly expensive.
Spotify described how RAP works by introducing an external index that maps lookup keys—such as user IDs—directly to Parquet files and row locations. Instead of scanning thousands of files, a query resolves the key through the index before issuing a targeted ranged read against object storage. As new data is written into Apache Iceberg tables, an index builder generates append-only index fragments without modifying immutable Parquet files. The company detailed several storage layout optimizations that reduce point query latency, including sorting data by lookup key to reduce the number of files accessed, grouping related records together, interleaving value columns so multiple attributes can be retrieved through a single contiguous read, and using covering indexes that can satisfy some queries without reading Parquet files. According to the company, these techniques trade modest increases in file or index size for fewer storage operations, allowing some point queries to be served through a single ranged read of only a few kilobytes.
The report states that modern data lakes have become the central repository for analytical and AI workloads, but retrieving individual records remains inefficient because distributed query engines such as Trino and BigQuery are optimized for analytical scans rather than key-based lookups. Spotify noted that although cloud object stores such as Google Cloud Storage now provide millisecond access latency, query planning, metadata traversal, and file discovery can add significant overhead for point queries. The company said the approach allows the same datasets to support analytical processing, machine learning pipelines, notebooks, AI agents, and latency-sensitive online applications without maintaining duplicate storage systems. Spotify also supports secondary indexes, enabling efficient querying across multiple lookup dimensions—such as buyer ID or seller ID—without rewriting Parquet files, with hash-based indexes supporting exact lookups while sorted indexes enable range queries.
The announcement reflects broader efforts to extend open data lake technologies beyond analytical processing. According to the report, Google Cloud recently described an Apache Iceberg-based lakehouse architecture for AI applications that similarly seeks to reduce data duplication while enabling operational access to data, though RAP introduces a dedicated external indexing layer optimized for point lookups while remaining compatible with existing Parquet files and Iceberg tables. The architecture generated discussion within the data engineering community, with Andrew Lamb highlighting RAP as an example of extending open data formats for interactive workloads. In a separate LinkedIn discussion, Vikas Singh argued that improvements in cloud object storage performance have shifted more of the latency associated with point queries toward query planning and metadata access, an area that RAP is designed to reduce through precomputed indexes. Spotify said secondary indexes are managed at the serving layer, allowing new access paths without changing data pipelines while continuing to use the same Parquet datasets for both analytical scans and interactive point lookups, with storage layout techniques such as Z ordering and Hilbert curves further improving data locality for secondary lookup dimensions. Organizations wrestling with the cost and complexity of maintaining separate systems for analytical and operational workloads may find themselves revisiting fundamental assumptions about where the boundary between those worlds actually needs to fall.