Perplexity has slashed median batch-read latency from 31.4 milliseconds to 5.60 milliseconds — a fivefold reduction — while cutting overall storage expenses by at least twenty percent, according to a case study published this week on InfoQ. The AI search company migrated its core search serving tier away from Amazon DynamoDB to CobbleDB, an internally developed distributed key-value store written in Rust. The shift addresses severe latency and cost bottlenecks that emerged from serving multi-kilobyte document batches to large language models under heavy query volumes.
In live production measurements, p90 latency fell from 56.7 milliseconds to 9.77 milliseconds, while p99 tail latency declined from 123 milliseconds to 24.2 milliseconds. Synthetic benchmarks handling payloads up to 100 kilobytes confirmed consistent throughput up to 500,000 requests per second. Each query dispatched to Perplexity generates between 100 and 120 target page keys, which the retrieval service splits into parallel batches of 10 to 20 keys. Unlike traditional search engines that return brief metadata snippets, retrieval for language models requires extracting full chunked passages and dense vector embeddings, yielding average record payloads of roughly 50 kilobytes. At production traffic scales exceeding 200,000 requests per second, DynamoDB usage-based pricing became financially unsustainable because AWS meters every byte transferred.
The report notes that DynamoDB operates as a black box that conceals internal partition placement, memory caching policies, and replica routing, preventing engineers from stopping tail-latency spikes caused by uncached reads, cross-zone networking hops, or lagging replicas. Aravind Srinivas, the CEO of Perplexity, noted that "the 40,000 lines of Rust comprising CobbleDB were built in two months by two systems engineers paired with an autonomous swarm of AI coding agents" that handled integration testing, build monitoring, and operational runbooks. The architecture incurs non-trivial trade-offs, according to the report: replacing a fully managed cloud database transfers node lifecycle management, backup verification, and partition rebalancing entirely onto internal site reliability engineers.
The report explains that Perplexity split its storage architecture into three specialized systems to resolve these constraints. Pillar runs on YTsaurus over high-capacity mechanical drives, maintaining versioned table families for web page metadata, passages, and vector representations. Lorry acts as a stateless queue consumer that groups Pillar exports into partition-aligned batch files, storing the payloads in Amazon S3 while posting metadata notices to CobbleDB. CobbleDB worker nodes pull and ingest these S3 batches independently, entirely isolating hot serving nodes from the write-heavy crawl pipeline. Each partition maintains three replicas distributed across independent compute nodes, with the core daemon using RocksDB as its embedded storage engine paired with memory-mapped caching and local NVMe solid-state disks. To minimize network overhead, a stateless query router maps hashed page identifiers to partitions and coordinates read execution by routing requests to node replicas located within the same availability zone. If a target replica exhibits elevated response times, the router speculatively hedges the request by issuing a concurrent read to an alternate replica on a different node. The database discards standard distributed transaction protocols and synchronous consensus algorithms because search serving tolerates slight replication lag, allowing replicas to apply updates asynchronously at their own rate and substantially reducing operational overhead.
Perplexity has indicated plans to open-source the CobbleDB codebase in an upcoming release. Applications must withstand eventual consistency, as replicas ingest batched files at different intervals. The migration demonstrates how specialized infrastructure can outperform general-purpose cloud services when workload characteristics diverge sharply from typical patterns. Companies building retrieval-augmented generation pipelines may face a strategic choice between accepting managed-service limitations and investing engineering resources into purpose-built systems that align precisely with their operational profile.