Spotify's engineering team has introduced a technique that compresses lengthy prompts for large language models into a much smaller set of learned tokens, cutting infrastructure costs and boosting processing speed without changing the underlying model. The method, called Gisting, replaces thousands of words with compact "gist" tokens that the model learns to interpret just as it would the original text. Spotify says the approach delivers significant gains in both speed and efficiency, allowing the company to scale back its GPU allocation while maintaining output quality.
The company applied Gisting to its Sidekick GraphQL agent, shrinking the system prompt from roughly 6,000 tokens down to 1,500 gist tokens—a 4:1 compression ratio—without losing prediction accuracy. At 350 requests per minute, the median time to first token dropped from 438 milliseconds to 354 milliseconds, while end-to-end request latency fell from 6.8 seconds to 4.2 seconds. Throughput climbed from 20.2 queries per second to 23.4 queries per second, and the performance improvements let Spotify reduce the number of GPUs it needed to run the system.
According to the engineering team, Gisting is rooted in a 2022 academic paper on prompt compression and works through a two-step training process. First, the model runs with the full original prompt to generate "teacher logits"—the expected output. Then it runs again with only the gist tokens to produce "student logits." The gist tokens are trained to minimize the difference between the two outputs, measured by KL divergence, until the compressed version behaves nearly identically to the full prompt. Once training wraps up, the gist embeddings are written directly into the model's embedding matrix and registered as special tokens in the tokenizer, so the model runs at inference time like any standard setup—no custom attention masks, extra encoders, or special serving requirements. Spotify emphasizes that Gisting also works alongside other optimizations: prefix caching skips recomputation of cached prompt sequences, but the model still processes those tensors during decoding, whereas Gisting cuts that overhead further by substituting a long prompt with a shorter learned sequence.
The core advantage is that the model doesn't process a conventional summary of the original prompt—it processes a learned representation specifically trained to replicate how the model would behave if it had seen the full text. That distinction matters because it preserves prediction quality while delivering measurable latency and throughput gains. Gisting is complementary to existing techniques, meaning the benefits stack: Spotify uses both prefix caching and Gisting together to compound the performance lift. The report notes that additional tuning—such as autosearch for hyperparameter optimization—and other implementation details played a significant role in achieving the final results, and interested readers should consult the full technical article for deeper coverage of those factors.
Spotify's results suggest that prompt compression through learned embeddings can meaningfully cut both cost and delay in production LLM systems, especially for applications with fixed, repetitive system prompts. The company's decision to scale down GPU allocation after deploying Gisting signals that the technique delivered not just faster responses but also real infrastructure savings. By writing the gist embeddings directly into the model and treating them as standard tokens, Spotify avoided the complexity of custom serving pipelines, making the approach easier to integrate into existing workflows. The company points to the synergy between Gisting and prefix caching as a key takeaway: organizations already using caching can layer Gisting on top to extract further gains without redesigning their stack. As models grow larger and prompts grow longer, techniques that compress context without sacrificing quality may become central to keeping inference costs manageable at scale. Organizations that rely on lengthy system instructions or repeated context across many queries stand to benefit most, since the training overhead of Gisting is offset by sustained reductions in per-request processing time and hardware requirements.