OpenAI has built its GPT-Live voice assistant by dividing the system into two distinct processing lanes: one that handles time-sensitive audio and inference, and another that manages everything else behind an asynchronous barrier. The company recently published an engineering account explaining how the architecture keeps conversations fluid even when background tasks encounter delays or rely on outside services. The design tackles a core problem for real-time AI applications—maintaining responsiveness when operations have unpredictable timing.
The live path runs only the media pipeline and the inference loop, according to the account. Delegation to other models, tool invocation, data persistence, and all other application logic sit behind an asynchronous remote procedure call boundary. Each session uses dedicated, stateful inference on an assigned instance that reserves capacity, but the context can shift to a different instance when the original is being drained or when a conversation hits its context ceiling. OpenAI kept WebRTC as its media foundation and added WebRTC Abridged Roundtrip Protocol improvements—called WARP—along with Instant Connect to shorten startup lag. Before the public launch, the team ran a "silent" test that routed real incoming Voice traffic through GPT-Live while throwing away the output, which surfaced load-related issues that synthetic tests hadn't caught.
Justin Uberti, head of Realtime AI at OpenAI, told InfoQ that the team adopted the principle that "the voice must flow," so they limited the live path to media delivery and inference and moved everything else outside. He explained that delegation to frontier models and feeding voice data to safety systems both required specific design attention, but optimizing those pieces in isolation proved easier than embedding them in the critical path. On the choice to extend WebRTC rather than adopt newer standards, Uberti said the protocol offers a proven low-latency stack with error recovery, and while efforts like RTP over QUIC show promise, they currently supply only the transport layer and lack features such as GCC congestion control and RTT-aware path selection. The silent test ran with the application service in a read-only mode and no user credentials, so incoming audio fed the model but the output was discarded. Uberti noted that unlike conventional load tests using scripted or synthetic speech, the silent test captured the variety and global spread of actual Voice sessions and revealed that system performance deteriorated under load in ways synthetic tests hadn't predicted—for instance, some GPUs in certain regions weren't colocated with the CPUs feeding them, introducing unexpected delay.
The architecture reflects a trade-off between consistency and flexibility. By isolating the media and inference loop, OpenAI can focus optimization work on the components that determine whether a conversation feels natural, without risking slowdowns from less urgent tasks. Stateful inference anchors each session to a specific instance, giving the model continuity, yet the ability to migrate context in real time means the system can spin capacity up and down as demand shifts. The WARP enhancements—SPED, DTLS 1.3, and SNAP—each deployed independently, letting the team verify benefits one at a time and delivering those improvements to existing WebRTC applications without code changes. Uberti expects WebRTC and QUIC to converge over time, removing the need to pick one over the other. The silent test's discovery of hardware colocation problems that went unnoticed in synthetic benchmarks underscores the value of validating fixes against production patterns before launch day. The result is a system designed to prioritize what users hear first, while letting everything else wait its turn. For organizations building conversational AI at scale, the implication is that architectural boundaries matter as much as raw speed—deciding what stays in the hot path determines whether latency targets hold under real-world strain.