A developer known as SlvDev has successfully run a small language model on an ESP32 microcontroller costing less than $10, achieving nearly 10 tokens per second—faster than the average person can read. The achievement, documented on GitHub and showcased on the Better Stack YouTube channel, demonstrates that generative AI models can operate on hardware primarily designed for remote sensors, IoT devices, and other embedded applications. While the practical applications remain limited, the technical feat shows how far model compression techniques have advanced.
Making a language model work on hardware this constrained required aggressive optimization. SlvDev used TinyStories, a 28.9 million-parameter model from Microsoft Research that's nearly 10,000 times smaller than typical large language models. Even so, the model initially demanded about 60 MB of memory at 16-bit precision—far exceeding the ESP32-S3's 520 KB of SRAM and 8 MB of pseudo SRAM. Through quantization, the developer compressed the weights from 16-bit to 8-bit precision, cutting memory requirements by 75 percent to just 14.9 MB. Then, borrowing a technique called per-layer-embedding from Google's Gemma model family, SlvDev offloaded roughly 25 million parameters—about 12 MB—to the chip's flash storage. The final configuration keeps only about 2 MB in the chip's PSRAM, with the output head, embeddings, and KV cache residing there while activations run in the 520 KB of SRAM. The result: 9.88 tokens per second from hardware that costs about the same as a fancy cup of coffee.
According to the developer, the achievement won't deliver much beyond "dumb simple pride." TinyStories can generate short, reasonably coherent stories on demand but can't power chatbots, generate code, or run agents. Another model called Barista performs roughly twice as fast but only answers questions about espresso. Both models are simply too small to handle broader tasks. Still, the fact these models run on an ESP32 at all is impressive in itself, the report notes.
The techniques that made this possible—quantization and weight offloading—have broader implications for devices with slightly more resources. Google's Gemma 4-E2B-it, launched in April, uses the same methods to squeeze a 5.1 billion-parameter vision language model into just over a gigabyte of memory with 4-bit weights, or around 500 MB with higher compression. On a Raspberry Pi or smartphone, that model can power local chatbots, orchestrate agents for calendar management, and reduce dependence on OpenAI or Anthropic—assuming users can tolerate occasional hallucinations. The ESP32 demonstration shows the floor of what's technically possible, even if practical applications for microcontrollers remain elusive for now. The path from proof-of-concept to production will depend on whether developers can find use cases where limited capability on ultra-cheap hardware beats no on-device intelligence at all. For makers and embedded engineers, that's a design challenge worth watching.