I Trained My Own 30M Parameter AI to Run on an ESP32

Summary

The presenter, Norina, trained a small decoder-only Transformer that predicts the next token based on preceding text. The model was designed for chatbot-style language and conversation, with 12 Transformer layers and a vocabulary of roughly 6,000 tokens.

Training data came from publicly available datasets and was tokenized using SentencePiece. The model was trained in batches using cross-entropy loss, backpropagation, and gradient clipping. Norina reports that its outputs improved during training, with conversation validation loss falling from 8.81 to 2.

To fit the model on an ESP32, most of its weights were quantized from 32-bit to 4-bit, while some remained at 16-bit to help preserve accuracy. The subtitles state that the model had 3.5 million parameters and would require about 122 MB at 32-bit precision. This conflicts with the video title’s claim of 30 million parameters.

ESP32-S3 DevKit

The model ran locally on an ESP32-S3 DevKit with 32 MB of flash and 16 MB of PS RAM, generating about 1.4 tokens per second. It produced meaningful answers but struggled with complex questions. Flash storage holds the weights, while PS RAM is used for temporary data.

XIAO ESP32S3

Norina also tested a XIAO ESP32S3 with 8 MB of flash and 8 MB of PS RAM. Since the full model would not fit, its weights were read from a microSD card. Double buffering and use of both processor cores helped manage the slower storage, but performance dropped to about 0.74 tokens per second.

Alternatives

The presenter demonstrated or described two simpler approaches:

• Google Gemini API: The ESP32 sends requests to Google’s servers and displays the returned answers.
• A locally hosted model: Run a model on a computer and connect the ESP32 to it through an API. The subtitles refer to this option as “Llama.”

Demonstrations and Further Material

The video demonstrates training progress, quantization, and sample responses from the model running on both ESP32 boards. It also outlines how to connect an ESP32 to Gemini or to a locally running model.

Norina says setup instructions and additional material will be published on GitHub.