Fitting a Neural Network onto a Microcontroller: Quantization, Pruning and Distillation

A model that runs comfortably on a workstation and a model that runs on a microcontroller are separated by roughly three orders of magnitude. The target has kilobytes of RAM rather than gigabytes, no operating system, and a power budget measured in milliwatts. Getting from one to the other is not a single technique but a choice among three, and choosing badly wastes weeks.

Quantization is the first and usually the largest win. Weights trained in 32-bit floating point are stored and computed in 8-bit integers instead, cutting memory by four and often speeding inference by more, because integer arithmetic is what these cores are good at. Post-training quantization takes an afternoon and typically costs a small amount of accuracy. When it costs too much, quantization-aware training — where the model learns with the rounding already applied — usually recovers most of the loss at the price of retraining.

Pruning attacks a different axis: it removes weights or whole channels that contribute little. Unstructured pruning zeroes individual weights and produces impressive sparsity figures that deliver nothing on a microcontroller, because a dense matrix multiply with zeros in it takes exactly as long. Structured pruning removes entire filters or channels, which makes the model genuinely smaller and genuinely faster, and it is the only variety worth pursuing unless your runtime specifically exploits sparse kernels.

Distillation is the least mechanical of the three and often the most effective. Rather than shrinking the trained model, you train a small model to imitate a large one, learning from its output distribution rather than only from hard labels. The small model frequently ends up better than the same architecture trained directly on the data, because the large model's soft outputs carry information about how confusable classes are that the labels do not. The cost is that you need the large model and a training pipeline, so it is a project rather than a step.

Which to reach for depends on which limit you have hit, and the limits are different. If the model does not fit in flash, quantization addresses storage directly. If it fits but inference is too slow, structured pruning removes work. If it fits and runs fast but is not accurate enough at that size, no compression technique will help — that is where distillation, or a different architecture, is the answer. Teams routinely apply all three in sequence without checking which one their problem needs, and then cannot explain which change caused the accuracy they lost.

One measurement usually reorders the whole plan: profile the memory the model needs at runtime, not just the size on disk. Intermediate activation buffers frequently dominate, and a model whose weights fit comfortably can still fail to run because two layers need to be resident at once. That number, taken early, tells you whether you are compressing the right thing.