Daedalus-150M: A Convolution-Attention Hybrid Designed for CPU Inference

Why CPU Inference Demands a Different Design

Most small language models start as scaled-down versions of large ones. Daedalus-150M inverts that logic: the authors fixed the CPU constraint first—4-bit weights, a single CPU, processing one token at a time—then architected around it. The result is a 150-million-parameter hybrid that outperforms several larger models trained on 3–6× more data, while decoding 1.76× faster at realistic conversation lengths.

The key insight is brutal: a traditional attention mechanism must cache every previous token in memory, and that cache grows indefinitely with conversation length. On a CPU with tight memory bandwidth, this becomes a bottleneck. The authors' solution: use full attention in only 6 of 18 blocks; replace the other 12 with short convolutions that maintain a memory window of exactly two timesteps, no matter how long the context gets. Two-thirds of the network never touches a growing cache.

How the Hybrid Works

In convolution layers, each token's representation depends only on nearby past tokens (via a sliding window), not the entire history. This radically shrinks memory traffic. The full-attention blocks—kept where they matter most—provide the global context that convolutions alone cannot capture. The architecture thus trades some representational flexibility for dramatic efficiency gains on the target hardware.

The authors trained from scratch on 59.9 billion tokens. Their model scored 47.31 on a five-task benchmark against a pre-announced threshold of 42.20—a meaningful margin. It beat GPT-2-124M, Pythia-160M, OPT-125M, and GPT-neo-125M despite those baselines seeing three to six times more data. It even exceeded MobileLLM-125M's published score, which was trained on one trillion tokens.

To isolate architecture from training recipe, the authors trained an identical-size all-attention model on the same 59.9B tokens with the same hyperparameters. They locked in the scoring metric before running either model. The hybrid won by 0.81% on the primary metric, matched the all-attention baseline on downstream tasks, produced a 6.3% smaller 4-bit file, and decoded 1.76× faster at 2048 tokens of context. Crucially, the speedup grew with context length—near zero when empty, substantial when full. A simple bandwidth calculation predicted only 1.17× faster, so the gap cannot be explained by sheer data volume savings alone.

What Matters, What Didn't

The speed gain scales predictably with context, exactly matching the mechanism's predictions. This rules out the chance that Daedalus is simply a leaner model that happens to be faster.

The authors also report failures. Quantizing to 4 bits cost roughly half a percent in quality—unavoidable. About half the convolution channels ended up inactive, impossible to prune away without retraining. The vocabulary was oversized for the model's parameter budget.

The result matters because it shows that CPU-first design, not just post-hoc compression, can yield real gains. For edge deployment and offline inference, this matters. The speedup is not free—it requires architectural co-design with the hardware target. But done right, it beats the brute-force alternative of training large and squeezing down.