Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models

The Core Finding

Hypernetworks—small networks that generate weights for larger networks—can reliably inject factual knowledge into LLMs during training, and their effectiveness follows predictable power-law scaling. Crucially, they outperform standard methods like LoRA fine-tuning and full fine-tuning when generalizing to out-of-distribution facts, and this advantage grows as the hypernetwork scales up.

How It Works

The approach is conceptually clean: instead of directly fine-tuning an LLM's weights to memorize facts, the authors train a hypernetwork to generate a LoRA adapter (a lightweight modification layer) customized for a given corpus of facts. The hypernetwork takes a set of facts as input and outputs adapter weights that, when plugged into the target LLM, enable it to answer questions about those facts.

This design has a key advantage—it separates the knowledge injection mechanism (the hypernetwork) from the base model's general reasoning abilities. This isolation allows the authors to study how hypernetwork architecture itself affects knowledge absorption, rather than conflating improvements with changes to the main model.

The authors tested this on MegaWikiQA, a new dataset of tens of millions of multi-hop question-answer examples spanning 39 domains, all derived from Wikidata5M. They then measured three things: training loss, reasoning accuracy on in-distribution facts, and generalization to out-of-distribution (OOD) facts the hypernetwork never saw during training.

Why This Matters

The scaling results are striking. First, hypernetwork performance follows predictable power laws across multiple architectural dimensions—depth, width, and target model size. This predictability is valuable for practitioners: it means you can estimate performance before training and allocate compute efficiently.

Second, hypernetworks show steeper scaling exponents than LoRA fine-tuning and full fine-tuning in OOD evaluations. In plain terms: as you make the hypernetwork bigger, its ability to generalize to new, unseen facts improves faster than competing methods. This suggests hypernetworks aren't just memorizing the training facts but learning something more general about how to inject knowledge.

This work addresses a real bottleneck in LLM deployment. Organizations need to inject domain-specific facts reliably without expensive full retraining. LoRA has become the go-to method, but the authors show a more principled alternative exists—one with better generalization properties at scale.

The MegaWikiQA dataset itself is a useful contribution: tens of millions of multi-hop examples provide a realistic, large-scale benchmark for evaluating knowledge injection methods.

By establishing the first empirical scaling laws for hypernetwork-based knowledge injection, this paper moves hypernetworks from an experimental curiosity into a tractable engineering tool for practical LLM adaptation.