DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation

The Core Problem and Solution

Large language model agents can tackle complex tasks by reasoning over multiple steps, but they need huge models, massive context windows, and repeated inference calls—making them impractical for phones or edge devices. The authors show that you can shrink these agents dramatically without losing capability by borrowing knowledge from a larger teacher model in two parallel ways.

DuoMem transfers problem-solving ability from a 72B-parameter teacher to a 4B student, boosting the student from 4.3% to 77.9% task success on ALFWorld (a benchmark where agents navigate virtual environments and manipulate objects). The student then runs 3× faster than the teacher while staying nearly as capable, opening the door to real-time deployment on resource-constrained hardware.

How Dual-Space Distillation Works

The framework operates on two fronts simultaneously.

Context-space distillation is the more direct transfer: instead of having the student generate its own reasoning chain from scratch, the teacher solves the task first and records its step-by-step memory or reasoning trace. The student then uses that pre-computed teacher trajectory as context prepended to its input, effectively learning to follow and extend better reasoning paths. This sidesteps a core bottleneck—the student doesn't waste inference on bad reasoning steps.

Parameter-space distillation fine-tunes lightweight LoRA adapters (low-rank modifications that add only a few million parameters) on successful teacher trajectories. LoRA is a parameter-efficient technique: instead of updating a model's weights wholesale, it learns small weight adjustments that can be added or removed without retraining. The student learns to mimic the teacher's decision-making by example.

Critically, both axes are complementary. The context-space approach gives the student immediate access to good reasoning; the parameter-space approach lets the student internalize and generalize that knowledge. Together they close most of the performance gap between a tiny model acting alone and a massive one.

Why This Matters

The results are striking in practical terms. On ALFWorld, the 4B student reaches 77.9% success versus the teacher's 87.1%—a performance cliff that would otherwise require deploying the 72B model. But the student uses fewer than 10M trainable parameters (just LoRA weights) and a few megabytes of cached teacher memories. Those are constraints that fit on an edge device. The 3× wall-clock speedup matters for real-time systems where latency is a constraint.

The authors validate their approach across eight models from 2B to 72B parameters and show both distillation axes contribute meaningfully. This suggests the framework is robust and not an artifact of one specific model size.

The practical implication is clear: you don't need to deploy the giant model to get near-giant performance. For a class of procedural reasoning tasks (navigation, manipulation, planning), you can offload the heavy thinking once—the teacher solves it and saves its reasoning—then let lightweight students on devices learn from those solutions. This splits deployment cost between a one-time teacher inference phase and many lightweight student inferences, which is economically and environmentally viable for edge AI.