Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR

The Problem: Uneven Learning Across Domains

When training a language model to reason across multiple domains—math, code, science—the typical approach samples from each domain with a fixed schedule. But reasoning skills don't transfer equally: improving math ability might boost coding performance, while improving coding might barely help science. Current "learnability-based" curricula adapt the schedule based on where the model is learning fastest locally, but they ignore whether those improvements help other domains. This creates inefficiency and leaves performance on the table.

Transfer-Aware Curriculum (TAC)

The authors propose TAC, a bandit-style algorithm that automatically adjusts domain sampling based on cross-domain transferability—how much a gradient step in one domain benefits the others.

The key insight is reusing computation already happening during RL training. During each GRPO (generalized reward-policy-optimization) step, the model calculates:

1. Per-domain advantages: How much the current domain is helping the model improve locally (existing signal, repurposed here).
2. Projected gradients: The actual gradient update computed for each domain. The authors measure alignment between these gradients using geometry—domains whose updates point in similar directions transfer better. Specifically, they estimate transferability via gradient-geometry alignment (how parallel the gradient directions are), extracting this information from gradients already being computed.

TAC then uses a multi-armed bandit framework to sample domains: it favors domains whose updates broadly help the rest of the suite, not just locally improve themselves. The overhead is negligible (<1% wall-clock time).

Why This Matters

The authors test TAC on a six-domain reasoning suite using two model sizes: Qwen3-1.7B and Llama3.2-3B. TAC achieves the best macro-averaged accuracy (equal weight across domains), outperforming:

• Proportional random sampling (baseline)
• Hand-designed schedules (what practitioners currently do)
• Learnability-only bandit (the previous state-of-the-art) by up to 2.8 percentage points—a 10% relative improvement

Ablation studies show the transferability term is critical: removing it causes sharp performance drops. TAC also remains robust when training data is imbalanced, whereas learnability-only curricula get stuck over-sampling easy domains.

The finding is straightforward but powerful: where a domain helps you learn matters as much as how quickly you're learning it. By steering training toward updates that strengthen multiple reasoning capabilities simultaneously, TAC achieves better generalization across the suite without hand-tuning or expensive computation.