Wnuan: Staged Post-Training for Question Answering over Proprietary Enterprise Knowledge

The Core Finding

Enterprise AI systems face a fundamental tradeoff: adding proprietary knowledge through fine-tuning erodes general reasoning abilities. Wnuan, a three-stage training pipeline, pushes the needle significantly toward solving enterprise question answering while quantifying the cost. On an internal 707-question benchmark (WnuanBench), the approach lifts acceptable-answer rates from 53% to 91.5%, but sacrifices 5.17 percentage points on general-purpose benchmarks.

How It Works

The pipeline has three distinct phases. Stage one extracts training signal directly from enterprise documents, converting raw knowledge into question-answer pairs that align with actual business use cases. This grounds the model in domain-specific information without relying on synthetic data generation.

Stage two applies supervised fine-tuning (SFT) on these task-oriented pairs, but with a critical twist: the authors replay general training data alongside enterprise examples. This mixed approach helps retain broad capabilities while absorbing new knowledge. The 32B model variant jumps from 52.76% to 80.06% acceptable-answer rate at this stage alone.

Stage three tackles the hardest remaining cases using reinforcement learning (RL), specifically targeting errors that supervised learning missed—so-called "residual errors." This refinement pushes performance to 91.51%. The authors show this residual-error sampling strategy outperforms simpler alternatives (random sampling, size-matched pools) by roughly 3 percentage points under equivalent computational budgets.

The Tradeoff Matters

The results reveal an asymmetry worth noting: enterprise gains concentrate in domain knowledge, while losses concentrate in instruction-following capabilities. A 5-point drop on general benchmarks is substantial if the system must handle both specialist and generalist tasks. However, the staged approach is intentional—the authors aren't hiding this cost; they measure it explicitly and show their method is better than alternatives at managing it.

An important validation detail: the authors don't rely solely on automated metrics. A stratified sample from their instruction-following validation set was evaluated by a domain expert, with 90.5% agreement between the automatic evaluation ensemble and human judgment. This gives confidence that the reported numbers reflect real capability shifts, not metric artifacts.

The paper also tests generalization within domain. Results on a same-domain validation set preserve the ordering of methods, suggesting the findings aren't overfit to the primary benchmark. Bootstrap confidence intervals confirm both the residual-error sampling advantage and the general-capability cost are statistically meaningful.

Why This Matters

Enterprise AI deployment is increasingly common but rarely discussed rigorously. Most companies either accept knowledge gaps, or they fine-tune and hope. Wnuan provides a concrete playbook: staged training with replay mechanisms can achieve ~91% accuracy on proprietary Q&A while limiting collateral damage. The quantification—naming exactly what breaks and by how much—is the real contribution. It lets practitioners decide whether the tradeoff is acceptable for their use case, and it shows that staged RL targeting hard cases is more efficient than brute-force fine-tuning.