How Fast Can Reward Models Score? A Systems Study of C++ and PyTorch Inference Runtimes for RLHF

The Real Bottleneck in RLHF Isn't What You Think

Reward models in RLHF pipelines block policy updates—no training step happens until every generated rollout gets scored. The authors built a native C++ inference engine on ONNX Runtime to measure whether faster scoring actually speeds up the full pipeline. Their counterintuitive finding: reward scoring itself is small relative to rollout generation, and the two compete for the same hardware resources. A faster scorer mainly frees capacity for generation to use, not a direct win.

This matters because most teams default to PyTorch eager mode or torch.compile without checking. The authors found the answer depends entirely on hardware and implementation details most people ignore.

What They Measured

The team compared four approaches: PyTorch eager mode, torch.compile, their custom C++ engine, and FastAPI serving. They tested on both CPU and GPU, running repeated independent trials to ensure reliability (single runs don't tell you much with inference variance).

First, they validated correctness against PyTorch. CPU outputs matched to 5.7 × 10⁻⁶ absolute error; GPU to 4.2 × 10⁻³—close enough to trust.

On CPU, their C++ engine decisively beat all baselines. Confidence intervals didn't overlap, meaning the win was statistically clear. PyTorch, torch.compile, and FastAPI all lost.

On GPU, the story inverted. Their engine beat PyTorch eager and FastAPI, but torch.compile came out ahead. Further investigation showed the speedup wasn't from C++ as a language—it came from ONNX Runtime itself, the underlying inference library they wrapped. The choice of runtime matters more than you'd expect.

The Surprise: Batching Beats Everything

Most striking: batching strategy outweighed both language choice and runtime selection. This suggests the conventional wisdom—"use this framework, compile this way"—misses the real lever. How you group requests into batches shapes latency and throughput more than whether you use C++ or Python, or PyTorch or ONNX.

Why does this matter? In a live RLHF pipeline, generation and scoring share GPU/CPU time. Faster scoring on paper doesn't shrink wall-clock step time unless it frees capacity generation can actually use. That depends on batching: if your scorer processes eight examples at a time but your generator produces sixteen, the scorer blocks even if it's theoretically "fast." The authors' work suggests teams should profile their own batching first before hunting for framework speedups.

The practical takeaway: there's no universal best answer. CPU users should consider C++ + ONNX. GPU users should try torch.compile first and measure batching behavior. But the measurement itself—repeated, independent runs with confidence intervals—is what matters most. Single-run timing claims in ML infrastructure are unreliable.