ARC: Fair Relative Advantage Comparison in Open-Ended Real-World Interaction

The Problem: Reward Models Break Fair Comparison

When agents interact with users in open-ended settings, many behaviors are equally valid. An assistant might answer immediately, ask clarifying questions, provide status updates, or confirm before taking action. Traditional reinforcement learning (RL) compares rollouts—sampled trajectories—within a group to compute relative performance. But here's the catch: if rollouts in the same group use different interaction styles, they're not truly comparable. A reward model trained on human preferences might favor one style (say, direct answers) over another (say, asking clarifications). This preference distorts the learning signal, steering agents toward reward-preferred behaviors rather than context-appropriate ones. The authors call this a reward fairness problem.

The Solution: Condition on Strategy

ARC (Advantage Regularization via Conditioning) fixes this by grouping rollouts not just by task, but by interaction strategy. Instead of mixing "ask clarification" and "answer directly" trajectories in the same comparison group, the method separates them. Each strategy gets its own conditioned advantage estimate—a way to measure "how much better is this rollout within its strategy class?" This ensures reward models aren't accidentally biased against valid alternative behaviors.

The training recipe combines three elements: strategy-conditioned grouping, hybrid rewards (blending multiple reward signals), and entropy regularization (encouraging exploration of diverse behaviors). The authors don't fully detail the mechanism in the abstract, but the core idea is preventing the reward model's stylistic preferences from masking genuine performance differences.

The Framework: Decoupled Communication

The authors introduce a new interaction paradigm called \inter that separates user-facing communication from internal reasoning and tool use. This matters because it lets agents think privately before responding—reducing latency without compromising response quality. They built \inter-86K, an annotated corpus of 86,000 examples with strategy labels, supporting both supervised learning and RL training.

Why It Matters

Two key results demonstrate impact:

1. Benchmark gains: ARC substantially improves τ and τ² scores on tool-use benchmarks, the standard metrics for multi-step agent tasks.

2. Speed gains: The \inter paradigm reduces time-to-first-token (latency before the user sees anything) from 4.91 seconds to 1.27 seconds compared to a "think-style" baseline—meaningful for real-time interaction.

The deeper insight is architectural: open-ended agent learning isn't just about what agents are rewarded for, but whether how they're compared during training is fair. By decoupling reward modeling from behavioral comparison, ARC removes a hidden bottleneck. The work suggests that many interactive RL failures stem not from weak rewards or poor architecture, but from unfair learning dynamics that pit valid strategies against each other.

This addresses a real gap in deployed systems. Current assistants often get locked into single interaction patterns (always verbose, always brief) because training conflated style with quality. ARC's fairer comparison should allow agents to adapt their approach to context.