Context-Aware RL for Agentic and Multimodal LLMs

## The Problem: LLMs Miss the Details

Large language models struggle with a specific but critical failure mode: they can't reliably find the needle in the haystack. When answering requires spotting a single crucial line in a tool output or noticing a subtle visual detail, LLMs often stumble—even when they have all the information in front of them. This matters because real-world tasks like debugging code or analyzing complex images demand this kind of fine-grained attention.

## ContextRL: Learning to Ground Answers in Evidence

The authors' insight is to train models not just to produce correct answers, but to *locate and select the supporting evidence*. Rather than supervising the final response directly, ContextRL teaches the model through an indirect auxiliary objective: given a query and an answer, choose which of two highly similar contexts actually supports that pair.

Think of it like teaching a student not only what the right answer is, but forcing them to justify *why* by identifying the exact passage or image region that proves it. This forces the model to develop fine-grained grounding—the ability to pinpoint decisive pieces of information.

The authors constructed two separate datasets to test this idea:

**Coding agents:** They collected ~1,000 contrastive pairs where contexts are execution trajectories (traces of a program running). Using condition filtering, they created pairs of similar but distinct trajectories and trained agents to pick the one matching the query-answer pair.

**Multimodal reasoning:** For visual tasks, they built ~7,000 pairs where contexts are images. They used generative editing (subtly altering images) and similarity search to create pairs of nearly identical images, one supporting a given question-answer pair and one not.

## Results and Proof of the Method

ContextRL yields modest but consistent gains: +2.2% average improvement over GRPO (a baseline RL method) on five long-horizon reasoning benchmarks, and +1.8% across twelve visual QA benchmarks. The improvements aren't dramatic, but they're systematic.

Critically, the authors test whether gains come from the *objective* or just from having more diverse data. They compared against baselines that use the exact same contrastive contexts but format them as standard query-context-answer triplets (not as a selection task). These data-augmentation baselines provided little to no gain, proving that the context-selection objective itself drives the improvement—not the contrastive data alone.

## Why It Matters

This work identifies and addresses a real limitation in how we train LLMs. Standard supervised learning and even standard RL optimize for the right answer but don't explicitly incentivize the model to identify *why* that answer is correct. ContextRL is a surgical intervention: it adds a lightweight auxiliary task that encourages the kind of evidence-grounding that real reasoning requires. The fact that similar data used differently produces no benefit is strong evidence that the method works because it changes how the model learns to attend to context, not because it simply adds training examples.