LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

The Problem: Prompts Can't Keep Up with Improving Policies

Most reinforcement learning systems that train language models to follow instructions rely on an LLM judge to score policy outputs. The judge uses a rubric (a scoring guideline) to decide which responses are better. Recent work has adapted these rubrics as the policy improves, but the prompts themselves—the instructions the policy is trying to follow—stay frozen in place. This creates a critical mismatch: as the policy gets stronger, the fixed prompts become too easy, and the judge can't distinguish between good and bad responses anymore. Without that discrimination, the reward signal collapses, and training stalls.

How LLM-as-a-Tutor Works

The authors' insight is to repurpose the judge itself. Instead of just scoring, the LLM becomes a tutor that simultaneously diagnoses and adapts prompts in real time.

The system works in two roles:

Examiner mode: The LLM compares pairs of policy rollouts (model outputs). When both responses are roughly equal in quality, the examiner flags the prompt as non-challenging—too easy for the current policy level. This is the diagnostic step.

Generator mode: Once a prompt is flagged, the LLM appends atomic constraints to it. Rather than rewriting the entire prompt (as prior methods do), this append-only design adds specific, discrete requirements that incrementally raise difficulty. For instance, if a prompt asks for a story, an atomic constraint might be "must include exactly three plot twists" or "written in iambic pentameter."

The append-only design matters because it guarantees monotonic difficulty increase: each constraint layer makes the task strictly harder, so the policy's improvement naturally stays synchronized with prompt difficulty. No external scheduler is needed.

Why It Matters

On three complex instruction-following benchmarks, the method outperforms both baselines that ignore policy state and prior work that adapts rubrics or rewrites prompts. The consistent wins suggest that prompt adaptation was a blind spot in policy-aware RL.

The practical payoff is a self-calibrating training loop. As the policy learns, prompts automatically become harder to match its capability. The judge always has a clear signal about which rollouts are better. This eliminates a common failure mode in non-verifiable RL: reward collapse due to prompt-policy misalignment.

The term "non-verifiable" matters here—these are tasks where you can't write down a ground-truth answer ahead of time. Instead, you rely on a learned evaluator (the LLM judge) to decide correctness on the fly. Instruction following (e.g., "write a poem about climate change") exemplifies this. In such settings, keeping the judge's signal sharp is essential, and prompt difficulty is a previously overlooked lever for doing so.