The Reward Was in Your Data All Along: Correcting Flow Matching with Discriminator-Guided RL

## The Core Problem: Why Flow Matching Needs Help It Shouldn't

Flow-matching and score-based generative models are trained to predict velocity or score fields—essentially learning how to gradually transform noise into images. The training loss (ℓ2 regression) measures prediction accuracy under the *training data distribution*, treating all errors equally. But this is a proxy mismatch: what makes a generated image good—visual realism, object coherence, semantic meaning—isn't what the loss directly optimizes for. The authors observe that these models then require preference-based RL (trained on human annotations) to recover perceptual quality that should have been learned from the training data itself. This indicates the matching objective and the actual quality metrics are fundamentally misaligned.

## The Solution: Use the Data Itself as the Reward Signal

Rather than ask humans what they prefer, the authors propose **Discriminator-Guided RL (DRL)**, which trains a discriminator to distinguish real data from model samples. The discriminator is trained in a pretrained representation space (like DINO or other frozen embeddings), which constrains it to learn only perceptually meaningful features. The discriminator's logit—its confidence that a sample is real rather than model-generated—serves as the reward signal in KL-regularized RL.

Why this works: the logit estimates the log-likelihood ratio between the data and model distributions, which is theoretically the optimal reward for steering a model toward the data distribution. By using the data directly as ground truth, the method sidesteps the expense and subjectivity of human preferences while targeting what actually matters: alignment with visual and semantic properties.

## Results Across Multiple Architectures

The method was tested on four flow-matching variants: SiT, JiT, REPA, and RAE. Across all backbones, DRL consistently improved image quality metrics without any guidance: - **FID (Fréchet Inception Distance)** dropped dramatically—from 9.38 to 2.62 on SiT, for example. This measures overall image fidelity. - **Semantic-space FD** (using DINOv3 embeddings) fell from 88.2 to 19.3 on SiT, showing improved semantic coherence and object structure.

Beyond guidance-free improvements, DRL also enhanced subsequent preference-based RL fine-tuning. Models trained with DRL first achieved a better Pareto frontier between preference-aligned quality and image fidelity, gaining human preference reward while reducing low-level artifacts like oversaturation and excessive brightness.

## Why This Matters

Generative models often require RL fine-tuning as a separate expensive step, yet the root issue—misalignment between training loss and perceptual quality—persists. By reframing the problem through the lens of data-model discrimination, the authors show that reward signals already exist in the data distribution itself. This is cheaper than human annotation, more grounded than arbitrary preferences, and theoretically sound. The method improves both guidance-free and preference-guided generation, suggesting that better initial alignment through discriminator-guided RL sets models on a firmer foundation before any downstream alignment.