AffectFlow-DINO: Uncertainty-Aware Multi-Task Affect Estimation via Conditional Rectified Flow

The Core Claim

Facial affect—emotions, expressions, and micro-movements—is inherently ambiguous in real-world images. Rather than forcing a single prediction, this work shows that modeling uncertainty as a learned distribution dramatically improves estimates of valence, arousal, facial expressions, and action units simultaneously. The approach boosts the multi-task performance metric by 2.6× over the official baseline.

How It Works

The authors start with a frozen DINOv3 vision transformer (a pre-trained image encoder) and add a conditional rectified flow head—think of it as a learnable probabilistic decoder. Standard approaches output one prediction per input. Here, the model instead learns to generate a distribution of plausible answers that respect the genuine ambiguity in facial behavior.

At inference time, the system samples multiple predictions from this learned distribution using Monte Carlo sampling (running the decoder many times with different random seeds) and aggregates them. For deterministic metrics (single best guess), this averaging often beats picking just one prediction. For uncertainty-aware metrics, it naturally captures the "I'm not sure" aspect of the problem.

The system jointly handles three affect tasks:

• Valence-arousal: two continuous dimensions (pleasure and intensity)
• Facial expressions: eight-way classification (angry, disgusted, fearful, etc.)
• Action Units: detecting twelve facial muscle movements

All three heads feed off the same frozen backbone, so the flow model must learn a conditional distribution that varies by task.

Why It Matters

Uncertainty as a feature, not a bug. Real facial images are ambiguous—a subtle expression might be fear or surprise. Forcing a single label wastes information. By learning distributions, the model captures this legitimate ambiguity and uses it to improve predictions.

Severe class imbalance solved without retraining. The challenge data has rare classes (Fear appears in only 3.8% of images). The authors show that adjusting decision thresholds after training recovers performance on rare classes—Fear accuracy jumps from 3.8% to 33.1%—without touching the model weights. This is a practical win for practitioners.

Consistent gains across tasks. The rectified-flow decoder improves the primary metric (valence-arousal CCC-V) by +0.058 over the deterministic baseline. More broadly, the final system achieves P_{MTL} = 1.177 versus the challenge baseline of 0.45, a 2.6× improvement. The ablations show the flow component is consistently helpful, not a one-off win on a particular task.

Frozen backbone, efficient scaling. By keeping the DINOv3 backbone frozen and only training the flow head, the method is computationally lean and avoids retraining a massive model. Fine-tuning the backbone further boosts results, but the gains are already substantial without it.

The technical insight is straightforward: when ground truth is inherently noisy or ambiguous, a generative model that learns what the ambiguity looks like outperforms deterministic regression. For the 11th ABAW challenge—a benchmark on facial affect in unconstrained video—this shift in perspective delivers state-of-the-art results.