OneEmo: A Unified Multimodal Reasoning Model for Emotion Perception, Understanding, and Interaction

The Core Claim

Emotion AI works better when you train a single model on multiple emotion-related tasks simultaneously, rather than building separate specialist models. The authors demonstrate this with OneEmo, which handles emotion recognition (perceiving emotions from audio/video), emotion understanding (reasoning about why emotions occur), and emotion interaction (generating empathetic responses)—all in one unified system that outperforms task-specific baselines despite using fewer parameters than commercial models.

The Method: Data + Training Strategy

The authors take two complementary steps to make unified emotion AI work.

First, they built EmoWorld-130K, a dataset of 130,000 examples pairing multimodal inputs (video, audio, text) with "reasoning trajectories"—explicit step-by-step explanations of emotional reasoning. Rather than labeling raw data, they used a human-in-the-loop workflow where annotators refined initial model outputs to create high-quality reasoning chains that teach the model why an emotion matters, not just what emotion is present. This dataset becomes the foundation for supervised fine-tuning, where the model learns that emotion perception, comprehension, and interaction inform each other.

Second, they designed Emo-Chord, a reinforcement learning strategy tailored for multi-task emotion learning. Standard reinforcement learning can destabilize training when juggling multiple objectives with conflicting gradients. Emo-Chord addresses this by introducing "unified multi-task reward allocation"—a mechanism that balances reward signals across the three emotion tasks so that optimizing one task doesn't undermine the others. Think of it as a conductor (the "Chord" in the name) ensuring each instrument (task) contributes harmoniously.

Why This Matters

Emotion perception, comprehension, and interaction are deeply interdependent. If a model recognizes sadness but doesn't understand its cause or generate appropriate responses, it fails as an affective system. Prior work treats these as separate problems; OneEmo treats them as facets of unified reasoning.

The efficiency gain is significant: OneEmo is "significantly smaller" than commercial models like GPT-4V or Claude while remaining competitive. This matters for deployment in real-world applications—healthcare chatbots, educational tools, accessibility systems—where latency and computational cost directly affect user experience. The focus on "reliable and interpretable" affective computing also addresses a real gap: emotion AI that explains its reasoning is more trustworthy than black-box predictions.

Experimentally, the multi-task setup yields mutual benefits during supervised fine-tuning; Emo-Chord then extracts additional gains through reinforcement learning by stabilizing the optimization landscape. The authors validate this across "most benchmarks," suggesting generalization beyond a single dataset or task.

The core insight is elegant: emotion is inherently multimodal and relational. A unified model that learns these connections simultaneously achieves better emotional intelligence than compartmentalized specialists. By pairing this insight with a thoughtfully designed dataset (reasoning trajectories) and a principled RL strategy (Emo-Chord), the authors show that emotional understanding in AI benefits from the same kind of holistic reasoning humans apply.