Adaptive Multi-Teacher Relational Distillation for Lightweight Speech Emotion Recognition

Speech emotion recognition (SER)—identifying whether someone sounds angry, happy, sad, or neutral—matters for conversational AI, mental health monitoring, and accessibility tools. Large self-supervised models trained on vast audio corpora perform this task well, but they're too memory-hungry and slow for phones, smartwatches, and embedded devices. The authors tackle the compression problem through knowledge distillation: a technique where a large "teacher" model teaches a small "student" model to mimic its behavior.

The Core Problem

Standard multi-teacher distillation has two blind spots. First, when you average predictions from multiple teachers, you treat them equally—but some teachers may be unreliable on a given batch of audio samples. A teacher trained on conversational speech might falter on music-backed dialogue; another trained on clean audio might struggle with noisy clips. Ignoring this variability wastes the student's learning capacity. Second, most distillation methods only match logits (the final layer scores before softmax), which ignores the relationships between samples. If the student learns that two audio clips sound emotionally similar, it captures structure that raw logit matching never encodes.

The Method in Plain Terms

AMRD addresses both issues. The authors introduce adaptive teacher weighting: for each batch, they compute a similarity matrix showing how consistent each teacher's predictions are (how much teachers agree on which samples are similar). They feed this matrix into a one-class SVM—a classical anomaly detector—which flags whether that teacher is behaving coherently on this batch. Teachers that pass the coherence check get higher weights; unreliable ones get downweighted. This happens dynamically, per batch, so a teacher that's reliable on clean audio can still contribute when processing noisy samples.

The second innovation is relational distillation. Instead of forcing the student's logits to match the teacher's logits directly, the authors make the student's similarity matrix (how much it agrees that two samples are emotionally alike) match the teacher's similarity matrix. This preserves the relational structure of the data—the geometry of the emotion space—rather than just the absolute predictions. Both components feed into the training loss jointly.

Why It Works and Why It Matters

On the IEMOCAP and CREMA-D benchmarks—standard emotional speech datasets—AMRD beats single-teacher baselines across four different student architectures (varying sizes and designs). Ablations show that removing either the adaptive weighting or the relational loss degrades performance, confirming both components earn their place.

The practical upshot: on-device emotion recognition becomes viable without sacrificing accuracy. Instead of shipping a multi-gigabyte model to an edge device, practitioners can now distill a lightweight version that runs in real time. This opens doors for offline emotion-aware applications in privacy-sensitive settings where sending audio to the cloud isn't an option. The adaptive weighting also hints at a broader lesson—in multi-expert learning, assuming all experts are equally reliable is naive; context matters.