Taste-aware music retrieval from audio embeddings

Taste and Sound Have a Measurable Link—Now in Music Retrieval

Psychology has long established that sounds evoke taste sensations: a high-pitched tone feels "sharp," a low rumble feels "heavy." The authors show this crossmodal link can be quantified and used to build a music search system. Their core claim is striking: machine learning models trained on audio can predict how music will be perceived along five basic taste dimensions (sweet, salty, sour, bitter, umami) with accuracy that exceeds a typical human listener's consistency.

How They Built It

The researchers started with a perceptually validated dataset—music samples rated by humans for taste associations—and tested ten different audio encoders (models pretrained on sound analysis tasks from the HEAR benchmark families). Rather than retraining these frozen encoders, they added a shared multi-task regression head on top to predict all five tastes simultaneously.

The key technical move was gated late-fusion, a configurable variant that combines predictions from multiple encoders before the final output. The idea is simple: take each encoder's learned representation, weight them intelligently based on the task, and merge them to reduce error.

They evaluated performance using two metrics: absolute error (how far predictions deviate from ground truth) and rank correlation (how well the predicted taste profile orders a set of songs the same way human raters would).

The Results Rewire Expectations

On held-out test songs, the best model achieved a macro RMSE of 0.13—meaning predictions deviated from consensus by roughly 0.13 units on the rating scale. Crucially, this is less than half the deviation of an average human rater (0.28), suggesting the model captures genuine consensus better than an individual listener. It also beats previous baselines (0.219).

A surprise: on absolute error, the ten encoders performed statistically similarly. A single VGGish encoder (a standard audio feature extractor) matched the performance of complex fusion systems. However, gated late-fusion showed its value in rank correlation, reaching 0.724 versus 0.666 for simpler approaches—meaning it ranked songs by taste far more reliably.

When operationalized as a retrieval index, their taste-space predictions ranked a 309-song pool far more faithfully than a CLAP-text baseline (which performed at chance level), suggesting audio-based taste is more predictive than text-based descriptions.

Why This Matters

This work bridges cognitive science and information retrieval. Taste-audio associations were documented in psychology but essentially ignored in music search systems. By creating a perceptually grounded benchmark and demonstrating that frozen encoders can capture these associations, the authors open a new retrieval dimension—one where users could theoretically search for music that "tastes sweet" or "tastes salty," enriching how we explore music beyond genre, mood, or acoustic properties.

The finding that the model tracks group consensus better than humans is particularly striking: it suggests the audio-taste link is real and robust enough to extract computationally, even without domain-specific retraining.