ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models

The Problem: Vision-Language Models Are Easily Distracted

Vision-language models (VLMs)—systems that process both images and text together—tend to be swayed by irrelevant, false, or nonsensical context, much like a person might change their answer if you slip misleading information into a question. This phenomenon, called contextual entrainment, has been studied in text-only language models, but no one has properly investigated whether and how it happens in multimodal systems. The authors argue that simply adapting existing text benchmarks to include images misses something fundamental: in VLMs, false context can pull outputs in ways unique to the multimodal setting.

Why VLMs Are Fundamentally Different

The leap from language-only models to vision-language systems isn't just an engineering upgrade—it's a conceptual shift. Entrainment becomes dual: irrelevant context can now come from the text stream (misleading instructions) or the visual stream (distracting image elements), independently or together. More importantly, VLMs introduce a distinction that text-only models don't face: context that is false of what's depicted yet plausible in the real world. For example, telling a model that a cat is a dog when looking at an actual cat is different from asking a pure language model something it has no grounding for. The visual ground truth creates new failure modes.

The Solution: A Purpose-Built Taxonomy and Dataset

Rather than claiming to measure entrainment in any specific model, the authors introduce ENTRAP-VL—a manually curated benchmark of 1,500 items designed specifically to probe this phenomenon rigorously. The dataset is organized around two axes: how the context associates with the item (whether it's directly related, weakly related, or unrelated) and its relationship to truth (true, false-but-plausible, or nonsensical). This yields eight categories in the textual-entrainment stream (testing how text can mislead) and three in the visual-entrainment stream (testing how visual elements distract). Each condition is constructed around a specific item—the actual image in text-based tests, the actual query in vision-based tests—so the benchmark measures entrainment with precision.

What This Enables

ENTRAP-VL is not presented as a final scorecard for any model. Instead, it's a foundation for the research community to investigate VLM vulnerabilities systematically. The taxonomy clarifies what entrainment looks like across modalities, the dataset provides concrete test cases, and the evaluation protocols make it possible to diagnose where and how these models fail. By releasing the dataset and documentation publicly, the authors create a shared language for studying a problem that will likely grow more important as VLMs are deployed in higher-stakes applications where robustness matters.