UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations

The Problem: Hallucinations Hide in Plain Sight

Large vision-language models (LVLMs) like GPT-4V confidently generate plausible-sounding text about images that isn't actually there. Detecting which tokens in a response are hallucinated matters because it lets you fix the problem surgically—reject just the bad tokens, not the entire answer. But current detection methods either require expensive retraining, rely on external fact-checkers that miss what the model itself "knows," or squash the model's rich internal signals into crude statistics, losing spatial structure and sequential context.

The authors introduce UniProbe, a lightweight detector that learns to spot hallucinated tokens by reading the LVLM's computational traces—the attention patterns, activations, and dependencies the model generates when producing each word.

How It Works: Three Views of One Computation

When an LVLM generates a response, it processes three kinds of tokens in parallel: image patches (what it sees), query tokens (what you asked), and response tokens (what it's generating). Attention weights between these tokens reveal which parts of the image influenced which words—information baked into the model's "reasoning."

UniProbe builds a directed graph where nodes are these three token types and edges are attention weights. Then it runs this graph through three complementary neural modules in alternating sequence:

1. Graph Neural Network (GNN): Reads relational structure—how tokens reference each other through attention. Captures which image regions "support" which response tokens.

2. Vision Transformer (ViT): Processes the 2D spatial layout of image patches. Understands whether hallucinated content clusters in regions the model attended to weakly or not at all.

3. GRU (Gated Recurrent Unit): Tracks sequential order of generated tokens. Catches inconsistencies—tokens that contradict earlier claims or appear out of context.

By interleaving these three, spatial, relational, and sequential evidence reinforce each other. The detector sees both the trees (individual attention links) and the forest (coherent visual and linguistic patterns).

Practical Gains and Real-Time Deployment

UniProbe works on frozen LVLMs—no expensive retraining. It also includes two deployment variants:

• Streaming variant: Runs during decoding. As the model generates each token, UniProbe flags it as hallucinated or real and can trigger resampling on the fly. This cuts object hallucinations by up to 55% with only a 1.06× slowdown.

• Self-adaptation: The detector aligns itself to the LVLM's own generation distribution, avoiding the brittleness of detectors trained on different models.

Across multiple LVLM backbones and benchmarks, UniProbe achieves state-of-the-art token-level and object-hallucination detection. The ~6% latency cost for real-time correction is acceptable for high-stakes applications like medical imaging or autonomous systems where hallucinations carry real consequences.

The key insight is simple but powerful: hallucinations leave traces. By preserving and interleaving multiple structural views of those traces—graph topology, visual geometry, and sequence order—the detector captures what isolated hand-crafted features miss.