Temporal Multi-Signal Fusion for Token-Level Hallucination Detection

The Core Problem

Token-level hallucination detectors fail at their core job: they score each word in isolation, missing exactly when a language model is confidently producing falsehoods. A model may sound plausible while being entirely wrong, and single-token signals can't catch that. This paper reframes hallucination as a span-level phenomenon—mistakes don't happen in isolation, they unfold across sequences—and detects them by treating the task as sequence labeling rather than independent classification.

Method: Temporal Fusion

The authors build a 33-dimensional feature stream that combines three orthogonal signals: text statistics (e.g., token frequency, position), Natural Language Inference scores (whether generated text is entailed by source documents), and language model surprisal (how unexpected each token is under a reference model). Crucially, they use no access to internal model activations, making the approach applicable to closed-source systems like GPT-4.

These features feed into a Bidirectional Gated Recurrent Unit (BiGRU), a recurrent neural network that reads text left-to-right and right-to-left simultaneously. The bidirectional flow lets confidence signals propagate from unambiguous tokens to nearby uncertain ones within a hallucinated span—the key insight. Where a logistic regression baseline treats each token independently, the BiGRU learns how evidence accumulates over time.

Why It Matters

On RAGTruth (a standard benchmark for hallucination detection), the BiGRU reaches 0.840 AUC, an 11-point absolute gain over the independent baseline (p = 0.002). Statistically rigorous testing across 10 random seeds confirms this isn't noise.

The decomposition analysis reveals something surprising: most of the gain comes from temporal order, not raw model capacity. Shuffling the feature sequence drops performance dramatically, while swapping the BiGRU for other architectures—Mamba (a modern state-space model) or Transformers—yields a consistent 0.845 ceiling. This pinpoints the bottleneck: it's the features, not the model.

The detector generalizes cleanly. Trained on one set of models, it loses less than 4% AUC on text from unseen language models, and it works on closed-source systems since it only reads the generated text and external signals (no model internals). This makes it practical for real-world deployment.

Why This Approach Works

The temporal framing matches how hallucinations actually behave in text. A single implausible word might be fluent and unsurprising in context; what betrays it is the pattern—neighboring tokens that collectively violate entailment or factual grounding. By fusing three independent signals and letting a recurrent network learn their interaction over spans, the method captures this pattern without needing access to the model's confidence scores or hidden states.

The practical upshot: you can deploy this detector against any text-generating system, without retraining, and it will catch hallucinations that single-signal baselines miss.