Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing

The Problem: Hidden Style Inconsistency

Modern automatic speech recognition (ASR) systems are trained on messy data—some transcripts capture exactly what was spoken (verbatim), including stutters and false starts, while others clean up the speech into what was intended to be said. This inconsistency isn't controlled; the model doesn't know which style to produce, leading to unpredictable outputs. The authors show this creates three concrete problems: the model's decisions become unstable, evaluation metrics like word error rate (WER) become misleading (up to 60% of measured WER comes from style mismatch rather than genuine errors), and word-level timing information becomes unreliable—critical when you need to know precisely when each word was spoken.

How They Activate Transcription Style

The key insight is that modern ASR models already learn both styles internally; they just can't be reliably controlled. The authors introduce a method using coverage-aware decoder task tokens—essentially learned control signals that tell the model which transcription style to use. These tokens are trained on parallel data where the same utterance has both a verbatim and intended transcription.

The method works in two stages. First, they train on English-only data using these task tokens to condition the decoder. Then they fine-tune with supervised cross-attention fine-tuning, which teaches the model to better align its internal attention mechanisms to word boundaries, improving timestamp accuracy on disfluent speech.

Results That Span Languages and Tasks

Despite training only on English, the model achieves a German disfluency F1 score of 79% in a zero-shot setting (no German training), jumping from a 10% baseline. When the model is fully fine-tuned on English, it outperforms all baselines on three fronts: verbatim transcription accuracy, disfluency detection (recognizing speech errors), and maintaining quality in intended-mode (cleaned-up) output across both English and German.

The cross-attention fine-tuning improves word-level timestamps beyond what forced-alignment (a standard post-processing technique) can achieve, which matters for applications requiring precise temporal information—video subtitles, transcription editing, or analyzing speech timing patterns.

Why This Matters

Current ASR evaluation conflates two different problems: actual speech recognition errors and inconsistent transcription choices. By separating these concerns and making transcription style controllable, the authors enable more honest evaluation and more reliable systems. The timestamps improvement is significant for downstream applications that depend on precise word timing.

They also propose verbatimize, a new task for creating and enriching speech datasets with high-quality verbatim annotations. This addresses a practical bottleneck: generating clean training data for controlled-style ASR at scale.

The work ultimately shows that ASR's latent inconsistency isn't a hard problem requiring fundamentally new models—it's a control problem solvable with task tokens and targeted fine-tuning.