Token Time Continuous Diffusion for Language Modeling

The Core Innovation

Diffusion models generate text by iteratively refining random noise into coherent tokens. Most existing approaches work in discrete space—sampling all tokens in parallel at each step. This creates a bottleneck: when you speed up generation (fewer refinement steps), parallel sampling breaks down and accuracy crashes. Token Time Continuous Diffusion (TTCD) solves this by working in continuous space and assigning different refinement speeds to different tokens, eliminating the need for parallel sampling entirely.

The key insight: not all tokens should take the same path from noise to clarity. Some tokens—those with high confidence predictions—can move faster. Others need more refinement steps. This per-token timing mimics how humans might write: committing quickly to obvious words while deliberating longer over less certain ones.

How It Works

TTCD operates in a continuous embedding space rather than jumping between discrete token categories. This deterministic approach maps Gaussian noise directly to a final token canvas without additional sampling steps—the model learns a continuous trajectory for each token's refinement.

The crucial addition is per-token times: each token gets its own schedule controlling how quickly it transitions from noise to a final prediction. Tokens the model is confident about receive shorter schedules (fewer refinement steps needed), while uncertain tokens get longer schedules (more refinement opportunities). During generation, tokens influence each other at different rates depending on these schedules, allowing for richer inter-token dependencies during the refinement process.

This design enables conditional generation tasks (like text completion or constraints) to work naturally: tokens that must satisfy external conditions can be given extended refinement time, while free-generation tokens proceed faster.

Results and Impact

The authors trained a 160M parameter TTCD model on OpenWebText, then further distilled it to improve efficiency. At high speedups (the practical regime where discrete models falter), TTCD achieves comparable unconditional generation quality to existing models of similar size trained on the same data. More importantly, it outperforms these baselines on conditional generation tasks—where per-token timing's flexibility shines.

The approach also transfers to structured domains: TTCD shows similar performance gains on Sudoku solving, suggesting the per-token timing mechanism generalizes beyond language.

Why this matters: modern language models prioritize speed during inference. As speedup requirements increase, discrete diffusion models degrade rapidly. TTCD's continuous space + adaptive timing framework offers a practical path to maintain quality while reducing sampling costs—critical for deployment constraints. The per-token timing mechanism also provides interpretability (which tokens needed refinement?) and fine-grained control over generation, useful for constrained decoding and conditional tasks where different parts of the output have different confidence or constraint requirements.