SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation

The Core Innovation

SANA-Video 2.0 achieves high-quality video generation (up to 720p) on a single GPU by mixing two attention mechanisms in a 3:1 ratio. The key insight: pure linear attention is fast but loses expressiveness, while full softmax attention is expressive but scales quadratically with sequence length. The authors' hybrid approach—Hybrid Linear-Softmax Attention—uses gated linear attention for most token interactions (the O(N) scaling advantage) but inserts periodic softmax "anchors" that restore the full-rank interactions linear attention misses. This design recovers softmax-quality outputs at a fraction of the computational cost.

At 480p with 40 sampling steps, the 5B model scores 84.30 on VBench in 13.2 seconds on an H100 GPU—competitive with much larger softmax-based video diffusion transformers while remaining 120× faster than comparable baselines.

How It Works

Hybrid attention mechanism: Instead of choosing between linear or softmax throughout the model, the authors interleave them strategically. Every fourth layer uses gated softmax (which preserves exact token relations), while the other three use gated linear attention (which approximates interactions in linear time). This 3:1 ratio was validated through reduced-resolution proxy studies to balance quality and efficiency.

Block Attention Residuals (AttnRes): A second innovation routes the outputs of completed attention blocks into later linear layers via residual connections. This allows the model to reuse the richer representations computed by softmax anchors deeper in the network, effectively boosting the expressiveness of subsequent linear attention layers. The authors show this increases deep-layer effective rank by ~12%—a proxy for representational capacity.

From-scratch training: Rather than converting a pretrained softmax-only model to hybrid form (a brittle process called linearization), SANA-Video 2.0 learns the complete hybrid architecture end-to-end from random initialization. This ensures the model fully exploits the architecture's design.

Why It Matters

Video generation is computationally brutal. Generating even short, high-resolution sequences involves millions of tokens, making quadratic attention prohibitively expensive. Prior work linearized pre-existing models, but this damages quality. SANA-Video 2.0 shows that hybrid designs trained from scratch can match softmax expressiveness while retaining linear scaling—a significant engineering win.

The practical impact is substantial: the model generates 720p videos in 13 seconds on consumer-grade hardware (single H100). With additional kernel-level optimizations (Sol-Engine), the pipeline reaches 13.06 seconds for 5-second 720p video. The 3.2× speedup of the hybrid backbone over full-softmax at 720p widens as videos get longer, showing the approach scales favorably.

The paper validates the design across two model sizes (5B and 14B parameters) under a unified architecture, suggesting the approach is general rather than a one-off trick. The VBench score of 84.30 is competitive with larger, slower baselines, indicating quality hasn't been sacrificed for speed.

This work matters because it removes a major bottleneck in video generation: you no longer need a data-center cluster to generate reasonable-quality video. The hybrid mechanism and attention residuals are likely to influence future efficient transformer designs beyond video.