Characterizing Warp Divergence from Pascal to Blackwell

The Stable Cost of Warp Divergence Across GPU Generations

A widely held assumption about modern NVIDIA GPUs—that Independent Thread Scheduling (ITS), introduced in Volta, fundamentally changed how warp divergence is handled—appears to be wrong. Testing Pascal, Ampere, Hopper, and Blackwell GPUs, the authors find that the programmer-visible performance cost of warp divergence has remained stable across all generations. When a warp (a group of 32 GPU threads executing in lockstep) encounters a branch where different threads take different paths, execution time grows linearly with the number of divergent paths k, following a simple formula: time ≈ sk. Warp execution efficiency drops to 32/k (meaning only 32/k percent of threads do useful work). Crucially, this cost model holds on Pascal too—predating ITS by several generations—suggesting the behavior is far older and more fundamental than previously understood.

How the Cost Model Works

When threads in a warp diverge at a branch, the GPU cannot execute all paths simultaneously. Instead, it serializes execution: it runs one path while other threads wait (masked off), then runs the next path, and so on. No path takes longer than any other, and there is no hidden "super-linear penalty" when many paths exist. The cost is simply proportional to the number of active paths.

One way to eliminate this cost entirely is through predication—the compiler converts the divergent branch into conditional execution where all threads run all code paths but selectively ignore results using per-thread flags. Predication removes serialization but at the cost of executing dead code.

The occupancy of the GPU (how many warps are active) does not affect this penalty, meaning the divergence cost is independent of how many other tasks are running.

The Hidden Architectural Evolution

Beneath this stable surface, NVIDIA has substantially redesigned the machinery that implements warp reconvergence (the mechanism that allows divergent threads to rejoin). Pascal uses a per-warp stack of SSY/SYNC instructions to track reconvergence points. Ampere and Hopper switch to barrier-register instructions. Blackwell introduces a two-tier barrier classification (uniform-branch vs. other instructions) and explicit partial-mask synchronization—features absent from earlier generations.

Despite these changes, controlled bit-flip experiments show that Blackwell's new barrier classification has no observable runtime effect in the authors' tests, suggesting it is purely a static compiler decision with no dynamic performance impact.

Deferred reconvergence—where threads wait to rejoin at a point beyond the immediate post-dominator of a divergent branch—dropped from 29 cases on Ampere to only 2 on Blackwell, indicating the compiler is becoming more aggressive about reconverging threads early.

Why This Matters

For GPU programmers and compiler writers, the stable cost model is good news: the performance of divergent code is predictable and follows simple rules, even as the underlying hardware evolves. However, the architectural changes reveal that NVIDIA continues to refine how it implements control flow. Understanding that the visible penalty is decoupled from the internal machinery allows developers to reason about performance without needing to retune for every new GPU generation—as long as they stick to the basic cost model that has held since at least Pascal.