Three-Body Scattering for Generative Modeling

The Core Insight

Most generative models—diffusion models, GANs, autoregressive systems—rely on one of three crutches: an adversarial critic to judge quality, a prescribed noise-to-data trajectory to follow, or a fixed factorization order for sequential generation. This paper shows those aren't necessary. Instead, a properly designed energy function can directly supervise a one-step generator without any of these scaffolds. The result: on ImageNet-256, their method achieves FID=2.23 using pixel-space models and FID=1.63 using latent-space models—both in a single forward pass (NFE=1, meaning one network evaluation).

How Three-Body Scattering Works

The key innovation is translating a global energy distance into local, per-sample interactions. Imagine each data point (a "projectile") experiences two forces: attraction toward a real example from the dataset and repulsion from an independently generated fake example. Both source examples are fixed during a forward pass, so computation stays constant per batch size.

Mathematically, the authors connect this scattering interaction to the 2-Wasserstein gradient-flow velocity—the direction that optimally moves the model's probability distribution toward the true data distribution. Crucially, the per-projectile loss depends only on its paired real and fake examples, not on all pairs within the minibatch. This reduces noise compared to prior work like Drifting Models, which compute global interactions across the entire batch.

The method trains on frozen image features (rather than raw pixels), leveraging pretrained encodings to keep one-step generation tractable. A backbone diffusion transformer (DiT) is repurposed as a one-step generator, conditioned on the projectile and supervised directly by the scattering loss.

Why This Matters

One-step generation is valuable for deployment: it's orders of magnitude faster than 50+ diffusion steps or iterative refinement. Yet achieving competitive quality in a single step has been notoriously hard. This work shows that elegant theory—the connection between energy distances and gradient flows—can translate into practical gains.

The paper also provides a "design map" relating diffusion-style supervision, drift-like dynamics (attracting toward data), and GAN-like objectives (repelling from fakes). This conceptual unification suggests the three paradigms are different views of the same underlying principle, rather than fundamentally incompatible approaches.

The results are empirically strong. FID=1.63 on ImageNet-256 with a single step rivals or beats many multi-step methods. The code release enables reproducibility and further exploration of scattering-based generation.

The main limitation—implicit in the framing—is that this approach requires frozen features rather than end-to-end training on raw images, which may constrain applicability to domains without good pretrained encoders. Nevertheless, the theoretical grounding and empirical validation establish scattering as a viable route to fast, high-quality image generation.