UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos

The Core Problem and Solution

Swapping faces and voices in talking videos is harder than it sounds. You need to change who appears on screen and who's speaking, but keep everything else intact—the head movements, lip sync, the background, the words being said, even the precise timing between audio and video. Current methods train separate models for face-swapping and voice-conversion, then stitch them together. That's a recipe for misalignment: the new face might not match the new voice's cadence, or the lips won't sync properly.

UniSwap tackles this with a single unified model that handles both appearance and voice simultaneously, using a diffusion transformer—a neural network trained to gradually refine noisy data into clean output. The key insight is that by optimizing audio and video together in one framework, you can enforce consistency between them naturally.

How It Works

The authors face a practical training obstacle: there aren't many videos available where two different people perform the exact same speaking act (same words, same head motions, different faces and voices). To work around this, they use a clever swap-and-reconstruct pipeline. They take real videos, artificially strip out the identity information from both the face and voice, then train the model to reconstruct the originals. This synthetic pairing problem is much easier to scale than collecting real cross-identity pairs.

Building the model involves three progressive refinements:

1. In-context Pretraining teaches the model to jointly replace appearance and voice by learning from the swap-and-reconstruct targets.

2. Conditional Streaming Adaptation optimizes the model for real-time generation. Normally, diffusion requires computing attention over the entire sequence, which is slow. The authors use KV-caching (storing pre-computed attention values) and block-causal masking (each output block only looks back at previous blocks), enabling streaming generation—you can start outputting video frames before seeing the whole input.

3. Efficient Self-forcing DMD (Diffusion Model Distillation) cuts sampling time dramatically. Instead of 30 denoising steps per video block, they reduce it to 3 while maintaining quality. Self-forcing means the model learns to use its own predictions as inputs during training, reducing the gap between training and inference.

A final trick, Feature-RoPE Decomposition, solves a position-tracking problem: as videos get longer, cached position embeddings can drift outside the range the model saw during training. They decompose these embeddings to keep them stable across long sequences.

Why It Matters

Video deepfakes are a dual-edged concern—one edge is misuse, the other is legitimate creative applications (dubbing, character replacement in film). Having a single, efficient, streaming-capable audio-visual swap method opens new possibilities for real-time editing and consistent synthetic media.

Experiments show UniSwap maintains strong audio-visual synchronization and identity transfer while being fast enough for practical use. The streaming design means you don't need the entire video in memory before starting synthesis. Most critically, unifying audio and video in one model sidesteps the synchronization failures that plagued prior two-model approaches.