Moving Alphabet: A Controlled Study of Training Data for Text-to-Video Generation

The Problem: We Don't Know What Makes Good Training Data for Video Generation

Text-to-video models have gotten impressive through sheer scale—bigger models, more compute, more data—but almost no one has systematically studied which data actually matters. Real-world dataset curation is messy: you clip raw videos, write captions, pair them up. But how do variations in what you include or how you describe it affect the final model? The authors found this question largely unexplored, so they built a controlled experimental setup to answer it.

The Method: Moving Alphabet as a Controllable Sandbox

Rather than wrestling with real-world video datasets (where dozens of variables change at once), the authors created Moving Alphabet: a synthetic testbed that renders animated letters on a black background. You control everything—font, color, size, position, direction, and speed. This lets you deliberately introduce variation or corruption while keeping everything else constant.

The key insight is that you can corrupt the captions (the text descriptions paired with videos) in reproducible ways: omit details, add noise, or make them vague. Then train models on these deliberately degraded datasets and measure the impact. It's the video-generation equivalent of controlled chemistry experiments.

Three Main Findings

First, diversity and balance matter. If your training videos cluster in one duration or visual style, the model generalizes poorly to other scenarios. A balanced distribution—mixing short and long clips, varied fonts, different speeds—produces more robust models. This seems obvious in hindsight, but it hadn't been quantified before.

Second, caption quality is a bottleneck. When the authors corrupted captions (removing details, introducing errors), model performance dropped sharply. More surprisingly, training efficiency also suffered—it took longer to converge. This suggests text-to-video models are fundamentally limited by how well they can understand video content, not just by architectural choices. Bad captions starve the model of the signal it needs to learn.

Third, downstream fixes only partly work. The authors tested whether you could recover from poor pre-training through classifier-free guidance (a technique that emphasizes text at generation time) or fine-tuning on high-quality data later. Both help, but neither fully compensates for learning on corrupted captions early on. Pre-training data quality is genuinely foundational—you can't patch it later.

Why This Matters

The authors argue for treating pre-training data as a scientific problem, not just a logistics problem. Most progress in generative models has come from scaling models and compute; data has been treated as a commodity to be collected cheaply. But this work shows caption quality and distribution are as important as volume. For practitioners building text-to-video systems, it suggests investing in careful curation and annotation quality pays off. For researchers, it's a call to study data systematically rather than assuming bigger is always better.