CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

The Gap: Nobody's Actually Testing Long-Form Video Understanding

Current benchmarks for video-language models—systems that understand both video and text—focus almost entirely on short clips and single-sentence outputs. This misses a crucial real-world capability: can these models generate detailed, paragraph-length descriptions of video content? The authors find that nobody has systematically evaluated this, and they introduce CLIP-CC-Bench to fill that void.

The Benchmark Design

CLIP-CC-Bench consists of 5 hours of movie footage split into 90-second clips, each with an expert-written paragraph (roughly 2–4 sentences) as the ground truth. Unlike databases of isolated clips, this uses continuous movie scenes, which demand coherent narrative understanding rather than frame-by-frame analysis.

The evaluation itself sidesteps a hard problem: movie scenes are complex and admit multiple valid descriptions, so there's no single "correct" answer. Rather than writing rigid metrics, the authors use large language models (LLMs) as judges. Specifically, they embed both the model-generated description and the reference paragraph using five different state-of-the-art embedding models (such as those based on sentence transformers). The ensemble approach—using five models rather than one—reduces bias and increases confidence in the rankings.

They then apply two evaluation approaches:

1. Coarse-grained semantic matching: Do the generated and reference descriptions cover the same high-level concepts and events?
2. Fine-grained semantic matching: How well do the detailed semantic relationships, actions, and object interactions align?

This two-level approach captures both whether a model "gets the gist" and whether it preserves nuance.

Results and Validation

The authors evaluate 17 state-of-the-art video-language models, reporting both individual scores and Borda-aggregated rankings—a voting method that ranks models by their pairwise comparisons rather than absolute scores, further improving robustness.

To validate the benchmark itself, they measure inter-judge agreement (do the five embedding models rank models consistently?) and bootstrap ranking stability (would the rankings hold if you resampled the dataset?). This transparency is critical: a benchmark is only useful if evaluators agree and results are reproducible.

Why This Matters

Video-language models are increasingly deployed for tasks like video summarization, accessibility captioning, and content moderation—domains where paragraph-level accuracy matters far more than single-sentence outputs. Existing benchmarks (typically QA or short-form caption tasks) don't stress-test these capabilities. A model might excel at selecting the right answer in multiple-choice but fail to construct coherent multi-sentence narratives.

By releasing evaluation scripts, model outputs, and aggregation tools alongside the benchmark, the authors enable other researchers to stress-test their models and iterate. This practical contribution—a reusable, robust framework—is as valuable as the initial results. CLIP-CC-Bench fills a genuine gap and provides a foundation for more realistic evaluation of video understanding at scale.