AVA-Encoder: Towards Agent-Native Video Representation Learning

The Core Problem and Solution

Video AI agents today struggle to work with raw footage the way human filmmakers do. They lack a structured representation that preserves cinematic intent while remaining interpretable and editable for agentic reasoning. AVA-Encoder solves this by converting videos into knowledge graphs—labeled networks of semantic relationships—that agents can query, understand, and manipulate directly.

The key insight is treating video representation learning as an autoencoding problem: compress a video into a structured knowledge graph, then reconstruct the video from that graph. The gap between original and reconstructed video becomes the training signal, but critically, this signal is expressed in natural language rather than pixel-level loss. An agent receives feedback like "the color grading is too warm" rather than raw numerical errors, making optimization more intuitive for agentic systems.

How It Works

AVA-Encoder's knowledge graph has three layers. Hierarchy and state nodes store structured text descriptions—scene metadata, shot information, dialogue. A linked asset layer then holds the actual generated images, audio clips, and video segments. Typed edges connect these text descriptions to their corresponding assets, forming a graph an agent can traverse and edit.

The training loop has two parts. In the outer loop, the system uses Data-Independent Encoding Policy Pseudo-Training: agents learn a policy for converting raw video into this KG structure using only natural-language feedback about reconstruction quality, without seeing the test data. This produces a reusable "shot-level Agentic Video Encoder" policy. Optionally, at test time, an inner loop performs Data-Dependent KG Representation Refinement, allowing agents to adapt the representation to specific inputs.

This textual-gradient optimization framework is the technical novelty—instead of backpropagating pixel-level reconstruction loss, the system uses language models to translate visual reconstruction differences into natural-language update directions that guide the agent's next encoding decisions.

Why This Matters

The results demonstrate practical improvements. AVA-Encoder achieves a 20.7 percentage point gain over the strongest external baseline in controlled experiments. More impressively, the learned policy outperforms a carefully hand-tuned baseline while using 74.3% fewer system-prompt tokens—meaning agents require less explicit instruction, suggesting they've genuinely learned the task.

For the video generation and creative AI community, this matters because it bridges a gap: agents can now work with video at a semantic level rather than pixel level. A filmmaker-AI system could reason about "cutting to a wide shot" and "adding warm lighting" as discrete, editable operations rather than pixel manipulations. The released dataset of film knowledge graph representations and the benchmark provide infrastructure for future work.

The approach also hints at scalability. Since the optimization uses natural-language gradients rather than computing through rendering pipelines, it's potentially more efficient and interpretable than prior methods that backpropagate through video generation models. For large-scale agentic video production, that efficiency could matter significantly.