MOSS-VL Technical Report

The Core Claim

Standard vision-language models process images first, then generate text—a pipeline that breaks down for real-time interaction. MOSS-VL flips this: it can perceive while speaking, making it genuinely responsive to incoming video frames during generation. The model achieves state-of-the-art performance on streaming benchmarks while being 5× faster at producing the first output token compared to similar-sized competitors.

How It Works

The key innovation is architectural. Instead of the vision encoder blocking the language decoder until all image processing finishes, MOSS-VL uses gated cross-attention: the decoder attends to visual information on-demand through a controlled gating mechanism. Incoming video frames feed directly into this attention layer while text is being generated, so the model doesn't have to wait.

To train this capability, the authors created a synthesized interaction corpus—synthetic training data that teaches the model when to speak, when to pause, and when to correct itself. This supervisory signal is crucial for learning genuine interactive behavior rather than just fast inference.

The training strategy is staged. First, MOSS-VL trains on offline data using standard vision-language tasks (image captioning, visual question-answering) to build a strong foundation. Then, in a final lightweight stage, all real-time-specific training happens. This curriculum design means real-time interaction doesn't destabilize the model; it's added on top of solid offline performance.

Why It Matters

Streaming benchmarks tell the story. On four evaluation sets designed for streaming models, MOSS-VL-Realtime ranks first on three and second on one. But the most striking result is on OmniMMI Proactive Alerting—a benchmark testing whether the model can proactively speak up without being asked. MOSS-VL scores 66.0; the best open-source baseline scores 37.5. This 76% gap shows the model has learned to behave like an actual conversational partner, not just answer questions reactively.

Latency matters in production. With 11.3B parameters, MOSS-VL reaches its first output token 5.1× faster than Qwen3-VL-8B (a smaller but competitive model) as visual context scales. Because visual tokens stay outside the decoded sequence—they're only accessed via gating—they don't slow down text generation. This is a genuine efficiency gain, not a trick; it means real-time systems can actually use this model without hardware overhead proportional to video resolution.

The offline-to-online transfer is clean. MOSS-VL-Instruct (the offline version) is competitive at its scale and leads on temporal reasoning tasks—proving the foundation is solid. The real-time version isn't a hack bolted onto a weaker base; it's a natural extension of a capable model.

Open releases of all five checkpoints, the training curriculum, and inference code lower the barrier for reproduction and deployment. For teams building conversational AI over video streams, this is immediately useful infrastructure.