Selective Synergistic Learning for Video Object-Centric Learning

## The Problem: Misaligned Learning in Video Object Segmentation

Video object-centric learning (VOCL) aims to decompose video frames into individual objects—a foundational task for understanding dynamic scenes. Current methods use slot-based architectures: an encoder produces attention maps (which regions to focus on), and a decoder generates object maps (where object boundaries are). The catch is that these two outputs have fundamentally different properties. Attention maps tend to be noisy and over-activate; object maps from decoders blur boundaries.

Recent work tried to fix this by forcing agreement between attention and object maps across every spatial and temporal patch using contrastive learning. Sounds logical, but it backfires: the approach amplifies weaknesses from both modules rather than correcting them. Additionally, comparing all patch pairs scales quadratically with video resolution—prohibitively expensive for high-resolution inputs.

## Selective Synergistic Learning: The Fix

The authors propose SSync, which takes a more targeted approach. Rather than aligning everything indiscriminately, SSync assigns each module a specialized role:

**The encoder (attention) is used only for boundary refinement.** Its strength is identifying where object edges lie, so the method leverages that strength and ignores its interior noise.

**The decoder (object map) handles interior denoising.** It excels at filling in object interiors cleanly, so SSync lets it denoiseConfidence predictions in those regions.

This selective distillation operates via a pseudo-labeling scheme—soft labels generated from confident model predictions rather than ground truth—that runs in linear time. No expensive quadratic patch comparisons needed.

A secondary innovation addresses slot redundancy. In slot-based systems, multiple slots sometimes activate on the same object, creating duplicates. SSync introduces transitive pseudo-label merging: it identifies overlapping slots by measuring their spatio-temporal activation consistency and consolidates redundant ones.

## Why This Matters

The practical impact is substantial. By eliminating quadratic complexity, SSync scales to realistic video resolutions. By selectively trusting each module's strengths, it avoids error propagation—a common failure mode when you enforce agreement between imperfect components.

The method is plug-and-play: it works on top of existing slot-based VOCL frameworks without architectural changes. Experiments show improved decomposition quality and robustness across different slot configurations (the number of objects the model can represent). The code's public availability enables reproducibility and adoption.

At its core, SSync embodies a simple but powerful principle: when components have different failure modes, don't force them into lockstep agreement. Instead, route information strategically—let each part do what it does best and prevent it from contaminating the others.