SAM-MT: Real-Time Interactive Multi-Target Video Segmentation

The Problem: Multi-Target Segmentation Bogs Down

Current video segmentation systems excel at tracking a single object in a video, but when asked to handle multiple targets simultaneously, they typically process each object independently—running the entire pipeline once per target. The result: a system that bogs down catastrophically. Tracking 10 objects becomes 10× slower than tracking 1. For interactive applications (where a user is clicking to specify new targets in real time), this cascading slowdown is unworkable.

SAM-MT solves this by redesigning how segmentation models process multiple targets. Instead of serial processing, the authors show you can handle many targets in parallel while maintaining frame rates above 36 FPS—matching single-target speed—without sacrificing accuracy.

How It Works: Decoupled Queries and Shared Context

The core insight is architectural: represent each target explicitly as a separate query (a learned token encoding "which object am I tracking?"), but process all targets through a shared representation of the global scene context. This avoids redundant computation.

The challenge: without careful handling, targets will "bleed into" each other's representations, causing identity confusion or overlap. SAM-MT uses three mechanisms to prevent this:

Decoupled masked attention. When the model processes target A, it masks out attention weights that would mix A's identity with other targets' information. Think of it as giving each target its own lane in the attention computation, even though they're running on the same highway of shared context.

Sparse memory. Rather than storing every frame's full representation, the model keeps a compact memory bank of key frames. This stabilizes predictions across time without exploding memory costs as video length increases.

Specialized occlusion and overlap handling. When target A disappears behind target B, or two masks start to merge, the model has explicit strategies to maintain correct boundaries and recover visibility.

Why It Matters

The paper builds on Segment Anything 2 (SAM2), a recent foundational model known for robust generalization. By keeping SAM2's core strengths while adding multi-target parallelism, SAM-MT makes interactive video editing and multi-object tracking practical at real-time speeds.

The decoupling of latency from target count is the key win: whether you're segmenting 2 objects or 10, the system runs at the same speed. This unlocks use cases where a user interactively selects and edits multiple objects in a video without waiting between clicks—crucial for editing software, robotics applications, and video analysis tools.

The reported >36 FPS for 10 targets (matching single-target SAM2 performance) is not trivial; it means the overhead of multi-target reasoning is negligible once the parallel architecture is in place.