EffectLearner: World-Aware Object-Effect Reasoning for Real-World Video Object Removal

The Core Problem

Video object removal isn't just about erasing a target from the frame—it's about erasing what that object does to the scene. A moving car leaves motion blur, dust, and shadows. A person casts reflections and disturbs water. Existing methods treat these "effects" as implicit side effects, learning them loosely from fixed categories and datasets. They fail when effects are spatially separated from the object, compositional (multiple overlapping effects), or rare in the training data. EffectLearner tackles this by making effect reasoning explicit and structured.

How It Works

The framework has two main components working in tandem.

Object-Effect Reasoner (VLM-based): This uses a vision-language model (a neural network trained to understand both images and text) to reason about what effects a target object induces. Rather than guessing from pixels alone, it analyzes a video where the target is highlighted and, guided by a detailed prompt asking "what visual changes does this object cause?", extracts structured effect-aware context. This reasoning step bridges the semantic gap—it knows why pixels should change, not just that they do.

Video Eraser (DiT-based): A diffusion transformer (a generative model that gradually refines noisy images into clean ones) takes the video and the effect descriptions and removes both the object and its effects. Two technical additions improve results: motion-aware mask guidance (using optical flow to track how things move frame-to-frame) ensures removal covers the object across all frames, and motion-consistency supervision enforces that the restored pixels move coherently with the scene.

The insight is that VLM reasoning and diffusion generation complement each other—semantic understanding guides pixel-level generation.

Why This Matters

Real-world videos are messy. Objects interact with scenes in compositional ways (a person near water creates splash, shadow, and ripple reflection simultaneously). Effects can be spatially detached (a car's dust cloud lingers meters behind). Rare phenomena don't appear often in training data. By explicitly reasoning about effects rather than implicitly learning them, the model generalizes better to unseen scenarios.

The authors validated this by building EffectWorld, a new paired dataset with complex object-induced effects, and an EffectWorld-Wild subset mimicking real-world difficulty. They also used a progressive training curriculum—starting with common effects, then gradually introducing complex ones.

Results

On the standard ROSE-Bench benchmark, EffectLearner outperforms existing baselines on most metrics. More importantly, it shows clear advantages on their new EffectWorld-Eval and the harder EffectWorld-Wild, confirming the framework handles compositional, spatially detached, and rare effects better than prior work.

The practical payoff: higher-quality video inpainting for content creators, safer autonomous vehicle datasets (removing obstacles while preserving scene physics), and more realistic video editing.