REBASE: Reference-Background Subspace Elimination for Training-Free In-Context Segmentation

The Problem: Background Noise Ruins One-Shot Segmentation

Imagine you show a model one photo of a cat and ask it to find cats in new images without any retraining. Recent methods try this by finding regions in the new image that "match" the reference cat using deep learning features—basically computing similarity maps. The catch: if both the reference and query images share similar backgrounds (say, grass), the model gets confused and marks the grass as a match too, even though grass isn't the target. This contextual bleed ruins the segmentation.

How REBASE Works

REBASE solves this by explicitly removing the background from similarity calculations. The key insight is that background features often occupy a low-rank subspace—they're repetitive and predictable. Here's the approach in plain terms:

1. Extract the background subspace: The method analyzes feature vectors from the reference image and identifies which dimensions capture "background-ness" rather than the target object. This involves computing a low-rank approximation of background features in closed form (mathematically tractable, no optimization needed).

2. Project onto orthogonal space: Instead of using raw features, the method projects both reference and query image features onto a perpendicular direction away from this background subspace. Think of it like removing a corrupted color channel from an image—what remains is cleaner.

3. Smarter prompt generation: The cleaned similarity map is then used to place positive point prompts (instructions for SAM, the segmentation model) using weighted farthest-point sampling—spreading prompts across diverse high-confidence regions rather than clustering them.

The entire process runs without training or updating any parameters, using only matrix algebra operations.

Why It Matters

One-shot segmentation is valuable for real-world scenarios where you can't afford to retrain models for every new object category—medical imaging, robotics, rare object detection. Prior training-free methods relied on SAM (a foundation model for segmentation) plus a correspondence model, but were fundamentally limited by spurious background matches.

REBASE's explicit background removal is a simple but effective principle. The authors demonstrate improvements on three major benchmarks: PACO-Part (diverse everyday objects), FSS-1000 (fine-grained categories), and cross-domain medical data (ISIC2018 skin lesion images). The fact that medical images—where backgrounds are especially distracting—show gains validates the core insight.

Critically, this works without training, meaning practitioners can apply it immediately to any new domain without data collection or compute overhead. The mathematical framework (low-rank decomposition and orthogonal projection) is elegant enough to be reproducible and interpretable—you can understand why the background suppression works rather than relying on a black-box neural network.

The method represents a shift in thinking: instead of trying to build better learned correspondences, explicitly reasoning about what not to match (the background) can be more effective.