GaussianSelector: Lightweight Human-Guided Object Selection in 3D Gaussian Splatting with Graph Optimization

The Core Innovation

Most methods for picking objects out of 3D scenes reconstructed with Gaussian splatting (a fast neural rendering technique) either force you to retrain the entire model or require dense multi-view annotations using segmentation tools. GaussianSelector sidesteps both burdens: it works on the raw Gaussian primitives without retraining and produces clean object selections from just a handful of sparse user scribbles drawn on a few camera angles.

How It Works

The authors' key insight is to treat the problem as a global optimization puzzle rather than a local labeling task. Here's the pipeline:

Coarsening and graph construction. The method starts with thousands of raw Gaussian primitives (the elementary units in Gaussian splatting) and groups them into fewer, geometrically coherent "superpoints"—think of these as clusters that respect spatial and appearance boundaries. These superpoints become nodes in a graph where edges encode continuity: two nearby superpoints with similar appearance and geometry get strong connections, while ones that differ get weak ones.

Lifting sparse scribbles to 3D. When a user draws a scribble on screen (e.g., a blue line on a car in one viewpoint), the method converts that 2D mark into 3D evidence. It uses visibility-aware transmittance—basically, it checks which Gaussians actually contribute to that pixel and how much—to assign the user's label to relevant 3D points without assuming full view coverage.

Graph-cut optimization. The selection problem becomes a graph-cut energy minimization: assign each superpoint a label (object or background) to minimize total energy. The energy pulls toward labeled nodes (where the user drew) while respecting continuity penalties between neighboring superpoints. Standard algorithms solve this globally, ensuring consistent, complete object boundaries.

Iterative refinement. If the result is imperfect, the user can draw more scribbles from a different camera angle, and the method refines the selection by re-running the graph-cut on the same graph—no retraining needed.

Why It Matters

Training-free operation is critical for real deployment. Retraining scene representations is slow and often impractical on consumer hardware. The authors' method runs efficiently on native Gaussians, delivering competitive quality to SAM-based approaches (which use large pre-trained segmentation models) while needing far fewer viewpoints and less computational overhead.

The sparse interaction model—just a few scribbles from sparse angles—matches realistic scenarios. You're not forced to annotate every viewpoint or maintain dense multi-view coverage. The visibility-aware lifting ensures that user input maps reliably to 3D, and the global graph-cut ensures coherent selections rather than fragmented pieces.

The result is a lightweight, human-in-the-loop tool suited for interactive 3D scene editing and asset extraction—practical workflows where speed and minimal user burden are essential. The method naturally supports iterative correction, letting users refine from new viewpoints as needed.