GS-Voxel: Fitting-Free Structured Latents for Large-Scale 3DGS Generation

The Core Problem

3D Gaussian Splatting (3DGS) is excellent at rendering 3D scenes—it's fast and produces high quality. But it's terrible for generation. A 3DGS reconstruction is a messy collection of thousands of unordered points scattered throughout space, with no fixed structure. Existing 3D generative models, by contrast, work on clean, structured grids (voxels). This mismatch means you can't easily use 3DGS with modern generative techniques like diffusion or flow models.

The authors solve this: GS-Voxel converts pre-optimized 3DGS reconstructions into sparse voxel grids without needing to retrain or reoptimize anything per-scene.

How It Works

The method has three steps.

Step 1: Voxelization (fitting-free). Take an existing 3DGS reconstruction and snap its Gaussian primitives onto a sparse voxel grid. The key trick: rather than discarding the sub-voxel positions and attributes, the method records them alongside the voxel assignment. This preserves the rendering quality of the original 3DGS without optimization.

Step 2: Latent encoding. A custom VAE (variational autoencoder) encodes the voxelized scene into a compact latent representation. Critically, it factorizes: geometry (which voxels are occupied) is encoded separately from appearance (color, opacity, and other Gaussian parameters). The resulting latent is sparse—its size scales with the number of occupied voxels, not a fixed scene budget.

Step 3: Generation. With latents in hand, the authors train image-conditioned flow models (a generative approach similar to diffusion) to synthesize new scenes. Crucially, these can be tiled across space: by conditioning on overlapping satellite imagery patches and carefully managing consistency at tile boundaries, the model can generate large aerial scenes that exceed the original training crop size.

Why This Matters

This unlocks large-scale aerial scene generation with 3DGS quality. Previous latent-based 3D generators either use fixed-size representations (limiting scene complexity) or ignore 3DGS entirely. GS-Voxel bridges the gap.

The "fitting-free" claim is important: no per-scene optimization means the pipeline is fast and scalable. You just convert and encode; inference is real-time.

The tiled generation is a practical win. Satellite imagery of cities or landscapes is huge. By generating scene-by-scene and stitching results with overlap awareness, the method sidesteps the memory and resolution ceilings of prior work.

The evaluation focused on aerial 3DGS scenes—challenging because they span large geographic areas with complex, irregular geometry. Results demonstrate that the sparse latent approach actually grows capacity as scenes expand, whereas fixed-latent methods would choke.

The structural insight is straightforward but powerful: 3DGS is already solving the hard rendering problem; the authors simply ask—how do we package it for generation?—and answer with minimal friction via voxel-based factorization.