AsySplat: Efficient Asymmetric 3D Gaussian Splatting for Long-Sequence Scene Modeling

The Core Claim

Current 3D Gaussian Splatting models waste computation by treating geometry and appearance learning equally, even though they have fundamentally different demands. AsySplat separates these tasks into an asymmetric architecture: a parameter-heavy geometry branch handles coarse spatial structure, while a lean appearance branch captures fine details. This division cuts redundant computation, shrinks model size, and accelerates inference by ~800x on 32-view 960P inputs without sacrificing quality.

How It Works

The key insight is that reconstructing precise 3D geometry across long image sequences is computationally expensive but doesn't directly drive visual quality—what matters is plausible shading and color. Appearance, by contrast, is easier to learn but needs fine detail resolution to look convincing.

AsySplat builds on this with two parallel branches. The geometry branch operates on coarse-grained (low-resolution) tokens and concentrates most parameters there, performing multi-view reconstruction to establish the spatial layout of 3D Gaussian primitives. The appearance branch works on fine-grained (high-resolution) tokens with far fewer parameters, extracting surface color, reflections, and other view-dependent effects.

Rather than running these in isolation, the branches exchange information through bilateral connections—the geometry branch can inform appearance about which regions need detail, and the appearance branch can feed back cues that refine geometry estimates. This mutual guidance prevents either branch from drifting too far off course.

The asymmetry is task-aware: geometry gets width and depth (more transformer layers, higher hidden dimensions) because reconstructing 3D structure is fundamentally harder; appearance gets precision (fine spatial resolution) but minimal parameters, since the job is largely pattern-matching against training examples.

Why It Matters

Long-sequence novel view synthesis—generating new camera angles from, say, 32 input images—is computationally brutal. Dense transformer encoders and large parameter counts have become standard to handle the volume of multi-view information. AsySplat demonstrates that this brute-force scaling is unnecessary; instead, you can achieve stronger results with a smaller, faster model by aligning architecture to task difficulty.

The reported results underline the efficiency gains. On standard benchmarks, AsySplat matches the visual quality of optimization-based methods (which train per-scene) while running ~800x faster, and outperforms zero-shot generalizable baselines (models that generalize to new scenes without per-scene fine-tuning) with meaningfully fewer parameters and lower training/inference costs. For practical deployment—especially on resource-constrained hardware or real-time applications—this trade-off is significant.

The finding also generalizes beyond 3D Gaussian Splatting. The principle that different reconstruction subtasks have different computational requirements could influence how other multi-task vision models are designed, suggesting that symmetric, fully-shared encoders may be leaving performance and efficiency on the table.