ReViV: Reconstructing the Viewer and the View in 4D from Monocular Egocentric Video

The Core Claim

ReViV reconstructs a complete 4D model of both the person wearing an egocentric camera and their environment from a single RGB video, doing so faster and without needing pre-computed camera paths. It's the first unified framework to jointly recover camera motion, gaze, full-body pose, hand motion, and depth all at once—tasks typically tackled separately.

Why This Matters

Egocentric cameras (mounted on glasses or headsets) capture how humans interact with their surroundings in first-person view. Reconstructing what's happening requires solving multiple interconnected problems: Where is the camera moving? Where is the person looking? What are their hands doing? What does the 3D scene look like?

Previous work treated these as independent puzzles and often demanded external help—pre-computed camera trajectories or motion capture data. That's impractical. More fundamentally, these problems are tightly coupled: hand position constrains where the camera can be, scene geometry influences where the person can move. Solving them separately leaves performance on the table.

How ReViV Works

The authors formulate the task as learning a joint probability distribution across all these signals simultaneously—video frames, camera trajectory, gaze, body pose, hand pose, and depth. The architecture is a Masked Generative Egocentric Transformer, inspired by masked language models. Think of it like filling in missing puzzle pieces: given video frames and partial information about the scene, the model predicts the missing 4D structure in a single forward pass.

The key design choice is unified, feed-forward inference. No iterative optimization, no separate pipelines for hands vs. body vs. camera tracking. One transformer processes everything together, which keeps computation lean and ensures all predictions pull from the same joint reasoning.

The "masked" part means the transformer learns by masking out random information (frames, body joints, depth) during training and predicting it back. This encourages the model to understand temporal consistency and the dependencies between viewer motion and scene structure.

What the Results Show

ReViV was evaluated on five diverse benchmarks: HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO. It achieved state-of-the-art accuracy on ego-body reconstruction, hand pose, gaze estimation, and camera tracking. Notably, it stayed competitive on depth estimation without task-specific hand-crafted priors—the joint modeling was enough.

The "holistic" framing is deliberate: ReViV doesn't isolate the human from the environment. Body motion, hand motion, camera motion, and depth are all inferred together, allowing the model to exploit correlations that siloed approaches miss. For instance, if the model detects hands interacting with an object, it can better localize that object in 3D, which constrains camera position, which refines body pose.

The authors also emphasize fast inference. Single feed-forward pass means practical deployment on wearable hardware, not batch processing on a server.

By open-sourcing code and models, the authors position ReViV as a foundation for downstream tasks—AR/VR interfaces, human activity understanding, or augmented reality overlays—that depend on accurate egocentric 4D reconstruction.