360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents

The Gap Between What We Can Build and What Agents Can Navigate

Urban navigation looks simple until you ask an AI to do it. Researchers have built several benchmarks for embodied agents—AI systems that move through and reason about environments—but most either look cartoonish or cover only small areas. 360CityArena closes this gap by creating a photorealistic city-scale navigation benchmark that exposes how far current AI still lags behind humans.

The authors reconstructed the Akihabara district in Tokyo from 602 segments of 360-degree video, covering 85 streets. This wasn't a render or simulation; it's built from real captured footage, so agents see genuine urban clutter: storefronts, pedestrians, poles, signage, shadows. Within this environment, the team designed 175 tasks grouped into three categories: Environment Understanding (recognizing landmarks, identifying locations), Path Reasoning (planning routes between points, following directions), and Spatial Reasoning (understanding relationships like "turn left at the convenience store and go straight for two blocks"). These tasks demand the core skills real navigation requires—localization, landmark spotting, route planning, and mental spatial mapping.

Why Photorealism Matters More Than You'd Think

The critical finding: even the strongest large multimodal model (LMM) tested—Gemini 2.5 Flash, which can process images and text—achieved only 17.1% success. Humans scored 77.3%. That's not a narrow gap; it's a chasm.

Why does photorealism matter for this result? Because simplified environments let agents exploit shortcuts. A stylized map or clean 3D simulation may allow pure geometric reasoning without parsing real visual noise. Akihabara's actual chaos—overlapping signs, reflections, similar-looking intersections, weather effects—requires agents to ground abstract reasoning in messy perception. An agent claiming it "found the landmark" must actually identify it among dozens of visual distractors.

The tasks span different difficulty tiers too. A basic task might ask: "Where are you?" (environment understanding). A harder one combines multiple reasoning steps: "Walk to the train station via the bookstore; how many shops did you pass?" This structure lets researchers diagnose whether failures stem from perception, planning, or reasoning.

Why This Matters

Embodied AI for real-world deployment needs a realistic proving ground. Existing outdoor benchmarks either simplify scenes too much (synthetic data, limited visual fidelity) or lack the spatial scale and task diversity needed to stress-test navigation systems. A 17% success rate on Akihabara suggests that deploying current LMM-based agents on real city streets would fail repeatedly.

This gap also clarifies what's missing: do agents struggle because they can't parse visual clues, because they can't plan coherent paths, or because they can't reason about spatial relationships under uncertainty? 360CityArena's structure lets researchers isolate these failures and focus development.

The benchmark is city-scale (85 streets) yet specific enough (175 human-designed tasks) to enable iterative improvement. For anyone building navigation or spatial-reasoning systems, it's a sobering reminder: photorealism and scale reveal what toy benchmarks hide.