VIABench: Testing AI Assistants for Real Blind Users

Current AI vision models fail at practical blind assistance because they're benchmarked on generic vision tasks, not the specific, real-time challenges visually impaired people face daily. The authors introduce VIABench—a video benchmark built from first-person footage recorded by blind individuals themselves—to expose these gaps and drive development of AI that actually works in the wild.

Why This Matters

Multimodal Large Language Models (MLLMs) have made headlines for their ability to understand images and answer questions about them. But there's a critical disconnect: these models are tested on curated datasets that don't reflect the messy reality of navigating the world without sight. A blind user needs an AI assistant that warns them about a pothole appearing ahead in real-time, not one that can describe a scene after the fact. VIABench closes this gap by grounding evaluation in genuine user needs and authentic video data.

Three Tasks, Three Real Problems

The benchmark defines three core scenarios:

Proactive Reminder asks the model to continuously watch video and predict upcoming hazards or navigation-critical events before they happen—then verbally alert the user. This is the hardest task because it demands both understanding of the current scene and genuine anticipation of what's next.

Visual Question Answering (VQA) is more familiar territory: the user asks questions about their surroundings ("What's in front of me?" "Is there a chair?"), and the model answers based on the video. This tests real-time comprehension but not prediction.

Vision-Guided Interaction requires the model to understand context and reason about how the user can accomplish a goal—for instance, helping them locate and interact with a specific object in their environment.

How It Works

The authors collected videos directly from visually impaired individuals, capturing authentic first-person perspectives of real navigation and interaction scenarios. They then built a rigorous evaluation pipeline that works in both online (streaming, real-time) and offline (full-video analysis) settings. This dual approach tests whether models can operate under the time constraints of actual deployment while also measuring their best-case performance with complete information.

The Result: Current MLLMs Fall Short

Experiments show that even state-of-the-art multimodal models struggle—particularly on Proactive Reminder tasks. The models can describe what they see, but they can't reliably anticipate danger or offer timely warnings. This finding is sobering but crucial: it reveals that scaling vision-language models on broad internet data doesn't automatically produce safe, responsive AI for assistive technology.

The benchmark is designed as both diagnosis and catalyst. By publishing VIABench with code and data, the authors are explicitly inviting the research community to build better models. The hope is that future work will develop MLLMs specifically tuned for the constraints and requirements of blind assistance—models that prioritize real-time responsiveness, accurate anticipation, and user safety over generic image understanding.