From Seeing to Acting: Smart Glasses as First-Person Intelligence Platforms

The Core Claim

Smart glasses aren't just cameras and displays—they're platforms for real-time, embodied AI that must perceive, remember, decide, and act while strapped to a person's body. The authors argue that existing research treats these problems in isolation, but a working system requires all pieces to work together reliably under severe constraints (battery life, heat, privacy, user safety). This survey is the first to treat smart glasses as an integrated perception-to-action loop rather than a collection of separate tasks.

Why This Matters

Smart glasses sit at an awkward boundary. They can see what the wearer sees, hear what they hear, and sense their hand movements—giving them genuine first-person context that a phone or robot cannot match. But they fail in ways that matter: a recommendation system that hallucinates could send someone down a wrong street; a translation app that lags by two seconds breaks conversation; a power drain that kills the battery mid-shift makes the device useless. Current benchmarks and papers don't measure these real-world constraints or how failures cascade through the pipeline.

The authors identify the core tension: it's not hard to build a model that recognizes objects, or answers questions about images, or remembers facts in isolation. The hard part is building a system that does all three in real time, on limited power, without overheating, while respecting privacy, and allowing users to correct mistakes or override decisions.

How the Framework Works

The authors propose two organizing structures:

L0–L5 capability stack: Starting from raw capture (L0), the system layers up through reactive perception (L1: what's in front of me?), contextual assistance (L2: what should I do?), persistent state (L3: what did I see before?), governed action (L4: what can I safely recommend?), and embodied coupling (L5: how do I learn from correction and feedback?).

Nine-dimensional deployment framework: They map smart glasses along hardware axes (camera resolution, compute power, battery capacity, latency, thermal budget, etc.) and evaluate systems on how well they handle nine real-world scenarios—from navigation assistance to workplace training to social interaction.

The authors also introduce a "claim-conditioned evaluation protocol" and an "evidence ladder" that requires claims to be tied to measurable deployment contexts. A system claiming to help someone recognize faces must specify in which lighting, at which distance, for which user population—and provide evidence from controlled tests through real-world field validation.

What's New

The contribution isn't a new algorithm or dataset. It's a conceptual infrastructure: a shared language for comparing systems, a checklist for designers to verify their constraints are met, and a roadmap for what evidence matters. The nine application scenes (navigation, workplace, accessibility, etc.) connect abstract capabilities to concrete failure modes and stakeholders who bear the consequences.

This framing reveals gaps: most papers optimize for accuracy on benchmarks, not for latency, power, or human-in-the-loop correction. Few systems handle the full loop from seeing to acting. The survey gives researchers and product teams a map of what a complete system needs and where evidence is missing.