PhysCaP: Grounding Code-as-Policy Agent with Physics-Informed Exploration

The Core Problem: Robots Can't Feel What They Need to Know

Vision-language-action policies—neural networks trained to imitate human demonstrations—have become powerful tools for robotic manipulation. But they have a fundamental blind spot: they rely entirely on what cameras see. When a robot needs to know whether a can is empty, whether an avocado is ripe, or whether an object is hidden behind something else, passive observation fails. These tasks require the robot to actively probe the environment and infer latent physical properties like mass and stiffness from the tactile and force feedback it receives through its joints and actuators (proprioception).

PhysCaP solves this by adding an active exploration layer on top of code-as-policy agents—systems that generate executable robot code rather than direct motor commands. The key insight is that a robot doesn't need extra sensors to learn physics; it can extract physical properties from its own body sensors if it explores strategically.

How PhysCaP Works

The system has three main components:

Physics-informed property extraction. The authors built modules that estimate object mass and stiffness directly from proprioceptive signals—joint torques, accelerations, and positions—during simple exploratory interactions. Crucially, these modules require no training; they apply standard physics principles (like dynamics equations) to infer properties from observed robot motion.

Dual-agent control. A Planner agent decides whether the robot should explore further or commit to a manipulation action. A Prioritizer filters out infeasible interactions and ranks remaining ones using a heuristic score, preventing wasteful over-exploration. Together, they balance the cost of gathering information against the benefit of reducing uncertainty.

Integration with code-as-policy. PhysCaP embeds these components into the code generation pipeline, so the policy can conditionally call exploration routines based on task demands.

Real-World Validation

The authors tested PhysCaP on three tabletop tasks where hidden physical properties matter: locating occluded objects, detecting empty cans (by weight), and identifying ripe avocados (by softness). They also evaluated on LIBERO, a simulated benchmark suite.

The results are clear: naive baselines either failed when properties were hidden or wasted time on unnecessary interactions. PhysCaP achieved comparable task success rates while using significantly fewer interactions and shorter execution times. Ablations confirmed that the physics extraction modules were essential—removing them degraded performance.

Why This Matters

Existing robotic manipulation systems assume the world is fully observable or resort to brute-force exploration. PhysCaP shows that intelligent, physics-grounded exploration can be both efficient and training-free. By leveraging a robot's inherent ability to sense through its own motion, it enables agents to discover hidden task-critical information without redesigning hardware or collecting new sensor data. This makes active perception practical for real-world deployment where exploration budgets are tight and latency matters.