SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation

The Problem: Success That Hides Bad Behavior

A robot can fold a cloth or manipulate a squishy object and technically "succeed" at the task while simultaneously slipping, compressing it too hard, or otherwise handling it poorly. Standard benchmarks celebrate task completion alone—they don't measure interaction quality. This gap matters: in real applications, you care not just that the job gets done, but how the robot does it. The authors identify why this gap persists: there are almost no datasets pairing what a policy can see (camera and touch sensor data) with ground-truth physical deformation measured independently throughout the manipulation.

What SoftVTBench Provides

The authors built a visuo-tactile dataset to fill this hole. It contains 4,000 expert demonstrations across 50+ deformable object assets, including both soft objects and rigid twins that look identical. Each episode, recorded at 20 Hz, synchronizes:

• Multi-view RGB video
• Tactile RGB from dual fingers (direct sensor imagery)
• Motion capture of object markers
• Robot proprioception (joint positions, etc.)
• Language descriptions
• Gripper actions (binary and continuous)
• Finite-element (FEM) ground truth (deformation state)—only available to evaluators, not the policy

The key innovation: FEM data acts as independent, objective measurement of how much the object actually deforms, decoupled from what cameras or touch sensors report. This lets researchers study whether policies learn to use tactile feedback effectively or just copy demonstrations.

A Tougher Success Metric

The authors then defined Deformation-aware Success Rate (DSR), a stricter metric than raw task success. A rollout counts as successful only if the policy both completes the task and keeps peak normalized deformation under a threshold. This catches the cases that matter: across three state-of-the-art policies (Diffusion Policy, π₀.₅, and FastWAM), 0.7–24% of in-distribution successes actually violated deformation tolerances. In other words, the policies were getting away with rough handling.

Why It Matters

The benchmark reveals a sobering finding: adding tactile data doesn't automatically make policies smarter. Under distribution shift, visuo-tactile variants did improve task success in all six policy–suite comparisons and raised DSR in five of six—but in-distribution, the benefits were mixed. Touch sensors alone don't guarantee effective multimodal fusion; policies must learn to integrate and act on that richer signal.

SoftVTBench is the first shared resource that lets the community move beyond asking "Did it work?" to asking "Did it work well?"—measuring both task success and the quality of physical interaction with deformable objects.