- Source
- arXiv
- Published
- Runtime
- 0:00
- Snippets
- 6
A conversation between
Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition
§02
Snippets
-
Models predict verbs via object-driven shortcuts, relying on labeled object class rather than temporal evidence of the action itself.
This explains why zero-shot compositional action recognition fails on unseen verb-object pairs—the model never learned what the verb actually looks like.
-
Sparse compositional supervision and verb-object learning asymmetry cause models to overfit to training co-occurrence patterns instead of learning robust temporal verb cues.
The model memorizes which objects go with which verbs rather than learning the underlying temporal structure of actions, breaking generalization.
-
Proposed diagnostic metrics reveal that existing ZS-CAR methods significantly underuse temporal verb cues and overrely on object recognition.
Quantifying the shortcut problem makes it measurable and addressable, rather than a vague generalization failure.
-
Co-occurrence Prior Regularization (CPR) explicitly supervises unseen compositions and treats frequent co-occurrence patterns as hard negatives.
Negative supervision on common pairings pushes the model toward temporal reasoning rather than relying on superficial object correlation.
-
Temporal Order Regularization for Composition (TORC) enforces temporal-order sensitivity so verb representations are grounded in action sequences, not objects.
By penalizing models that ignore sequence, TORC ensures verbs are learned from motion dynamics, the true signal of action identity.
-
RCORE reduces shortcut diagnostics and improves compositional generalization across Sth-com and EK100-com datasets.
The method is validated on real compositional tasks, showing the shortcut-mitigation strategy translates to measurable generalization gains.
§03
Synthesis
The Problem: Models Are Cheating
When recognizing unseen verb-object combinations (like "open drawer"), state-of-the-art models don't actually understand the action—they cheat by relying on the object alone. If a model has seen "drawer" labeled with common verbs in training, it predicts those same verbs for any new combination involving drawers, ignoring whether the video actually shows opening, closing, or pushing. This object-driven shortcut works on familiar combinations but catastrophically fails on novel ones, which is precisely what zero-shot compositional recognition must handle.
The authors diagnose why this happens: training data provides sparse supervision (only a few verb-object pairs per object), and verbs and objects are learned asymmetrically (the model picks up object patterns faster than temporal verb patterns). Together, these pressures push models toward the lazy shortcut.
Detecting and Measuring the Shortcut
Before fixing the problem, the authors introduce diagnostic metrics to quantify how much a model relies on objects versus temporal evidence. By analyzing model predictions on held-out compositions, they show that existing methods indeed overfit to training co-occurrence patterns (certain verb-object pairs that frequently appeared together) and underuse the temporal cues—the actual motion in the video—that distinguish one verb from another.
The Solution: RCORE
The proposed method, Robust COmpositional REpresentations (RCORE), attacks the shortcut problem with two components:
Co-occurrence Prior Regularization (CPR): This component explicitly teaches the model to recognize unseen compositions by adding direct supervision for held-out verb-object pairs. Crucially, it treats frequent co-occurrence patterns as hard negatives—penalizing the model when it relies on common training pairs to predict novel ones. This forces the model away from surface-level object associations.
Temporal Order Regularization for Composition (TORC): This component enforces that the model becomes sensitive to temporal order. By making the model pay attention to when events happen in a video, TORC grounds verb representations in actual motion rather than static object features. A verb like "open" has a characteristic temporal signature that differs fundamentally from "close," and TORC ensures the model learns this.
Results
Tested on Sth-com and EK100-com (two standard benchmarks for compositional action recognition), RCORE reduces the diagnostic metrics indicating shortcut reliance and improves generalization to unseen verb-object combinations. The gains demonstrate that explicitly combating object-driven shortcuts and enforcing temporal grounding matters in practice.
The work matters because compositional generalization is central to building vision systems that can recognize novel combinations of familiar concepts—a key requirement for real-world robustness. By identifying a concrete failure mode and proposing targeted fixes, the authors make progress on a problem that affects how well action recognition scales to open worlds.
Mine your own.
Lode is a workbench, not a feed. Paste a YouTube URL. The model proposes a transcript, a set of quote-grounded snippets, a synthesis essay, and the fan-out. You decide what stays.