EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment

The Core Problem and Solution

Deep face recognition models work remarkably well—they can identify people with near-perfect accuracy—but they're black boxes. A practitioner using one cannot explain why a similarity score between two faces is high or low. Which facial attributes did the model actually rely on? EXPL-FR solves this by translating the model's internal representations into human-readable semantic concepts, all without needing to retrain or modify the face recognition model itself.

The key insight is elegant: take a vision-language model (VLM)—a system trained to understand both images and text, like CLIP—and build a lightweight adapter that maps its image encoder into the frozen face recognition embedding space. Train this adapter only on face images, never on text. Because the VLM's image and text encoders share the same space, the same adapter automatically works for text too. Feed in 978 attribute prompts organized into 22 categories (e.g., age, ethnicity, facial hair), and they become anchors in the face recognition space at essentially no computational cost. The vocabulary is extensible—users can add new attributes as needed.

Why This Actually Works (and Why It Doesn't Fully)

The authors don't assume this transfer magically works. They measure it using a face-verification protocol: does ranking faces by their distance to an attribute anchor actually correlate with having that attribute? The ablation—varying only the adapter while keeping everything else fixed—isolates its contribution.

Here's the subtlety: not every concept survives the transfer. A face recognition model achieves its invariance to identity-irrelevant factors (like lighting, pose, expression) by actively discarding them. So some attributes simply aren't recoverable from the embedding space. The authors introduce a label-free detectability measure that compares each concept's separability in FR space versus the VLM space. The top 100 most detectable attributes form a "semantic signature" of the model—and strikingly, these 100 concepts actually separate identities better than the full vocabulary of 978.

Breadth and Impact

The method is architecture-agnostic; it requires no access to the face recognition model's internals, only its outputs. The authors test four different FR backbones and two VLM encoders. It supports three explanation levels: identity-level (which attributes distinguish two identities), per-image (what attributes does this face image have), and differential (how does changing one attribute affect verification).

The practical payoff: the authors benchmark model auditing under three regimes—using human labels (the current gold standard), using VLM pseudo-labels (cheaper), and using only prompts with no labels at all. With zero ground-truth labels, the prompt-driven audit ranks four FR models by their per-ethnicity errors on a real benchmark (RFW) and ranks controlled attribute changes by their actual verification cost. This matters because it enables fairness audits and transparency audits without expensive human annotation.