Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations

The Core Finding

Demographic bias in large language models concentrates in a tiny, surgically targetable set of neurons. By zeroing fewer than 40 neurons in a 1 billion-parameter model—less than 0.03% of the network—the authors can alter how the model responds to demographic attributes while retaining 99.49% of its reasoning and general knowledge. This suggests bias and capability rely on separate neural circuits.

How Fairness Pruning Works

The method identifies biased neurons using a simple but clever approach: minimally contrastive prompt pairs. These are pairs of prompts that differ only in a demographic attribute—for instance, "John applied for the job" versus "Jamal applied for the job." The authors feed both prompts through the model and capture neuron activations at the input to the down_proj layer (a key component in GLU-MLP architectures, a common design in modern LLMs). Neurons that respond differently to these paired prompts—those showing "differential activation"—are flagged as bias-sensitive.

Once identified, zeroing these neurons changes the model's behavior around the demographic variable. The intervention is inference-time only, requiring no retraining.

The Complication: Bidirectional Bias

Here's where the findings get interesting—and messier than a simple fix. When neurons are zeroed, the bias doesn't flatten evenly. Instead, it becomes "destabilized" bidirectionally. The reason: neurons operate in both directions. Some amplify stereotypes; others counteract them. The method's detection metric (BiasScore) is unsigned—it doesn't distinguish between pro-stereotype and anti-stereotype neurons. So when you zero a mixed set, whether bias increases or decreases depends on which type dominates in your candidate set. This is not a bug but a feature: it reveals that the model contains competing bias circuits, and simple ablation mixes them in unpredictable ways.

Why It Matters

The work establishes two critical insights. First, it shows that demographic bias is localized rather than diffuse—concentrated enough to modify with surgical precision without cratering model performance. Second, it proves this localization is meaningful: the fact that you can remove bias-related processing while preserving reasoning suggests these aren't tangled together at a fundamental level.

The method is also practical. It requires only inference-time activation capture and minimal compute, making it applicable to deployed models without fine-tuning. Testing on models up to 3 billion parameters (Llama-3.2 family and Salamandra-2B) with both standardized benchmarks and qualitative text generation experiments grounds the claims empirically.

The main limitation is that current results don't yet achieve directional bias mitigation—reliably pushing bias in one direction. The authors frame this honestly: the paper establishes methodological foundations. Future work would need to distinguish pro- from anti-stereotype neurons to move beyond blind zeroing toward controlled behavior modulation. That distinction is the next frontier.