Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion

The Core Finding

Mapping vulnerabilities to attacker techniques is valuable for security teams, but the usual chain of existing security standards (CWE→CAPEC→ATT&CK) introduces errors. The authors built a supervised classifier trained on 1,207 expert-annotated CVEs and found it roughly doubles recall compared to simple embedding lookup. However, when they tried to expand the training set using large language models (LLMs), the apparent gains evaporated under scrutiny—revealing a cautionary tale about evaluation practices in machine learning.

How the Method Works

The authors took CVE vulnerability descriptions and trained a multi-label classifier to predict which MITRE ATT&CK techniques they map to. Instead of chaining through three intermediate taxonomies (each introducing noise), they worked directly from text to techniques using a curated ground-truth set reviewed by the MITRE Center for Threat-Informed Defense. The model substantially outperformed a baseline that simply looked for semantic similarity between descriptions and technique definitions.

The interesting part came next: could LLMs automatically label more CVEs to enlarge the training set? Naïvely, yes—the initial results looked promising. But when the authors ran the same experiment five times with different random seeds, the improvements shrank. Independent replication by others confirmed this suspicious pattern. Further investigation uncovered the culprit: evaluation noise. On a small test set, selecting which model checkpoint to keep based on performance metrics is like running many noisy experiments simultaneously. By chance alone, one run picks a lucky checkpoint that scores 0.05 higher on recall@5 than it should.

Once the authors fixed their checkpoint-selection protocol—using a separate validation split instead of peeking at the test set—the LLM expansion showed no real gain. LLM labels agreed with experts only 39% of the time, and adding 100–984 LLM-labeled CVEs did not improve performance. Worse, at around 1,000 added examples, rare techniques suffered, with macro-F1 dropping 0.04.

Why It Matters

This work advances CVE-to-technique mapping, a practical tool for threat intelligence and vulnerability triage. More broadly, it documents a subtle but widespread pitfall in machine learning: how small evaluation datasets and multiple random runs can create the illusion of progress. Researchers comparing runs on different seeds, or tuning hyperparameters on a held-out test set, risk overfitting to evaluation noise rather than capturing real signal.

The authors demonstrate that the classifier is bottlenecked by label quality, not quantity. Adding expert annotations consistently helps; adding low-agreement LLM labels does not. This finding resonates beyond this domain—it suggests caution when using LLMs to scale annotation pipelines for tasks where ground truth is expensive but critical.

All code, data, and logs are released, making the pipeline reproducible and the evaluation pitfalls verifiable. That transparency is rare and valuable.