Benchmarking Sensor Robustness in Plasma Diagnostic Models: A Systematic Evaluation on TokaMark

The Real Problem With Fusion AI Models

Plasma diagnostic models trained on machine learning perform well in lab conditions—but fusion reactors are messy. Sensors fail constantly: they start recording late, individual channels die mid-shot, and dropouts cluster right when plasma is about to disrupt (catastrophically lose confinement). This paper is the first to systematically test how robust these models actually are when sensors fail in realistic ways.

The authors benchmark four architectures (XGBoost, LSTM, Transformer, CNN) on 11,573 real tokamak shots from the MAST reactor using the TokaMark dataset, injecting six physically-grounded failure modes and testing three imputation (data-filling) strategies. The core finding is stark: sequence models like LSTMs are brittle when sensors fail near the end of the measurement window—exactly when predictions matter most—while simpler statistical models degrade more gracefully.

Why This Matters

Disruption prediction is safety-critical: predicting tokamak instabilities seconds ahead allows feedback control to prevent damage. But if your sensor fails in the final window—the most informative part of the prediction window—an LSTM's performance collapses catastrophically (+212% normalized mean squared error). XGBoost, which treats each timestep as independent features rather than a sequence, only degrades +37% under the same failure.

The imputation strategy choice flips the outcome. Forward-fill (repeating the last known value) nearly erases degradation from random dropout for LSTMs (+57% error down to ~0%), but it provides no help when the end-of-window corruption is the problem. Conversely, mean-fill imputation recovers LSTM alarm detection to perfect sensitivity (TPR=1.00) under proximate failure—reversing the pattern seen in raw prediction error metrics.

The authors introduce a Robustness Score (RS) for standardized cross-architecture comparison, enabling fair benchmarking despite different model types. They also identify plasma current as the single most critical diagnostic across all architectures, with its removal causing +73% to +140% error increases.

The Practical Gap

This work exposes why lab-trained fusion ML models fail in the field: standard benchmarking uses complete, clean data. Real tokamaks don't. The MAST dataset lets the authors test realistic scenarios—late-starting acquisition, individual sensor death, clustered dropouts—rather than synthetic random corruption. Transformer and CNN models fall somewhere between XGBoost stability and LSTM brittleness, suggesting architectural choice matters for robustness, not just accuracy.

The reversal between NRMSE (error magnitude) and TPR (alarm detection) under mean-fill imputation is particularly important: a model with slightly higher prediction error may still reliably detect dangerous conditions, which is what operators actually need. The work shows that robustness evaluation cannot rely on a single metric or assume imputation strategies generalize across all failure types.

Code, data, and checkpoints are publicly available, making this a reproducible benchmark for future fusion diagnostics research.