Teaching LLMs to Recommend and Defer in Underrepresented Epilepsy Care

LLMs can help frontline clinicians in resource-scarce settings manage epilepsy treatment, but they make systematic errors rooted in training data bias rather than inability to read patient records. The authors show that a small dataset of local prescribing patterns can fix this through "prompt memories"—auditable instructions that retrain the model's decision-making without modifying its weights.

The Problem: Mismatch Between Global Training and Local Practice

Standard LLM prompting achieves reasonable agreement with Ugandan pediatric epilepsy prescriptions, but neurologist review reveals the errors aren't random. Instead, the model defaults to prescribing patterns from its training data, which reflect wealthy-country practice and drug availability. An LLM asked to recommend an anti-seizure medication for a child in Uganda may suggest a drug unavailable locally or contraindicated under resource constraints—a distribution mismatch that occurs even when the model correctly parses the patient's clinical history.

Equally critical: the system must know when not to recommend. Deferring uncertain cases to specialist review is often safer than confident wrong guesses, especially where expertise is scarce.

The Method: MANANA and Bayesian Prompt Averaging

MANANA (the acronym is not defined in the abstract, but the framework works as follows) learns local prescribing guidance from a small training set of patient visits with known prescriptions. Instead of fine-tuning parameters, it converts prescription errors into interpretable prompt instructions—rules or examples that guide the LLM's next prediction. These "prompt memories" are auditable; clinicians can read and validate them.

The framework has two variants: single-agent (one LLM instance with accumulated memories) and multi-agent (multiple specialized instances, presumably for different decision points). Both outperform classical machine learning baselines, standard LLM prompting, and other prompt-optimization methods on independently collected cohorts.

The key innovation is Bayesian prompt averaging. Rather than treating the learned prompts as hard rules, the authors convert them into probability distributions over medication options. This quantifies uncertainty: high confidence for one drug, ambiguity across several, or genuine doubt. Low-confidence predictions are automatically deferred to a specialist, while high-confidence predictions proceed.

Results and Impact

On a held-out Ugandan cohort, Bayesian prompt averaging improved top-3 prescription accuracy by 4–8 percentage points over prompt-optimization baselines. More importantly, selective prediction worked:

• Auto-handle the 50% most confident cases at 95% precision.
• Auto-handle the 25% most confident cases at 99% precision.
• Defer the remainder for human review.

This is practically valuable. A system that correctly handles half the routine cases and escalates complex ones reduces specialist bottlenecks without sacrificing safety.

The work directly addresses underrepresentation: Ugandan pediatric epilepsy data is scarce in LLM training sets, so generic models fail. MANANA works with dozens of local examples—feasible for a clinic to collect over months. The auditable prompts also build clinician trust, critical for adoption in settings where skepticism of algorithmic systems is reasonable.