CardioState-JEPA: Delay-Aware Cross-Modal Learning of a Shared Cardiac Representation

The Core Insight

Three different heart sensors—ECG (electrical), PPG (blood flow), and PCG (sound)—measure the same heartbeat from different angles. Existing cardiac AI models treat each sensor independently, missing the fact that they're fundamentally measuring the same biology. CardioState-JEPA learns a single shared representation across all three by training the model to predict what one sensor "says" given another, without requiring that the sensors be perfectly synchronized in time.

How It Works

The model faces two main challenges: heterogeneity and delay.

Handling heterogeneity. ECG, PPG, and PCG are fundamentally different data types—different sampling rates, different physical phenomena, different noise patterns. The authors solve this by converting all three into tokens (discrete representations) using modality-specific encoders, then feeding them into a shared Transformer. Rather than predicting raw waveforms, the model learns by masking latent "cardiac states"—the abstract representation of what's happening in the heart cycle—and predicting them from other modalities. This focuses learning on shared physiology, not sensor-specific noise.

Handling delay. Electrical events (ECG) happen before mechanical events (PCG), which happen before blood-pressure changes (PPG). These delays vary slightly between heartbeats and across individuals. Instead of assuming perfect alignment, the model includes a learned "delay aligner"—essentially a trainable module that learns to shift signals into alignment in latent time before making cross-modal predictions.

Two-stage training. Synchronized multi-sensor recordings are rare and expensive. So the authors first pretrain the model on abundant single-sensor data (learning the structure within ECG, within PPG, within PCG separately), then fine-tune alignment using the smaller pool of paired multi-sensor recordings. This pragmatic approach leverages both cheap unimodal and expensive multimodal data.

Why It Matters

The results are substantial. When frozen as a feature extractor and evaluated on 25 downstream classification tasks:

• PPG classification: +8.2 AUROC points over self-supervised baselines
• PCG murmur detection: +18.8 AUROC points—a large gain for a clinically important task
• ECG classification: +15.5 AUROC points, matching or exceeding models trained with supervised labels or clinical text on several benchmarks

These numbers demonstrate that the model genuinely learns shared cardiac physiology, not just memorizing sensor-specific patterns. A murmur detection improvement of nearly 19 points is substantial in clinical terms.

The work sidesteps a practical bottleneck: instead of requiring massive synchronized multi-sensor datasets (which are expensive to collect), it combines the abundance of single-sensor recordings with smaller amounts of paired data. This makes the approach scalable.

The deeper contribution is conceptual: it proves that heterogeneous cardiac signals can act as mutual supervisors. Each modality teaches the model about cardiac state, constraining and enriching what the others learn. For wearable health monitoring and clinical diagnosis, this could enable richer phenotyping from mixed-sensor systems without needing to train separate models per device.