Neuro-JEPA: A Sparse Latent-Predictive Foundation Model for Multimodal Brain MRI
— by Vivax
On 12 June 2026, researchers at NYU Langone Health released Neuro-JEPA — a sparse, multimodal foundation model for brain MRI that extends Yann LeCun's…
Brain MRI is rarely a single image. A typical exam stacks several complementary sequences — T1-weighted anatomy, fluid-sensitive T2-weighted contrast, and FLAIR — each highlighting different tissue. Yet most learning methods treat these sequences in isolation, and the field has lacked a way to learn one unified representation across MRI contrast mechanisms at health-system scale. Neuro-JEPA, introduced in a June 2026 preprint from NYU Langone Health, sets out to fix that with a single foundation model that encodes T1w, T2w and FLAIR together.
The model's design is what makes it interesting to us. Instead of reconstructing every voxel — the generative approach — Neuro-JEPA learns by prediction in latent space: it masks part of a scan and predicts the hidden region's representation, the Joint-Embedding Predictive Architecture (JEPA) idea Yann LeCun has championed. Neuro-JEPA extends Meta's V-JEPA 2 video model to 3D brain volumes and pairs it with a sparse Mixture-of-Experts vision transformer (ViT-Base-MoE), so different 'experts' can specialize while keeping inference efficient. The authors also run a systematic study of the masking, objective and sparsity choices that make multimodal neuroimaging representations robust.
The scale is substantial for medical imaging. Neuro-JEPA was pretrained on 1,551,862 scans drawn from 428,647 imaging studies across 282,693 unique patients at NYU Langone, after modality-specific preprocessing and curation across the three core structural sequences. That kind of health-system-scale, real-world clinical data — rather than small curated research sets — is exactly the regime where foundation models are supposed to shine, and where grounded medical AI ultimately has to work.
Crucially, the team evaluated honestly. They assembled one of the broadest neuroimaging benchmarks to date — 25 clinical tasks across three health systems (NYU Langone, NYU Long Island, and Massachusetts General Hospital) plus 22 tasks from twelve public datasets, 47 in total, covering diagnosis, prognosis, regression, multimodal and time-to-event prediction. Their finding is sobering: existing neuroimaging foundation models showed inconsistent gains over a simple convolutional-network baseline, while Neuro-JEPA delivered stronger and more consistent performance across every setting. The paper argues, rightly, that foundation-model evaluations must include simple baselines and clinically heterogeneous cohorts.
We follow Neuro-JEPA closely at Vivax because it sits exactly where our thesis lives: predictive-embedding, world-model ideas applied to grounded clinical AI. A latent-predictive model that learns how brain anatomy and pathology hang together — rather than memorizing pixels — is the kind of representation that can support decision-grade reasoning across messy, multimodal patient data. The code and a gated model are openly available for non-commercial research (CC-BY-NC-ND 4.0), and we're keen to study how this approach compares with our own multimodal medical-imaging work.