AdaJEPA — Yann LeCun's Lab Builds a World Model That Keeps Learning While It Acts
— by Vivax
A world model that stops learning the moment it is deployed will plan against the wrong imagined future the moment reality drifts.
NYU's Agentic Learning AI Lab and AMI Labs have released AdaJEPA: An Adaptive Latent World Model — new work by Ying Wang, Oumayma Bounou, Yann LeCun and Mengye Ren, with the paper on arXiv and code on GitHub. The one-line idea: instead of freezing a world model after training and hoping it generalizes, AdaJEPA adapts the model during deployment, inside the planning loop itself. Every action the agent executes produces an observed transition, and AdaJEPA uses that transition as a self-supervised training signal before the next replan. Plan, act, adapt, replan.
AdaJEPA starts from a pretrained JEPA world model — a sensory encoder, an action encoder, and a latent predictor — and plans with model-predictive control in latent space. After each executed step, the observed transition goes into a small online buffer and the model takes one gradient step on the latent prediction error before the next replan, updating only the final layers of the visual encoder and predictor over a five-transition buffer. Under distribution shift — held-out object shapes, corrupted observations, altered physics, unseen maze layouts — the gains are consistent, and in low-data regimes test-time adaptation can more than double the frozen model's success rate.
The takeaway is bigger than the benchmarks: world models should continue learning during deployment rather than remain frozen after training. For medicine that is the core problem restated — a clinical world model is deployed into permanent distribution shift, and AdaJEPA is a blueprint for grounded models that recalibrate from the transitions they actually observe under tight, auditable update budgets.