LeVJEPA Makes Video Pretraining Cheaper by Removing JEPA’s Collapse-Avoidance Scaffolding
— by Vivax
Kuhn et al.’s arXiv preprint asks whether video JEPA training needs its usual anti-collapse machinery.
Video carries temporal structure, but self-supervised video learning can be expensive: many frames become many patch tokens, and common JEPA-style recipes add separate target networks and prediction machinery. LeVJEPA, introduced in the arXiv preprint by Lukas Kuhn and colleagues, asks whether a simpler formulation can learn useful video representations. It is a video encoder for representation learning, not an action-conditioned simulator, planner, or complete world model.
Prior V-JEPA-family training commonly uses an asymmetric student and target setup, with an EMA target encoder, stop-gradient, and a predictor helping avert a trivial constant representation. Those components are practical collapse-avoidance scaffolding, not merely decorative complexity. LeVJEPA tests a different proposition: a single shared encoder can be trained with gradients flowing through both views when the objective includes an explicit regularizer.
The recipe makes global and local views of a video, passes both through one shared block-causal encoder and a projector, then aligns their representations with an invariance objective. SIGReg—Sketched Isotropic Gaussian Regularization—provides the anti-collapse term. The authors theoretically claim SIGReg excludes representation collapse under their setup; that is a claim about the paper’s objective, not proof of semantic understanding, downstream reliability, or safety.
Its most striking systems choice is to discard tokens before encoding. The paper drops 95% of tokens per view, reducing work in the encoder; in its ablation, frozen ImageNet-1K accuracy rises from 33.9% to 47.6% as the drop rate moves from ρ=0 to 0.95. Block-causal attention means a frame representation can use present and past frames rather than future frames. The reported gain is dataset- and protocol-specific, not a rule that more dropping always helps.
The benchmark record is encouraging but mixed. At fixed FLOPs with ViT-B frozen probes, LeVJEPA reports 61.0 on ImageNet-1K, 40.4 on Something-Something v2, and 44.6 on Kinetics-400; V-JEPA 2 reports 51.6, 42.5, and 40.7. Against DINOv2 on the same video data and equal FLOPs, DINOv2 leads ImageNet-1K (53.8% versus 50.7%) while LeVJEPA leads SSv2 (30.4% versus 16.9%). These are frozen-probe transfer results, not clinical, robotic-control, or safety evaluations.
For Vivax, cheaper temporally ordered self-supervised representations are relevant research ingredients because longitudinal events and changing patient state matter in clinical world-model work. Vivax’s position is to evaluate representation ideas such as LeVJEPA only with clinically grounded, privacy-conscious, task-specific, and safety-oriented evidence—not to claim that this preprint models patients, diagnoses disease, or is ready for care. Its honest contribution is a possible reduction in the cost of experiments on temporal data; clinical utility remains unestablished.