SiamJEPA — On the Role of Siamese Student Encoders in JEPA
— by Vivax
Makoto Yamada (OIST; CEO of FlatMinima) has published "On the Role of Siamese Student Encoders in JEPA" — SiamJEPA, a JEPA framework that replaces the single…
Makoto Yamada — professor at the Okinawa Institute of Science and Technology (OIST) and CEO of FlatMinima Inc. — has posted SiamJEPA: On the Role of Siamese Student Encoders in JEPA to arXiv. JEPA, the Joint-Embedding Predictive Architecture, has become a leading recipe for self-supervised representation learning, and its best-known instances — I-JEPA for images, V-JEPA for video — all share one structural choice: a single student encoder paired with an exponential-moving-average (EMA) teacher. Yamada asks the question nobody had isolated: what is the role of a Siamese student encoder in JEPA? To answer it he builds SiamJEPA — a JEPA with masked Siamese student encoders and an EMA teacher — and turns the Siamese constraint into a dial he can switch off and on.
The architecture adds one mirrored branch: two identical student encoders each see disjoint masked views of the same image, a first similarity loss (Sim-1) aligns the two students, and a second (Sim-2) aligns the student prediction to the EMA teacher. A KL-divergence regularizer between the students, weighted by a coefficient lambda_KL, controls whether the Siamese pair matters at all — at a tiny 1e-5 the students never converge and the system behaves like ordinary JEPA, while at 0.01 the Sim-1 loss collapses and the Siamese constraint kicks in. With it on, the Siamese students act as an effective regularizer: representations become more separable, early training accelerates, and under limited compute SiamJEPA beats single-encoder JEPA and clears MAE's linear-probe accuracy despite MAE needing far longer training.
On ImageNet-1K linear probing with a ViT-Base, SiamJEPA climbs from 62.0% at 101 epochs to 70.7% at 400, against MAE's 61.9% at 400 and 68.0% at 1600 and a tiny-lambda_KL JEPA-like control at 68.9%. Yamada frames SiamJEPA as a JEPA formulation of PhiNet, his brain-inspired temporal-prediction model, so the conclusion is a claim about inductive bias. For a lab building a clinical world model, a Siamese prior that extracts more representation quality per epoch is exactly the kind of lever that matters where compute is scarce and every epoch on real clinical signal is expensive.