LeVJEPA README

The project's README from github.com/MLO-lab/LeVJEPA. As one markdown file: README.md.

LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

Official implementation of LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective.

Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner

Paper on arXiv Project Page Model on Hugging Face

What is LeVJEPA?

Self-supervised video pretraining has so far relied on architectural machinery to avoid representation collapse: an EMA target encoder, a stop-gradient, a capacity-limited predictor (V-JEPA), or pixel-space reconstruction with a dedicated decoder (VideoMAE). LeVJEPA dispenses with all of it. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg (the Sketched Isotropic Gaussian Regularizer from LeJEPA), which constrains the embedding distribution to an isotropic Gaussian and thereby excludes collapse with a provable guarantee. The trainable architecture reduces to an encoder and a small projector, and the objective carries a single hyperparameter (λ = 0.02).

Because every operation contributes directly to the objective — there is no target-encoder forward pass and no predictor — the cost of a training step is governed only by the number of tokens the encoder observes. The paper develops two consequences of this:

Two further simplifications fall out of the same analysis: temporal patch aggregation at the input (tubelet 2) is unnecessary — per-frame tokenization matches or beats it at an equal token budget, including on motion-centric SSv2 — and dense, semantically organized patch representations emerge even though only the [cls] token is supervised, without the auxiliary patch-level objectives of prior work.

Results

All comparisons retrain the baselines on identical data with their official implementations, evaluated with frozen attentive probing following V-JEPA's protocol.

This repository contains the full pretraining recipe. The default configuration trains a ViT-B/16 on the Walking Tours dataset (Venkataramanan et al., Is ImageNet worth 1 video?, ICLR 2024), so you can reproduce video pretraining end to end from public data.

Pretrained checkpoint

If you just want features, skip the pretraining pipeline entirely: the ViT-L/16 from the data-scaling experiments is released as LeVJEPA-VideoMix-Large — 303M parameters, trained on VideoMix (1.8M clips from Kinetics-710, Something-Something v2, Walking Tours, and PE-Video) with tubelet 1, RoPE, and block-causal attention. It reaches 69.5% top-1 on ImageNet-1K and 55.0% on Something-Something-v2 under frozen attentive probing. The released tensors are the EMA copy of the encoder, which is what we evaluate.

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "galilai-group/LeVJEPA-VideoMix-Large", trust_remote_code=True
).eval()

# (B, C, T, H, W), ImageNet-normalised, 16 frames at 224px
video = torch.randn(1, 3, 16, 224, 224)

with torch.no_grad():
    out = model(pixel_values=video)

out.last_hidden_state   # (1, 3137, 1024) -- CLS + 16*14*14 patch tokens
out["pooler_output"]    # (1, 1024)       -- the CLS token

trust_remote_code=True is required because the modeling code (RoPE + block-causal attention) ships with the weights. For a single image, repeat it along the temporal axis: image.unsqueeze(2).repeat(1, 1, 16, 1, 1). Two things to keep in mind: normalise with the ImageNet statistics (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]; training sampled frames at ~7.5 fps, so 16 frames cover about two seconds), and leave config.attn_mode at its "block_causal" default — the weights were trained causal, and running them under full attention won't error but will quietly return worse features. The model card has the full usage guide and training details.

The checkpoint is intended for frozen feature extraction — attentive or linear probing, retrieval, or as a backbone for downstream heads; it is a self-supervised encoder with no classification head. Like module.py, the released weights are under CC BY-NC 4.0.

To see the features rather than just extract them, notebooks/feature_visualization.ipynb loads the checkpoint and reproduces the paper's dense-feature visualizations — patch-token PCA on images and query-patch cosine similarity on videos (uv sync --extra notebook, then uv run jupyter lab notebooks/feature_visualization.ipynb).

Setup

The project is managed with uv:

uv sync               # training environment (CUDA 12.8 wheels on Linux)
uv sync --extra data  # + decord/yt-dlp, needed only to build the dataset

Everything runs from the repo root; there is nothing to install as a package.

Getting Walking Tours

Walking Tours is 10 long (1-7 h) continuous first-person city walk videos. The HuggingFace dataset ships only the YouTube URLs, so the videos are downloaded from YouTube at 720p60 (25 GB) and then encoded into a frame-per-row Lance store at 15 fps, short edge 384, JPEG q90, in 128-frame episodes (33 GB):

# 1. download the 10 videos (needs outbound HTTPS; a login or data-transfer
#    node is fine, the step is network-bound)
bash scripts/download_walking_tours.sh

# 2. encode the Lance store (CPU-bound; use many cores)
uv run python scripts/build_lance_walking_tours.py --workers 16

On a slurm cluster, the same two steps as batch jobs (add your --partition/--account to the headers first):

dl=$(sbatch --parsable slurm/download_walking_tours.slurm)
sbatch --dependency=afterok:$dl slurm/build_walking_tours_lance.slurm

Both default to data/walking_tours/ under the repo root. To keep the data elsewhere, pass the paths explicitly and point training at it with LEVJEPA_DATA_ROOT=/path/to/data (the loader expects $LEVJEPA_DATA_ROOT/walking_tours/train.lance).

Training

If you'd rather start small, notebooks/training_workshop.ipynb walks through the whole method in miniature — building the model from this repo's components and training it inside the notebook on a few minutes of a Walking Tours video, with the same multi-crop transform and forward pass as the full pipeline. Both notebooks need only uv sync --extra notebook and run on a CUDA GPU, an Apple Silicon Mac (MPS), or plain CPU.

The full recipe lives in conf/config.yaml and follows our large-scale ViT-L video-mixture pretraining runs: ViT-B/16, 16-frame clips at stride 2 (7.5 fps), per-frame tokenization (tubelet 1) with 95% random token drop, block-causal attention, one global 224 crop + 10 local 96 crops, projector 256, SIGReg λ = 0.02 on the CLS token, effective batch 3072 clips (16 GPUs x 96 x grad-accum 2), lr 4e-4 with 1200-step warmup then flat, constant weight decay 0.04, and an EMA of the encoder saved into every checkpoint under state_dict_ema (evaluation checkpoint only — it plays no role in the objective). That is ~395 optimizer steps per epoch, so the default 26 epochs is ~10k steps; since the schedule is flat after warmup, trainer.max_epochs can be raised to train longer without changing anything else.

Token drop applies during training only — at 95% the encoder sees ~158 of the clip's 3137 tokens per step, which is what makes tubelet 1 affordable — while evaluation and inference always run on the full token set.

Slurm (2 nodes x 8 GPUs, matching the config defaults):

sbatch slurm/train_walking_tours_vitb.slurm

Single node x 8 GPUs — keep the effective batch by doubling the accumulation:

sbatch --nodes=1 slurm/train_walking_tours_vitb.slurm \
  trainer.num_nodes=1 accumulate_grad_batches=4

Locally (e.g. one GPU, for a smoke run — this changes the effective batch, so it is not the paper recipe):

uv run python main.py \
  trainer.devices=1 trainer.num_nodes=1 \
  loader.batch_size=8 loader.num_workers=6 accumulate_grad_batches=1

Any config key can be overridden the same way (optimizer.lr=2e-4, model.name=vit_large, ...). The encoder trains with the paper's block-causal attention by default; pass model.attn_mode=full for a fully bidirectional encoder. Hydra writes each run into outputs/<date>/<time>/, with checkpoints under checkpoints/ inside it. Logging goes to Weights & Biases when enabled: wandb.enabled=true wandb.config.entity=<you>.

To fine-tune or continue from a checkpoint with a fresh schedule, pass resume.ckpt_path=/path/to/last.ckpt (with the default resume.weights_only=true only the weights are loaded); set resume.weights_only=false to resume optimizer state and schedule as well.

Repository layout

Citation

@misc{kuhn2026levjepaefficientscalable,
      title={LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics}, 
      author={Lukas Kuhn and Lucas Maes and Giuseppe Serra and Quentin Le Lidec and Yann LeCun and Randall Balestriero and Florian Buettner},
      year={2026},
      eprint={2608.27395},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.27395}, 
}

License

MIT (see LICENSE), except module.py, which is adapted from Meta's V-JEPA and remains under CC BY-NC 4.0.

LeVJEPA (MLO-lab/LeVJEPA)

Searching Database…

270 ·GitHub
LeVJEPA
Что это такое?Официальная реализация LeVJEPA — видеомодели с самостоятельным (self-supervised) обучением, которая тренируется с помощью свободной от коллапса (collapse-free) функции потерь из LeJEPA. В отличие от V-JEPA или VideoMAE, ей не нужны целевой энкодер с EMA, stop-gradient, предиктор или декодер для восстановления пикселей: только энкодер, небольшой проектор и один гиперпараметр функции потерь.
Это библиотека, которую можно установить через pip, или кодовая база, которую я запускаю сам?Это исследовательская кодовая база, которую вы запускаете прямо из репозитория, а не устанавливаемый пакет: окружением управляет uv, обучение идёт через точку входа Hydra (main.py), и есть два Jupyter-ноутбука — один для визуализации признаков, другой — мини-воркшоп по обучению с пошаговым руководством.
Как его запустить?Склонируйте репозиторий и выполните 'uv sync' (для ноутбуков добавьте '--extra notebook'). Самый быстрый старт — выпущенная контрольная точка (checkpoint): загрузите LeVJEPA-VideoMix-Large с HuggingFace с trustremotecode=True или откройте notebooks/featurevisualization.ipynb, чтобы изучить её признаки.
Есть ли подводные камни при использовании выпущенной контрольной точки?Несколько: нужен trustremotecode=True, так как код модели поставляется вместе с весами; на вход подаются 16 кадров (~2 секунды при 7.5 кадрах/с) с нормализацией ImageNet, а статичное изображение повторяется вдоль временной оси; оставьте блоково-каузальное (blockcausal) внимание по умолчанию — полное внимание тоже работает, но незаметно ухудшает признаки.
Какие-то части некоммерческие?Репозиторий под лицензией MIT, но module.py (адаптирован из V-JEPA от Meta) и веса выпущенной контрольной точки — под CC BY-NC 4.0, то есть эти части можно использовать только в некоммерческих целях.
Могу ли я пропустить обучение и просто скачать предобученные веса?Да — выпущена LeVJEPA-VideoMix-Large (ViT-L/16, 303 млн параметров), обученная на 1,8 млн клипов из Kinetics-710, Something-Something v2, Walking Tours и PE-Video. При замороженном внимательном пробировании (attentive probing) она даёт 69,5% top-1 на ImageNet-1K и 55,0% на SSv2, и предназначена для извлечения признаков в замороженном виде — классификационной головы у неё нет.
На каких данных обучается рецепт по умолчанию?Конфигурация по умолчанию обучает ViT-B/16 на Walking Tours: 10 видео городских прогулок от первого лица (по 1–7 часов каждое), скачанных с YouTube в 720p60 (~25 ГБ) и закодированных в хранилище кадров Lance размером ~33 ГБ, так что предобучение воспроизводится от начала до конца на общедоступных данных. Для других данных достаточно yaml-файла в conf/data/ для хранилища Lance с той же схемой.
Как он сравнивается с V-JEPA 2?При равном числе эпох он не уступает или превосходит V-JEPA 2 на всех размерах ViT-S/B/L, затрачивая в 5,6–20,8 раза меньше суммарных вычислений на предобучение — LeVJEPA ViT-L обходится дешевле, чем половина вычислений V-JEPA 2 ViT-S. При равном бюджете FLOP (ViT-B) он выигрывает +7,6 пункта на ImageNet-1K (61,0%) и показывает лучший линейный пробинг на Kinetics-400 (44,6%).
Почему во время обучения отбрасывается 95% токенов патчей?С функцией потерь, свободной от коллапса, отбрасывание токенов работает как аугментация, а не как приближение: равномерное случайное отбрасывание монотонно повышает точность пробинга на замороженном ImageNet (с 33,9% при всех токенах до 47,6% при отбрасывании 95%), одновременно снижая стоимость каждого шага до 20 раз, тогда как структурированное трубчатое маскирование (tube masking) заметно вредит (с 50,7% до 39,6%).
What are similar projects?dinov3, dinov2, sam2, Swin-Transformer, pointdit, OraRL, CLIP, pytorch-image-models
Which topics is it in?video-understanding, self-supervised-learning, vision-transformer, attention, pretrained-models, embeddings, jupyter notebook, transformer, video, huggingface, computer-vision, cuda
Creator
MLO-lab
Size
1.4 MB
Stars
270
Updated
CommitSep 02
Age
1mo
Commits · 90 Days