LeVJEPA README

The project's README from github.com/MLO-lab/LeVJEPA. As one markdown file: README.md.

LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

Official implementation of LeVJEPA, the first video encoder trained under LeJEPA's collapse-free objective.

Lukas Kuhn, Lucas Maes, Giuseppe Serra, Quentin Le Lidec, Yann LeCun, Randall Balestriero, Florian Buettner

Paper on arXiv Project Page Model on Hugging Face

What is LeVJEPA?

Self-supervised video pretraining has so far relied on architectural machinery to avoid representation collapse: an EMA target encoder, a stop-gradient, a capacity-limited predictor (V-JEPA), or pixel-space reconstruction with a dedicated decoder (VideoMAE). LeVJEPA dispenses with all of it. A single encoder is trained with an invariance loss over global and local views of a clip, regularized by SIGReg (the Sketched Isotropic Gaussian Regularizer from LeJEPA), which constrains the embedding distribution to an isotropic Gaussian and thereby excludes collapse with a provable guarantee. The trainable architecture reduces to an encoder and a small projector, and the objective carries a single hyperparameter (λ = 0.02).

Because every operation contributes directly to the objective — there is no target-encoder forward pass and no predictor — the cost of a training step is governed only by the number of tokens the encoder observes. The paper develops two consequences of this:

Two further simplifications fall out of the same analysis: temporal patch aggregation at the input (tubelet 2) is unnecessary — per-frame tokenization matches or beats it at an equal token budget, including on motion-centric SSv2 — and dense, semantically organized patch representations emerge even though only the [cls] token is supervised, without the auxiliary patch-level objectives of prior work.

Results

All comparisons retrain the baselines on identical data with their official implementations, evaluated with frozen attentive probing following V-JEPA's protocol.

This repository contains the full pretraining recipe. The default configuration trains a ViT-B/16 on the Walking Tours dataset (Venkataramanan et al., Is ImageNet worth 1 video?, ICLR 2024), so you can reproduce video pretraining end to end from public data.

Pretrained checkpoint

If you just want features, skip the pretraining pipeline entirely: the ViT-L/16 from the data-scaling experiments is released as LeVJEPA-VideoMix-Large — 303M parameters, trained on VideoMix (1.8M clips from Kinetics-710, Something-Something v2, Walking Tours, and PE-Video) with tubelet 1, RoPE, and block-causal attention. It reaches 69.5% top-1 on ImageNet-1K and 55.0% on Something-Something-v2 under frozen attentive probing. The released tensors are the EMA copy of the encoder, which is what we evaluate.

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "galilai-group/LeVJEPA-VideoMix-Large", trust_remote_code=True
).eval()

# (B, C, T, H, W), ImageNet-normalised, 16 frames at 224px
video = torch.randn(1, 3, 16, 224, 224)

with torch.no_grad():
    out = model(pixel_values=video)

out.last_hidden_state   # (1, 3137, 1024) -- CLS + 16*14*14 patch tokens
out["pooler_output"]    # (1, 1024)       -- the CLS token

trust_remote_code=True is required because the modeling code (RoPE + block-causal attention) ships with the weights. For a single image, repeat it along the temporal axis: image.unsqueeze(2).repeat(1, 1, 16, 1, 1). Two things to keep in mind: normalise with the ImageNet statistics (mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]; training sampled frames at ~7.5 fps, so 16 frames cover about two seconds), and leave config.attn_mode at its "block_causal" default — the weights were trained causal, and running them under full attention won't error but will quietly return worse features. The model card has the full usage guide and training details.

The checkpoint is intended for frozen feature extraction — attentive or linear probing, retrieval, or as a backbone for downstream heads; it is a self-supervised encoder with no classification head. Like module.py, the released weights are under CC BY-NC 4.0.

To see the features rather than just extract them, notebooks/feature_visualization.ipynb loads the checkpoint and reproduces the paper's dense-feature visualizations — patch-token PCA on images and query-patch cosine similarity on videos (uv sync --extra notebook, then uv run jupyter lab notebooks/feature_visualization.ipynb).

Setup

The project is managed with uv:

uv sync               # training environment (CUDA 12.8 wheels on Linux)
uv sync --extra data  # + decord/yt-dlp, needed only to build the dataset

Everything runs from the repo root; there is nothing to install as a package.

Getting Walking Tours

Walking Tours is 10 long (1-7 h) continuous first-person city walk videos. The HuggingFace dataset ships only the YouTube URLs, so the videos are downloaded from YouTube at 720p60 (25 GB) and then encoded into a frame-per-row Lance store at 15 fps, short edge 384, JPEG q90, in 128-frame episodes (33 GB):

# 1. download the 10 videos (needs outbound HTTPS; a login or data-transfer
#    node is fine, the step is network-bound)
bash scripts/download_walking_tours.sh

# 2. encode the Lance store (CPU-bound; use many cores)
uv run python scripts/build_lance_walking_tours.py --workers 16

On a slurm cluster, the same two steps as batch jobs (add your --partition/--account to the headers first):

dl=$(sbatch --parsable slurm/download_walking_tours.slurm)
sbatch --dependency=afterok:$dl slurm/build_walking_tours_lance.slurm

Both default to data/walking_tours/ under the repo root. To keep the data elsewhere, pass the paths explicitly and point training at it with LEVJEPA_DATA_ROOT=/path/to/data (the loader expects $LEVJEPA_DATA_ROOT/walking_tours/train.lance).

Training

If you'd rather start small, notebooks/training_workshop.ipynb walks through the whole method in miniature — building the model from this repo's components and training it inside the notebook on a few minutes of a Walking Tours video, with the same multi-crop transform and forward pass as the full pipeline. Both notebooks need only uv sync --extra notebook and run on a CUDA GPU, an Apple Silicon Mac (MPS), or plain CPU.

The full recipe lives in conf/config.yaml and follows our large-scale ViT-L video-mixture pretraining runs: ViT-B/16, 16-frame clips at stride 2 (7.5 fps), per-frame tokenization (tubelet 1) with 95% random token drop, block-causal attention, one global 224 crop + 10 local 96 crops, projector 256, SIGReg λ = 0.02 on the CLS token, effective batch 3072 clips (16 GPUs x 96 x grad-accum 2), lr 4e-4 with 1200-step warmup then flat, constant weight decay 0.04, and an EMA of the encoder saved into every checkpoint under state_dict_ema (evaluation checkpoint only — it plays no role in the objective). That is ~395 optimizer steps per epoch, so the default 26 epochs is ~10k steps; since the schedule is flat after warmup, trainer.max_epochs can be raised to train longer without changing anything else.

Token drop applies during training only — at 95% the encoder sees ~158 of the clip's 3137 tokens per step, which is what makes tubelet 1 affordable — while evaluation and inference always run on the full token set.

Slurm (2 nodes x 8 GPUs, matching the config defaults):

sbatch slurm/train_walking_tours_vitb.slurm

Single node x 8 GPUs — keep the effective batch by doubling the accumulation:

sbatch --nodes=1 slurm/train_walking_tours_vitb.slurm \
  trainer.num_nodes=1 accumulate_grad_batches=4

Locally (e.g. one GPU, for a smoke run — this changes the effective batch, so it is not the paper recipe):

uv run python main.py \
  trainer.devices=1 trainer.num_nodes=1 \
  loader.batch_size=8 loader.num_workers=6 accumulate_grad_batches=1

Any config key can be overridden the same way (optimizer.lr=2e-4, model.name=vit_large, ...). The encoder trains with the paper's block-causal attention by default; pass model.attn_mode=full for a fully bidirectional encoder. Hydra writes each run into outputs/<date>/<time>/, with checkpoints under checkpoints/ inside it. Logging goes to Weights & Biases when enabled: wandb.enabled=true wandb.config.entity=<you>.

To fine-tune or continue from a checkpoint with a fresh schedule, pass resume.ckpt_path=/path/to/last.ckpt (with the default resume.weights_only=true only the weights are loaded); set resume.weights_only=false to resume optimizer state and schedule as well.

Repository layout

Citation

@misc{kuhn2026levjepaefficientscalable,
      title={LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics}, 
      author={Lukas Kuhn and Lucas Maes and Giuseppe Serra and Quentin Le Lidec and Yann LeCun and Randall Balestriero and Florian Buettner},
      year={2026},
      eprint={2608.27395},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.27395}, 
}

License

MIT (see LICENSE), except module.py, which is adapted from Meta's V-JEPA and remains under CC BY-NC 4.0.

LeVJEPA (MLO-lab/LeVJEPA)

Searching Database…

270 ·GitHub
LeVJEPA
What is this?The official implementation of LeVJEPA — a self-supervised video encoder trained with LeJEPA's collapse-free objective. Unlike V-JEPA or VideoMAE it needs no EMA target encoder, stop-gradient, predictor, or pixel-reconstruction decoder: just an encoder, a small projector, and one loss hyperparameter.
Is this a library I can pip install, or a codebase I run myself?It's a research codebase you run from the repo, not an installable package: uv manages the environment, training goes through a Hydra entry point (main.py), and there are two Jupyter notebooks — one for feature visualization, one a guided mini training workshop.
How do I get it running?Clone the repo and run 'uv sync' (add '--extra notebook' for the notebooks). The fastest start is the released checkpoint: load LeVJEPA-VideoMix-Large from HuggingFace with trustremotecode=True, or open notebooks/featurevisualization.ipynb to inspect its features.
Any gotchas when using the released checkpoint?A few: it needs trustremotecode=True since the modeling code ships with the weights; inputs are 16 frames (~2 s at 7.5 fps) with ImageNet normalization, and a still image is repeated along the temporal axis; keep the blockcausal attention default — full attention runs but silently degrades features.
Is any of it non-commercial?The repository is MIT, but module.py (adapted from Meta's V-JEPA) and the released checkpoint weights are CC BY-NC 4.0 — so those parts are non-commercial.
Can I skip training and just download pretrained weights?Yes — LeVJEPA-VideoMix-Large (ViT-L/16, 303M parameters) is released, trained on 1.8M clips from Kinetics-710, Something-Something v2, Walking Tours, and PE-Video. It hits 69.5% top-1 on ImageNet-1K and 55.0% on SSv2 under frozen attentive probing, and is intended for frozen feature extraction — it has no classification head.
What data does the default recipe train on?The default config trains a ViT-B/16 on Walking Tours: 10 first-person city-walk videos (1-7 h each) downloaded from YouTube at 720p60 (~25 GB) and encoded into a ~33 GB Lance frame store, so pretraining reproduces end to end from public data. Other data only needs a yaml under conf/data/ for a Lance store with the same schema.
How does it compare to V-JEPA 2?Epoch-matched it matches or surpasses V-JEPA 2 across ViT-S/B/L at 5.6-20.8x less total pretraining compute — the LeVJEPA ViT-L costs less than half the compute of the V-JEPA 2 ViT-S. At an equal FLOP budget (ViT-B) it leads by +7.6 points on ImageNet-1K (61.0%) and takes the best Kinetics-400 linear probe (44.6%).
Why does it drop 95% of patch tokens during training?With the collapse-free objective, token dropping works as an augmentation rather than an approximation: uniform random dropping raises frozen ImageNet probing accuracy monotonically (33.9% with all tokens to 47.6% at 95% drop) while cutting per-step cost by up to 20x, whereas structured tube masking actively hurts (50.7% to 39.6%).
What are similar projects?dinov3, dinov2, sam2, Swin-Transformer, pointdit, OraRL, CLIP, pytorch-image-models
Which topics is it in?video-understanding, self-supervised-learning, vision-transformer, attention, pretrained-models, embeddings, jupyter notebook, transformer, video, huggingface, computer-vision, cuda
Creator
MLO-lab
Size
1.4 MB
Stars
270
Updated
CommitSep 02
Age
1mo
Commits · 90 Days