SCoPE README

The project's README from github.com/TencentARC/SCoPE. As one markdown file: README.md.

SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers

Minghao Yin · Jiahao Lu · Wenbo Hu · Wang Zhao · Ying Shan · Kai Han

Project Page arXiv Demo Model License

SCoPE method overview Six camera-controlled SCoPE video results in a 3-by-2 grid

SCoPE adds camera sightlines as positional coordinates to a pretrained video diffusion transformer. Given a first frame, a text prompt, and a camera trajectory, it generates a video that follows the requested camera motion while preserving the original image-to-video prior.

🚀 Try It Online

Try SCoPE in your browser — no setup required: 🤗 Hugging Face Space

📑 Table of Contents

Overview

Video diffusion transformers address their tokens by position on the pixel-time grid: an address in the tensor, not in the world. The world point a token depicts lies on a surface that has not been generated yet, but its camera ray is fixed the moment the user specifies a trajectory. SCoPE therefore treats the ray as a second positional coordinate, so camera control becomes a property of the coordinate system rather than an added module.

The ray is added to the pretrained attention's queries and keys, and the score gains a term that reads the two rays alone; its canonical form — the reciprocal product of line geometry — measures how nearly two lines of sight meet. A Normalize-Gate-Inject scheme makes a single encoding trainable across both metric and up-to-scale pose sources. The retrofit keeps RoPE bit-exact, starts from the unchanged pretrained DiT, and adds under 0.1% new parameters.

In practice this means:

Repository structure

Path Description
inference.py Inference entry: manifest, case + external pose, custom, or all-trajectory generation.
train.py Training entry for the RDPO high-only recipe.
scope/encoding.py Sightline-Coordinate positional encoding (Normalize-Gate-Inject).
scope/geometry.py Camera-ray and Plücker-coordinate utilities.
scope/camera.py Converts camera trajectories into SCoPE coordinates.
scope/modeling.py Wan self-attention augmented with the SCoPE encoding.
scope/patch.py Installs SCoPE attention into both Wan2.2-A14B experts.
scope/pipeline.py Wan2.2-A14B inference pipeline with camera conditioning.
scope/weights.py Loads the complete SCoPE model from sharded weights.
scope/config.py Inference defaults matching the released checkpoint.
scope/training.py Config-driven RDPO high-only training loop.
scope/data/ Four native dataset loaders + the shared pose convention.
configs/ Training YAML and the default negative prompt.
scripts/ overlay_camera.py, estimate_near_depth.py, build_omniworld_index.py, train.sh, inference.sh.
examples/ manifest.json, example first frames, and camera trajectories.
assets/ Teaser image and demo GIFs.
diffsynth/ Vendored DiffSynth code required by inference.
tests/ Unit tests.

Installation

SCoPE requires Python 3.11 and a CUDA-capable GPU. The released weights were trained and evaluated with PyTorch 2.9.1 (CUDA 12.8); because changing the PyTorch version can change the numerical output, we recommend reproducing this exact environment.

Recommended — uv (resolves the pinned CUDA 12.8 torch build):

git clone https://github.com/TencentARC/SCoPE.git
cd SCoPE
uv sync
source .venv/bin/activate

Alternative — pip with the PyTorch CUDA 12.8 wheel index:

conda create -n scope python=3.11 -y
conda activate scope
pip install -e . --extra-index-url https://download.pytorch.org/whl/cu128

Optional extras:

Model download

Download the SCoPE model from Hugging Face:

pip install -U huggingface_hub
hf download TencentARC/SCoPE --local-dir checkpoints/SCoPE

The checkpoint is approximately 67 GB. Keep both the checkpoint and the Hugging Face cache on local storage.

Inference

All inference is driven by inference.py. The repository ships example first frames, prompts, and camera trajectories under examples/, so every command below runs out of the box. See Inference options for the full flag list.

Quick start — choose a trajectory

Input: one bundled case.

Output: one 81-frame video using the camera trajectory selected in the terminal (480 × 832, seed 42).

Pass a case without --trajectory to see its available camera poses and choose one by number or trajectory id:

python inference.py \
  --model_path checkpoints/SCoPE \
  --case omni-misty-forest \
  --output_path outputs/omni-misty-forest__truck_right.mp4

The terminal will prompt before loading the model:

Available trajectories for 'omni-misty-forest':
  [1] real_split0_000041  (poses/omni-misty-forest/real_split0_000041.npy)
  [2] real_split0_000081  (poses/omni-misty-forest/real_split0_000081.npy)
  [3] real_split0_000161  (poses/omni-misty-forest/real_split0_000161.npy)
  [4] snake_fwd  (poses/omni-misty-forest/snake_fwd.npy)
  [5] truck_right  (poses/omni-misty-forest/truck_right.npy)
Select a trajectory [1-5] or enter its id: 5
Input first frame Output (truck_right)

The output is shown with the camera-control HUD overlay (see Visualizing camera control); the bottom-right inset traces the driving camera path.

Specify a trajectory directly

This is the non-interactive form of the same bundled-case inference. Pass the trajectory id directly for scripts, cluster jobs, and exactly reproducible commands:

python inference.py \
  --model_path checkpoints/SCoPE \
  --case omni-misty-forest \
  --trajectory truck_right \
  --output_path outputs/omni-misty-forest__truck_right.mp4

bash scripts/inference.sh wraps this explicit form. It generates the same result as choosing truck_right in the Quick Start selector.

All trajectories for one scene

Input: one first frame + every trajectory bundled for that scene. Output: a series of videos (one per trajectory), generated with a single model load.

python inference.py \
  --model_path checkpoints/SCoPE \
  --case omni-misty-forest \
  --all_trajectories \
  --output_dir outputs/omni-misty-forest

The same first frame driven by all five bundled trajectories. truck_right and snake_fwd are synthetic camera paths; GT 1–3 are the scene's own recorded OmniWorld camera paths (use their --trajectory ids to reproduce them).
Input first frame truck_right snake_fwd
truck_right snake_fwd
GT 1 · real_split0_000041 GT 2 · real_split0_000081 GT 3 · real_split0_000161
gt1 gt2 gt3

Existing MP4 files are skipped, so interrupted runs can be resumed with the same command.

Existing case with an external trajectory

Use a bundled case's first frame, prompt, x_fov, and xi, but drive it with any compatible local pose file. This avoids copying case metadata or searching for a matching pose directory.

python inference.py \
  --model_path checkpoints/SCoPE \
  --case omni-misty-forest \
  --camera_path path/to/external_camera_poses.npy \
  --output_path outputs/omni-misty-forest__external.mp4

The pose must still use the same [81,3,4] or [81,4,4] OpenCV camera-to-world convention. SCoPE validates its shape and finite values before loading model weights. Use --x_fov or --xi only when the external camera requires values different from the selected case. --trajectory and --camera_path are alternative pose sources and cannot be used together.

Custom image and trajectory

Input: your own first frame, prompt, OpenCV camera-to-world poses, and horizontal FOV. Output: one video following your trajectory.

python inference.py \
  --model_path checkpoints/SCoPE \
  --input_image path/to/first_frame.png \
  --prompt "A person walks along a misty forest trail." \
  --camera_path path/to/camera_poses.npy \
  --x_fov 1.11847 \
  --output_path outputs/custom.mp4

camera_poses.npy must have shape [81, 3, 4] or [81, 4, 4] and use OpenCV camera-to-world coordinates. x_fov is the horizontal field of view in radians. Pinhole cameras use the default xi=0; unified camera models can set --xi explicitly.

Visualizing camera control

Input: a generated video + the camera trajectory used to produce it. Output: the same video with a camera-frustum HUD (and optional WASD keys) composited in.

scripts/overlay_camera.py derives the overlay from the pose: it renders the accumulated camera frustum in the bottom-right corner and, for keyboard-style trajectories, the active WASD keys in the bottom-left corner. At runtime it also prints a lightweight WASD assessment; when horizontal motion is too small or too few frames map to an active key, it recommends --hide_wasd. The assessment is advisory and does not change the output automatically. Every demo video in this README was produced with this tool.

# Camera-frustum HUD only (recommended for scenic / dolly / orbit motions):
python scripts/overlay_camera.py outputs/omni-misty-forest__truck_right.mp4 \
  --pose examples/poses/omni-misty-forest/truck_right.npy \
  --out_dir outputs/overlay --hide_wasd

# Add the WASD key indicator (for keyboard-style / drone trajectories):
python scripts/overlay_camera.py outputs/clip.mp4 --pose path/to/pose.npy --out_dir outputs/overlay

Requires the viz extra (pip install -e .[viz]) and ffmpeg on PATH.

Inference options

Option Default Description
--model_path TencentARC/SCoPE Local path or Hugging Face id of the SCoPE model.
--manifest examples/manifest.json Example manifest for --case / --trajectory.
--case first case Example case id; when supplied alone, choose its trajectory interactively.
--trajectory – Optional bundled trajectory id; bypasses the interactive selector and is required for non-interactive jobs using --case.
--all_trajectories off Generate every trajectory of --case (uses --output_dir).
--input_image – Custom first frame (with --prompt, --camera_path, --x_fov).
--prompt – Custom text prompt.
--camera_path – External pose .npy; use with --case or all custom-input flags.
--x_fov case value / required for custom Horizontal field of view in radians.
--xi case value / 0.0 for custom Unified-camera distortion parameter.
--output_path outputs/sample.mp4 Output file for a single generation.
--output_dir outputs Output directory for --all_trajectories.
--overwrite off Regenerate existing outputs (--all_trajectories).
--negative_prompt configs/negative_prompt.txt Negative prompt file.
--seed 42 Random seed.
--cache_dir – Hugging Face cache directory.
--vram_limit_gb – VRAM budget for offloading (enables VRAM management).

Training

The public training entry reproduces the RDPO high-noise recipe used for SCoPE: only the high-noise expert is optimized, and training timesteps are sampled from [0.9, 1.0). The released mixture concatenates four datasets — RealEstate10K, DL3DV, PanShot, and OmniWorld — each read by its own native loader in scope/data/.

A run is described by a single YAML config:

python train.py --config configs/train_rdpo_high_only.yaml --num_gpus 8

scripts/train.sh wraps this command. Multi-GPU training uses FSDP automatically; the model is large, so multi-GPU training is strongly recommended.

Training options

Option Default Description
--config configs/train_rdpo_high_only.yaml Training YAML (data mixture, trainer, optimizer).
--model_path from config Override the config model_path.
--output_dir from config Override the checkpoint/output directory.
--num_gpus from config (8) Override GPU count; >1 enables FSDP full-shard.
--max_steps from config (10000) Override the number of training steps.
--resume_from_checkpoint – Resume from an existing checkpoint.

Camera convention

Every dataset is normalized with the same convention as RealEstate10K:

  1. Poses are OpenCV camera-to-world matrices, expressed relative to the first camera (the first frame becomes the identity pose).
  2. Translation is preprocessed by trajectory_scale / near_depth, where near_depth is a per-clip near-distance depth estimate. This normalization only makes the translation magnitude comparable across datasets — absolute scale is handled inside the model by the learned scale gate, so trajectory_scale stays 1.0 for all datasets.

Data preparation

Point the config at your local dataset roots, then precompute the per-clip near_depth for each dataset with the shared estimator (RAFT optical flow + two-view triangulation):

# OmniWorld first needs a validity index over its training windows:
python scripts/build_omniworld_index.py --data_root /path/to/OmniWorld \
  --output /path/to/OmniWorld/valid_entries.json

# Estimate per-clip near-depth for each dataset (repeat per dataset):
python scripts/estimate_near_depth.py --dataset realestate10k \
  --data_root /path/to/RealEstate10K --split train \
  --output /path/to/RealEstate10K/near_depth_train.json

The same estimate_near_depth.py handles all four datasets via --dataset {realestate10k,dl3dv, panshot,omniworld}. See configs/train_rdpo_high_only.yaml for the full list of per-dataset paths and options (sample_stride is random ≤4 for RealEstate10K and 1 for the others).

Citation

@article{yin2026scope,
  title={SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformers},
  author={Yin, Minghao and Lu, Jiahao and Hu, Wenbo and Zhao, Wang and Shan, Ying and Han, Kai},
  year={2026}
}

Acknowledgements

SCoPE is built on Wan2.2 and DiffSynth-Studio. We thank the authors and contributors of these projects.

License

SCoPE is released under the Apache-2.0 License.

SCoPE (TencentARC/SCoPE)

Searching Database…

214 ·GitHub
SCoPE
Что это такое?SCoPE — это исследовательский релиз, который добавляет управление камерой в предобученный видеодиффузионный трансформер (Wan2.2-I2V-A14B): по первому кадру, текстовому промпту и траектории камеры он генерирует видео с заданным движением камеры.
То есть управление камерой — это отдельный модуль, прикрученный к модели сбоку?Нет, отдельная управляющая ветка не обучается: лучи камеры подаются как вторые позиционные координаты (координаты Плюккера) прямо в предобученное внимание. Такая переделка сохраняет RoPE бит-в-бит, стартует из нетронутого предобученного DiT и добавляет меньше 0,1% новых параметров.
Как мне запустить это у себя локально?Установите через uv (он подтянет зафиксированную сборку PyTorch 2.9.1 + CUDA 12.8) или через pip, затем скачайте чекпоинт SCoPE с Hugging Face — он полностью самодостаточный, отдельно скачивать Wan2.2 не нужно. Инференс запускается через inference.py с приложенными примерами, которые работают сразу из коробки.
Какое железо мне понадобится?Нужна GPU с поддержкой CUDA, а чекпоинт весит около 67 ГБ — его лучше держать на локальном диске вместе с кэшем Hugging Face. Для инференса можно задать --vramlimitgb, чтобы включить выгрузку в VRAM; FlashAttention — опционально.
Есть ли ограничения на траектории камеры, которые можно подать на вход?Свои позы должны быть массивами .npy формы [81, 3, 4] или [81, 4, 4] в координатах camera-to-world из OpenCV, в соответствии с выводом из 81 кадра; форма и значения проверяются до загрузки модели. --trajectory и --camerapath — альтернативные источники поз, их нельзя комбинировать.
Можно попробовать до установки чего-либо, и где это запускается локально?Да — Hugging Face Space позволяет попробовать прямо в браузере без какой-либо настройки. Локально нужны Python 3.11 и NVIDIA GPU с поддержкой CUDA и зафиксированной сборкой PyTorch 2.9.1 (CUDA 12.8), так что на практике это Linux или Windows.
Как увидеть траекторию камеры на своих сгенерированных видео?Скрипт scripts/overlaycamera.py накладывает на видео, используя его траекторию, HUD с конусом камеры (справа внизу) и активные клавиши WASD для траекторий в стиле клавиатуры (слева внизу). Требуется доп. зависимость viz (pip install -e .[viz]) и ffmpeg в PATH; все демо-видео в README сделаны именно с его помощью.
Откуда взяты примеры траекторий камеры?В приложенных примерах смешаны синтетические траектории вроде truckright и snakefwd с реальными записанными траекториями камеры из OmniWorld (GT 1-3). Можно выбрать одну интерактивно, передать id траектории для воспроизводимых запусков или отрендерить все траектории сцены за одну загрузку модели с флагом --alltrajectories.
Можно ли обучать или дообучать её на своих данных?Да. train.py воспроизводит рецепт RDPO с высоким уровнем шума — оптимизируется только high-noise эксперт — на смеси датасетов RealEstate10K, DL3DV, PanShot и OmniWorld, для каждого из которых есть свой загрузчик данных. Мульти-GPU обучение автоматически использует FSDP, а конфиг по умолчанию рассчитан на 8 GPU.
Как она справляется с позами из разных источников и разными масштабами сцен?Схема Normalize-Gate-Inject объединяет нормализацию ближней глубины по клипу с обучаемым гейтом масштаба, поэтому одно кодирование может принимать позы из разных пайплайнов реконструкции и для разных масштабов сцен — как метрических, так и up-to-scale.
What are similar projects?Evoke, LTX-Video, SolarWM, SparkDiffusion, cosmos, FramePack, generative-models, LongCat-Video
Which topics is it in?wan, world-models, image-to-video, diffusion-models, video-generation, 3d, transformer, computer-vision, cuda, pytorch, deep-learning, research
Creator
TencentARC
Size
75 MB
Stars
214
Updated
CommitSep 27
Age
2mo
Commits · 90 Days