OraRL README

The project's README from github.com/HVision-NKU/OraRL. As one markdown file: README.md.

简体中文

OraRL

Annotations as Rollouts

Efficient and scalable reinforcement learning for unified video MLLMs

Yunheng Li · Guohong Mu · Hao Li · Shengsheng Qian · Dingwen Zhang · Qibin Hou · Ming-Ming Cheng

📄 Paper   ·   🌐 Project Page   ·   🎮 Live Demo   ·   🤗 Models (4B / 9B)

⚙️ Environment   ·   🚀 Training   ·   📊 Evaluation   ·   ⚖️ License

Papers with Code: SOTA on ActivityNet-TimeLens Papers with Code: SOTA on Charades-TimeLens Papers with Code: SOTA on MeViS Papers with Code: SOTA on QVHighlights-TimeLens Validation Papers with Code: SOTA on VideoHolmes Papers with Code: #2 on ReasonVOS

View all 11 verified evaluations on Papers with Code

Animated OraRL method preview

▶ Click the image to watch the 1:38 project overview.

Why OraRL

OraRL in One Update

Animated OraRL framework

An OraRL update separates reliable annotation guidance from on-policy normalization:

  1. Build the group: append one serialized annotation rollout to the policy samples generated for the same prompt.
  2. Keep the baseline on-policy: estimate the group baseline from policy rewards only.
  3. Guide and select: convert the annotation-policy reward gap into a correction, then retain a sign-balanced subset for the update.

This design uses task-native annotations directly and requires no chain-of-thought supervision or decoding.

Video-ORA Results

Video-ORA-9B results across seven task families

Dataset-Level Results

Dataset-level benchmark matrix comparing Video-ORA with multimodal baselines

Video-ORA-9B leads the matched seven-family comparison without CoT decoding. Best and second-best values are highlighted per row; † denotes an original-report value whose frame, prompt, split, or decoding settings may differ. Averages require complete family coverage.

Model Scaling

Animated Video-ORA model scaling from 0.8B to 9B

Data Scaling

Animated OraRL data scaling and reward dynamics

Models

Model Backbone Released recipe Weights
Video-ORA-9B Qwen3.5-9B orarl_9b.yaml Hugging Face
Video-ORA-4B Qwen3.5-4B orarl_4b.yaml Hugging Face

vLLM Serving

Both Video-ORA checkpoints load directly with vLLM 0.19.1 for OpenAI-compatible serving:

MODEL=OraRL/Video-ORA-9B

vllm serve "$MODEL" \
  --served-model-name Video-ORA-9B \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 131072 \
  --limit-mm-per-prompt '{"image": 1, "video": 1}'

Set --tensor-parallel-size to the GPU count for multi-GPU deployment and lower --max-model-len on smaller-memory devices. Use enable_thinking=false in the chat template for answer-only inference.

Use OraRL

The release is organized around three user-facing workflows:

  1. Environment: install the pinned CUDA stack that covers both the bundled trainer and the evaluators.
  2. Training: prepare licensed local training data and launch GRPO or OraRL on one or multiple nodes.
  3. Evaluation: download Video-ORA and OraRL-Data, then run a smoke test or the complete paper suite.

Training and evaluation are dry runs by default; inspect the resolved command before adding --run. Checkpoints and evaluation media are hosted under the OraRL Hugging Face organization.

Acknowledgements

OraRL is built on veRL — a high-performance RL framework with HybridEngine. We thank its authors and contributors for open-sourcing the training infrastructure.

License

OraRL source is released under Apache-2.0. Datasets, models, benchmarks, and optional dependencies retain their original licenses; see NOTICE.

Citation

If you find OraRL useful, please consider giving this repository a ⭐ and citing our paper.

@article{li2026orarl,
  title   = {Annotations as Rollouts: Efficient and Scalable
             Reinforcement Learning for Video MLLMs},
  author  = {Li, Yunheng and Mu, Guohong and Li, Hao and
             Qian, Shengsheng and Zhang, Dingwen and Hou, Qibin
             and Cheng, Ming-Ming},
  journal = {arXiv preprint arXiv:2608.20492},
  year    = {2026},
  url     = {https://arxiv.org/abs/2608.20492}
}

OraRL (HVision-NKU/OraRL)

Searching Database…

166 ·GitHub
这是什么?OraRL 是一个强化学习框架,它把标注当作推演(rollout),用于高效训练覆盖七大视频任务族的统一视频多模态大语言模型(MLLM)。
它是开箱即用的应用,还是研究用训练框架?它是一个基于 veRL 构建的研究代码库,捆绑了训练方案、评估器和已发布的 Video-ORA 检查点,而非即插即用的应用。
怎么把 OraRL 跑起来?遵循三个工作流:为 Environment 安装锁定版本的 CUDA 软件栈,准备已获授权的本地训练数据并在单节点或多节点上启动 GRPO 或 OraRL,然后下载 Video-ORA 和 OraRL-Data 运行评估。
部署已发布的模型需要什么?你需要 CUDA GPU 和 vLLM 0.19.1 来加载 Video-ORA-4B/9B 检查点(Qwen3.5 主干)以进行 OpenAI 兼容推理,并根据你的硬件调整 tensor-parallel-size 和 max-model-len。
开始训练或评估前有什么要注意的坑吗?训练和评估命令默认是试运行(dry run),你必须检查解析后的命令并添加 --run 才会真正执行;此外你需要先获得授权并准备好自己的本地训练数据。
「标注即推演」实际是怎么做的?它把一条序列化标注作为可靠的正向推演样本追加到同一提示的策略生成样本中,使 RL 基线保持纯 on-policy,同时仍利用标注与策略之间的奖励差距来引导和筛选更新。
一次 OraRL 更新能处理哪些视频任务?单条更新规则覆盖七个任务族:时序定位、空间定位、分割、跟踪、时空定位、视频问答和空间智能。
高效训练模式能提速多少?对 4B 模型进行符号均衡剪枝,使更新速度提升 1.48 倍(92.5 降至 62.4 秒/步),并把单 GPU 峰值显存从 62.4 GB 降到 50.9 GB。
有哪些模型规模可用?发布了两个 Video-ORA 检查点:4B 和 9B,均基于 Qwen3.5 主干构建,托管在 Hugging Face 上。
Size
9.5 MB
Stars
166
Updated
CommitAug 28
Age
1mo
Commits · 90 Days