StreamEval README

项目的 README,来自 github.com/StreamEval/StreamEval。合并为一个 markdown 文件:README.md。

StreamEval

Evaluating Interactive Streaming Video Generation Models

Hugging Face ModelScope Paper Leaderboard

English | 简体中文

StreamEval evaluates interactive streaming video generation in sessions lasting 15 to 300 seconds.

Our benchmark has four key features:

Tasks and metrics

Task The system receives Scored on
Digital-human dialogue a portrait and a persona; the user's speech (or text) turns Quality, A–V Sync, Stability, Dialogue, Identity
Avatar performance a portrait; speech to perform and live directions Quality, A–V Sync, Stability, Following, Identity
World generation an opening prompt (or image); live prompts and camera controls Quality, Stability, Following, Continuity (+ Reference for image-conditioned inputs)
Instruction editing a live source video; text edit instructions Quality, Stability, Following, Edit Success, Preservation
Reference editing a live source video; reference images to apply, patch, remove or reset Quality, Stability, Reference, Edit Success, Preservation

Metrics are scored 0–100. A task's Overall is the equal-weight mean of its applicable content metrics; tasks are never merged into one total. Definitions.

Benchmark

Benchmark construction and verification

Model-assisted construction and modality-specific validation.

Benchmark composition

Benchmark subset: task composition, input types, input counts and timing.

The complete benchmark in benchmark/v1/full covers all five tasks and duration strata. Each condition-set folder contains cases.jsonl (input timelines), oracles.jsonl (expected outcomes, for the judge only) and manifest.json (media paths and SHA-256 hashes). See the benchmark documentation for the file format.

Quick start

Python ≥ 3.10 on Linux or macOS. Run the following steps from the code repository root in the same activated Python environment.

Installation and CPU example

This example generates its own media and needs no API key, GPU or benchmark download.

git clone https://github.com/StreamEval/StreamEval.git streameval && cd streameval
python -m venv .venv && source .venv/bin/activate
python -m pip install -e ".[runtime,judge,dev]"
python -m streameval.benchmark.synthetic --out .smoke
streameval run --model streameval.models.mock:MockModel --track instruction_edit \
  --case sa.v1.901.instruction_edit --benchmark .smoke/set --data .smoke/data --out .smoke/attempt
streameval validate-run .smoke/attempt --benchmark .smoke/set --data .smoke/data
pytest -q

Download and validation

Hugging Face and ModelScope provide the same benchmark definitions and media (≈3.1 GB).

python -m pip install -U huggingface_hub modelscope

Choose either platform. Download from Hugging Face:

from huggingface_hub import snapshot_download
path = snapshot_download("StreamEval/StreamEval", repo_type="dataset")
print(path)

Or download from ModelScope:

from modelscope import dataset_snapshot_download
path = dataset_snapshot_download(
    "StreamEval/StreamEval",
    endpoint="https://www.modelscope.ai",
)
print(path)

Replace <path> with the directory returned by the download command.

export STREAMEVAL_DATA="<path>/data"
streameval validate --benchmark benchmark/v1/full

Model evaluation

Install the selected model's dependencies using its adapter instructions, then create the credentials file:

cp .env.example .env

Fill in the model and judge credentials in .env. Choose ID from streameval list, then evaluate a registered self-hosted model:

streameval list
streameval evaluate --system ID --launch --out runs_full \
  --benchmark benchmark/v1/full --population full

Use --launch for a registered self-hosted server, or replace it with --url ws://HOST:PORT/session to connect to a running server. Follow the adapter's setup instructions for hosted services.

Results are saved under runs_full/. For comparable runs, follow the settings in the evaluation protocol.

Documentation

Document Content
docs/add_a_model.md the model interface, system.json, servers, composed avatars, vendor adapters
docs/protocol.md the WebSocket protocol and what each session writes
docs/evaluation.md metrics, aggregation and the judge
docs/reproduce.md running a set, individual stages, judge replay, result files
benchmark/v1/README.md the benchmark and its rosters
models/README.md the v1 systems: upstream code, weights and licences
Repository layout
streameval/          Python package: benchmark, runtime (harness), models, providers, judge, scoring, cli
benchmark/v1/          the complete benchmark (full), compatible component rosters and attribution
models/                v1 systems: self-hosted model servers and vendor adapters
examples/my_model/     template model, in-process and as a WebSocket server
evaluation/protocol/   evaluation policy, scoring specification, frozen source manifest
assets/                benchmark construction and composition figures
tests/                 CPU test suite; generates its own media; no keys or GPU needed

Citation

@article{team2026streameval,
  title={StreamEval: Evaluating Interactive Streaming Video Generation Models},
  author={Team, StreamEval},
  journal={arXiv preprint arXiv:2610.28402},
  year={2026}
}

License

StreamEval (StreamEval/StreamEval)

正在搜索数据库…

这是什么?StreamEval 是一个基准测试,用于评估交互式流式视频生成模型。它包含五类任务,每次会话时长 15 到 300 秒,模型在生成的同时会逐步接收输入。
这是模型还是评测框架?它是一个评测框架加基准数据集,本身并不是视频生成模型。它会按时间顺序向被测模型输入内容,记录输出和时间戳,然后用自动化评判器为结果打分。
不配置真实模型,怎么先试一下?你可以先运行内置的 CPU 示例。它会自己生成所需媒体,不需要 API 密钥、GPU,也不需要下载基准数据。之后再去评测真实模型。
运行 StreamEval 需要安装什么?你需要在 Linux 或 macOS 上安装 Python 3.10 或更高版本。如果要进行完整评测,还需要从 Hugging Face 或 ModelScope 下载约 3.1GB 的基准数据,安装目标模型的适配器依赖,并准备一个 .env 凭据文件。
内置的第三方模型有什么需要注意的吗?基准数据本身采用 CC BY 4.0 协议。不过,评测中包含的几个第三方模型仍保留非商业许可证;另外,用于编辑的源视频来自第三方素材,需要遵守其各自的署名要求。
能在 Windows 上运行吗?不能。快速入门明确面向 Linux 或 macOS 上的 Python 3.10+,文档中没有提到支持 Windows。
StreamEval 测试的五类任务是什么?数字人对话、虚拟形象表现、世界生成、指令编辑和参考编辑。每类任务都有自己的指标集,例如 A-V Sync、Stability、Following 和 Edit Success。
一次运行的结果是怎么打分的?每个适用的指标按 0 到 100 分打分。某类任务的 Overall 分数是其内容指标的等权平均值。五类任务不会合并成一个总分。
基准数据文件去哪里下载?完整基准数据(包括用例、预期结果和媒体清单,约 3.1GB)在 Hugging Face 和 ModelScope 上内容完全相同,你可以任选其一下载。
如何把我自己的模型服务接入评测?你可以使用 --launch 启动已注册的自托管模型;也可以使用 --url ws://HOST:PORT/session 指向任意正在运行的服务器,但需遵循项目的 WebSocket 协议。
大小
1.5 MB
星标
622
更新
提交Oct 25
时长
1个月
提交 · 90 天