pyrowave README

The project's README from github.com/Themaister/pyrowave. As one markdown file: README.md.

PyroWave

PyroWave is an intra-only video codec (practially speaking a still-image codec) that is optimized for extremely fast GPU encode (< ~0.1 ms encode and decode at 1080p, < ~0.2 ms at 4K). It is fully implemented in Vulkan compute shaders.

The targeted bit-rates are quite high (~200+ mbit/s) and the intended use case is local network game streaming over ethernet with absolute minimum latency where bandwidth is less of a concern.

It is integrated as part of my pyrofling project.

It is similar in scope to my master thesis from 2014, except the use case now is game streaming instead of adaptation of raw video to ethernet links.

Overview

Colorspace

Currently implemented YCbCr 4:2:0 and 4:4:4.

Wavelets

The images are transformed with the Discrete Wavelet Transform using CDF 9/7 filter. This is basically the exact same as JPEG2000.

Exact rate control

The encoder can target exact maximum bitrate for an encoded image.

Trivial "entropy" coding

The encoding of coefficients is trivial compared to normal image codecs, and is responsible for a large increase in bit-rate, especially at higher compression ratios. This simplicity massively improves encode/decode performance, since it's extremely parallelizable. It also makes it possible to do a single-pass exact rate control for an entire image.

It should be possible in theory to add a more proper entropy coder to the encoded bit-planes to achieve somewhat competent compression, but that has not been explored and is out of scope. High Throughput JPEG2000 is likely a good place to look for that anyway.

Robustness against packet loss

Being intra-only and encoding 64x64 blocks of coefficients in isolation ensures great recovery against packet loss. PyroWave has been battled tested over long distance streaming over fiber links.

Bitstream definition

See docs/bitstream.md

Building

PyroWave is intended to be built alongside PyroFling with Granite in the normal case.

Standalone C API

NOTE: This API is still under development and the API/ABI is not yet quite stable.

A small portion of Granite needs to be checked out.

bash checkout_granite.sh

Build normally with CMake and a C API is installed.

$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=output -DCMAKE_BUILD_TYPE=Release -G Ninja
$ ninja install

[0/1] Install the project...
-- Install configuration: "Release"
-- Installing: ...../build/output/include/pyrowave/pyrowave.h
-- Installing: ...../build/output/lib/libpyrowave-shared.so.0.0.0
-- Installing: ...../build/output/lib/libpyrowave-shared.so.0
-- Installing: ...../build/output/lib/libpyrowave-shared.so
-- Installing: ...../build/output/share/pyrowave-shared/cmake/pyrowave-sharedConfig.cmake
-- Installing: ...../build/output/share/pyrowave-shared/cmake/pyrowave-sharedConfig-release.cmake
-- Installing: ...../build/output/share/pkgconfig/pyrowave-shared.pc

The build is tested on Linux, MinGW, msys2 and MSVC. See pyrowave-c-test which unit tests the shared C API. That test also serves as a basic user guide for the API.

build-steamrt.sh builds against the Sniper SDK and is also supported.

Android standalone build

This assumes that NDK is installed somewhere.

# Replace as needed. Any NDK should work in theory, but this is the one I checked with.
$ NDK_VERSION=29.0.14206865

$ cmake .. -DCMAKE_TOOLCHAIN_FILE=$ANDROID_HOME/ndk/$NDK_VERSION/build/cmake/android.toolchain.cmake \
    -DCMAKE_BUILD_TYPE=Release \
    -DANDROID_ABI=arm64-v8a \
    -G Ninja -DCMAKE_INSTALL_PREFIX=output
$ ninja install

Local development and CLI

For the sample and test applications in this repo however, check out the full https://github.com/Themaister/Granite before invoking CMake. Build with -DPYROWAVE_DEVEL=ON to get the "full" build.

git clone --depth 1 --recursive --shallow-submodules https://github.com/Themaister/Granite Granite

Basic encoder/decoder CLI

The basic CLI takes a y4m and dumps out a raw bitstream. This is just intended for local testing.

pyrowave-encode test.y4m out.wave 400000

This encodes a raw bitstream where each frame consumes maximum 400 KB. To decode back to y4m:

pyrowave-decode out.wave out.y4m

pyrowave (Themaister/pyrowave)

Searching Database…

772 ·GitHub
Pyrowave
这是什么？一个纯帧内视频编码器——实际上相当于静态图片编码器——完全在 GPU 上以 Vulkan 计算着色器的方式运行：1080p 下每帧耗时不到 0.1 毫秒，4K 下约 0.2 毫秒。它是为局域网游戏串流打造的。
这是一个独立的软件，还是需要链接进我自己代码里的库？这是一个通过 C API 集成的编码器库，仓库里还带有示例和命令行测试程序。它已经作为作者的 PyroFling 串流项目的一部分发布了。
我要怎么把它构建到自己的项目里？用 CMake 构建——通常是和 PyroFling 以及 Granite 一起构建。也可以单独构建这个 C API（需要先检出一部分 Granite），不过 API/ABI 目前还不稳定。
构建之前需要装好哪些东西？需要 CMake 和 Granite 库的部分代码；Android 独立构建默认已安装 NDK。由于整个编码器都是计算着色器，运行它需要一块支持 Vulkan 的显卡。
有什么限制——为什么它只适用于局域网？它有意用压缩率换取速度：熵编码保持得非常简单，因此目标码率很高（约 200 Mbit/s 以上）。它假设使用的是带宽不是瓶颈的以太网连接。
它可以在哪些平台上构建？已在 Linux、MinGW、msys2 和 MSVC 上测试过构建，通过 NDK 还有一个 Android 独立构建，build-steamrt.sh 则面向 Steam Runtime Sniper SDK。
它到底是怎么压缩一张图片的？图片先用 CDF 9/7 滤波器做离散小波变换——本质上就是 JPEG2000 用的那种变换——采用 YCbCr 4:2:0 或 4:4:4 格式，然后系数被放在互相独立的 64x64 块里，用一个刻意保持简单的熵编码器进行编码。
编码器能精确达到指定的码率吗？能——码率控制是精确的，而且只需一遍编码：它会为每一张编码后的图片设定一个码率上限。命令行示例把每一帧都限制在 400 KB 以内。
传输过程中丢包了会怎么样？画面损伤只影响局部：纯帧内编码加上互相独立的 64x64 块，让它对丢包有很好的恢复能力，而且在光纤线路上经过远距离串流的长期实战检验。
不写集成代码能不能先试用一下？可以——用 -DPYROWAVEDEVEL=ON 构建可以得到开发者工具，其中包括一个命令行程序，能把 y4m 文件编码成原始码流，再解码回 y4m。
What are similar projects?android-airplay-server, Sunshine, OpenNOW, srs, filmcraft, SmarterStreaming, TurboBench, taskflow
Which topics is it in?game-streaming, codec, parallel-computing, low-latency, compression, cmake, vulkan, video-processing, high-performance, gpu-acceleration, real-time, streaming
Creator
Themaister
Size
7.2 MB
Stars
772
Updated
CommitOct 17
Age
1y
Commits · 90 Days