REDox README

The project's README from github.com/CAPCOM-TD-OSS/REDox. As one markdown file: README.md.

RE:Dox

1. What is RE:Dox?

RE:Dox is a high-performance, token-based structured data engine for .NET, developed as one of the core components of 'REX' Technology for CAPCOM's next-generation game engine.

RE:Dox is not just a JSON serializer. It parses structured data into a compact, fixed-size token DOM / IR, then uses that structural index for reading, editing, format conversion, serialization, deserialization, and automatic parallel deserialization.

JSON / JSON5 / CBOR / MessagePack / TOML / XML / HTML / CSV / INI / DOX
        ↓
Compact token DOM / IR
        ↓
Reader / Writer / Serializer / Deserializer
        ↓
.NET objects, JSON, CBOR, MessagePack, TOML, XML, HTML, DOX, ...

The same token representation acts as both a compact parsed document model and a common structural layer shared by serializers, converters, and supported formats.

2. Why should I care?

RE:Dox combines three goals that usually conflict:

What you get:

3. 30-second example

Install the core package:

dotnet add package CAPCOM.REDox

Serialize, deserialize, parse, and edit through the same API surface:

using REDox.Json;

var player = new Player
{
    Name = "Leon",
    Level = 42,
    Items = ["Handgun", "Green Herb"]
};

// Serialize / deserialize
var json = JsonSerializer.Serialize(player);
var restored = JsonSerializer.Deserialize<Player>(json);

// Parse into a token DOM and edit it in place
using var doc = JsonDocument.Parse(json);

var root = doc.RootElement.AsObject();
root["Name"] = "Claire";            // replace
root.Add("Hp", 100);                // add
root.Remove("Level");               // remove

var items = root["Items"].AsArray();
items.Add("First Aid Spray");       // append
items.Insert(0, "Knife");           // insert
items.RemoveAt(1);                  // remove by index

var edited = doc.RootElement.ToJsonString();

public sealed class Player
{
    public string? Name { get; set; }
    public int Level { get; set; }
    public string[] Items { get; set; } = [];
}

4. Performance

Environment: BenchmarkDotNet, .NET 10 (X64 RyuJIT x86-64-v3), AMD Ryzen Threadripper PRO 5975WX, Windows 11.

Summary: speedup vs System.Text.Json JsonSerializer

Speedup vs System.Text.Json

Dataset Deserialize Deserialize (parallel) Serialize
canada.json 1.68x 2.84x 1.06x
citm_catalog.json 1.77x 2.25x 1.62x
twitter.json 1.36x 2.34x 1.42x

These ratios describe the benchmark datasets shown below; they are not universal performance guarantees.

Deserialize: canada.json (numeric-heavy, ~2.2 MB)

canada.json deserialize

Deserialize: Mean / Allocated

citm_catalog.json deserialize

twitter.json deserialize

Dataset RE:Dox RE:Dox (parallel) System.Text.Json STJ (JsonTypeInfo) Utf8Json
canada.json 9,113.0 us / 2,619.92 KB 5,384.6 us / 2,685.11 KB 15,303.1 us / 8,734.66 KB 15,005.9 us / 8,734.59 KB 15,864.0 us / 6,655.67 KB
citm_catalog.json 2,280.5 us / 553.55 KB 1,795.8 us / 631.87 KB 4,041.1 us / 1,175.38 KB 3,940.8 us / 1,175.38 KB 2,605.5 us / 1,124.13 KB
twitter.json 1,008.8 us / 504.91 KB 587.4 us / 555.84 KB 1,371.6 us / 557.09 KB 1,401.6 us / 557.09 KB 1,370.6 us / 530.05 KB

Serialize: Mean / Allocated

canada.json serialize

citm_catalog.json serialize

twitter.json serialize

Dataset RE:Dox System.Text.Json STJ (JsonTypeInfo) Utf8Json
canada.json 13,615.5 us / 2,041.41 KB 14,388.3 us / 2,042.85 KB 14,433.1 us / 2,042.85 KB 11,714.2 us / 6,042.21 KB
citm_catalog.json 668.7 us / 494.41 KB 1,085.4 us / 494.85 KB 1,073.8 us / 494.87 KB 707.3 us / 1,389.64 KB
twitter.json 557.3 us / 465.15 KB 789.0 us / 467.95 KB 800.3 us / 467.91 KB 709.4 us / 1,361.37 KB

Reproduce with:

dotnet run -c Release --project benchmarks/REDox.Json.Benchmarks

Benchmark results depend on data shape, target type, runtime, CPU, and serializer options. Always benchmark with your own workload.

5. Architecture

A 64-bit dual-mode token at the core

Every value is described by a fixed-size 64-bit token. An extension bit selects how the payload is interpreted:

extension bit = 0  →  payload interpretation is delegated to the document/format layer
                      (source-backed, format-specific payload; many formats, one token shape)

extension bit = 1  →  payload interpretation is fixed by RE:Dox, and control tokens enable
                      format-agnostic editing (insert / remove / replace)

This allows one token structure to represent both a source-backed view of format data and a RE:Dox-owned editable DOM. Where the document/format layer can retain the original payload, unchanged values can continue to reuse source slices instead of being eagerly materialized.

A token can represent null, boolean, integer, floating point, string, binary, timestamp, big number, array, map/object, trivia (comment/whitespace), or format-specific extension data. It may store inline data, a source offset/length, a container count, link information, or an extension id.

Asymmetric read/write design

Reading and writing have different information requirements, so RE:Dox uses different paths for each direction:

Serialize    : DataWriter, one pass, written directly to the output buffer
               (information is known → no look-ahead, no intermediate DOM)

Deserialize  : DataReader over the token DOM, random access by token id
               (information is unknown → look-ahead, out-of-order, context lookup)

On the write path the value is already known, so RE:Dox writes straight to the buffer using bulk (WriteValues) and fused-property (WriteProperty*) helpers — no intermediate object tree is built.

On the read path the structure is unknown, so RE:Dox first builds a compact token DOM. Because the structure is then fully addressable, RE:Dox can:

Lazy decoding and on-demand materialization

Tokens can store offsets and lengths into the original source buffer. Strings, numbers, timestamps, and binary values are decoded only when requested. Elements are lightweight handles over a document and token id, while object and array views are materialized only when needed.

Mutable tape DOM

A tape DOM is normally read-only because its logical structure is represented by a flat token sequence. RE:Dox adds indirection at container views: DArray, DObject, and DMap keep value slots that can be re-linked, a free list reuses emptied token slots, and extension/control tokens carry edit state.

This preserves the cache-friendly characteristics of a sequential token representation while providing node-like operations such as insert, remove, and replace without rebuilding the entire document for each edit.

Unified converters

Converters (DataConverter<T>) are written against the format-agnostic DataReader / DataWriter abstractions rather than a specific format implementation.

Conceptually:

                 DataConverter<T>
                        │
               DataReader / Writer
                /       |        \
             JSON      CBOR    MessagePack   ...

This allows converter logic to be shared by formats that participate in the common reader/writer model. Instead of duplicating equivalent conversion logic for each type-format pair, RE:Dox can reuse one converter implementation across those formats — also providing a foundation for future AOT / source-generator support.

6. Formats/packages

NuGet package IDs use the CAPCOM.REDox.* prefix. C# namespaces use REDox.*.

Package Status Description
CAPCOM.REDox Public Core + JSON + JSON5 + DOX + common serializer infrastructure
CAPCOM.REDox.Cbor Public CBOR parser/writer integration
CAPCOM.REDox.MessagePack Public MessagePack parser/writer integration
CAPCOM.REDox.Dynamic Public dynamic access to the token DOM
CAPCOM.REDox.Serialization.SystemTextJson Preview System.Text.Json compatibility layer
CAPCOM.REDox.Serialization.NewtonsoftJson Preview Newtonsoft.Json compatibility layer
CAPCOM.REDox.Serialization.DataContractJson Public DataContractJsonSerializer compatibility layer
CAPCOM.REDox.Toml Preview TOML parser/writer with comment-aware token DOM integration
CAPCOM.REDox.Xml Preview XML structural parser/writer
CAPCOM.REDox.Html Preview Practical HTML structural parser/writer
CAPCOM.REDox.Ini Public INI support
CAPCOM.REDox.Csv Preview CSV support

Preview packages use NuGet pre-release versions such as 0.1.0-preview.1; the package name itself does not include -preview. Preview components may change API behavior before stable release.

Install public packages:

dotnet add package CAPCOM.REDox
dotnet add package CAPCOM.REDox.Cbor
dotnet add package CAPCOM.REDox.MessagePack
dotnet add package CAPCOM.REDox.Dynamic
dotnet add package CAPCOM.REDox.Serialization.DataContractJson
dotnet add package CAPCOM.REDox.Ini

Install preview packages with --prerelease:

dotnet add package CAPCOM.REDox.Serialization.SystemTextJson --prerelease
dotnet add package CAPCOM.REDox.Serialization.NewtonsoftJson --prerelease
dotnet add package CAPCOM.REDox.Toml --prerelease
dotnet add package CAPCOM.REDox.Xml --prerelease
dotnet add package CAPCOM.REDox.Html --prerelease
dotnet add package CAPCOM.REDox.Csv --prerelease

CBOR

using REDox.Cbor;

byte[] cbor = CborSerializer.Serialize(new Player
{
    Name = "Jill",
    Level = 30,
    Items = ["Lockpick"]
});

var player = CborSerializer.Deserialize<Player>(cbor);

// Parse into the token DOM and re-encode
using var doc = CborDocument.Parse(cbor);
byte[] encoded = CborDocument.Encode(doc.RootElement);

MessagePack

using REDox.MessagePack;

byte[] msgpack = MessagePackSerializer.Serialize(new Player
{
    Name = "Chris",
    Level = 35,
    Items = ["Knife", "First Aid Spray"]
});

var player = MessagePackSerializer.Deserialize<Player>(msgpack);

using var doc = MessagePackDocument.Parse(msgpack);
byte[] encoded = MessagePackDocument.Encode(doc.RootElement);

Dynamic

using REDox;
using REDox.Dynamic;
using REDox.Json;

using var doc = JsonDocument.Parse("""{"name":"Leon","level":40,"stats":{"alive":true}}""");

var player = doc.RootElement.AsDynamic()!;
var name = (string)player.name;
var level = (int)player.level;
var alive = (bool)player["stats"]["alive"];

// Build a new object dynamically
var obj = new DObject().AsDynamic();
obj.Name = "Ada";
obj.Items = new[] { "Hookshot" };

string json = JsonSerializer.Serialize(obj); // {"Name":"Ada","Items":["Hookshot"]}

Cross-format conversion

Formats that share the common token IR can be converted through the document model:

using REDox.Cbor;
using REDox.Json;

using var doc = JsonDocument.Parse(json);
byte[] cbor = CborDocument.Encode(doc.RootElement);

JSON5  → token IR → JSON
JSON   → token IR → CBOR
CBOR   → token IR → MessagePack
TOML   → token IR → JSON
XML    → token IR → JSON-like structural representation

Formats with different native data models may use a structural projection rather than a lossless semantic round-trip.

7. Advanced features

Automatic parallel deserialization

The deserializer inspects each array's element count and token range before materializing objects. When an array is large enough, element token ids are collected and each element is deserialized into its corresponding index in parallel. Small arrays stay sequential to avoid parallel scheduling overhead.

using REDox;
using REDox.Json;
using REDox.Serialization;

var settings = new DoxSerializerSettings
{
    ParallelOptions = new ParallelDeserializeOptions
    {
        ParallelDeserializeEnabled = true,
        MinimumNumberOfElements = 1024,
        MinimumNumberOfTokens = 4096
    }
};

var players = JsonSerializer.Deserialize<Player[]>(json, settings);

JSON5 with comment and trivia preservation

using REDox.Json;

var json5 = """
{
  // Player name
  name: 'Leon',

  // Inventory
  items: [
    'Handgun',
    'Green Herb',
  ],
}
""";

using var doc = Json5Document.Parse(json5, options: new Json5DocumentOptions
{
    PreserveTrivia = true,
    EnableValueValidation = true
});

var root = doc.RootElement.AsObject();
root["name"] = "Claire";

var output = Json5Document.EncodeToString(doc.RootElement, new Json5WriteOptions
{
    PreserveTrivia = true
});

Async streaming of NDJSON / JSON sequences

JsonSequence reads a stream of JSON values (NDJSON / JSON Lines, or a large top-level array) asynchronously, one value at a time, without loading the whole input into memory. For NDJSON, enable UseNewlineDelimitedFormat in JsonDocumentOptions.

using REDox.Json;

var options = new JsonDocumentOptions { UseNewlineDelimitedFormat = true };

await using var stream = File.OpenRead("players.ndjson");

// Deserialize each line into a .NET object
await foreach (var player in JsonSequence.DeserializeAsync<Player>(stream, options: options, cancellationToken: ct))
{
    Console.WriteLine(player?.Name);
}

To work with the token DOM directly, use ParseAsync. Without UseNewlineDelimitedFormat, a top-level JSON array is streamed element by element. The yielded DElement is backed by a reused document and is valid only until the next iteration; call Clone() or convert it with To<T>() if it must be retained.

using REDox;
using REDox.Json;

await using var stream = File.OpenRead("events.json"); // [ { "level": "info", ... }, { "level": "error", ... }, ... ]

var errors = new List<DElement>();

await foreach (var element in JsonSequence.ParseAsync(stream, cancellationToken: ct))
{
    if (element.TryGetProperty("level", out var level) && level.GetString() == "error")
    {
        errors.Add(element.Clone());
    }
}

DOX format and deep clone

DOX is a binary layout designed to mirror the in-memory token array closely. Constructing a document from DOX requires little structural reconstruction, which also makes DOX suitable for serialization-based deep clone: the result is an independent, editable copy rather than a shared read-only view.

using REDox;

byte[] dox = DoxSerializer.Serialize(player);
var clone = DoxSerializer.Deserialize<Player>(dox);

Compatibility layers

Compatibility packages allow existing serializer options to be adapted to RE:Dox.

System.Text.Json

using REDox.Serialization.SystemTextJson;

var stjSettings = new SystemTextJsonSerializerSettings(new System.Text.Json.JsonSerializerOptions
{
    PropertyNamingPolicy = System.Text.Json.JsonNamingPolicy.CamelCase
});

Newtonsoft.Json

using Newtonsoft.Json;
using REDox.Serialization.NewtonsoftJson;

var nsjSettings = new NewtonsoftJsonSerializerSettings(new JsonSerializerSettings
{
    NullValueHandling = NullValueHandling.Ignore
});

DataContractJsonSerializer

using REDox.Serialization.DataContractJson;

var dcjSettings = new DataContractJsonSerializerSettings(
    new System.Runtime.Serialization.Json.DataContractJsonSerializerSettings());

Requirements

How to build

1. Clone with submodules

Test and benchmark data (simdjson-data, JSONTestSuite, json5-tests, toml-test) live under external/ as git submodules.

git clone --recurse-submodules https://github.com/CAPCOM-TD-OSS/REDox.git
cd REDox

If you already cloned without submodules:

git submodule update --init --recursive

2. Build

dotnet build REDox.slnx -c Release

3. Run tests

dotnet test REDox.slnx -c Release

4. Run benchmarks

Benchmarks use BenchmarkDotNet and must be run in Release configuration.

dotnet run -c Release --project benchmarks/REDox.Json.Benchmarks

Third-party components

The RE:Dox library packages do not bundle third-party code. The following third-party projects are referenced by tests and benchmarks. Each is distributed under its own license.

Test

Repository Used for License
simdjson/simdjson-data Benchmark datasets (canada.json, citm_catalog.json, twitter.json, ...) See repository
nst/JSONTestSuite JSON conformance tests MIT
json5/json5-tests JSON5 conformance tests MIT
toml-lang/toml-test TOML conformance tests MIT

License

RE:Dox is released under the Apache License 2.0.

Status

RE:Dox is under active development. APIs, package boundaries, and preview format support may evolve before the first stable public release.

RE:Dox is developed as part of 'REX' Technology for CAPCOM's next-generation game engine.

REDox (CAPCOM-TD-OSS/REDox)

Searching Database…

1.6K ·GitHub
REDox
这是什么？RE:Dox 是一个面向 .NET 的高性能、基于令牌的结构化数据引擎，作为 CAPCOM 面向下一代游戏引擎的“REX”技术的核心组件而构建。它把结构化数据解析为紧凑的令牌 DOM，用于读取、编辑、格式转换、序列化和并行反序列化。
这是独立应用，还是添加到 .NET 项目里的东西？它是一组 .NET NuGet 库包（前缀为 CAPCOM.REDox.*），从你自己的 C# 项目中引用；没有独立应用程序。
怎么把它加进我的项目？安装公开的 NuGet 包（例如核心的 CAPCOM.REDox 包）；TOML、XML、HTML、CSV 以及兼容层等仅限预览的包需要 --prerelease 标志，因为它们以预发布 NuGet 版本的形式发布。
需要先安装什么？构建和使用 RE:Dox 需要 .NET 10 或更高版本；从源码构建还需要用 git submodules 克隆仓库以获取测试/基准数据。
所有格式包都达到生产可用了吗？没有——JSON、JSON5、CBOR、MessagePack、INI 和 DOX 是公开/稳定版，但 TOML、XML、HTML 和 CSV 支持，以及 System.Text.Json 和 Newtonsoft.Json 兼容层，都是 API 仍可能变化的预览包。
RE:Dox 比 System.Text.Json 快多少？在基准数据集上，RE:Dox 顺序反序列化最快约 1.8 倍，自动并行反序列化约 2.8 倍，序列化约 1.6 倍，同时内存分配也少得多（例如 canada.json 约 2.56MB 对比约 8.53MB）。
它能读写哪些数据格式？JSON、JSON5、CBOR、MessagePack、INI 和原生 DOX 二进制格式是公开且稳定的；TOML、XML、HTML 和 CSV 解析器/写入器目前是预览包。
可以在不重建整个文档的情况下编辑已解析的数据吗？可以，RE:Dox 提供可变的令牌 DOM（DArray/DObject/DMap），支持原地插入、删除和替换操作，JSON5 编辑还能在重新编码时保留注释和其他琐碎信息。
可以继续使用现有的 System.Text.Json 或 Newtonsoft.Json 代码吗？可以，RE:Dox 提供 System.Text.Json、Newtonsoft.Json 和 DataContractJsonSerializer 的兼容层，可以把现有的序列化器选项适配到 RE:Dox 令牌模型。
What are similar projects?Newtonsoft.Json, BRUTAL, BenchmarkDotNet, MonoGame, sbox-public, QuestPDF, roslyn, cheerio
Which topics is it in?nuget, toml, xml, parsing, data, csv, dom, parser, game-engine, gamedev, high-performance, game-development
Creator
CAPCOM-TD-OSS
Size
0.9 MB
Stars
1.6K
Updated
CommitOct 16Releasev1.0.3Oct 16
Age
18d
Commits · 90 Days