arc-task-gen README

The project's README from github.com/pathwaycom/arc-task-gen. As one markdown file: README.md.

ARC-AGI-1 Task Generator

Distribution-Matched Tasks for Model Evaluation

This repository generates fresh ARC-AGI-1-style tasks that resemble the public dataset. These tasks can be used to evaluate frontier models such as BDH-CQ on problems they are unlikely to have encountered before.

Overview

BDH-CQ is a reasoning model that combines in-context learning through evolving recurrent memory with iterative reasoning in a structured, continuous latent space.

It builds on Dragon Hatchling (BDH), a post-Transformer recurrent architecture in which neuron-like units communicate through low-rank interactions and maintain context in an evolving associative state. BDH-CQ extends this architecture: at inference time, its recurrent memory is continuously updated, and queries are solved through iterative computation in latent space rather than by generating an intermediate, text-based chain of thought.

We evaluate BDH-CQ on the public ARC-AGI-1 evaluation set and use controlled ARC-like interventions to study what the model learns from demonstrations.

A 150M-parameter configuration achieves 29.5% pass@2 on the public ARC-AGI-1 evaluation set at a computed inference cost of $0.0007 per task. This is 11× cheaper per task than GPT-5.6 Luna (Low), following an 80% price reduction on July 30.

The architecture has also been tested in pretraining experiments ranging from 1B to 600B parameters, demonstrating Transformer-like scaling while retaining BDH-CQ's recurrent latent-reasoning capabilities.

Paper and Blog Post

This repository accompanies the following paper:

B. Engdahl, A. Kosowski, J. Chorowski, Z. Stamirowska, P. Uznański, J. Jiang, R. Phadke, R. Kinas, and R. Zhong. BDH-CQ: Introducing In-Context Learning with Recurrent Latent Reasoning.

For a broader introduction and discussion, read the accompanying blog post.

ARC-AGI-1 Efficiency Frontier

Pathway ARC-AGI efficiency chart

What You'll Find in This Repository

As a public benchmark, ARC-AGI-1 cannot fully isolate few-shot rule induction from potential prior familiarity with its tasks.

To provide a complementary measure, arc-task-gen creates a private evaluation set with similar properties, enabling meaningful comparisons between public-benchmark performance and performance on newly generated tasks.

The generated tasks.json file follows the standard ARC format:

{
  "train": [],
  "test": []
}

It is compatible with existing ARC evaluation harnesses.

Follow the instructions in instructions.md to start generating tasks.

Independent Reproduction

BDH-CQ's ARC-AGI-1 results were evaluated and reproduced by Łukasz Kaiser, a co-author of the Transformer architecture and TensorFlow.

The results were also independently reproduced by Remigiusz Kinas, a contributor to Bielik, and Richard Zhong, an NYU researcher. Their work focused on model evaluation and benchmark robustness.

arc-task-gen (pathwaycom/arc-task-gen)

Searching Database…

11K ·GitHub
Arc-task-gen
这是什么？一个用于生成全新原创 ARC-AGI-1 风格谜题任务的生成器，这些任务的统计分布与公开的 ARC-AGI-1 评估集相匹配，因此可以在模型从未见过的问题上对模型进行测试。
这是一个数据集，还是一个任务生成工具？它是一个生成脚本，而不是静态数据集：它会按需生成符合标准 ARC 格式的 tasks.json 文件，可直接接入现有的 ARC 评估框架。
我该如何生成这些任务？按照仓库中 instructions.md 文件的分步说明操作，即可生成一组 tasks.json。
为什么不直接复用现有的公开 ARC-AGI-1 测试集？公开测试集无法完全排除模型事先接触过其中任务的可能性，因此该工具会生成一个私有的、分布相匹配的任务集，以提供更干净的对比基准。
这个生成器是给谁用的？面向那些想在 ARC 风格问题上对 BDH-CQ（或其他）等推理模型进行基准测试、又不希望被公开已知任务污染的研究人员和模型评估者。
什么是 BDH-CQ？BDH-CQ 是一个构建在 Dragon Hatchling（BDH）后 Transformer 循环架构之上的推理模型；它使用持续更新的循环记忆，并通过潜在空间中的迭代计算来求解查询，而不是生成文本形式的思维链。
BDH-CQ 在 ARC-AGI-1 上的表现如何？一个 150M 参数的 BDH-CQ 配置在公开的 ARC-AGI-1 评估集上达到 29.5% 的 pass@2，每个任务的成本约为 0.0007 美元，比 GPT-5.6 Luna（Low）每个任务便宜约 11 倍。
报告的结果经过独立验证了吗？是的，这些结果已由 Łukasz Kaiser（Transformer/TensorFlow 共同作者）、Remigiusz Kinas（Bielik 贡献者）和 Richard Zhong（纽约大学研究员）复现，重点验证了评估和基准测试的稳健性。
有没有解释这项工作的论文或文章？有，该仓库附带了论文《BDH-CQ: Introducing In-Context Learning with Recurrent Latent Reasoning》以及一篇配套的博客文章。
What are similar projects?bdh, MSA, Attention-Residuals, math, APSD, ParseBench, timeseries-atlas, High-Angle_Robust_Fast_FaceAlignment
Which topics is it in?llm-evaluation, reasoning, paper, dataset, evaluation, neural-network, academic-research, transformer, benchmark, deep-learning, research, machine-learning
Creator
pathwaycom
Size
0.6 MB
Stars
11K
Updated
CommitAug 11
Age
2mo
Commits · 90 Days