arc-task-gen README

The project's README from github.com/pathwaycom/arc-task-gen. As one markdown file: README.md.

ARC-AGI-1 Task Generator

Distribution-Matched Tasks for Model Evaluation

This repository generates fresh ARC-AGI-1-style tasks that resemble the public dataset. These tasks can be used to evaluate frontier models such as BDH-CQ on problems they are unlikely to have encountered before.

Overview

BDH-CQ is a reasoning model that combines in-context learning through evolving recurrent memory with iterative reasoning in a structured, continuous latent space.

It builds on Dragon Hatchling (BDH), a post-Transformer recurrent architecture in which neuron-like units communicate through low-rank interactions and maintain context in an evolving associative state. BDH-CQ extends this architecture: at inference time, its recurrent memory is continuously updated, and queries are solved through iterative computation in latent space rather than by generating an intermediate, text-based chain of thought.

We evaluate BDH-CQ on the public ARC-AGI-1 evaluation set and use controlled ARC-like interventions to study what the model learns from demonstrations.

A 150M-parameter configuration achieves 29.5% pass@2 on the public ARC-AGI-1 evaluation set at a computed inference cost of $0.0007 per task. This is 11× cheaper per task than GPT-5.6 Luna (Low), following an 80% price reduction on July 30.

The architecture has also been tested in pretraining experiments ranging from 1B to 600B parameters, demonstrating Transformer-like scaling while retaining BDH-CQ's recurrent latent-reasoning capabilities.

Paper and Blog Post

This repository accompanies the following paper:

B. Engdahl, A. Kosowski, J. Chorowski, Z. Stamirowska, P. Uznański, J. Jiang, R. Phadke, R. Kinas, and R. Zhong. BDH-CQ: Introducing In-Context Learning with Recurrent Latent Reasoning.

For a broader introduction and discussion, read the accompanying blog post.

ARC-AGI-1 Efficiency Frontier

Pathway ARC-AGI efficiency chart

What You'll Find in This Repository

As a public benchmark, ARC-AGI-1 cannot fully isolate few-shot rule induction from potential prior familiarity with its tasks.

To provide a complementary measure, arc-task-gen creates a private evaluation set with similar properties, enabling meaningful comparisons between public-benchmark performance and performance on newly generated tasks.

The generated tasks.json file follows the standard ARC format:

{
  "train": [],
  "test": []
}

It is compatible with existing ARC evaluation harnesses.

Follow the instructions in instructions.md to start generating tasks.

Independent Reproduction

BDH-CQ's ARC-AGI-1 results were evaluated and reproduced by Łukasz Kaiser, a co-author of the Transformer architecture and TensorFlow.

The results were also independently reproduced by Remigiusz Kinas, a contributor to Bielik, and Richard Zhong, an NYU researcher. Their work focused on model evaluation and benchmark robustness.

arc-task-gen (pathwaycom/arc-task-gen)

Searching Database…

11K ·GitHub
Arc-task-gen
What is this?A generator for fresh, original ARC-AGI-1-style puzzle tasks whose statistical distribution matches the public ARC-AGI-1 evaluation set, so models can be tested on problems they haven't seen before.
Is this a dataset or a task-generation tool?It's a generator script, not a static dataset: it produces a tasks.json file on demand that follows the standard ARC format and plugs into existing ARC evaluation harnesses.
How do I generate the tasks?Follow the step-by-step instructions in the repo's instructions.md file to produce a tasks.json set.
Why not just reuse the existing public ARC-AGI-1 test set?The public set can't fully rule out models having prior familiarity with its tasks, so this tool creates a private, distribution-matched set to give a cleaner comparison point.
Who is this generator meant for?Researchers and model evaluators who want to benchmark reasoning models like BDH-CQ (or others) on ARC-style problems without contamination from publicly known tasks.
What is BDH-CQ?BDH-CQ is a reasoning model built on the Dragon Hatchling (BDH) post-Transformer recurrent architecture; it uses continuously updated recurrent memory and solves queries through iterative latent-space computation instead of generating a text chain of thought.
How well does BDH-CQ perform on ARC-AGI-1?A 150M-parameter BDH-CQ configuration reaches 29.5% pass@2 on the public ARC-AGI-1 evaluation set at roughly $0.0007 per task, about 11x cheaper per task than GPT-5.6 Luna (Low).
Have the reported results been independently checked?Yes, they were reproduced by Łukasz Kaiser (Transformer/TensorFlow co-author), Remigiusz Kinas (Bielik contributor), and Richard Zhong (NYU researcher), focusing on evaluation and benchmark robustness.
Is there a paper or write-up explaining this work?Yes, the repo accompanies the paper 'BDH-CQ: Introducing In-Context Learning with Recurrent Latent Reasoning' plus a companion blog post.
What are similar projects?bdh, MSA, Attention-Residuals, math, APSD, ParseBench, timeseries-atlas, High-Angle_Robust_Fast_FaceAlignment
Which topics is it in?llm-evaluation, reasoning, paper, dataset, evaluation, neural-network, academic-research, transformer, benchmark, deep-learning, research, machine-learning
Creator
pathwaycom
Size
0.6 MB
Stars
11K
Updated
CommitAug 11
Age
2mo
Commits · 90 Days