bdh README

The project's README from github.com/pathwaycom/bdh. As one markdown file: README.md.

BDH (Dragon Hatchling)

Bridging the Gap Between Transformers and the Brain

BDH (Dragon Hatchling) is a biologically inspired large language model architecture that connects principles of deep learning with the foundations of neuroscience. Developed by researchers at Pathway, BDH provides a theoretical and practical framework for understanding the emergence of reasoning and generalization in artificial systems.

This repository contains the official implementation from the paper:

A. Kosowski, P. Uznański, J. Chorowski, Z. Stamirowska, M. Bartoszkiewicz. The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain, arXiv (2025).

Overview

BDH represents a scale-free, locally interacting network of neurons capable of intrinsic reasoning dynamics. BDH scales like a Transformer on performance benchmarks—yet retains full interpretability and theoretical grounding in the fine-grained dynamics of neuron interactions.

Key properties:

BDH formalizes a bridge between neural computation and machine-based language understanding. It shows how macro reasoning behavior in large AI models emerges from micro-level neuron dynamics, guided by principles of graph theory and local computation.

Empirically, BDH matches GPT-2–scale Transformers across language and translation tasks at equivalent parameter scales (10M–1B).

Architecture

Relation to Transformers

BDH and the Transformer share attention-inspired computation; however, BDH’s graph-based architecture makes its attention emerge naturally from neuron-level interactions, reflecting attention as seen in biological systems.

Scaling Laws

BDH follows Transformer-like scaling laws, maintaining parameter efficiency while achieving interpretability at any scale.

Latest research update: Sudoku Benchmark

Note: The Sudoku Extreme result refers to Pathway’s internal BDH implementation, not to the current open-source repository. This repository contains the implementation of the baseline variant as described in our public paper and does not reproduce the 97.4% benchmark result out of the box. See the dedicated Extreme Sudoku research blog post for additional benchmark context and the reported results.

On Sudoku Extreme, BDH reaches 97.4% accuracy across roughly 250,000 difficult puzzles, without chain-of-thought, solution backtracking, or external tool use, while leading LLMs struggle to perform on the benchmark at all.

Language is not enough for intelligence. Transformers process information token by token with limited internal state, which makes search-heavy, non-linguistic reasoning tasks like Sudoku awkward. BDH uses a larger latent reasoning space with intrinsic memory that supports learning and adaptation during use.

We believe that the future of AI will belong to systems that can reason natively across domains, that can hold multiple possibilities in a rich latent space, and that can converge on solutions without needing to verbalize every step. BDH is our answer to that challenge. It is designed to be a universal reasoning system that can speak our language without being trapped inside it. And yes, it solves Sudoku.

Read more: Post-transformers: Sudoku Bench

Performance Comparison

Model Sudoku Extreme Accuracy Relative Cost
Pathway BDH 97.4% 10× lower, No chain-of-thought
Leading LLMs (O3-mini, DeepSeek R1, Claude 3.7 8K) ~0% High (chain-of-thought)

Table 1: Performance comparison on extreme Sudoku benchmarks (~250,000 difficult puzzles).
Source: Pathway internal data and https://arxiv.org/pdf/2506.21734 for the Leading LLMs’ accuracy score. Pathway’s approach reflects top-1 accuracy and does not rely on chain-of-thought nor solution backtracking.

Installation and Training

# install dependencies
pip install -r requirements.txt

# train BDH on a toy dataset
python train.py

Learn and Discuss

Community Projects

Acknowledgements

We thank Andrej Karpathy for the nanoGPT code and the tiny Shapespeare dataset used in this demonstration.

BDH research stands at the intersection of AI architecture, biological learning models, and theoretical computer science—an effort to map the equations of reasoning between artificial and biological intelligence.

bdh (pathwaycom/bdh)

Searching Database…

3.6K ·GitHub
Bdh
What is this?BDH (Dragon Hatchling) is a biologically inspired large language model architecture that connects deep learning with neuroscience, aiming to explain how reasoning and generalization emerge in AI systems at the level of individual neuron interactions.
Is this a pre-trained model I can download, or source code for an architecture?This is the official source code implementing the BDH architecture described in the paper, not a downloadable pre-trained checkpoint; you train and experiment with it yourself.
Does this repo reproduce the famous 97.4 percent Sudoku result?No. The published 97.4 percent Sudoku Extreme result comes from Pathway's internal BDH implementation; this open-source repo only contains the baseline variant and does not reproduce that benchmark out of the box.
What makes BDH more interpretable than a standard Transformer?BDH uses locally interacting neuron particles with excitatory/inhibitory dynamics and Hebbian working memory that produce sparse, positive, monosemantic activations, in contrast to the opaque attention patterns of standard Transformers.
How does BDH's performance compare to GPT-2-scale Transformers?BDH empirically matches GPT-2-scale Transformers on language and translation tasks at equivalent parameter counts from 10M to 1B, while following similar scaling laws.
Why did Pathway build BDH instead of just using Transformers?The authors argue that token-by-token Transformers struggle with search-heavy, non-linguistic reasoning like Sudoku, so BDH was designed as a universal reasoning system with a larger latent space and intrinsic memory that works across domains without needing chain-of-thought.
Are there community forks or ports of BDH?Yes, community projects include a Burn framework port (mosure/burndragonhatchling), an MLX port (severian42/bdh), and forks adding dynamic vocabulary and stateful attention (adamskrodzki/bdh).
What are similar projects?Attention-Residuals, MSA, Engram, llama3-from-scratch, DMI, minGPT, gpt-2, gpt-3
Which topics is it in?interpretability, arxiv, reasoning, attention-mechanism, language-model, neural-networks, paper, neural-network, academic-research, large-language-models, transformer, artificial-intelligence
Creator
pathwaycom
Size
1 MB
Stars
3.6K
Updated
CommitMay 16
Age
1y
Commits · 90 Days