lm_understand-logs README

项目的 README,来自 github.com/RikiyaT/lm_understand-logs。合并为一个 markdown 文件:README.md。

Logs: LLMs can understand and write neural network weights

The complete logs of every experiment in the blog post Sparks of Weight Space Programming: LLMs can understand and write neural network weights. For each experiment you get:

Each experiment folder has its own README. It says which run is which tab or row in the post, and which file each number comes from.

Folder In the post Runs Tools
exp1_which_game/ Experiment 1: which game was this network trained on? Breakout, Pong, Pac-Man, Freeway (open answer) none
exp2_diffusion/ Experiment 2: what does this diffusion model generate? circle, square, triangle none, no forward pass
exp3_language/ Experiment 3: which language a language model learned, from its middle blocks Python, romaji, English, two conversations each Python sandbox, no forward pass
exp4_fix_car/ Experiment 4: fix a failing network by editing its weights one one-shot edit none
exp5_write_from_scratch/ Experiment 5: write networks from scratch Breakout, Seaquest, Asterix (MinAtar) none
exp6_go_from_scratch/ Experiment 6: write a 1.5M-weight Go network 0, 10, 25 and 50 passes Python sandbox
exp7_sycophancy/ Experiment 7: which language model is sycophantic? four weights-only attempts, then the attention tool (3 pairs each) Python sandbox
exp8_math/ Experiment 8: make a language model better at math Qwen3.5-0.8B with 10, 25 and 50 passes, and the Gemma 3 270M follow-up Python sandbox

Only the runs the post reports are here. Trial runs of earlier prompt versions are left out.

How the model was run

"Astra" in the post is OpenAI's gpt-6-astra.

One turn. Every turn is one call of the Codex CLI: codex exec, or codex exec resume <thread> to continue a conversation. The prompt goes in on stdin. The reasoning effort is set per call and recorded in each turn's run.json:

Effort Experiments
high 1, 2, 4
xhigh 3, 5, 6, 7
max 8

No tools of its own. Every built-in Codex capability that gives the model a tool is switched off: shell, file editing, web search, image viewing, sub-agents, plugins and the rest. The configuration is in harness/codex_config.toml, and the system instructions are one neutral line, harness/instructions.md. Each call names the model and the effort itself, overriding the defaults in that file.

Every turn was audited. We checked each turn's event stream. A turn counts as tool-free when the stream holds only the model's messages, its reasoning summaries, and Codex's notice that code mode is disabled. The result is the audit.tool_free field of the turn's run.json. All 1,125 turns in this release are tool-free.

Where code was allowed (Experiments 3, 6, 7, 8):

Files

Not every file appears in every run.

File What it is
prompt.txt, message.txt What we sent: the first message, and later ones per turn.
events.jsonl The raw Codex event stream of one turn: the model's messages and reasoning summaries, plus token usage.
answer.md, turn_text.md, final_answer.md The model's reply in one turn, and its final reply.
explanations.md The model's explanation of each step, collected over the run, where the prompt asked for one.
tool_calls.jsonl One line per sandbox run: the code, its printed output, and the time it took.
run.json One turn's record: model, effort, tokens, wall time, and the audit.
state.json One conversation's record: turns, runs used, and whether it finished.
loop.log, driver.log, run.log The harness's log of the run.
eval/ The scoring, with the answer key or ground truth.
networks/ The networks the model read or wrote.

Placeholders. Internal paths are replaced by placeholders:

Placeholder Stands for
<data> our data disk
<repo> our experiment repository
<codex> the Codex binary
<codex_home> Codex's state folder
<venvs> Python environments
<home> home directory
<remote> a rented GPU node

Codex thread ids are replaced by id- and 8 hex digits. The same thread keeps the same id in every file, so turns of one conversation still match up.

Not included

Licenses

See LICENSE.

Content License
Logs, data and networks CC BY 4.0
Our scripts MIT
MinAtar source in Experiment 5's prompts GPL-3.0, see exp5_write_from_scratch/LICENSE-MinAtar

Third-party benchmark data keep their own licenses: MATH, GSM8K, IFEval, MMLU-Pro and the others in Experiment 8.

lm_understand-logs (RikiyaT/lm_understand-logs)

正在搜索数据库…

这是什么?这是博客文章 'Sparks of Weight Space Programming: LLMs can understand and write neural network weights' 中每个实验的完整日志。包括发给模型的提示词、模型的回复、它运行的代码、每个数字背后的评估,以及它读取或写入的网络。
这是工具或库,还是只是数据?这是一个数据发布,不是程序。没有需要安装的软件包,只有一些文件,包括提示词、模型回复、沙箱代码和输出、评分结果以及网络。
我怎样找到我关心的那次运行?打开你想看的实验文件夹,例如 exp4fixcar/,先读它的 README。README 会把每次运行对应到文章中的标签页或表格行,并说明每个数字来自哪个文件。
用了哪些模型和工具?'Astra' 是 OpenAI 的 gpt-6-astra,通过 Codex CLI 运行,每次调用只进行一轮对话,并关闭了所有内置工具。代码只在实验 3、6、7 和 8 中运行,运行环境是作者自己的沙箱,且没有网络。
发布内容里缺少什么?未包含:实验 7 中的 17 GB Gemma 3 270M 微调模型、模型工作时产生的中间文件、实验 3 的 Python 语料文本,以及实验 8 中的 GPQA 题目文本。其中部分内容提供了重建脚本和哈希值。
这些实验中模型用了工具吗?没有使用 Codex 内置工具:全部 1,125 轮对话经审计都未使用工具。代码执行只发生在实验 3、6、7 和 8 的自定义 Python 沙箱中。
什么是权重空间编程?这些实验测试 LLM 能否读取神经网络的权重,并修改或编写这些权重。例如,实验 4 修复了一个出错的网络,实验 6 编写了一个含 150 万个权重的 Go 网络。
模型能从零编写玩游戏的网络吗?实验 5 要求模型在不使用任何工具的情况下,为 MinAtar 中的 Breakout、Seaquest 和 Asterix 从零编写网络。日志展示了每次尝试及其评估结果。
有哪些类似的项目?jeeves, NanoJev, bdh, detectorscope, math, Valen, MSA, pointdit
作者
RikiyaT
大小
135.4 MB
星标
1.3K
更新
提交Oct 29
时长
20天
提交 · 90 天