SWE-bench README

The project's README from github.com/SWE-bench/SWE-bench. As one markdown file: README.md.

Kawi the SWE-Llama

[ Read the Docs ]

日本語 | 中文简体 | 中文繁體

Build License

Code and data for the following works:

📰 News

👋 Overview

SWE-bench is a benchmark for evaluating large language models on real world software issues collected from GitHub. Given a codebase and an issue, a language model is tasked with generating a patch that resolves the described problem.

To access SWE-bench, copy and run the following code:

from datasets import load_dataset
swebench = load_dataset('princeton-nlp/SWE-bench', split='test')

🚀 Set Up

SWE-bench uses Docker for reproducible evaluations. Follow the instructions in the Docker setup guide to install Docker on your machine. If you're setting up on Linux, we recommend seeing the post-installation steps as well.

Finally, to build SWE-bench from source, follow these steps:

git clone [email protected]:SWE-bench/SWE-bench.git
cd SWE-bench
pip install -e .

# Required for local image builds with the v5 CLI. This path is only an example.
git clone --depth 1 https://github.com/SWE-bench/swe-bench-tasks.git ./swe-bench-tasks
swebench dataset check ./swe-bench-tasks

You can replace this path with any local checkout location.

Test your installation by running:

swebench eval verified --gold \
    -i sympy__sympy-20590 \
    --run-id validate-gold \
    --task-repo ./swe-bench-tasks

[!NOTE] The current v5 CLI builds images from a task repo. On an M-series Mac or another ARM-based system, use --task-repo so the images are built locally with Docker Buildx.

💽 Usage

Evaluate patch predictions with the following command:

swebench eval verified -p <path_to_predictions> --run-id <run_id> -j <num_workers>

DATASET accepts an alias (full, verified, multimodal, multilingual), a HuggingFace id, or a local path. Anything else is passed through as given, so SWE-bench/SWE-bench_Lite works too:

swebench eval verified --gold                 # reference patches
swebench eval multimodal --gold -i carbon-design-system__carbon-10188
swebench report <run_id> -d verified          # re-grade saved logs, no containers

Other commands:

swebench infer verified -m gpt-5 -o preds -w 8        # generate predictions with mini-SWE-agent
swebench images build verified -j 8           # build/pull images ahead of time
swebench images check multilingual            # verify images exist on the registry
swebench images clean --run-id <run_id>       # remove leftover containers
swebench submit hf <run_id> -b <user/bucket>      # publish results to an HF bucket
swebench --help                               # all commands

[!NOTE] The previous python -m swebench.harness.run_evaluation ... form still works and takes the same arguments as before.

This command will generate docker build logs (logs/build_images) and evaluation logs (logs/evaluation) in the current directory.

The run summary is written to logs/evaluation/<run_id>/results.json.

[!NOTE] Result Caching: The evaluation harness caches results by run_id and instance_id only. If you run the same instance with the same run_id multiple times, even with different prediction diffs, the harness will reuse the cached results from the first run and will not re-evaluate. To re-evaluate an instance with a different prediction diff, you must use a different run_id.

[!WARNING] SWE-bench evaluation can be resource intensive We recommend running on an x86_64 machine with at least 120GB of free storage, 16GB of RAM, and 8 CPU cores. We recommend using fewer than min(0.75 * os.cpu_count(), 24) for --max_workers.

If running with Docker desktop, make sure to increase your virtual disk space to ~120 free GB. Set max_workers to be consistent with the above for the CPUs available to Docker.

Support for arm64 machines is experimental.

To see the full list of arguments for the evaluation harness, run:

swebench eval --help

See the evaluation tutorial for the full rundown on datasets you can evaluate. If you're looking for non-local, cloud based evaluations, check out...

Additionally, you can also:

⬇️ Downloads

Datasets Models RAG
💿 SWE-bench 🦙 SWE-Llama 13b 🤗 "Oracle" Retrieval
💿 SWE-bench Lite 🦙 SWE-Llama 13b (PEFT) 🤗 BM25 Retrieval 13K
💿 SWE-bench Verified 🦙 SWE-Llama 7b 🤗 BM25 Retrieval 27K
💿 SWE-bench Multimodal 🦙 SWE-Llama 7b (PEFT) 🤗 BM25 Retrieval 40K
🤗 BM25 Retrieval 50K (Llama tokens)

💫 Contributions

We would love to hear from the broader NLP, Machine Learning, and Software Engineering research communities, and we welcome any contributions, pull requests, or issues! To do so, please either file a new pull request or issue and fill in the corresponding templates accordingly. We'll be sure to follow up shortly!

Contact person: Carlos E. Jimenez and John Yang (Email: [email protected], [email protected]).

✍️ Citation & license

MIT license. Check LICENSE.md.

If you find our work helpful, please use the following citations.

For SWE-bench (Verified):

@inproceedings{
    jimenez2024swebench,
    title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
    author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
    booktitle={The Twelfth International Conference on Learning Representations},
    year={2024},
    url={https://openreview.net/forum?id=VTF8yNQM66}
}

For SWE-bench Multimodal

@inproceedings{
    yang2024swebenchmultimodal,
    title={{SWE}-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?},
    author={John Yang and Carlos E. Jimenez and Alex L. Zhang and Kilian Lieret and Joyce Yang and Xindi Wu and Ori Press and Niklas Muennighoff and Gabriel Synnaeve and Karthik R. Narasimhan and Diyi Yang and Sida I. Wang and Ofir Press},
    booktitle={The Thirteenth International Conference on Learning Representations},
    year={2025},
    url={https://openreview.net/forum?id=riTiq3i21b}
}

For SWE-bench Multilingual

@misc{yang2025swesmith,
    title={SWE-smith: Scaling Data for Software Engineering Agents},
    author={John Yang and Kilian Lieret and Carlos E. Jimenez and Alexander Wettig and Kabir Khandpur and Yanzhe Zhang and Binyuan Hui and Ofir Press and Ludwig Schmidt and Diyi Yang},
    year={2025},
    eprint={2504.21798},
    archivePrefix={arXiv},
    primaryClass={cs.SE},
    url={https://arxiv.org/abs/2504.21798},
}

Our Other Projects

sb-cli    SWE-smith    SWE-agent    CodeClash    Mini-SWE-Agent    SWE-ReX

SWE-bench (SWE-bench/SWE-bench)

Searching Database…

6K ·GitHub
SWE-bench
What is this?SWE-bench is a benchmark that tests large language models on real GitHub issues: the model gets a codebase plus an issue and must produce a patch that resolves it. An automated harness scores the patches.
Is it a dataset or a tool I run?Both. The repo ships several benchmark datasets (full, Lite, Verified, Multimodal, Multilingual) plus a Python evaluation harness that scores patch predictions in Docker, and sb-cli for cloud-based runs.
How do I get it running?Install Docker first (Linux users should also do the post-installation steps), then build SWE-bench from source and run the evaluation CLI against a dataset alias like 'verified' plus your patch predictions.
What does my machine need to run evaluations locally?It is resource intensive: x86_64 with at least 120GB free storage, 16GB of RAM and 8 CPU cores is recommended, with --max_workers kept below min(0.75 x CPU count, 24). Docker Desktop users should raise the virtual disk to about 120GB.
Will it run on my M-series Mac?Yes, but arm64 support is experimental: on an M-series Mac or other ARM system use the --task-repo flag so images build locally with Docker Buildx. x86_64 machines remain the recommended setup.
Can I evaluate in the cloud instead of on my own machine?Yes. sb-cli submits runs to the SWE-bench leaderboards with evaluation handled automatically on AWS, and you can also run the harness on Modal.
What is the difference between SWE-bench, Lite and Verified?They are slices of the same benchmark. Verified is a 500-problem subset that real software engineers, in collaboration with OpenAI's Preparedness team, confirmed are solvable; Multimodal adds visually-grounded tasks and its v2 is fully open source with 480 tasks (September 2026). You pick a set via a DATASET alias: full, verified, multimodal, multilingual.
Why do re-runs skip instances even though I changed my predictions?The harness caches results by run_id and instance_id only. Re-running the same instance under the same run_id reuses the first result even with a different diff, so use a new run_id when your predictions change.
Besides scoring models, what else can I do with it?Train models on the pre-processed datasets (SWE-smith is the team's toolkit for creating SWE training data), run inference with local or API models, or collect new SWE-bench tasks from your own repositories — though support for instance-creation queries is temporarily paused.
What is SWE-agent?A coding agent from the same team, released in April 2024, which set the state of the art on the full SWE-bench test set at the time. The repo also provides SWE-Llama models (7b/13b, with PEFT variants) and BM25/Oracle retrieval resources.
What are similar projects?Open-Assistant, XRepoTest, seamless_communication, open-r1, gpt-2, lm-evaluation-harness, EvoTrace, DeepSpec
Which topics is it in?software-engineering, language-model, code-generation, aws, dataset, coding-agents, evaluation, nlp, huggingface, benchmark, cloud, research
Creator
SWE-bench
Size
78.3 MB
Stars
6K
Updated
CommitSep 02
Age
3y
Commits · 90 Days