lm-evaluation-harness README

The project's README from github.com/EleutherAI/lm-evaluation-harness. As one markdown file: README.md.

Language Model Evaluation Harness

DOI

Latest News 📣

Announcement

A new v0.4.0 release of lm-evaluation-harness is available !

New updates and features include:

Please see our updated documentation pages in docs/ for more details.

Development will be continuing on the main branch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub, or in the EleutherAI discord!

Overview

This project provides a unified framework to test generative language models on a large number of different evaluation tasks.

Features:

The Language Model Evaluation Harness is the backend for 🤗 Hugging Face's popular Open LLM Leaderboard, has been used in hundreds of papers, and is used internally by dozens of organizations including NVIDIA, Cohere, BigScience, BigCode, Nous Research, and Mosaic ML.

Install

To install the lm-eval package from the github repository, run:

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

Installing Model Backends

The base installation provides the core evaluation framework. Model backends must be installed separately using optional extras:

For HuggingFace transformers models:

pip install "lm_eval[hf]"

For vLLM inference:

pip install "lm_eval[vllm]"

For API-based models (OpenAI, Anthropic, etc.):

pip install "lm_eval[api]"

Multiple backends can be installed together:

pip install "lm_eval[hf,vllm,api]"

A detailed table of all optional extras is available at the end of this document.

Basic Usage

Documentation

Guide Description
CLI Reference Command-line arguments and subcommands
Configuration Guide YAML config file format and examples
Python API Programmatic usage with simple_evaluate()
Task Guide Available tasks and task configuration

Use lm-eval -h to see available options, or lm-eval run -h for evaluation options.

List available tasks with:

lm-eval ls tasks

Hugging Face transformers

[!Important] To use the HuggingFace backend, first install: pip install "lm_eval[hf]"

To evaluate a model hosted on the HuggingFace Hub (e.g. GPT-J-6B) on hellaswag you can use the following command (this assumes you are using a CUDA-compatible GPU):

lm_eval --model hf \
    --model_args pretrained=EleutherAI/gpt-j-6B \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

Additional arguments can be provided to the model constructor using the --model_args flag. Most notably, this supports the common practice of using the revisions feature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:

lm_eval --model hf \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
    --tasks lambada_openai,hellaswag \
    --device cuda:0 \
    --batch_size 8

Models that are loaded via both transformers.AutoModelForCausalLM (autoregressive, decoder-only GPT style models) and transformers.AutoModelForSeq2SeqLM (such as encoder-decoder models like T5) in Huggingface are supported.

Batch size selection can be automated by setting the --batch_size flag to auto. This will perform automatic detection of the largest batch size that will fit on your device. On tasks where there is a large difference between the longest and shortest example, it can be helpful to periodically recompute the largest batch size, to gain a further speedup. To do this, append :N to above flag to automatically recompute the largest batch size N times. For example, to recompute the batch size 4 times, the command would be:

lm_eval --model hf \
    --model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
    --tasks lambada_openai,hellaswag \
    --device cuda:0 \
    --batch_size auto:4

[!Note] Just like you can provide a local path to transformers.AutoModel, you can also provide a local path to lm_eval via --model_args pretrained=/path/to/model

Evaluating GGUF Models

lm-eval supports evaluating models in GGUF format using the Hugging Face (hf) backend. This allows you to use quantized models compatible with transformers, AutoModel, and llama.cpp conversions.

To evaluate a GGUF model, pass the path to the directory containing the model weights, the gguf_file, and optionally a separate tokenizer path using the --model_args flag.

🚨 Important Note:
If no separate tokenizer is provided, Hugging Face will attempt to reconstruct the tokenizer from the GGUF file — this can take hours or even hang indefinitely. Passing a separate tokenizer avoids this issue and can reduce tokenizer loading time from hours to seconds.

✅ Recommended usage:

lm_eval --model hf \
    --model_args pretrained=/path/to/gguf_folder,gguf_file=model-name.gguf,tokenizer=/path/to/tokenizer \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 8

[!Tip] Ensure the tokenizer path points to a valid Hugging Face tokenizer directory (e.g., containing tokenizer_config.json, vocab.json, etc.).

Evaluating GGUF Models Served by llama.cpp

The gguf model type evaluates models hosted by a llama.cpp server (llama-server) through its OpenAI-compatible /v1/completions endpoint:

lm_eval --model gguf \
    --model_args base_url=http://127.0.0.1:8080 \
    --tasks hellaswag

Multi-GPU Evaluation with Hugging Face accelerate

We support three main ways of using Hugging Face's accelerate 🚀 library for multi-GPU evaluation.

To perform data-parallel evaluation (where each GPU loads a separate full copy of the model), we leverage the accelerate launcher as follows:

accelerate launch -m lm_eval --model hf \
    --tasks lambada_openai,arc_easy \
    --batch_size 16

(or via accelerate launch --no-python lm_eval).

For cases where your model can fit on a single GPU, this allows you to evaluate on K GPUs K times faster than on one.

WARNING: This setup does not work with FSDP model sharding, so in accelerate config FSDP must be disabled, or the NO_SHARD FSDP option must be used.

The second way of using accelerate for multi-GPU evaluation is when your model is too large to fit on a single GPU.

In this setting, run the library outside the accelerate launcher, but passing parallelize=True to --model_args as follows:

lm_eval --model hf \
    --tasks lambada_openai,arc_easy \
    --model_args parallelize=True \
    --batch_size 16

This means that your model's weights will be split across all available GPUs.

For more advanced users or even larger models, we allow for the following arguments when parallelize=True as well:

The third option is to use both at the same time. This will allow you to take advantage of both data parallelism and model sharding, and is especially useful for models that are too large to fit on a single GPU.

accelerate launch --multi_gpu --num_processes {nb_of_copies_of_your_model} \
    -m lm_eval --model hf \
    --tasks lambada_openai,arc_easy \
    --model_args parallelize=True \
    --batch_size 16

To learn more about model parallelism and how to use it with the accelerate library, see the accelerate documentation

Warning: We do not natively support multi-node evaluation using the hf model type! Please reference our GPT-NeoX library integration for an example of code in which a custom multi-machine evaluation script is written.

Note: we do not currently support multi-node evaluations natively, and advise using either an externally hosted server to run inference requests against, or creating a custom integration with your distributed framework as is done for the GPT-NeoX library.

Tensor Parallelism (native PyTorch)

For models that support PyTorch's native Tensor Parallelism (via DTensor), you can shard model weights across GPUs without accelerate's device-map by passing tp_plan=auto in --model_args. Launch with torchrun or accelerate launch:

torchrun --nproc-per-node=4 -m lm_eval \
    --model hf \
    --model_args pretrained=google/gemma-4-31B-it,tp_plan=auto \
    --tasks lambada_openai,arc_easy \
    --batch_size 16

Constraints:

Steered Hugging Face transformers models

To evaluate a Hugging Face transformers model with steering vectors applied, specify the model type as steered and provide the path to either a PyTorch file containing pre-defined steering vectors, or a CSV file that specifies how to derive steering vectors from pretrained sparsify or sae_lens models (you will need to install the corresponding optional dependency for this method).

Specify pre-defined steering vectors:

import torch

steer_config = {
    "layers.3": {
        "steering_vector": torch.randn(1, 768),
        "bias": torch.randn(1, 768),
        "steering_coefficient": 1,
        "action": "add"
    },
}
torch.save(steer_config, "steer_config.pt")

Specify derived steering vectors:

import pandas as pd

pd.DataFrame({
    "loader": ["sparsify"],
    "action": ["add"],
    "sparse_model": ["EleutherAI/sae-pythia-70m-32k"],
    "hookpoint": ["layers.3"],
    "feature_index": [30],
    "steering_coefficient": [10.0],
}).to_csv("steer_config.csv", index=False)

Run the evaluation harness with steering vectors applied:

lm_eval --model steered \
    --model_args pretrained=EleutherAI/pythia-160m,steer_path=steer_config.pt \
    --tasks lambada_openai,hellaswag \
    --device cuda:0 \
    --batch_size 8

NVIDIA nemo models

NVIDIA NeMo Framework is a generative AI framework built for researchers and pytorch developers working on language models.

To evaluate a nemo model, start by installing NeMo following the documentation. We highly recommended to use the NVIDIA PyTorch or NeMo container, especially if having issues installing Apex or any other dependencies (see latest released containers). Please also install the lm evaluation harness library following the instructions in the Install section.

NeMo models can be obtained through NVIDIA NGC Catalog or in NVIDIA's Hugging Face page. In NVIDIA NeMo Framework there are conversion scripts to convert the hf checkpoints of popular models like llama, falcon, mixtral or mpt to nemo.

Run a nemo model on one GPU:

lm_eval --model nemo_lm \
    --model_args path=<path_to_nemo_model> \
    --tasks hellaswag \
    --batch_size 32

It is recommended to unpack the nemo model to avoid the unpacking inside the docker container - it may overflow disk space. For that you can run:

mkdir MY_MODEL
tar -xvf MY_MODEL.nemo -c MY_MODEL

Multi-GPU evaluation with NVIDIA nemo models

By default, only one GPU is used. But we do support either data replication or tensor/pipeline parallelism during evaluation, on one node.

  1. To enable data replication, set the model_args of devices to the number of data replicas to run. For example, the command to run 8 data replicas over 8 GPUs is:
torchrun --nproc-per-node=8 --no-python lm_eval \
    --model nemo_lm \
    --model_args path=<path_to_nemo_model>,devices=8 \
    --tasks hellaswag \
    --batch_size 32
  1. To enable tensor and/or pipeline parallelism, set the model_args of tensor_model_parallel_size and/or pipeline_model_parallel_size. In addition, you also have to set up devices to be equal to the product of tensor_model_parallel_size and/or pipeline_model_parallel_size. For example, the command to use one node of 4 GPUs with tensor parallelism of 2 and pipeline parallelism of 2 is:
torchrun --nproc-per-node=4 --no-python lm_eval \
    --model nemo_lm \
    --model_args path=<path_to_nemo_model>,devices=4,tensor_model_parallel_size=2,pipeline_model_parallel_size=2 \
    --tasks hellaswag \
    --batch_size 32

Note that it is recomme

lm-evaluation-harness (EleutherAI/lm-evaluation-harness)

llm-evaluation
Searching Database…

14.3K ·GitHub
Lm-evaluation-harness
What is this?It is a unified framework for few-shot evaluation of language models across a huge number of standard academic benchmarks.
Is this a library or a CLI tool?Both: it ships a CLI (lm-eval with run, ls, validate subcommands) and a Python API (simple_evaluate()) for programmatic use.
How do I install it and run an evaluation?Install the base package from the GitHub repo, then add the backend you need as an extra (e.g. pip install lmeval[hf] or lmeval[vllm]), and run lm-eval -h to see options, e.g. evaluating a HF model on hellaswag.
Which model backends do I need to install separately?The base package no longer bundles transformers/torch; you must install the specific backend extra you plan to use, such as hf, vllm, sglang, or API-based backends.
Any pitfalls when evaluating GGUF models?If you don't pass a separate tokenizer path for GGUF models, Hugging Face will try to reconstruct the tokenizer from the GGUF file, which can take hours or hang indefinitely.
Who actually uses this in practice?It's the backend for Hugging Face's Open LLM Leaderboard and is used internally by organizations like NVIDIA, Cohere, BigScience, BigCode, Nous Research, and Mosaic ML.
Can I evaluate GGUF models served by llama.cpp directly?Yes, the gguf model type can hit a running llama-server's OpenAI-compatible /v1/completions endpoint, but it requires a llama.cpp release from December 2024 or newer that returns modern logprobs format.
Does it support multi-GPU setups?Yes, via Hugging Face accelerate for data-parallel evaluation across GPUs, or parallelize=True in --model_args to shard a model too large for one GPU, and even native PyTorch Tensor Parallelism via tp_plan=auto.
Can I extend it with my own model backends or metrics?Yes, a new plugin system lets you register custom model backends, filters, metrics and aggregations from your own package via an lmeval entry point, or by pointing --plugins at a local module.
What kinds of models can it evaluate?It supports HuggingFace transformers (including LoRA adapters via PEFT and quantized GPTQ models), vLLM, SGLang, GPT-NeoX, Megatron-DeepSpeed, and commercial APIs like OpenAI and TextSynth.
What are similar projects?decider, MiniCPM, open-r1, Index-Translate, DeepSeek-Coder, DMI, Model-Optimizer, jeeves
Which topics is it in?evaluation-framework, llm-evaluation, language-model, vllm, large-language-models, transformer, nlp, huggingface, pytorch, benchmark, command-line-tool, deep-learning
Creator
EleutherAI
Size
35.1 MB
Stars
14.3K
Updated
CommitSep 14Releasev0.4.13Aug 31
Age
6y
Commits · 90 Days