Language Model Evaluation Harness
Latest News 📣
- [2026/09] Plugins: model backends, filters, metrics and aggregations can now be registered from your own package instead of forking — declare an
lm_eval.* entry point for zero-config discovery, or point --plugins at a local module. See the Plugin Guide.
- [2025/12] CLI refactored with subcommands (
run, ls, validate) and YAML config file support via --config. See the CLI Reference and Configuration Guide.
- [2025/12] Lighter install: Base package no longer includes
transformers/torch. Install model backends separately: pip install lm_eval[hf], lm_eval[vllm], etc.
- [2025/07] Added
think_end_token arg to hf (token/str), vllm and sglang (str) for stripping CoT reasoning traces from models that support it.
- [2025/03] Added support for steering HF models!
- [2025/02] Added SGLang support!
- [2024/09] We are prototyping allowing users of LM Evaluation Harness to create and evaluate on text+image multimodal input, text output tasks, and have just added the
hf-multimodal and vllm-vlm model types and mmmu task as a prototype feature. We welcome users to try out this in-progress feature and stress-test it for themselves, and suggest they check out lmms-eval, a wonderful project originally forking off of the lm-evaluation-harness, for a broader range of multimodal tasks, models, and features.
- [2024/07] API model support has been updated and refactored, introducing support for batched and async requests, and making it significantly easier to customize and use for your own purposes. To run Llama 405B, we recommend using VLLM's OpenAI-compliant API to host the model, and use the
local-completions model type to evaluate the model.
- [2024/07] New Open LLM Leaderboard tasks have been added ! You can find them under the leaderboard task group.
Announcement
A new v0.4.0 release of lm-evaluation-harness is available !
New updates and features include:
- New Open LLM Leaderboard tasks have been added ! You can find them under the leaderboard task group.
- Internal refactoring
- Config-based task creation and configuration
- Easier import and sharing of externally-defined task config YAMLs
- Support for Jinja2 prompt design, easy modification of prompts + prompt imports from Promptsource
- More advanced configuration options, including output post-processing, answer extraction, and multiple LM generations per document, configurable fewshot settings, and more
- Speedups and new modeling libraries supported, including: faster data-parallel HF model usage, vLLM support, MPS support with HuggingFace, and more
- Logging and usability changes
- New tasks including CoT BIG-Bench-Hard, Belebele, user-defined task groupings, and more
Please see our updated documentation pages in docs/ for more details.
Development will be continuing on the main branch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub, or in the EleutherAI discord!
Overview
This project provides a unified framework to test generative language models on a large number of different evaluation tasks.
Features:
- Over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented.
- Support for models loaded via transformers (including quantization via GPTQModel and AutoGPTQ), GPT-NeoX, and Megatron-DeepSpeed, with a flexible tokenization-agnostic interface.
- Support for fast and memory-efficient inference with vLLM.
- Support for commercial APIs including OpenAI, and TextSynth.
- Support for evaluation on adapters (e.g. LoRA) supported in HuggingFace's PEFT library.
- Support for local models and benchmarks.
- Evaluation with publicly available prompts ensures reproducibility and comparability between papers.
- Easy support for custom prompts and evaluation metrics.
The Language Model Evaluation Harness is the backend for 🤗 Hugging Face's popular Open LLM Leaderboard, has been used in hundreds of papers, and is used internally by dozens of organizations including NVIDIA, Cohere, BigScience, BigCode, Nous Research, and Mosaic ML.
Install
To install the lm-eval package from the github repository, run:
git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .
Installing Model Backends
The base installation provides the core evaluation framework. Model backends must be installed separately using optional extras:
For HuggingFace transformers models:
pip install "lm_eval[hf]"
For vLLM inference:
pip install "lm_eval[vllm]"
For API-based models (OpenAI, Anthropic, etc.):
pip install "lm_eval[api]"
Multiple backends can be installed together:
pip install "lm_eval[hf,vllm,api]"
A detailed table of all optional extras is available at the end of this document.
Basic Usage
Documentation
Use lm-eval -h to see available options, or lm-eval run -h for evaluation options.
List available tasks with:
lm-eval ls tasks
[!Important]
To use the HuggingFace backend, first install: pip install "lm_eval[hf]"
To evaluate a model hosted on the HuggingFace Hub (e.g. GPT-J-6B) on hellaswag you can use the following command (this assumes you are using a CUDA-compatible GPU):
lm_eval --model hf \
--model_args pretrained=EleutherAI/gpt-j-6B \
--tasks hellaswag \
--device cuda:0 \
--batch_size 8
Additional arguments can be provided to the model constructor using the --model_args flag. Most notably, this supports the common practice of using the revisions feature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:
lm_eval --model hf \
--model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
--tasks lambada_openai,hellaswag \
--device cuda:0 \
--batch_size 8
Models that are loaded via both transformers.AutoModelForCausalLM (autoregressive, decoder-only GPT style models) and transformers.AutoModelForSeq2SeqLM (such as encoder-decoder models like T5) in Huggingface are supported.
Batch size selection can be automated by setting the --batch_size flag to auto. This will perform automatic detection of the largest batch size that will fit on your device. On tasks where there is a large difference between the longest and shortest example, it can be helpful to periodically recompute the largest batch size, to gain a further speedup. To do this, append :N to above flag to automatically recompute the largest batch size N times. For example, to recompute the batch size 4 times, the command would be:
lm_eval --model hf \
--model_args pretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float" \
--tasks lambada_openai,hellaswag \
--device cuda:0 \
--batch_size auto:4
[!Note]
Just like you can provide a local path to transformers.AutoModel, you can also provide a local path to lm_eval via --model_args pretrained=/path/to/model
Evaluating GGUF Models
lm-eval supports evaluating models in GGUF format using the Hugging Face (hf) backend. This allows you to use quantized models compatible with transformers, AutoModel, and llama.cpp conversions.
To evaluate a GGUF model, pass the path to the directory containing the model weights, the gguf_file, and optionally a separate tokenizer path using the --model_args flag.
🚨 Important Note:
If no separate tokenizer is provided, Hugging Face will attempt to reconstruct the tokenizer from the GGUF file — this can take hours or even hang indefinitely. Passing a separate tokenizer avoids this issue and can reduce tokenizer loading time from hours to seconds.
✅ Recommended usage:
lm_eval --model hf \
--model_args pretrained=/path/to/gguf_folder,gguf_file=model-name.gguf,tokenizer=/path/to/tokenizer \
--tasks hellaswag \
--device cuda:0 \
--batch_size 8
[!Tip]
Ensure the tokenizer path points to a valid Hugging Face tokenizer directory (e.g., containing tokenizer_config.json, vocab.json, etc.).
Evaluating GGUF Models Served by llama.cpp
The gguf model type evaluates models hosted by a llama.cpp server (llama-server) through its OpenAI-compatible /v1/completions endpoint:
lm_eval --model gguf \
--model_args base_url=http://127.0.0.1:8080 \
--tasks hellaswag
- Requires a llama.cpp release from December 2024 or newer, which returns logprobs in the modern OpenAI format (
logprobs.content, see llama.cpp#10783). The deprecated legacy format (token_logprobs) is not supported.
- If the server runs in router mode (multiple models), pass the model name or alias:
--model_args base_url=http://127.0.0.1:8080,model=my-model-alias.
- Requests are issued concurrently. The default degree of parallelism is auto-detected from the server's slot count (
/props → total_slots, i.e. llama-server's --parallel setting); override it with parallel=<N>. loglikelihood requests are additionally pinned to slots via llama.cpp's id_slot parameter, with consecutive requests sharing a context (e.g. the candidate continuations of one multiple-choice question) assigned to the same slot so they reuse its cached prompt prefix. generate_until requests are left unpinned, since the server's dynamic idle-slot assignment load-balances variable-length generations better.
loglikelihood is implemented via exact teacher forcing: llama.cpp ignores echo and never returns prompt logprobs, so the continuation is tokenized with the server's /tokenize endpoint (context and continuation encoded separately, matching the HF backend), and each continuation token is scored with its natural prefix as a token-id prompt plus a logit_bias that forces the server to sample it. llama.cpp reports the forced token's pre-sampling logprob and an unbiased top_logprobs, yielding loglikelihoods that match the HF backend to within quantization noise. (A naive alternative — forcing the continuation with a GBNF grammar — does not work: grammar-constrained decoding can pick a fragmented tokenization whose logprobs are not the natural-tokenization loglikelihoods, which systematically deflates scores.)loglikelihood_rolling (perplexity tasks such as wikitext) is not implemented for this model type.
Multi-GPU Evaluation with Hugging Face accelerate
We support three main ways of using Hugging Face's accelerate 🚀 library for multi-GPU evaluation.
To perform data-parallel evaluation (where each GPU loads a separate full copy of the model), we leverage the accelerate launcher as follows:
accelerate launch -m lm_eval --model hf \
--tasks lambada_openai,arc_easy \
--batch_size 16
(or via accelerate launch --no-python lm_eval).
For cases where your model can fit on a single GPU, this allows you to evaluate on K GPUs K times faster than on one.
WARNING: This setup does not work with FSDP model sharding, so in accelerate config FSDP must be disabled, or the NO_SHARD FSDP option must be used.
The second way of using accelerate for multi-GPU evaluation is when your model is too large to fit on a single GPU.
In this setting, run the library outside the accelerate launcher, but passing parallelize=True to --model_args as follows:
lm_eval --model hf \
--tasks lambada_openai,arc_easy \
--model_args parallelize=True \
--batch_size 16
This means that your model's weights will be split across all available GPUs.
For more advanced users or even larger models, we allow for the following arguments when parallelize=True as well:
device_map_option: How to split model weights across available GPUs. defaults to "auto".max_memory_per_gpu: the max GPU memory to use per GPU in loading the model.max_cpu_memory: the max amount of CPU memory to use when offloading the model weights to RAM.offload_folder: a folder where model weights will be offloaded to disk if needed.
The third option is to use both at the same time. This will allow you to take advantage of both data parallelism and model sharding, and is especially useful for models that are too large to fit on a single GPU.
accelerate launch --multi_gpu --num_processes {nb_of_copies_of_your_model} \
-m lm_eval --model hf \
--tasks lambada_openai,arc_easy \
--model_args parallelize=True \
--batch_size 16
To learn more about model parallelism and how to use it with the accelerate library, see the accelerate documentation
Warning: We do not natively support multi-node evaluation using the hf model type! Please reference our GPT-NeoX library integration for an example of code in which a custom multi-machine evaluation script is written.
Note: we do not currently support multi-node evaluations natively, and advise using either an externally hosted server to run inference requests against, or creating a custom integration with your distributed framework as is done for the GPT-NeoX library.
Tensor Parallelism (native PyTorch)
For models that support PyTorch's native Tensor Parallelism (via DTensor), you can shard model weights across GPUs without accelerate's device-map by passing tp_plan=auto in --model_args. Launch with torchrun or accelerate launch:
torchrun --nproc-per-node=4 -m lm_eval \
--model hf \
--model_args pretrained=google/gemma-4-31B-it,tp_plan=auto \
--tasks lambada_openai,arc_easy \
--batch_size 16
Constraints:
tp_plan and parallelize=True are mutually exclusive — use one or the other.- The number of key-value heads in the model must be divisible by
--nproc-per-node (the TP degree).
- Requires PyTorch >= 2.4 and a
transformers version that exposes a TP plan for the model (v4.47+).
To evaluate a Hugging Face transformers model with steering vectors applied, specify the model type as steered and provide the path to either a PyTorch file containing pre-defined steering vectors, or a CSV file that specifies how to derive steering vectors from pretrained sparsify or sae_lens models (you will need to install the corresponding optional dependency for this method).
Specify pre-defined steering vectors:
import torch
steer_config = {
"layers.3": {
"steering_vector": torch.randn(1, 768),
"bias": torch.randn(1, 768),
"steering_coefficient": 1,
"action": "add"
},
}
torch.save(steer_config, "steer_config.pt")
Specify derived steering vectors:
import pandas as pd
pd.DataFrame({
"loader": ["sparsify"],
"action": ["add"],
"sparse_model": ["EleutherAI/sae-pythia-70m-32k"],
"hookpoint": ["layers.3"],
"feature_index": [30],
"steering_coefficient": [10.0],
}).to_csv("steer_config.csv", index=False)
Run the evaluation harness with steering vectors applied:
lm_eval --model steered \
--model_args pretrained=EleutherAI/pythia-160m,steer_path=steer_config.pt \
--tasks lambada_openai,hellaswag \
--device cuda:0 \
--batch_size 8
NVIDIA nemo models
NVIDIA NeMo Framework is a generative AI framework built for researchers and pytorch developers working on language models.
To evaluate a nemo model, start by installing NeMo following the documentation. We highly recommended to use the NVIDIA PyTorch or NeMo container, especially if having issues installing Apex or any other dependencies (see latest released containers). Please also install the lm evaluation harness library following the instructions in the Install section.
NeMo models can be obtained through NVIDIA NGC Catalog or in NVIDIA's Hugging Face page. In NVIDIA NeMo Framework there are conversion scripts to convert the hf checkpoints of popular models like llama, falcon, mixtral or mpt to nemo.
Run a nemo model on one GPU:
lm_eval --model nemo_lm \
--model_args path=<path_to_nemo_model> \
--tasks hellaswag \
--batch_size 32
It is recommended to unpack the nemo model to avoid the unpacking inside the docker container - it may overflow disk space. For that you can run:
mkdir MY_MODEL
tar -xvf MY_MODEL.nemo -c MY_MODEL
Multi-GPU evaluation with NVIDIA nemo models
By default, only one GPU is used. But we do support either data replication or tensor/pipeline parallelism during evaluation, on one node.
- To enable data replication, set the
model_args of devices to the number of data replicas to run. For example, the command to run 8 data replicas over 8 GPUs is:
torchrun --nproc-per-node=8 --no-python lm_eval \
--model nemo_lm \
--model_args path=<path_to_nemo_model>,devices=8 \
--tasks hellaswag \
--batch_size 32
- To enable tensor and/or pipeline parallelism, set the
model_args of tensor_model_parallel_size and/or pipeline_model_parallel_size. In addition, you also have to set up devices to be equal to the product of tensor_model_parallel_size and/or pipeline_model_parallel_size. For example, the command to use one node of 4 GPUs with tensor parallelism of 2 and pipeline parallelism of 2 is:
torchrun --nproc-per-node=4 --no-python lm_eval \
--model nemo_lm \
--model_args path=<path_to_nemo_model>,devices=4,tensor_model_parallel_size=2,pipeline_model_parallel_size=2 \
--tasks hellaswag \
--batch_size 32
Note that it is recomme
这是一个统一的框架，用于在大量标准学术基准测试上对语言模型进行少样本（few-shot）评测。