evals README

The project's README from github.com/openai/evals. As one markdown file: README.md.

OpenAI Evals

You can now configure and run Evals directly in the OpenAI Dashboard. Get started →

Evals provide a framework for evaluating large language models (LLMs) or systems built using LLMs. We offer an existing registry of evals to test different dimensions of OpenAI models and the ability to write your own custom evals for use cases you care about. You can also use your data to build private evals which represent the common LLMs patterns in your workflow without exposing any of that data publicly.

If you are building with LLMs, creating high quality evals is one of the most impactful things you can do. Without evals, it can be very difficult and time intensive to understand how different model versions might affect your use case. In the words of OpenAI's President Greg Brockman:

https://x.com/gdb/status/1733553161884127435?s=20

Setup

To run evals, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the OPENAI_API_KEY environment variable. Please be aware of the costs associated with using the API when running evals. You can also run and create evals using Weights & Biases.

Minimum Required Version: Python 3.9

Downloading evals

Our evals registry is stored using Git-LFS. Once you have downloaded and installed LFS, you can fetch the evals (from within your local copy of the evals repo) with:

cd evals
git lfs fetch --all
git lfs pull

This will populate all the pointer files under evals/registry/data.

You may just want to fetch data for a select eval. You can achieve this via:

git lfs fetch --include=evals/registry/data/${your eval}
git lfs pull

Making evals

If you are going to be creating evals, we suggest cloning this repo directly from GitHub and installing the requirements using the following command:

pip install -e .

Using -e, changes you make to your eval will be reflected immediately without having to reinstall.

Optionally, you can install the formatters for pre-committing with:

pip install -e .[formatters]

Then run pre-commit install to install pre-commit into your git hooks. pre-commit will now run on every commit.

If you want to manually run all pre-commit hooks on a repository, run pre-commit run --all-files. To run individual hooks use pre-commit run <hook_id>.

Running evals

If you don't want to contribute new evals, but simply want to run them locally, you can install the evals package via pip:

pip install evals

You can find the full instructions to run existing evals in run-evals.md and our existing eval templates in eval-templates.md. For more advanced use cases like prompt chains or tool-using agents, you can use our Completion Function Protocol.

We provide the option for you to log your eval results to a Snowflake database, if you have one or wish to set one up. For this option, you will further have to specify the SNOWFLAKE_ACCOUNT, SNOWFLAKE_DATABASE, SNOWFLAKE_USERNAME, and SNOWFLAKE_PASSWORD environment variables.

Writing evals

We suggest getting starting by:

Please note that we are currently not accepting evals with custom code! While we ask you to not submit such evals at the moment, you can still submit model-graded evals with custom model-graded YAML files.

If you think you have an interesting eval, please open a pull request with your contribution. OpenAI staff actively review these evals when considering improvements to upcoming models.

FAQ

Do you have any examples of how to build an eval from start to finish?

Do you have any examples of evals implemented in multiple different ways?

When I run an eval, it sometimes hangs at the very end (after the final report). What's going on?

There's a lot of code, and I just want to spin up a quick eval. Help? OR,

I am a world-class prompt engineer. I choose not to code. How can I contribute my wisdom?

Disclaimer

By contributing to evals, you are agreeing to make your evaluation logic and data under the same MIT license as this repository. You must have adequate rights to upload any data used in an eval. OpenAI reserves the right to use this data in future service improvements to our product. Contributions to OpenAI evals will be subject to our usual Usage Policies: https://platform.openai.com/docs/usage-policies.

evals (openai/evals)

Searching Database…

19.6K ·GitHub
Evals
Что это?Evals — это фреймворк от OpenAI для оценки LLM и систем на их основе, а также открытый реестр бенчмарков, которые можно запускать или дополнять.
Это библиотека, которую я импортирую, или CLI-инструмент, который я запускаю?Это Python-пакет, который устанавливается через pip и запускается из командной строки для выполнения эвалов над моделями.
Как запустить готовый эвал?Установите пакет через pip, затем следуйте инструкциям в run-evals.md, чтобы запустить один из готовых шаблонов эвалов, перечисленных в eval-templates.md.
Нужен ли мне ключ API OpenAI, чтобы этим пользоваться?Прямо не указано, but так как эвалы оценивают модели OpenAI, вам понадобится доступ к API; дополнительно можно настроить базу данных Snowflake для записи результатов.
Могу ли я создавать приватные эвалы на своих данных?Да, можно создавать приватные эвалы на своих данных, чтобы моделировать типичные сценарии использования LLM в вашем рабочем процессе, не раскрывая эти данные публично.
Что такое Completion Function Protocol?Это механизм, который поддерживает Evals для более сложных сценариев, например цепочек промптов или агентов, использующих инструменты, то есть выходящих за рамки простых эвалов с одним вызовом.
Где посмотреть примеры эвалов перед написанием своих?В папке examples есть полные разобранные примеры, а файл evals/registry/evals/coqa.yaml показывает, как один и тот же датасет реализован сразу в нескольких шаблонах эвалов.
OpenAI вообще смотрит эвалы, которые присылает сообщество?Да, сотрудники OpenAI активно просматривают эвалы из пул-реквестов, когда думают об улучшениях будущих моделей.
What are similar projects?jevals, ASI-Bench, livenerf, ragas, EvalForge, jev-align, arc-code, ParseBench
Which topics is it in?evals, llm-evaluation, registry, yaml, gpt, evaluation, json, testing, benchmark, openai, research, llm
Creator
openai
Size
6.5 MB
Stars
19.6K
Updated
CommitApr 14
Age
3y
Commits · 90 Days