🤗 Datasets is a lightweight library providing two main features:
one-line dataloaders for many public datasets: one-liners to download and pre-process any of the major public datasets (image datasets, audio datasets, text datasets in 467 languages and dialects, 3D medical images, video datasets, agent traces, etc.) provided on the HuggingFace Datasets Hub. With a simple command like squad_dataset = load_dataset("rajpurkar/squad"), get any of these datasets ready to use in a dataloader for training/evaluating a ML model (Numpy/Pandas/PyTorch/TensorFlow/JAX/Polars),
efficient data pre-processing: simple, fast and reproducible data pre-processing for the public datasets as well as your own local datasets in CSV, JSON, JSONL, Parquet, HDF5, XML, text, PNG, JPEG, WAV, MP3, PDF, NIfTI, and more. With simple commands like processed_dataset = dataset.map(process_example), efficiently prepare the dataset for inspection and ML model evaluation and training.
🤗 Datasets is made to be very simple to use — the API is centered around a single function, datasets.load_dataset(dataset_name, **kwargs), that instantiates a dataset.
Here is a quick example:
from datasets import load_dataset
# Load a dataset and print the first example in the training set
squad_dataset = load_dataset('rajpurkar/squad')
print(squad_dataset['train'][0])
# Process the dataset - add a column with the length of the context texts
dataset_with_length = squad_dataset.map(lambda x: {"length": len(x["context"])})
# Tokenize the context texts (using a tokenizer from the 🤗 Transformers library)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
tokenized_dataset = squad_dataset.map(lambda x: tokenizer(x['context']), batched=True)
# Tokenize chat conversations with a chat template (using a model that supports chat templates)
# This is useful for fine-tuning instruction/chat models
# Load a popular chat dataset (ultrachat_200k contains ~200k AI assistant conversations)
chat_dataset = load_dataset('HuggingFaceH4/ultrachat_200k', split='train_sft')
chat_tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-7B-Instruct')
def tokenize_chat(examples):
# Apply the chat template and tokenize in one step
return chat_tokenizer.apply_chat_template(examples["messages"])
tokenized_chat_dataset = chat_dataset.map(tokenize_chat, batched=True)
Streaming mode
If your dataset is bigger than your disk or if you don't want to wait to download the data, you can use streaming:
# Stream the dataset without downloading anything
image_dataset = load_dataset('timm/imagenet-1k-wds', streaming=True)
for example in image_dataset["train"]:
print(example["image"])
break
Multi-modal data
🤗 Datasets supports a wide variety of data types out of the box:
# Load from local CSV
dataset = load_dataset('csv', data_files='my_data.csv')
# Load from local Parquet
dataset = load_dataset('parquet', data_files='data/*.parquet')
# Load from a local directory (auto-detect format)
dataset = load_dataset('./path/to/data')
From Python objects
from datasets import Dataset
# From a dictionary
dataset = Dataset.from_dict({"text": ["Hello world", "How are you?"]})
# From a list
dataset = Dataset.from_list([{"text": "Hello world"}, {"text": "How are you?"}])
# From Pandas
import pandas as pd
df = pd.DataFrame({"col1": [1, 2, 3], "col2": ["a", "b", "c"]})
dataset = Dataset.from_pandas(df)
# From a generator
def gen():
for i in range(10):
yield {"value": i}
dataset = Dataset.from_generator(gen)
For more details on using the library, check the quick start guide and the specific pages on:
You can use 🤗 Datasets to load datasets based on versioned git repositories maintained by the dataset authors. For reproducibility reasons, we ask users to pin the revision of the repositories they use.
If you're a dataset owner and wish to update any part of it (description, citation, license, etc.), or do not want your dataset to be included in the Hugging Face Hub, please get in touch by opening a discussion or a pull request in the Community tab of the dataset page. Thanks for your contribution to the ML community!
Contributing
We welcome contributions! Please see our Contributing Guide for details on:
If you want to cite our 🤗 Datasets library, you can use our paper:
@inproceedings{lhoest-etal-2021-datasets,
title = "Datasets: A Community Library for Natural Language Processing",
author = "Lhoest, Quentin and
Villanova del Moral, Albert and
Jernite, Yacine and
Thakur, Abhishek and
von Platen, Patrick and
Patil, Suraj and
Chaumond, Julien and
Drame, Mariama and
Plu, Julien and
Tunstall, Lewis and
Davison, Joe and
{\v{S}}a{\v{s}}ko, Mario and
Chhablani, Gunjan and
Malik, Bhavitvya and
Brandeis, Simon and
Le Scao, Teven and
Sanh, Victor and
Xu, Canwen and
Patry, Nicolas and
McMillan-Major, Angelina and
Schmid, Philipp and
Gugger, Sylvain and
Delangue, Cl{\'e}ment and
Matussi{\`e}re, Th{\'e}o and
Debut, Lysandre and
Bekman, Stas and
Cistac, Pierric and
Goehringer, Thibault and
Mustar, Victor and
Lagunas, Fran{\c{c}}ois and
Rush, Alexander and
Wolf, Thomas",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
month = nov,
year = "2021",
address = "Online and Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-demo.21",
pages = "175--184",
abstract = "The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets.",
eprint={2109.02846},
archivePrefix={arXiv},
primaryClass={cs.CL},
}
If you need to cite a specific version of our 🤗 Datasets library for reproducibility, you can use the corresponding version Zenodo DOI from this list.
datasets (huggingface/datasets)
Choose a tag and it will appear here. With no tags selected, all catalog projects are shown.
Searching Database…
Choose a tag and it will appear here. With no tags selected, all catalog projects are shown.
Hugging Face 出品的一个轻量级 Python 库，只需一行代码就能加载 HF Hub 上海量现成公共数据集，还能对 Hub 和本地数据做快速、可复现的预处理。