🤗 Datasets is a lightweight library providing two main features:

• one-line dataloaders for many public datasets: one-liners to download and pre-process any of the major public datasets (image datasets, audio datasets, text datasets in 467 languages and dialects, 3D medical images, video datasets, agent traces, etc.) provided on the HuggingFace Datasets Hub. With a simple command like squad_dataset = load_dataset("rajpurkar/squad"), get any of these datasets ready to use in a dataloader for training/evaluating a ML model (Numpy/Pandas/PyTorch/TensorFlow/JAX/Polars),
• efficient data pre-processing: simple, fast and reproducible data pre-processing for the public datasets as well as your own local datasets in CSV, JSON, JSONL, Parquet, HDF5, XML, text, PNG, JPEG, WAV, MP3, PDF, NIfTI, and more. With simple commands like processed_dataset = dataset.map(process_example), efficiently prepare the dataset for inspection and ML model evaluation and training.

🎓 Documentation 🔎 Find a dataset in the Hub 🌟 Share a dataset on the Hub

🚀 Key Features

🤗 Datasets is designed to let the community easily add and share new datasets, and provides powerful capabilities for data manipulation:

Feature	Description
📦 One-line dataset loading	Load AI-ready datasets from the Hugging Face Hub or local files with load_dataset()
🔍 Multiple formats	Native support for CSV, JSON, JSONL, Parquet, Arrow, XML, Text, Webdataset, and more
🖼️ Multi-modal data	Built-in support for text, audio, image, video, PDF, and NIfTI (3D medical) data
🚀 Streaming mode	Stream datasets without downloading — iterate over data on-the-fly with streaming=True (now up to 100x faster with Xet backend)
💾 HF Storage Buckets	Read and write directly from/to Hugging Face Storage Buckets for mutable, large-scale raw data
🧠 AI Agent Traces	Load and process AI agent traces (prompts, tool calls, responses) from the Hub
⚡ Apache Arrow backend	Zero-copy memory-mapped storage — datasets naturally free you from RAM limitations
🔄 Smart caching	Never wait for your data to process twice — cached results are automatically reused
📊 Multi-framework interoperability	Native conversion to/from NumPy, Pandas, Polars, Arrow, PyTorch, TensorFlow, JAX, and Spark
🏎️ Multi-processing	Fast parallel data processing with map(num_proc=N)
🔎 Search & index	Built-in FAISS and Elasticsearch index support for similarity search
📦 JSON type	Flexible JSON/structured data support with Json() feature type

Installation

With pip

🤗 Datasets can be installed from PyPi and should be installed in a virtual environment (venv or conda for instance):

pip install datasets

For the latest development version:

pip install "datasets @ git+https://github.com/huggingface/datasets.git"

With conda

conda install -c huggingface -c conda-forge datasets

Optional dependencies

🤗 Datasets supports various optional features via extras:

# For audio (torchcodec)
pip install datasets[audio]

# For image/video (Pillow, torchcodec)
pip install datasets[vision]

# For PDFs/NIfTI (pdfplumber, nibabel)
pip install datasets[pdfs,nibabel]

# For PyTorch/TensorFlow/JAX integration
pip install datasets[torch,tensorflow,jax]

For more details on installation, check the installation page.

Quick Start

🤗 Datasets is made to be very simple to use — the API is centered around a single function, datasets.load_dataset(dataset_name, **kwargs), that instantiates a dataset.

Here is a quick example:

from datasets import load_dataset

# Load a dataset and print the first example in the training set
squad_dataset = load_dataset('rajpurkar/squad')
print(squad_dataset['train'][0])

# Process the dataset - add a column with the length of the context texts
dataset_with_length = squad_dataset.map(lambda x: {"length": len(x["context"])})

# Tokenize the context texts (using a tokenizer from the 🤗 Transformers library)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')

tokenized_dataset = squad_dataset.map(lambda x: tokenizer(x['context']), batched=True)

# Tokenize chat conversations with a chat template (using a model that supports chat templates)
# This is useful for fine-tuning instruction/chat models

# Load a popular chat dataset (ultrachat_200k contains ~200k AI assistant conversations)
chat_dataset = load_dataset('HuggingFaceH4/ultrachat_200k', split='train_sft')

chat_tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-7B-Instruct')

def tokenize_chat(examples):
    # Apply the chat template and tokenize in one step
    return chat_tokenizer.apply_chat_template(examples["messages"])

tokenized_chat_dataset = chat_dataset.map(tokenize_chat, batched=True)

Streaming mode

If your dataset is bigger than your disk or if you don't want to wait to download the data, you can use streaming:

# Stream the dataset without downloading anything
image_dataset = load_dataset('timm/imagenet-1k-wds', streaming=True)
for example in image_dataset["train"]:
    print(example["image"])
    break

Multi-modal data

🤗 Datasets supports a wide variety of data types out of the box:

This is the beginning of the README. Read the rest at the original. Read the full README ↗