DataTrove
DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt commonly used processing blocks with a framework to easily add custom functionality.
DataTrove processing pipelines are platform-agnostic, running out of the box locally or on a slurm cluster. Its (relatively) low memory usage and multiple step design makes it ideal for large workloads, such as to process an LLM's training data.
Local, remote and other file systems are supported through fsspec.
Table of contents
Installation
Requires Python 3.11+.
uv sync
Available flavours (combine them by repeating --extra, e.g. uv sync --extra processing --extra s3):
all installs everything: uv sync --extra allio dependencies to read warc/arc/wet files and arrow/parquet/Optimized-parquet formats: uv sync --extra ioprocessing dependencies for text extraction, filtering and tokenization: uv sync --extra processings3 s3 support: uv sync --extra s3cli for command line tools: uv sync --extra cliray for distributed compute engine: uv sync --extra rayinference for LLM inference pipelines: uv sync --extra inferencedecont for decontamination with lighteval: uv sync --extra decontmultilingual for multilingual text processing: uv sync --extra multilingual
Quickstart examples
You can check the following examples:
- fineweb.py full reproduction of the FineWeb dataset
- process_common_crawl_dump.py full pipeline to read commoncrawl warc files, extract their text content, filters and save the resulting data to s3. Runs on slurm
- tokenize_c4.py reads data directly from huggingface's hub to tokenize the english portion of the C4 dataset using the
gpt2 tokenizer
- estimate_tokens.py estimate total token counts for large HF datasets — needed to set the correct
SamplerFilter rate when creating a random shuffled subsample (e.g. 100B tokens from a multi-trillion-token dataset). Streams a small sample per dataset, converges on the average tokens/doc, and multiplies by the total row count.
- smol_data.py builds ~100B token subsets, 50-30-20 mixtures, and shuffled variants for several large Hugging Face datasets
- minhash_deduplication.py full pipeline to run minhash deduplication of text data
- sentence_deduplication.py example to run sentence level exact deduplication
- exact_substrings.py example to run ExactSubstr (requires this repo)
- finephrase.py standalone example to generate a synthetic dataset at scale with multiple prompt templates
Terminology
pipeline: a list of processing steps to execute (read data, filter, write to disk, etc)executor: runs a specific pipeline on a given execution environment (slurm, multi cpu machine, etc)job: the execution of a pipeline on a given executortask: a job is comprised of multiple tasks, and these are used to parallelize execution, usually by having each task process a shard of data. Datatrove keeps track of which tasks have completed and when you relaunch only incomplete tasks will run.file: an individual input file (.json, .csv, etc).
[!TIP]
Note that each file will be processed by a single task. Datatrove does not automatically split a file into multiple parts, so to fully parallelize you should have multiple medium sized files rather than a single large file)
shard: a group of input data (usually a group of files), which will be assigned to a specific task. Each task will process a different non overlapping shard of data, from the full list of input filesworker: compute resource that will execute a single task at a time, e.g., if you have 50 cpu cores you can run a LocalPipelineExecutor with workers=50, to execute 50 tasks simultaneously (one per cpu). Once a worker is done with a task, it will start processing another waiting task
This is the beginning of the README. Read the rest at the original. Read the full README ↗
