DataChain: The Context Layer for Unstructured Data

A Python library that turns files in S3, GCS, and Azure into versioned, typed datasets, queryable at warehouse speed.

• Compute Engine: parallel Python over files, distributed on Studio. Async I/O, checkpoint recovery, incremental updates.
• Dataset DB: Pydantic schemas, versioning, file pointers, automatic lineage. Sub-second filter, join, and group_by over millions of typed records locally, hundreds of millions on Studio. Vector search over the same rows, no separate store.

Optional, for agent workflows:

• Knowledge Base: markdown summaries derived from the Dataset DB and enriched by LLM. Readable by humans and LLMs.
• Agent Harness: a skill that plugs all three into Claude Code, Cursor, Codex, GitHub Copilot, and Pi, so they understand your data. On Studio, agents reach the same datasets over MCP.

Bytes never leave your storage. Every run deposits a typed dataset the next pipeline (or agent) reads instead of recomputing.

1. Install

pip install datachain

To add the agent skill for data processing and saved-result queries:

datachain skill install --target claude     # also: cursor, codex, copilot, pi

Works with S3, GCS, Azure, and local filesystems.

2. Quickstart: agent-driven pipeline

Task: find dogs in S3 similar to a reference image, filtered by breed, mask availability, and image dimensions.

Grab a reference image and run Claude Code (or other agent):

datachain cp --anon s3://dc-readme/fiona.jpg .

claude

Prompt:

Find dogs in s3://dc-readme/oxford-pets-micro/ similar to ./fiona.jpg:
  - Pull breed metadata and mask files from annotations/
  - Exclude images without mask
  - Exclude Cocker Spaniels
  - Only include images wider than 400px

Result:

  ┌──────┬───────────────────────────────────┬────────────────────────────┬──────────┐
  │ Rank │               Image               │           Breed            │ Distance │
  ├──────┼───────────────────────────────────┼────────────────────────────┼──────────┤
  │    1 │ shiba_inu_52.jpg                  │ shiba_inu                  │    0.244 │
  ├──────┼───────────────────────────────────┼────────────────────────────┼──────────┤
  │    2 │ shiba_inu_53.jpg                  │ shiba_inu                  │    0.323 │
  ├──────┼───────────────────────────────────┼────────────────────────────┼──────────┤
  │    3 │ great_pyrenees_17.jpg             │ great_pyrenees             │    0.325 │
  └──────┴───────────────────────────────────┴────────────────────────────┴──────────┘

  Fiona's closest matches are shiba inus (both top spots), which makes sense given her
  tan coloring and pointed ears.

The agent decomposed the task into steps - embeddings, breed metadata, mask join, quality filter - and saved each as a named, versioned dataset. Next time you ask a related question, it starts from what's already built.

For shared dataset descriptions and search, connect Studio MCP. Local agents inspect saved datasets directly.

3. Data Harness

Code harnesses (Claude Code, Cursor, Codex, GitHub Copilot, Pi) give agents repo context, dedicated tools, and memory across sessions. DataChain adds the same for data: typed datasets the agent reads, chain operations the agent calls (read_storage, map, save), a Dataset DB where its results persist.

A dataset is the unit of work - a named, versioned result of a pipeline step like [email protected]. Every .save() registers one.

For the data-flow architecture (Compute Engine, Dataset DB, Knowledge Base) and how the components connect, see Architecture.

4. Core concepts

4.1. Dataset

A dataset is a versioned data reasoning step - what was computed, from what input, producing what schema. DataChain indexes your storage into one: no data copied, just typed metadata and file pointers. Re-runs only process new or changed files.

Create a dataset manually create_dataset.py:

from PIL import Image
import io
from pydantic import BaseModel
import datachain as dc


class ImageInfo(BaseModel):
    width: int
    height: int


def get_info(file: dc.File) -> ImageInfo:
    img = Image.open(io.BytesIO(file.read()))
    return ImageInfo(width=img.width, height=img.height)


ds = (
    dc.read_storage(
        "s3://dc-readme/oxford-pets-micro/images/**/*.jpg",
        anon=True,
        update=True,
        delta=True,  # re-runs skip unchanged files
    )
    .settings(prefetch=64)
    .map(info=get_info)
    .save("pets_images")
)
ds.show(5)

[email protected] is now the shared reference to this data - schema, version, lineage, and metadata.

Every .save() registers the dataset in the Dataset DB, DataChain's persistent store for schemas, versions, lineage, and processing state, kept locally in SQLite DB .datachain/db. Pipelines reference datasets by name, not paths. When the code or input data changes, the next run bumps dataset version.

This is what makes a dataset a management unit: owned, versioned, and queryable by everyone on the team.

4.2. Schemas and types

DataChain uses Pydantic to define the shape of every column. The return type of your UDF becomes the dataset schema - each field a queryable column in the Dataset DB.

This is the beginning of the README. Read the rest at the original. Read the full README ↗