Ginkgo

Ginkgo is a scientific workflow orchestrator built for the 21st century.

• @flow and @task() — define workflows in plain Python, no DSL to learn
• An intuitive, aesthetic API built specifically for data science and bioinformatics
• natively dynamic workflows — expand workflows during runtime from resolved tasks
• content-addressed caching — never recompute what hasn't changed
• isolated environments — pixi or containers, per task
• agent-friendly — built from the ground up for workflows to be built and operated by AI agents
• cloud-native I/O — stream inputs directly from S3, GCS, or Azure and stage outputs back, without local copies
• deep observability — provenance and CLI tooling

See it in action

A bacterial GWAS, from 60 assemblies to resistance-associated unitigs: dynamic task graphs, pixi environments, caching, provenance, measured resource usage, and resuming after a failure. The workflow is in examples/bacterial_gwas.

Documentation

📖 sanjaynagi.github.io/ginkgo

It covers installation, quickstart, core concepts, environments, notebook tasks, caching, CLI usage, and a canonical example workflow.

Installation

Quick install (curl)

Install the ginkgo CLI in one line. Requires uv to be installed:

curl -LsSf https://raw.githubusercontent.com/sanjaynagi/ginkgo/main/install.sh | sh

This installs the ginkgo-workflow package from PyPI into an isolated environment via uv tool install. Re-run the same command to upgrade. The package is ginkgo-workflow; the command and the import are both ginkgo:

pip install ginkgo-workflow

Plugins

The installer gives you core plus ginkgo-pixi, which runs tasks' bare env="name" Pixi environments. Other optional integrations ship as separate plugins in this repository, each one an extra of ginkgo-workflow:

• ginkgo-aws: the s3:// object store and the mountpoint-s3 FUSE driver
• ginkgo-gcp: the gs:// object store, the gcsfuse FUSE driver and the GCP Batch executor
• ginkgo-oci: the oci:// object store and the rclone FUSE driver
• ginkgo-pixi: Pixi environments for tasks' env="name", and ginkgo env
• ginkgo-k8s: the Kubernetes executor
• ginkgo-notebook: @task("notebook"), which runs marimo notebooks
• ginkgo-report: the ginkgo report command, for HTML run reports
• ginkgo-slack: Slack run notifications

To install ginkgo with every first-party plugin:

uv tool install --force "ginkgo-workflow[all]"

Or name only the plugins you want:

uv tool install --force "ginkgo-workflow[pixi,k8s]"

ginkgo plugins lists what is installed. A remote_file() or remote_folder() URI needs the plugin for its scheme; without it, ginkgo names the plugin to install.

Pixi

For local development:

pixi install
pixi run test
pixi run typecheck

The default environment installs every first-party plugin editable, and pixi run test runs core's tests and the plugins' (tests/ and plugins/). To check that core works with no plugin installed, as CI does:

pixi run -e core test-core

After pixi install, build the uncoded symbol index used by AI coding tools (regenerated automatically by the pre-commit hook, but useful to seed up front):

pixi run uncoded sync

If your workflows use Pixi-backed task environments, pixi must also be available on PATH when you run them.

Run the CLI with either:

pixi run python -m ginkgo.cli --help

or:

ginkgo --help

Editable install

If you prefer a plain Python environment:

pip install -e .

Minimal Example

A population-genetics workflow that filters a VCF, computes per-population allele frequencies, and renders a summary marimo notebook (the notebook task needs the ginkgo-notebook plugin).

from pathlib import Path

import numpy as np

from ginkgo import file, flow, shell, task
from ginkgo_notebook import notebook

POPULATIONS = ["YRI", "CEU", "CHB"]
NOTEBOOKS_DIR = Path(__file__).resolve().parent / "notebooks"


# shell task — runs bcftools in a subprocess
@task("shell", env="genomics_tools")
def filter_snps(vcf_path: file, min_maf: float) -> file:
    """Filter to biallelic SNPs above a minor-allele-frequency threshold."""
    output = "results/filtered.vcf.gz"
    return shell(
        cmd=(
            f"bcftools view -m2 -M2 -v snps -i 'MAF>={min_maf}' "
            f"{vcf_path} -Oz -o {output} && bcftools index {output}"
        ),
        output=output,
    )


# python task — uses scikit-allel, fanned out per population via .map()
@task()
def allele_frequencies(vcf_path: file, population: str) -> file:
    """Compute per-SNP alt-allele frequencies for one population."""
    import allel

    callset = allel.read_vcf(str(vcf_path), fields=["calldata/GT"])
    ac = allel.GenotypeArray(callset["calldata/GT"]).count_alleles()
    freqs = ac.to_frequencies()[:, 1]  # alt allele frequency

    output = f"results/af_{population}.npy"
    np.save(output, freqs)
    return file(output)


# notebook task — runs a marimo notebook and renders it to HTML
@task("notebook")
def population_structure(af_files: list[file], populations: list[str]) -> file:
    """Render an HTML population-genetics summary notebook."""
    return notebook(NOTEBOOKS_DIR / "population_structure.py")


# flow
@flow
def main():
    filtered = filter_snps(vcf_path="data/chr22.vcf.gz", min_maf=0.05)
    af_results = allele_frequencies(vcf_path=filtered).map(population=POPULATIONS)
    return population_structure(af_files=af_results, populations=POPULATIONS)

Run it with:

ginkgo run flow.py

All three tasks above share one file, and a task's cache identity covers the whole module it is defined in — so editing any one of them invalidates the cached results of all three. That is what makes an edit to module-level state such as POPULATIONS invalidate the tasks that read it. Once a step is expensive enough that you mind re-running it, give it its own module; see the caching guide.

Canonical Example

The docs and examples are centered on examples/bioinfo, which demonstrates:

• Pixi-backed shell tasks
• a container-backed shell task
• .map() fan-out across samples
• a local Python aggregation task that returns a DataFrame

Run it with:

cd examples/bioinfo
ginkgo run

Core CLI Commands

To confirm a workflow is wired correctly, run ginkgo run --dry-run: it previews the plan for the entrypoint you actually run without executing any task body. Validation workflows a project keeps under tests/workflows/ are workflows too, so they run by path: ginkgo run tests/workflows/smoke.py. See the CLI guide.

• ginkgo run
• ginkgo doctor
• ginkgo debug
• ginkgo init
• ginkgo inspect workflow
• ginkgo runs (runs ls, runs show)
• ginkgo history
• ginkgo query
• ginkgo export (export events, export manifest)
• ginkgo report (from the ginkgo-report plugin)
• ginkgo why
• ginkgo secrets (secrets list, secrets validate)
• ginkgo cache ls
• ginkgo cache clear
• ginkgo cache prune
• ginkgo cache explain
• ginkgo env ls (from the ginkgo-pixi plugin)
• ginkgo db (db path, db check, db prune, db vacuum)

For full details on every command, see the CLI reference.

Repository Layout

ginkgo/
├── core/
├── runtime/
├── envs/
└── cli/

• core/ contains the user-facing DSL
• runtime/ contains evaluation, scheduling, caching, provenance, and value transport
• envs/ contains execution backends
• cli/ contains the ginkgo command-line interface

---

Ginkgo is licensed under the Apache License, Version 2.0. See LICENSE.