Cereyan

Cereyan is a minimal, local-first orchestrator for Python data pipelines. A Rust core (SQLite store, state machine, scheduler, HTTP server) sits behind a thin layer of Python decorators, and the whole thing ships as one wheel with no runtime dependencies. Decorate functions as flows and tasks, run them as plain scripts, and every run is recorded locally; cereyan serve adds live monitoring, one queue with processors you size while it runs, schedules (including continuous loops), retries, backfills, rules, and a built-in MCP server for agents on top of the same store. When one machine is not enough, cereyan worker on another machine adds its processors to the same queue.

pip install cereyan

# pipeline.py
from datetime import date
from cereyan import flow, task, get_run_logger

@task
def extract(day: date) -> list[int]:
    get_run_logger().info("extracting %s", day)
    return [1, 2, 3]

@task
def load(rows: list[int]) -> int:
    return sum(rows)

@flow(run_name="etl-{day}")
def etl(day: date = date(2026, 9, 6)) -> int:
    return load(extract(day))

if __name__ == "__main__":
    print(etl())

python pipeline.py                                    # records a run in ~/.cereyan/db.sqlite
cereyan run pipeline.py:etl --param day=2026-01-02    # same, with parameters and a summary
cereyan serve . --token $TOKEN                        # API, UI, scheduler, engines at http://127.0.0.1:4200
cereyan worker . --host https://cereyan.internal      # on another machine: its processors join the queue

• Offline first. A script records runs into a local SQLite file; nothing else needs to run.
• One process to serve. The API, the web UI, the scheduler, one queue, a warm pool of engine processes, rules, and MCP, in one cereyan serve. The Queue page shows every run in line and why it waits, and lets you add or remove processors while the server runs.
• Workers when you need them. cereyan worker runs on another machine with its own checkout and connects out to the server: when the server's processors are full, runs spill over to workers whose code matches. The store, results and secrets stay on the server, every hand-off carries a lease, the Workers tab shows each machine's status, host and schedule, and each worker serves a status page of its own.
• Data-pipeline semantics. Targets make reruns idempotent, backfills cover date ranges or listed values, resources are named semaphores that can be keyed by parameter, flows chain and fan in by key, unique keys stop duplicate runs, continuous schedules run a flow again a set time after each run ends, and rules react to events or to their absence.
• Durable runs. Every completed task is checkpointed, so a crash rerun or a retry from failure resumes after the last finished task; a run can sleep, wait for an event, a target, or a message sent through the API, or snooze without holding an engine; and a task keeps notes that survive its retries.

Screenshots

Every page below is described in the tour; click a shot to jump to it.

Runs — scope by project and group, filter, act in bulk	Run detail — logs, tasks and retries
Flows — groups, schedules, skips, dependencies, history	Timeline — the task graph of a run
Events — what happened, as it happens	Rules — react to events, or to their absence

The UI follows your system theme; the documentation shows the dark variants too.

Documentation

The site is at https://sercanatalik.github.io/cereyan/ (built from docs/ with just docs):

• Quickstart: from install to a scheduled, retried, backfilled pipeline in ten minutes.
• Concepts: the model behind flows, runs, states, schedules, targets, resources, backfills, dependencies, events, rules, artifacts, and variables.
• Guides: one goal per page, from retries to running the server as a service, running across machines, and using cereyan with an AI agent.
• Reference: the Python API, CLI, HTTP API, MCP tools, events, states, and configuration.
• Design and limitations: what cereyan does not do, and why.
• For agents: llms.txt and llms-full.txt.
• Changelog.

Status

Production ready, and published to PyPI. The pieces described here work and are tested on macOS, Linux and Windows, and the interfaces — names, signatures, defaults, routes, event payloads and the database schema — are settled. Windows lacks the Unix socket and engine niceness, and cannot interrupt a flow blocked in most I/O; Install lists what that changes. Read the changelog before upgrading: 3.0 changes the engine protocol (servers and workers of 2.x and 3.0 do not mix), migrates the store, and starts a server with one processor instead of one per CPU. Issues and questions are welcome.

Development

Rust stable, Python 3.11 or newer with uv, Node 22, and just. just ui builds the UI, just dev builds the extension in place, just test and just lint run everything, just demo serves the examples. Details, the repository layout, and the release checklist are in the contributing guide. Design decisions are on the Design pages, and changes not yet released are under Unreleased in CHANGELOG.md.

Credits

Cereyan owes its shape to two projects that came first.

Luigi contributed the idea the data-pipeline semantics rest on: a task declares the target it produces, and work is skipped when that target already exists. Target and LocalTarget keep Luigi's names because they are Luigi's concept, and idempotent reruns and backfills follow from it.

Prefect contributed the authoring model — flows and tasks as decorated functions, runs carrying explicit states, a server watching them — and the look of the UI. Six UI components were adapted from Prefect's, rewritten in React from the Vue originals

Neither is a dependency, and cereyan deliberately does far less than either: one queue, one process, one wheel, no database to run, and workers that are plain processes on machines you already have rather than a platform of pools and images. If you need what they do, use them. Migrating from Prefect or Luigi says what carries over and what does not.

License

MIT, see LICENSE.