Flowfile

---

Build pipelines on a visual canvas with a live preview at every node, or write them in Python with a Polars-like API — code and visual are two views of the same graph. Around it: a Delta-backed catalog, a SQL editor, a scheduler, sandboxed Python kernels, and sharing for teams.

• Just want to see it? demo.flowfile.org.
• Transforming files on your laptop? pip install Flowfile (Python 3.10–3.13).
• Running it for a team? Docker Compose, which adds accounts, groups and a shared catalog. See the Quick Start.

 

---

What's in Flowfile

Canvas and code

A visual canvas with 49 node types — joins, fuzzy matching, bill-of-materials explosion, filters, pivots, aggregations, text-to-rows, window functions, conditional branching. Beyond the nodes, the formula editor brings 95 transformation functions, and a Polars code node gives you full Polars for anything the palette doesn't cover — all running in-process, no external engine.

It connects to local files (CSV, Parquet, Excel, JSON and friends), five databases (PostgreSQL, MySQL, SQL Server, SQLite, DuckDB), cloud storage (S3, ADLS, GCS — Delta included, Iceberg read-only), Kafka (consumer only), Google Analytics, and REST APIs. That's the whole list — no Snowflake, BigQuery or Oracle driver yet, and no CDC; if your data lives there, land it somewhere Flowfile can reach first.

A Python API with Polars-like syntax — write a pipeline, call open_graph_in_editor(), and it opens as an editable flow. Porting an existing Polars script over is mostly mechanical.

 

An AI assistant. Tell it what you want and it builds the flow with you, on the canvas. Bring your own key — Anthropic, OpenAI, Google, Groq, OpenRouter — or point it at Ollama or a local model.

 

Code generation. Prototype visually, ship a plain script: flows export as Python, and save as human-readable YAML so version control works. A flow of standard transforms on local files exports as pure Polars, usually with import polars as pl as its only import.

Fuzzy matching, graph solving, hierarchy explosion, and formulas that don't translate to a native Polars expression pull in the helper packages Flowfile itself is built on (pl-fuzzy-frame-match, polars-grouper, polars-expr-transformer). They're normal pip installs and don't drag Flowfile along.

Database and REST nodes export as flowfile calls, so their stored connections and secrets resolve at run time. The platform nodes (catalog, cloud storage, Kafka, ML) export against the FlowFrame API rather than raw Polars. And a few nodes (Google Analytics, SQL query, API response) have no code generation yet.

 

Around the canvas

Everything a flow produces can land in the data catalog — a catalog > schema > table hierarchy on Delta Lake, so tables get version history and time travel. Flow outputs can also register as virtual tables: Flowfile stores the Polars query plan instead of the data, and a consumer's filters push down straight through the flow boundary.

There's a SQL editor on top (Polars SQLContext): query any registered table, chart the result in the embedded Graphic Walker, and save a useful ad-hoc query as a flow in one click.

 

Flows can run on a schedule — on an interval, or when catalog tables update — with run history and logs in the UI. Any node setting takes ${variable} parameters, overridable at run time with --param.

The Docker deployment makes it multi-user: accounts, groups, and sharing of connections, flows and catalog namespaces at "use" or "manage" level. The desktop app is single-user.

Python kernels. User code runs in isolated Docker containers with their own package environments (so Docker needs to be running locally), with a Jupyter-style notebook editor — cell execution, autocompletion, rich output.

Custom nodes. Build your own in the visual Node Designer and share them through the community registry — publishing opens a pull request straight from the app; installing is one click, with sha256-pinned downloads and a consent dialog.

An embeddable editor. The browser editor also ships as a standalone Vue component, flowfile-editor: a Polars-powered canvas in any web app, zero backend.

Templates and clipboard import. Start from built-in templates, or paste data from Excel / Google Sheets straight onto the canvas.

---

Quick Start

In the browser — demo.flowfile.org runs a 23-node subset on Pyodide; good for a first look and small files.

On your laptop (Python 3.10–3.13):

pip install Flowfile
flowfile run ui

Or start from code:

import flowfile as ff
from flowfile import col, open_graph_in_editor

df = ff.from_dict({
    "id": [1, 2, 3, 4, 5],
    "category": ["A", "B", "A", "C", "B"],
    "value": [100, 200, 150, 300, 250]
})

result = (
    df.filter(col("value") > 150)
      .with_columns((col("value") * 2).alias("double_value"))
      .group_by("category")
      .agg(col("value").sum().alias("total"))
)

open_graph_in_editor(result.flow_graph)

For a team — the Docker stack runs core, worker, and the web UI with user accounts and a shared catalog:

git clone https://github.com/edwardvaneechoud/Flowfile.git
cd Flowfile
docker compose up -d   # UI at http://localhost:8080

The compose file in this repo builds from source. To run the published Docker Hub images on a server, behind HTTPS, use flowfile-hosting.

Desktop app — installers for Windows, macOS, and Linux on the Releases page.

This is the beginning of the README. Read the rest at the original. Read the full README ↗