pipectl

---

What is pipectl?

pipectl is a CLI for running YAML-defined data pipelines. You define an ordered list of steps in a YAML file, pipe your data in, and get transformed data out.

It supports JSON, JSONL, and CSV payloads — including conversions between them — and ships with built-in steps for filtering, normalising, redacting, casting, sorting, validating, and more.

Quick example

# customer-intake.yaml
id: customer-intake
input:
  format: csv
steps:
  - normalize:
      fields:
        first_name: capitalize
        last_name: capitalize
        email: lower
  - filter:
      field: country
      equals: AU
  - redact:
      fields: [credit_card]
      strategy: mask
  - select:
      fields: [first_name, last_name, email, plan]
output:
  format: jsonl

pipectl run customer-intake.yaml < customers.csv

{"first_name":"Alice","last_name":"Smith","email":"[email protected]","plan":"pro"}
{"first_name":"Bob","last_name":"Jones","email":"[email protected]","plan":"free"}

Features

• Built-in steps — filter, normalize, redact, cast, sort, dedupe, validate, convert, and more
• Multiple payload formats — JSON, JSONL, and CSV with automatic conversion between them
• Composable pipelines — chain any number of steps; output of one feeds the next
• JSON Schema validation — validate records against a schema at any point in the pipeline
• HTTP transforms — POST/PUT your payload to an HTTP endpoint and continue with the response
• Dry-run mode — validate and preview a pipeline without reading stdin or executing steps
• Minimal dependencies — a single binary with no runtime requirements

Install

macOS (Homebrew)

brew tap pipectl/pipectl
brew install --cask pipectl

Windows

Direct download (no package manager required): download pipectl_<version>_windows_amd64.zip from the Releases page, extract, and add the folder to your PATH.

Scoop (for developers who have Scoop installed):

scoop bucket add pipectl https://github.com/pipectl/scoop-pipectl
scoop install pipectl

Linux

Download a .deb or .rpm from the Releases page, or use the .tar.gz archive on any Linux distribution.

Debian/Ubuntu:

sudo dpkg -i pipectl_<version>_linux_amd64.deb

Fedora/RHEL:

sudo rpm -i pipectl_<version>_linux_amd64.rpm

Docker

docker pull ghcr.io/pipectl/pipectl:latest

Run a pipeline with stdin:

echo '[...]' | docker run --rm -i ghcr.io/pipectl/pipectl:latest run pipeline.yaml

Run with local files mounted:

docker run --rm -i \
  -v $(pwd):/data \
  ghcr.io/pipectl/pipectl:latest run /data/pipeline.yaml --input /data/input.json

Direct download

Pre-built binaries for all platforms are on the Releases page. Verify with the included checksums.txt:

sha256sum --check checksums.txt

Go install (from source)

go install github.com/pipectl/pipectl/cmd/pipectl@latest

Usage

pipectl init                      # interactively generate a starter pipeline YAML

pipectl run <pipeline.yaml> [flags]

Flags:
  -i, --input <path>    Read pipeline input from file (alternative to stdin)
  -o, --output <path>   Write output to file instead of stdout
  -v, --verbose         Enable verbose logging
  -q, --quiet           Suppress all diagnostic output
      --dry-run         Validate pipeline and print plan without executing
      --timing          Print per-step timing table to stderr after execution
      --var KEY=VALUE   Substitute ${VAR} tokens in pipeline YAML (repeatable)
      --max-input-size SIZE  Maximum size of stdin/--input data before failing (default 256MB)
                              Warns to stderr if raised too high relative to system RAM

pipectl validate <pipeline.yaml>
pipectl docs [step]

Core concepts

Concept	Description
Pipeline	A YAML file with an id, input format, ordered steps, and output format
Step	A single operation that reads the payload, transforms it, and passes it on
Payload	The data flowing through the pipeline — a JSON object/array, JSONL stream, or CSV
Defaults	Optional pipeline-level values (e.g. HTTP proxy/headers/timeout, text case-sensitivity) inherited by matching steps unless overridden

Steps

Step	What it does
assert	Assert record count, field existence, or field value
cast	Convert field types — int, float, bool, string, time
convert	Convert payload format — json ↔ jsonl ↔ csv
count	Print current record count to stderr
dedupe	Remove duplicate records by key fields
default	Fill missing or empty fields with a default value
filter	Keep records matching a condition or nested all/any group
http-request	Send payload to an HTTP endpoint as a side effect, continue with the same payload
http-transform	POST/PUT/PATCH payload to an HTTP endpoint, continue with the response
limit	Truncate to N records
log	Print current record count and samples to stderr
normalize	Normalise string fields (lower, upper, trim, capitalize, collapse-spaces)
redact	Replace field values (including nested fields) with mask, sha256, partial-last[:N], partial-first[:N], nullify, or REDACTED
rename	Rename fields
select	Keep only the specified fields
sort	Sort records by one or more fields, ascending or descending
validate-json	Validate records against a JSON Schema

Documentation

Full documentation, step reference, and example pipelines are at pipectl.github.io/pipectl.

The examples/ directory contains ready-to-run pipelines with sample data.

Contributing

The project is still establishing its foundation and is not actively seeking contributions at this stage. Once things stabilise, contributions will be welcome — check back then or watch the repo for updates.

Releasing

To publish a new release, create and push a semver tag:

git tag v1.2.3
git push origin v1.2.3

s This triggers the release.yml GitHub Actions workflow, which runs GoReleaser to build binaries for Linux, macOS, and Windows (amd64 and arm64), publish .deb/.rpm packages, push Docker images to GHCR, and update the Homebrew tap and Scoop bucket.

License

MIT — see LICENSE.