soda-core

Soda Core is a data quality and data contract verification engine. It lets you define data quality contracts in YAML and automatically validate both schema and data across your data stack.

Soda Core provides the Soda Command-Line Interface (CLI) and Python API, which you can use to generate, test, publish, and verify contracts. These operations can be executed locally during development, embedded programmatically within your data pipelines (Airflow, Dagster, Prefect, etc.), or executed remotely when connected to Soda Cloud.

Highlights

• Define data contracts using a clean, human-readable YAML syntax
• Run checks on PostgreSQL, Snowflake, BigQuery, Databricks, DuckDB, and more
• Use 50+ built-in data quality checks for common and advanced validations
• Integrate with Soda Cloud for centralized management and anomaly detection monitoring

Setup

This repository hosts the open source Soda Core packages which are installable using the Public PyPI installation flow described in Soda's documentation.

Requirements

To use Soda, you must have installed the following on your system.

• Python 3.9, 3.10, 3.11, or 3.12 To check your existing version, use the CLI command: python --version or python3 --version. If you have not already installed Python, consider using pyenv to manage multiple versions of Python in your environment. Note: While Python 3.12 is the highest officially supported version, there are no known issues preventing use of Python 3.13+.

• UV (recommended) or Pip 21.0 or greater We recommend using UV for faster and more reliable package management. To install UV, see the UV installation guide. Alternatively, you can use pip (version 21.0+). To check your pip version: pip --version

Installation

Soda Core v4 open source packages are available on public PyPI and have the form soda-{data source}.

Using UV (recommended)

uv pip install soda-postgres  # install latest version 4 package

Using pip

pip install soda-postgres  # install latest version 4 package

Replace soda-postgres with the appropriate package for your data source. For a list of supported data sources, packages, and configurations, see the data source reference for Soda Core.

Working with legacy Soda Core v3

Soda package names have changed with the release of version 4. Legacy version 3 open source packages have the form soda-core-{data source}. For example, to install Soda Core v3 for Postgres, pinning the version at 3.5.x:

uv pip install soda-core-postgres~=3.5.0   # install legacy version 3 package (using UV)
# or
pip install soda-core-postgres~=3.5.0      # install legacy version 3 package (using pip)

For a list of supported data sources and other details, see the v3 documentation within this repository. For information about Soda Core v3, see the v3 README file and the Soda v3 online documentation.

Development

Prerequisites

• Python 3.10+
• UV (recommended) or pip 21.0+

Setup (UV — recommended)

Clone the repo and install all workspace packages with dev dependencies:

uv sync --all-packages --group dev

Setup (pip — alternative)

If you don't have UV, you can still set up a dev environment with pip:

python -m venv .venv
source .venv/bin/activate  # or .venv\Scripts\activate on Windows
pip install -e soda-core -e soda-tests -e soda-postgres  # add other packages as needed
pip install pytest pre-commit pydantic python-dotenv freezegun

Running tests

uv run pytest soda-tests/
# or, if using pip:
pytest soda-tests/

Pre-commit checks

uv run pre-commit run --all-files
# or, if using pip:
pre-commit run --all-files

Quickstart

The examples show minimal configurations. For more detailed examples and explanations, see the Soda Cloud documentation.

To see detailed logs, add --verbose or -v to any of these commands.

Configure a data source

To define a local configuration for your data source and validate the connection, run the following commands.

Create data source config

soda data-source create -f ds_config.yml

Parameter	Required	Description
-f, --file	Yes	Output file path for the data source YAML configuration file.

By default, the YAML file generated as ds_config.yml is a template for PostgreSQL connections. To learn how to populate a data source configuration file, see the data source reference for Soda Core.

Test data source config

soda data-source test -ds ds_config.yml

Parameter	Required	Description
-ds, --data-source	Yes	Path to a data source YAML configuration file.

Create a contract

Create and populate new contract YAML file. To understand how to write a contract, see the online documentation, or the example below, which is configured to test a table or view with qualified name db.schema.dataset within a data source named postgres_ds. This table is assumed to have columns named id, name, and size. The data source name postgres_ds must match the name property in the data source configuration file.

This is the beginning of the README. Read the rest at the original. Read the full README ↗