Why SQLFrame?
- Familiar API. Write transformations with PySpark DataFrame operations, joins, and window functions.
- Direct execution. Run your transformations on your database engine.
- Visible SQL. Inspect, debug, and share the queries behind your DataFrame code.
Try it locally
Install SQLFrame with DuckDB. You need Python 3.10 or later; no cloud account or Spark installation is required.
pip install "sqlframe[duckdb]"
from sqlframe.duckdb import DuckDBSession
session = DuckDBSession()
df = session.createDataFrame(
[("US", 120), ("UK", 80), ("US", 30)],
["country", "revenue"],
)
df.groupBy("country").sum("revenue").orderBy("country").show()
+---------+--------------+
| country | sum(revenue) |
+---------+--------------+
| UK | 80 |
| US | 150 |
+---------+--------------+
To inspect a transformation's SQL, call df.sql(optimize=True). Follow the quickstart for a complete walkthrough.
Choose your engine
Standalone sessions generate SQL without a database connection. For local Parquet support, install sqlframe[standalone].
ClickHouse and Trino are preview integrations; see their guides for supported workflows and limits.
Community integrations: GizmoSQL · Redshift (pip install "sqlframe[redshift]").
SQLFrame is also available on conda-forge: conda install -c conda-forge sqlframe.
Already using PySpark?
Activate SQLFrame before importing PySpark to run supported transformations on another engine.
from sqlframe import activate
activate("duckdb")
from pyspark.sql import SparkSession
session = SparkSession.builder.getOrCreate()
You can also import SQLFrame directly, as shown in the quickstart. See the activation guide and your engine's API compatibility reference for details.
Keep exploring
SQLFrame is open source under the MIT license.
