sqlframe

Why SQLFrame?

• Familiar API. Write transformations with PySpark DataFrame operations, joins, and window functions.
• Direct execution. Run your transformations on your database engine.
• Visible SQL. Inspect, debug, and share the queries behind your DataFrame code.

Try it locally

Install SQLFrame with DuckDB. You need Python 3.10 or later; no cloud account or Spark installation is required.

pip install "sqlframe[duckdb]"

from sqlframe.duckdb import DuckDBSession

session = DuckDBSession()
df = session.createDataFrame(
    [("US", 120), ("UK", 80), ("US", 30)],
    ["country", "revenue"],
)
df.groupBy("country").sum("revenue").orderBy("country").show()

+---------+--------------+
| country | sum(revenue) |
+---------+--------------+
|    UK   |      80      |
|    US   |     150      |
+---------+--------------+

To inspect a transformation's SQL, call df.sql(optimize=True). Follow the quickstart for a complete walkthrough.

Choose your engine

Engine	Install
DuckDB	pip install "sqlframe[duckdb]"
BigQuery	pip install "sqlframe[bigquery]"
Snowflake	pip install "sqlframe[snowflake]"
Postgres	pip install "sqlframe[postgres]"
ClickHouse (preview)	pip install "sqlframe[clickhouse]"
Databricks	pip install "sqlframe[databricks]"
Spark	pip install "sqlframe[spark]"
Standalone	pip install sqlframe
Trino (preview)	pip install "sqlframe[trino]"

Standalone sessions generate SQL without a database connection. For local Parquet support, install sqlframe[standalone].

ClickHouse and Trino are preview integrations; see their guides for supported workflows and limits.

Community integrations: GizmoSQL · Redshift (pip install "sqlframe[redshift]").

SQLFrame is also available on conda-forge: conda install -c conda-forge sqlframe.

Already using PySpark?

Activate SQLFrame before importing PySpark to run supported transformations on another engine.

from sqlframe import activate

activate("duckdb")

from pyspark.sql import SparkSession

session = SparkSession.builder.getOrCreate()

You can also import SQLFrame directly, as shown in the quickstart. See the activation guide and your engine's API compatibility reference for details.

Keep exploring

• Examples: longer pipelines with aggregations and window functions.
• Generated SQL: formatting, optimization, dialect overrides, and OpenAI enrichment.
• Input and output dialects: configure SQLFrame for your engine's conventions.
• Standalone SQL generation: generate queries without connecting to an engine.

SQLFrame is open source under the MIT license.