Apache Auron

Apache Auron is an accelerator for big data engines, leveraging native vectorized execution to accelerate query processing. It combines the power of the Apache DataFusion library and the scale of the distributed computing framework.

Auron takes a fully optimized physical plan from a distributed computing framework, mapping it into DataFusion's execution plan, and performs native plan computation.

The key capabilities of Auron include:

• Native execution: Implemented in Rust, eliminating JVM overhead and enabling predictable performance.
• Vectorized computation: Built on Apache Arrow's columnar format, fully leveraging SIMD instructions for batch processing.
• Pluggable architecture: Seamlessly integrates with Apache Spark while designed for future extensibility to other engines.
• Production-hardened optimizations: Multi-level memory management, compacted shuffle formats, and adaptive execution strategies developed through large-scale deployment.

Based on the inherent well-defined extensibility of DataFusion, Auron can be easily extended to support:

• Various object stores.
• Operators.
• Simple and Aggregate functions.
• File formats.

We encourage you to extend DataFusion capability directly and add support in Auron with simple modifications in plan-serde and extension translation.

Build from source

To build Auron from source, follow the steps below:

1. Install Rust

Auron's native execution lib is written in Rust. You need to install Rust (nightly) before compiling.

We recommend using rustup for installation.

2. Install JDK

Auron is regularly tested on JDK 8, 11, and 17, and is also tested on JDK 21.

Make sure JAVA_HOME is properly set and points to your desired version.

3. Check out the source code.

4. Build the project.

You can build Auron either locally or inside Docker using one of the supported OS images via the unified script: auron-build.sh.

Run ./auron-build.sh --help to see all available options.

After the build completes, a fat JAR with all dependencies will be generated in either the target/ directory (for local builds) or target-docker/ directory (for Docker builds), depending on the selected build mode.

Run Spark Job with Auron Accelerator

This section describes how to submit and configure a Spark Job with Auron support.

1. Move the Auron JAR to the Spark client classpath (normally spark-xx.xx.xx/jars/).

2. Add the following configs to spark configuration in spark-xx.xx.xx/conf/spark-default.conf:

spark.auron.enable true
spark.sql.extensions org.apache.spark.sql.auron.AuronSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.auron.shuffle.AuronShuffleManager
spark.memory.offHeap.enabled false

# suggested executor memory configuration
spark.executor.memory 4g
spark.executor.memoryOverhead 4096

3. submit a query with spark-sql, or other tools like spark-thriftserver:

spark-sql -f tpcds/q01.sql

For ORC files with large String/Binary values, set spark.auron.orc.batchSize to a smaller positive row count (for example, --conf spark.auron.orc.batchSize=1024) when starting the application. It defaults to spark.auron.batchSize and only controls native ORC decoding; downstream operators may coalesce these batches. Smaller batches reduce the risk of offset overflow but cannot accommodate an individual value that exceeds Arrow's offset limit.

Native conversion diagnostics in the Spark UI

After configuring Auron as described above, enable the Spark UI and Auron tab in conf/spark-defaults.conf before starting the application. This file accepts whitespace-separated keys and values:

spark.ui.enabled true
spark.auron.ui.enabled true

Alternatively, add --conf spark.ui.enabled=true and --conf spark.auron.ui.enabled=true to your spark-submit or spark-shell command. The --conf option requires the key=value format.

The Auron UI option defaults to true. Open the application's Spark UI and select the Auron tab. For direct access, the URL is http[s]://<spark-ui-host>:<spark-ui-port>/auron/. Use the application's actual Spark UI address from its startup output or cluster manager. If the UI is accessed through a proxy, open the provided Spark UI link and select the Auron tab. Keep the application running while inspecting its live UI.

Alongside build information, the tab provides a paginated SQL execution list. Select a SQL ID to inspect conversion snapshots, per-operator fallback reasons, physical plans, selected configuration values captured at conversion time, and available Spark SQL operator metrics. A link opens the corresponding Spark SQL execution page for jobs and stages.

Basic concepts

Term	Meaning
SQL execution / SQL ID	A Spark SQL execution tracked by Spark and identified by its execution ID. It can originate from SQL or DataFrame/Dataset operations and can involve multiple Spark jobs and stages.
Physical plan / operator	A physical plan describes how Spark executes an operation. Its nodes are operators, such as scans, filters, joins, and aggregations, or wrappers around other nodes.
Native conversion	Auron converts supported Spark operators into operators that use its native execution engine. A plan can contain both native and Spark operators.
Fallback	An operator remains in Spark, with a recorded reason explaining why native conversion was not used. This does not by itself mean that the SQL execution failed.
Conversion snapshot	A diagnostic record captured after a plan conversion, associated with a SQL execution ID. It contains a capture timestamp, physical plan text, per-node conversion statuses and recorded fallback reasons, and selected configuration values.
Adaptive Query Execution (AQE)	Spark's mechanism for adapting an execution plan using runtime information. It can lead to additional plan conversions and snapshots during the same SQL execution.

A snapshot describes the plan at a particular conversion point. It contains no query result data and is not a checkpoint for recovering execution. One SQL execution can have multiple snapshots, and a snapshot can cover an individual stage rather than the entire query. The Snapshots column reports the number of retained diagnostic snapshots, not the number of jobs, stages, or query runs.

The operator metrics shown on the detail page are retrieved separately from Spark SQL's status store. They are not a copy of metric values at the time each conversion snapshot was captured.

Interpreting conversion results

Status	Meaning
Native	The operator was converted to native execution.
Fallback	The operator remains in Spark with a recorded reason, such as disabled conversion or an unsupported operator or expression.
Other Spark	No fallback reason was recorded for the Spark operator in this snapshot; this does not imply successful native conversion.
Wrapper	A planning or execution wrapper, excluded from native and fallback counts. Inspect its child operators for conversion results.

AQE may convert individual stages separately. The execution list shows counts from the latest snapshot, not totals across the entire query. Up to 20 recent snapshots are retained per execution, with consecutive identical snapshots collapsed. Shared plan objects are counted once per snapshot, and diagnostic node IDs are local to each snapshot rather than Spark SQL plan-graph IDs.

Conversion snapshots require Auron to be enabled and a SQL execution ID to be available during conversion. Executions without snapshots show Not available instead of zero conversion counts. Finished indicates that execution ended; it does not distinguish success from failure.

Metrics include timing, row counts, memory, spill, and shuffle information when reported by the operators. Values retain Spark's units and aggregation semantics and are not summed across operators. Metrics depend on Spark SQL's own retention and may be unavailable even when a diagnostic snapshot is retained.

Retention and historical applications

Diagnostic records use spark.sql.ui.retainedExecutions for cleanup, preserving active executions. Stored physical plan text is limited to 65,536 characters and individual fallback reasons to 8,192 characters. These limits bound stored text, not the cost of generating the plan string. Plan text and fallback reasons follow Spark's configured string redaction pattern; configuration capture uses an allowlist of diagnostic settings.

For historical applications, enable spark.eventLog.enabled=true before running the application and configure Spark's event-log storage. Make the matching Auron JAR available on the Spark History Server classpath so its bundled history plugin can replay diagnostic events. Event logs created without those events cannot reconstruct conversion snapshots.

Performance

TPC-DS 1TB Benchmark Results:

For methodology and additional results, please refer to benchmark documentation.

We also encourage you to benchmark Auron and share the results with us. 🤗

Community

Subscribe Mailing Lists

Mail List is the most recognized form of communication in the Apache community. Contact us through the following mailing list.

Name	Scope
[email protected]	Development-related discussions	Subscribe	Unsubscribe

Contributing

Interested in contributing to Auron? Please read our Contributing Guide for detailed information on how to get started.

License

Auron is licensed under the Apache 2.0 License. A copy of the license can be found here.