Kubeflow Spark Operator

What is Spark Operator?

The Kubernetes Operator for Apache Spark aims to make specifying and running Spark applications as easy and idiomatic as running other workloads on Kubernetes. It uses Kubernetes custom resources for specifying, running, and surfacing status of Spark applications.

Quick Start

For a more detailed guide, please refer to the Getting Started guide.

Installation with Helm

# Add the Helm repository
helm repo add --force-update spark-operator https://kubeflow.github.io/spark-operator

# Install the operator into the spark-operator namespace and wait for deployments to be ready
helm install spark-operator spark-operator/spark-operator \
    --namespace spark-operator \
    --create-namespace \
    --wait

# Create an example application in the default namespace
kubectl apply -f https://raw.githubusercontent.com/kubeflow/spark-operator/refs/heads/master/examples/spark-pi.yaml

# Get the status of the application
kubectl get sparkapp spark-pi

# Delete the application
kubectl delete sparkapp spark-pi

Installation with Kustomize

An alternative install method is using kustomize manifests. For more information go to the kustomize installation docs.

git clone https://github.com/kubeflow/spark-operator.git && cd spark-operator
kubectl apply -k config/default --server-side --force-conflicts

# If you use kustomize to install, also install Spark application RBAC in the namespace where you will run Spark applications. 
# The example below runs in the default namespace and uses the spark-operator-spark service account.
kubectl -n default apply -k config/spark-rbac

# Create an example application in the default namespace
kubectl apply -f https://raw.githubusercontent.com/kubeflow/spark-operator/refs/heads/master/examples/spark-pi.yaml

# Get the status of the application
kubectl get sparkapp spark-pi

# Delete the application
kubectl delete sparkapp spark-pi

Overview

For a complete reference of the custom resource definitions, please refer to the API Definition. For details on its design, please refer to the Architecture. It requires Spark 2.3 and above that supports Kubernetes as a native scheduler backend.

The Kubernetes Operator for Apache Spark currently supports the following list of features:

• Supports Spark 2.3 and up.
• Enables declarative application specification and management of applications through custom resources.
• Automatically runs spark-submit on behalf of users for each SparkApplication eligible for submission.
• Provides native cron support for running scheduled applications.
• Supports customization of Spark pods beyond what Spark natively is able to do through the mutating admission webhook, e.g., mounting ConfigMaps and volumes, and setting pod affinity/anti-affinity.
• Supports automatic application re-submission for updated SparkApplication objects with updated specification.
• Supports automatic application restart with a configurable restart policy.
• Supports automatic retries of failed submissions with optional linear back-off.
• Supports collecting and exporting application-level metrics and driver/executor metrics to Prometheus.

Project Status

Project status: beta

Current API version: v1beta2

If you are currently using the v1beta1 version of the APIs in your manifests, please update them to use the v1beta2 version by changing apiVersion: "sparkoperator.k8s.io/<version>" to apiVersion: "sparkoperator.k8s.io/v1beta2". You will also need to delete the previous version of the CustomResourceDefinitions named sparkapplications.sparkoperator.k8s.io and scheduledsparkapplications.sparkoperator.k8s.io, and replace them with the v1beta2 version either by installing the latest version of the operator or by running kubectl create -f config/crd/bases.

Prerequisites

• Kubernetes >= 1.16
• Helm >= 3
• kubectl >= 1.16

Getting Started

For getting started with Spark operator, please refer to Getting Started.

User Guide

For detailed user guide and API documentation, please refer to User Guide and API Specification.

If you plan to run Spark workloads with Spark operator on Alibaba Cloud Container Service for Kubernetes (ACK), also refer to the ACK guide for optimized configuration practices.

If you are running Spark operator on Google Kubernetes Engine (GKE) and want to use Google Cloud Storage (GCS) and/or BigQuery for reading/writing data, also refer to the GCP guide.

Blog Posts

• [2025-03] Announcing the Kubeflow Spark Operator Benchmarking Results
• [2024-04] Announcing the Kubeflow Spark Operator: Building a Stronger Spark on Kubernetes Community

Version Matrix

The following table lists the most recent few versions of the operator.

Operator Version	API Version	Kubernetes Version	Base Spark Version
v2.3.x	v1beta2	1.16+	4.0.0
v2.2.x	v1beta2	1.16+	3.5.5
v2.1.x	v1beta2	1.16+	3.5.3
v2.0.x	v1beta2	1.16+	3.5.2
v1beta2-1.6.x-3.5.0	v1beta2	1.16+	3.5.0
v1beta2-1.5.x-3.5.0	v1beta2	1.16+	3.5.0
v1beta2-1.4.x-3.5.0	v1beta2	1.16+	3.5.0
v1beta2-1.3.x-3.1.1	v1beta2	1.16+	3.1.1
v1beta2-1.2.3-3.1.1	v1beta2	1.13+	3.1.1
v1beta2-1.2.2-3.0.0	v1beta2	1.13+	3.0.0
v1beta2-1.2.1-3.0.0	v1beta2	1.13+	3.0.0
v1beta2-1.2.0-3.0.0	v1beta2	1.13+	3.0.0
v1beta2-1.1.x-2.4.5	v1beta2	1.13+	2.4.5
v1beta2-1.0.x-2.4.4	v1beta2	1.13+	2.4.4

Developer Guide

For developing with Spark Operator, please refer to Developer Guide.

Contributor Guide

For contributing to Spark Operator, please refer to Contributor Guide.

Community

• Join the CNCF Slack Channel and then join #kubeflow-spark-on-kubernetes Channel.
• Check out our blog post Announcing the Kubeflow Spark Operator: Building a Stronger Spark on Kubernetes Community.
• Join our monthly community meeting Kubeflow Spark Operator Meeting Notes.

Adopters

Check out adopters of Spark Operator.