karpenter-provider-azure

Table of contents:

• Features Overview
• Node Auto Provisioning (NAP) vs. Self-hosted
• Node memory reservations
• Known limitations
• Installation (self-hosted)
  • Install utilities
  • Create a cluster
  • Configure Helm chart values
  • Install Karpenter
• Using Karpenter (self-hosted)
  • Create NodePool
  • Scale up deployment
  • Scale down deployment
  • Maintain spare capacity
  • Delete Karpenter nodes manually
• Cleanup (self-hosted)
  • Delete the cluster
• Source Attribution
• Community, discussion, contribution, and support

Features Overview

The AKS Karpenter Provider enables node autoprovisioning using Karpenter on your AKS cluster. Karpenter improves the efficiency and cost of running workloads on Kubernetes clusters by:

• Watching for pods that the Kubernetes scheduler has marked as unschedulable
• Evaluating scheduling constraints (resource requests, node selectors, affinities, tolerations, and topology-spread constraints) requested by the pods
• Provisioning nodes that meet the requirements of the pods
• Removing the nodes when they are no longer needed
• Consolidating existing nodes onto cheaper nodes with higher utilization per node

Node Auto Provisioning (NAP) vs. Self-hosted Karpenter

Karpenter provider for AKS can be used in two modes:

• Node Auto Provisioning (NAP) mode: Karpenter is run by AKS as a managed addon similar to managed Cluster Autoscaler. This is the recommended mode for most users as it has more test coverage, improved scale-up speed, automatic maintenance window integration, support for some additional SKUs, and various other improvements over self-hosted Karpenter. NAP also leverages Node Provisioning Service for optimized performance and scalability. Follow the instructions in Node Auto Provisioning documentation to use Karpenter in that mode. NAP manages:
  • Token rotation
  • Helm charts
  • Karpenter version updates
  • VM OS disk updates
  • node image upgrades (Linux)
• Self-hosted mode: Karpenter is run as a standalone deployment in the cluster. This mode is useful for advanced users who want to customize or experiment with Karpenter's deployment, use custom Helm charts, or integrate non-standard workflows. Self-hosted mode requires users to directly manage upgrades, token rotation, and helm charts.

Node memory reservations

All provisioning modes (aksscriptless, bootstrappingclient, aksmachineapi, and aksmachineapiheaderbatch) use the same memory reservation policy for non-hardened nodes: min(20 * maxPods + 50, totalMemoryMiB / 4) MiB of kube-reserved memory and a 100 MiB hard-eviction threshold. totalMemoryMiB is the nominal SKU memory in MiB. This policy applies without a Kubernetes version gate and follows the AKS node resource reservations documentation.

Scriptless provisioning uses these values both for scheduling estimates and for the generated kubelet configuration on newly created nodes. Other provisioning modes retain their AKS-owned bootstrap defaults; the provider's estimates align with those defaults. Existing nodes are not reconfigured by this change.

Node-hardening reservations, CPU and system reservations, storage thresholds, and the separate VM_MEMORY_OVERHEAD_PERCENT safety margin remain unchanged.

Known limitations

The following AKS features are not supported:

• Windows nodes require the AKS Machine API provision mode. This mode is used by Node Auto Provisioning and can also be used with self-hosted Karpenter; the scriptless and bootstrapping-client provision modes do not support Windows. Windows2025 requires Kubernetes 1.32 or newer and AKS Windows2025 availability for the subscription and region.
• Kubenet and Calico.
• IPv6 clusters.
• Service Principal based clusters. A system-assigned or user-assigned managed identity must be used.
• Clusters running Karpenter should not be stopped.
• All cluster egress outbound types are supported, however the type can't be changed after the cluster is created.

Installation (NAP)

Follow the instructions in the Node Auto Provisioning documentation.

Installation (self-hosted Karpenter)

⚠️ Warning ⚠️ We strongly recommend using Node Auto Provisioning (aka managed Karpenter) instead of self-hosted. Official support via Microsoft support channels is limited to NAP. Self-hosted Karpenter is supported primarily via GitHub issues, and subject to best-effort response time.

This guide shows how to get started with Karpenter by creating an AKS cluster and installing Karpenter.

Install utilities

Install these tools before proceeding:

• Azure CLI
• kubectl
• Helm
• jq - used by some of the scripts below
• yq - used by some of the scripts below

Create a cluster

Create a new AKS cluster with the required configuration, and ready to run Karpenter using workload identity.

Note: You can use hack/deploy/create-cluster.sh <cluster-name> <resource-group> <namespace> to automate the following steps.

Set environment variables:

export CLUSTER_NAME=karpenter
export RG=karpenter
export LOCATION=westus3
export KARPENTER_NAMESPACE=kube-system

Login and select a subscription to use:

az login

Create the resource group:

az group create --name ${RG} --location ${LOCATION}

Create the workload MSI that backs the karpenter pod auth:

KMSI_JSON=$(az identity create --name karpentermsi --resource-group "${RG}" --location "${LOCATION}" --output json)

Create the AKS cluster compatible with Karpenter, with workload identity enabled:

AKS_JSON=$(az aks create \
  --name "${CLUSTER_NAME}" --resource-group "${RG}" \
  --node-count 3 --generate-ssh-keys \
  --network-plugin azure --network-plugin-mode overlay --network-dataplane cilium \
  --enable-managed-identity \
  --enable-oidc-issuer --enable-workload-identity \
  --output json)
az aks get-credentials --name "${CLUSTER_NAME}" --resource-group "${RG}" --overwrite-existing

Create federated credential linked to the karpenter service account for auth usage:

az identity federated-credential create --name KARPENTER_FID --identity-name karpentermsi --resource-group "${RG}" \
  --issuer "$(jq -r ".oidcIssuerProfile.issuerUrl" <<< "$AKS_JSON")" \
  --subject system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter-sa \
  --audience api://AzureADTokenExchange

Create role assignments to let Karpenter manage VMs and Network resources:

KARPENTER_USER_ASSIGNED_CLIENT_ID=$(jq -r '.principalId' <<< "$KMSI_JSON")
RG_MC=$(jq -r ".nodeResourceGroup" <<< "$AKS_JSON")
RG_MC_RES=$(az group show --name "${RG_MC}" --query "id" --output tsv)
for role in "Virtual Machine Contributor" "Network Contributor" "Managed Identity Operator"; do
  az role assignment create --assignee "${KARPENTER_USER_ASSIGNED_CLIENT_ID}" --scope "${RG_MC_RES}" --role "$role"
done

Note: If you experience any issues creating the role assignments, but should have the given ownership to do so, try going through the Azure portal:

1. Navigate to your MSI.
2. Give it the following roles "Virtual Machine Contributor", "Network Contributor", and "Managed Identity Operator" at the scope of the node resource group.

Configure Helm chart values

The Karpenter Helm chart requires specific configuration values to work with an AKS cluster. While these values are documented within the Helm chart, you can use the configure-values.sh script to generate the karpenter-values.yaml file with the necessary configuration. This script queries the AKS cluster and creates karpenter-values.yaml using karpenter-values-template.yaml as the configuration template. Although the script automatically fetches the template from the main branch, inconsistencies may arise between the installed version of Karpenter and the repository code. Therefore, it is advisable to download the specific version of the template before running the script.

# Select version to install
export KARPENTER_VERSION=1.10.2

# Download the template for the selected version
curl -sO https://raw.githubusercontent.com/Azure/karpenter-provider-azure/v${KARPENTER_VERSION}/karpenter-values-template.yaml

# use configure-values.sh to generate karpenter-values.yaml
# (in repo you can just do ./hack/deploy/configure-values.sh ${CLUSTER_NAME} ${RG})
curl -sO https://raw.githubusercontent.com/Azure/karpenter-provider-azure/v${KARPENTER_VERSION}/hack/deploy/configure-values.sh
chmod +x ./configure-values.sh && ./configure-values.sh ${CLUSTER_NAME} ${RG} karpenter-sa karpentermsi

Install Karpenter

Using the generated karpenter-values.yaml file, install Karpenter using Helm:

helm upgrade --install karpenter oci://mcr.microsoft.com/aks/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" \
  --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
  --values karpenter-values.yaml \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

Check karpenter deployed successfully:

kubectl get pods --namespace "${KARPENTER_NAMESPACE}" -l app.kubernetes.io/name=karpenter

Check its logs:

kubectl logs -f -n "${KARPENTER_NAMESPACE}" -l app.kubernetes.io/name=karpenter -c controller

Note: Snapshot versions can be installed in a similar way for development:

export KARPENTER_NAMESPACE=kube-system
export KARPENTER_VERSION=0-f83fadf2c99ffc2b7429cb40a316fcefc0c4752a

helm upgrade --install karpenter oci://ksnap.azurecr.io/karpenter/snapshot/karpenter \
  --version "${KARPENTER_VERSION}" \
  --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
  --values karpenter-values.yaml \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

Using Karpenter (self-hosted)

Create NodePool

A single Karpenter NodePool is capable of handling many different pod shapes. Karpenter makes scheduling and provisioning decisions based on pod attributes such as labels and affinity. In other words, Karpenter eliminates the need to manage many different node groups.

Create a default NodePool using the command below. (Additional examples available in the repository under examples/v1.) The consolidationPolicy set to WhenUnderutilized in the disruption block configures Karpenter to reduce cluster cost by removing and replacing nodes. As a result, consolidation will terminate any empty nodes on the cluster. This behavior can be disabled by setting consolidateAfter to Never, telling Karpenter that it should never consolidate nodes.

Note: This NodePool will create capacity as long as the sum of all created capacity is less than the specified limit.

cat <<EOF | kubectl apply -f -
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
  annotations:
    kubernetes.io/description: "General purpose NodePool"
spec:
  template:
    metadata:
      labels:
        # required for Karpenter to predict overhead from cilium DaemonSet
        kubernetes.azure.com/ebpf-dataplane: cilium
    spec:
      nodeClassRef:
        group: karpenter.azure.com
        kind: AKSNodeClass
        name: default
      startupTaints:
        # https://karpenter.sh/docs/concepts/nodepools/#cilium-startup-taint
        - key: node.cilium.io/agent-not-ready
          effect: NoExecute
          value: "true"
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: karpenter.azure.com/sku-family
          operator: In
          values: [D]
        # Optional: constrain Azure placement to zonal capacity.
        # Use "regional" instead to request non-zonal capacity, labeled with topology.kubernetes.io/zone=0.
        # - key: karpenter.azure.com/placement-scope
        #   operator: In
        #   values: ["zonal"]
      expireAfter: Never
  limits:
    cpu: 100
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 0s
---
apiVersion: karpenter.azure.com/v1beta1
kind: AKSNodeClass
metadata:
  name: default
  annotations:
    kubernetes.io/description: "General purpose AKSNodeClass for running Ubuntu nodes"
spec:
  imageFamily: Ubuntu
EOF

Karpenter is now active and ready to begin provisioning nodes.

By default, Azure NodePools can use both zonal and regional VM placement when the selected SKU supports it. Add a karpenter.azure.com/placement-scope requirement with zonal or regional to choose one explicitly; regional nodes are represented with topology.kubernetes.io/zone=0.

Scale up deployment

This deployment uses the pause image and starts with zero replicas.

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: mcr.microsoft.com/oss/kubernetes/pause:3.6
          resources:
            requests:
              cpu: 1
EOF

kubectl scale deployment inflate --replicas 5
kubectl logs -f -n "${KARPENTER_NAMESPACE}" -l app.kubernetes.io/name=karpenter -c controller

Scale down deployment

Now, delete the deployment. After a short amount of time, Karpenter should terminate the empty nodes due to consolidation.

kubectl delete deployment inflate
kubectl logs -f -n "${KARPENTER_NAMESPACE}" -l app.kubernetes.io/name=karpenter -c controller

Maintain spare capacity

The alpha CapacityBuffer feature can maintain schedulable headroom for workload spikes. It is disabled by default for self-hosted installations and has explicit cost, ownership, and workload-shape limits. See the CapacityBuffer guide and bounded example.

Delete Karpenter nodes manually

If you delete a node with kubectl, Karpenter will gracefully cordon, drain, and shutdown the corresponding instance. Under the hood, Karpenter adds a finalizer to the node object, which blocks deletion until all pods are drained and the instance is terminated. Keep in mind, this only works for nodes provisioned by Karpenter.

kubectl delete node $NODE_NAME

Cleanup (self-hosted)

Delete the cluster

To avoid additional charges, remove the demo infrastructure from your AKS account.

helm uninstall karpenter --namespace "${KARPENTER_NAMESPACE}"
az aks delete --name "${CLUSTER_NAME}" --resource-group "${RG}"

---

Source Attribution

Notice: Files in this source code originated from a fork of https://github.com/aws/karpenter which is under an Apache 2.0 license. Those files have been modified to reflect environmental requirements in AKS and Azure.

Many thanks to @ellistarn, @jonathan-innis, @tzneal, @bwagner5, @njtran, and many other developers active in the Karpenter community for laying the foundations of a Karpenter provider ecosystem!

Many thanks to @Bryce-Soghigian, @rakechill, @charliedmcb, @jackfrancis, @comtalyst, @aagusuab, @matthchr, @gandhipr, @dtzar for contributing to AKS Karpenter Provider!

---

Community, discussion, contribution, and support

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Come discuss Karpenter in the #karpenter channel in the Kubernetes slack!

Check out the Docs to learn more.

Check out our contributing guide.

This open-source project is subject to best-effort support by project maintainers and not covered by Microsoft support channels. File an issue in this project for any questions, feature requests, or issues.