Cluster API Runtime Extensions - Nutanix (CAREN)
CAREN is a Cluster API Runtime Extension. It patches
ClusterClass-managed clusters during topology mutation, and it installs cluster
addons from lifecycle hooks. The cluster resource is the place to choose those
options, including generic Kubernetes settings such as audit policy and HTTP
proxy, and essential addons such as CNI.
Supported providers
| Provider | Role |
|---|
| Nutanix | Clusters on Nutanix infrastructure |
| AWS | Self-managed clusters on AWS |
| EKS | Managed control plane with worker configuration |
| Docker | Local development and tests |
| Generic | Provider-independent Kubernetes settings |
Documentation
User docs: https://nutanix-cloud-native.github.io/cluster-api-runtime-extensions-nutanix/.
Install with clusterctl
or Helm.
Community
Development
Implementing Topology Mutation Handler
See examples of existing topology mutation handlers in the pkg/handlers/../mutation/ directory.
When adding a new handler, or modifying an existing one, pay close attention to what happens to existing clusters
when a new version of this extension is deployed in the management cluster,
and avoid rollouts of Machines in those existing clusters.
If a change to a handler is unavoidable, you must:
- Update the version of the handlers, e.g. from
v4 to v5.
This is done by updating the ..ClusterV4ConfigPatch in the handler files in pkg/handlers/../mutation/.
And ..clusterv4configpatch in hack/examples/overlays/clusterclasses/../kustomization.yaml.tmpl
- Update the version of the handler in the
pkg/handlers/v3 package, e.g. from v3 to v4.
- Copy the existing implementation of the handler to
pkg/handlers/../v4/.
During CAPI provider upgrades, and periodically, all managed clusters are reconciled and mutation handler patches
are applied.
Any new handlers that return a new set of patches, or updated handlers that return a different set of patches,
will be applied causing a rollout of Machines in all managed clusters.
For example, when adding a new handler, a handler that is enabled by default and returns CAPI resources patches,
will cause a rollout of Machines.
Similarly, if a handler is modified to return a different set of patches, it will also cause a rollout of Machines.
Run Locally
Install tools
To deploy a local build, either initial install to update an existing deployment, run:
make dev.run-on-kind
eval $(make kind.kubeconfig)
Pro-tip: to redeploy without rebuilding the binaries, images, etc (useful if you have only changed the Helm chart for
example), run:
make SKIP_BUILD=true dev.run-on-kind
You can just update the image in the webhook Deployment on an existing KIND cluster:
make KIND_CLUSTER_NAME=<> dev.update-webhook-image-on-kind
Generate a cluster definition from the file specified in the --from flag
and apply the generated resource to actually create the cluster in the API.
For example, the following command will create a Docker cluster with Cilium CNI applied via the Helm addon provider:
export CLUSTER_NAME=docker-cluster-cilium-helm-addon
export CLUSTER_FILE=examples/capi-quick-start/docker-cluster-cilium-helm-addon.yaml
export KUBERNETES_VERSION=v1.35.3
clusterctl generate cluster ${CLUSTER_NAME} \
--from ${CLUSTER_FILE} \
--kubernetes-version ${KUBERNETES_VERSION} \
--worker-machine-count 1 | \
kubectl apply --server-side -f -
Wait until control plane is ready:
kubectl wait clusters/${CLUSTER_NAME} --for=condition=ControlPlaneInitialized --timeout=5m
To get the kubeconfig for the new cluster, run:
clusterctl get kubeconfig ${CLUSTER_NAME} > ${CLUSTER_NAME}.conf
If you are not on Linux, you will also need to fix the generated kubeconfig's server, run:
kubectl config set-cluster ${CLUSTER_NAME} \
--kubeconfig ${CLUSTER_NAME}.conf \
--server=https://$(docker container port ${CLUSTER_NAME}-lb 6443/tcp)
Wait until all nodes are ready (this indicates that CNI has been deployed successfully):
kubectl --kubeconfig ${CLUSTER_NAME}.conf wait nodes --all --for=condition=Ready --timeout=5m
Show that Cilium is running successfully on the workload cluster:
kubectl --kubeconfig ${CLUSTER_NAME}.conf get daemonsets -n kube-system cilium
Deploy kube-vip to provide service load-balancer functionality for Docker clusters:
helm repo add --force-update kube-vip https://kube-vip.github.io/helm-charts
helm repo update
kind_subnet_prefix="$(docker network inspect kind -f '{{ (index .IPAM.Config 0).Subnet }}' | \
grep -o '^[[:digit:]]\+\.[[:digit:]]\+\.')"
kubectl create configmap \
--namespace kube-system kubevip \
--from-literal "range-global=${kind_subnet_prefix}100.0-${kind_subnet_prefix}100.20" \
--dry-run=client -oyaml |
kubectl --kubeconfig ${CLUSTER_NAME}.conf apply --server-side -n kube-system -f -
helm upgrade kube-vip-cloud-provider kube-vip/kube-vip-cloud-provider --version 0.2.2 \
--install \
--wait --wait-for-jobs \
--namespace kube-system \
--kubeconfig ${CLUSTER_NAME}.conf \
--set-string=image.tag=v0.0.6
helm upgrade kube-vip kube-vip/kube-vip --version 0.4.2 \
--install \
--wait --wait-for-jobs \
--namespace kube-system \
--kubeconfig ${CLUSTER_NAME}.conf \
--set-string=image.tag=v0.6.0
Deploy traefik as a LB service:
helm --kubeconfig ${CLUSTER_NAME}.conf repo add traefik https://helm.traefik.io/traefik
helm repo update &>/dev/null
helm --kubeconfig ${CLUSTER_NAME}.conf upgrade --install traefik traefik/traefik \
--version v10.9.1 \
--wait --wait-for-jobs \
--set ports.web.hostPort=80 \
--set ports.websecure.hostPort=443 \
--set service.type=LoadBalancer
Watch for traefik LB service to get an external address:
watch -n 0.5 kubectl --kubeconfig ${CLUSTER_NAME}.conf get service/traefik
To delete the workload cluster, run:
kubectl delete cluster ${CLUSTER_NAME}
Notice that the traefik service is deleted before the cluster is actually finally deleted.
Check the pod logs:
kubectl logs deployment/cluster-api-runtime-extensions-nutanix -f
To delete the dev KinD cluster, run:
make kind.delete
