OpenShift Wrapper

This plugin is a very opinionated OpenShift wrapper designed to simplify the execution of different workloads in this Kubernetes distribution.

Executed with kube-burner-ocp, it looks like:

$ kube-burner-ocp --help
kube-burner plugin designed to be used with OpenShift clusters as a quick way to run well-known workloads

Usage:
  kube-burner-ocp [command]

Available Commands:
  anp-density-pods           Runs anp-density-pods workload
  berserker-load             Runs berserker-load workload
  build-farm                 Runs build-farm workload
  cluster-density-ms         Runs cluster-density-ms workload
  cluster-density-v2         Runs cluster-density-v2 workload
  cluster-health             Checks for ocp cluster health
  completion                 Generate the autocompletion script for the specified shell
  crd-scale                  Runs crd-scale workload
  cudn-density               Runs cudn-density workload with tiered cross-namespace communication
  dv-clone                   Runs dv-clone workload
  egressip                   Runs egressip workload
  evpn                       Runs evpn workload
  help                       Help about any command
  index                      Runs index sub-command
  init                       Runs custom workload
  kueue-operator-jobs        Runs kueue-operator-jobs workload
  kueue-operator-jobs-shared Runs kueue-operator-jobs-shared workload
  kueue-operator-pods        Runs kueue-operator-pods workload
  maas-gateway-perf          Runs maas-gateway-perf workload
  network-policy             Runs network-policy workload
  node-density               Runs node-density workload
  node-density-cni           Runs node-density-cni workload
  node-density-heavy         Runs node-density-heavy workload
  node-scale                 Runs node-scale workload
  olm                        Runs olm workload
  pvc-density                Runs pvc-density workload
  rds-core                   Runs rds-core workload
  udn-bgp                    Runs udn-bgp workload
  udn-density-pods           Runs udn-density-pods workload
  version                    Print the version number of kube-burner
  virt-capacity-benchmark    Runs capacity-benchmark workload
  virt-clone                 Runs virt-clone workload
  virt-clone-multi           Runs virt-clone-multi workload
  virt-density               Runs virt-density workload
  virt-ephemeral-restart     Runs virt-ephemeral-restart workload
  virt-migration             Runs virt-migration workload
  virt-parallel              Runs virt-parallel workload
  virt-udn-density           Runs virt-udn-density workload
  web-burner-cluster-density Runs web-burner-cluster-density workload
  web-burner-init            Runs web-burner-init workload
  web-burner-node-density    Runs web-burner-node-density workload
  whereabouts                Runs whereabouts workload

Flags:
      --alerting                  Enable alerting (default true)
      --burst int                 Burst (default 20)
      --enable-file-logging       Enable file logging (default true)
      --es-index string           Elastic Search index
      --es-server string          Elastic Search endpoint
      --extract                   Extract workload in the current directory
      --gc                        Garbage collect created resources (default true)
      --gc-metrics                Collect metrics during garbage collection
  -h, --help                      help for kube-burner-ocp
      --ignore-health-check       Run cluster health check, but ignore failures
      --local-indexing            Enable local indexing
      --log-level string          Allowed values: debug, info, warn, error, fatal (default "info")
      --metrics-endpoint string   YAML file with a list of metric endpoints, overrides the es-server and es-index flags
      --profile-type string       Metrics profile to use, supported options are: regular, reporting or both (default "both")
      --prometheus-token string   Prometheus bearer token to use with --prometheus-url
      --prometheus-url string     Prometheus endpoint URL, overrides OpenShift Prometheus discovery
      --qps int                   QPS (default 20)
      --set strings               Set arbitrary key=value pairs to override values in the config file
      --timeout duration          Benchmark timeout (default 4h0m0s)
      --user-metadata string      User provided metadata file, in YAML format
      --uuid string               Benchmark UUID (default "46d08f73-1b12-4137-8e00-67d9c7f069b7")

Documentation

Documentation is available here

Quick start

Install latest stable version with:

curl -Ls https://raw.githubusercontent.com/kube-burner/kube-burner-ocp/refs/heads/main/hack/install.sh | sh

[!NOTE] Default installation path is ${HOME}/.local/bin/, you can change it by setting INSTALL_DIR environment variable before running the script

Usage

Some of the benefits the OCP wrapper provides are:

• Simplified execution of the supported workloads. (Only some flags are required)
• Adds OpenShift metadata to generated jobSummary and a small subset of metadata fields to the remaining metrics.
• Prevents modifying configuration files to tweak some of the parameters of the workloads.
• Discovers the OpenShift Prometheus URL and authentication token, or accepts an explicit Prometheus endpoint for environments where discovery is not available.
• Workloads configuration is directly embedded in the binary.

Running node-density with 100 pods per node

kube-burner-ocp node-density --pods-per-node=100

With the command above, the wrapper will calculate the required number of pods to deploy across all worker nodes of the cluster.

Workload Flags Metadata

All workload-specific flags (local flags defined on each workload command) are automatically captured and added to the SummaryMetadata in the workloadFlags field. This allows you to track exactly which flags were used when running a workload, making it easier to correlate performance results with specific configurations.

The flags are stored as a map where:

• Keys are flag names converted from kebab-case to camelCase (e.g., pods-per-node becomes podsPerNode)
• Values are the flag values as strings

For example, when running:

kube-burner-ocp node-density --pods-per-node=100 --churn-cycles=5 --churn-percent=10

The jobSummary document will include:

{
  "workloadFlags": {
    "podsPerNode": "100",
    "churnCycles": "5",
    "churnPercent": "10",
    ...
  }
}

This metadata can be used for filtering and analysis in your metrics backend.

!!! Note Only local flags specific to each workload are captured. Persistent flags from the parent command (such as --uuid, --qps, --burst, etc.) are not included in workloadFlags to avoid duplication..

Multiple endpoints support

The flag --metrics-endpoint can be used to interact with multiple Prometheus endpoints For example:

kube-burner-ocp cluster-density-v2 --iterations=1 --churn-duration=2m0s --churn-cycles=2 --es-index kube-burner --es-server https://www.esurl.com:443 --metrics-endpoint metrics-endpoints.yaml

metrics-endpoints.yaml

- endpoint: prometheus-k8s-openshift-monitoring.apps.rook.devshift.org
  metrics:
    - metrics.yml
  alerts:
    - alerts.yml
  indexer:
      esServers: ["{{.ES_SERVER}}"]
      insecureSkipVerify: true
      defaultIndex: {{.ES_INDEX}}
      type: opensearch
- endpoint: https://prometheus-k8s-openshift-monitoring.apps.rook.devshift.org
  token: {{ .TOKEN }}
  metrics:
    - metrics.yml
  indexer:
      esServers: ["{{.ES_SERVER}}"]
      insecureSkipVerify: true
      defaultIndex: {{.ES_INDEX}}
      type: opensearch

.TOKEN can be captured by running TOKEN=$(oc create token -n openshift-monitoring prometheus-k8s)

!!! Note

Avoid passing absolute path of the file with --metrics-endpoint option

Metric profile names specified against `metrics` key should be unique and shouldn't overlap with the existing ones. A metric profile will be looked up in this directory [config](https://github.com/kube-burner/kube-burner-ocp/tree/main/cmd/config) first for the sake of simplicity and if it doesn't exist, will fallback to our specified path. So in order for our own metric profile to get picked up, we will need to specify its absolute path or name differently whenever there is an overlap with the existing ones.

Cluster density workloads

This workload family is a control-plane density focused workload that that creates different objects across the cluster. There are 2 different variants cluster-density-v2 and cluster-density-ms.

Each iteration of these create a new namespace, the three support similar configuration flags. Check them out from the subcommand help.

!!! Info Workload churning of 1h is enabled by default in the cluster-density workloads; you can disable it by passing --churn-duration=0 to the workload subcommand.

cluster-density-v2

Each iteration creates the following objects in each of the created namespaces:

• 1 image stream.
• 1 build. The OCP internal container registry must be set-up previously because the resulting container image will be pushed there.
• 3 deployments with two pod 2 replicas (nginx) mounting 4 secrets, 4 config maps, and 1 downward API volume each.
• 2 deployments with two pod 2 replicas (curl) mounting 4 Secrets, 4 config maps and 1 downward API volume each. These pods have configured a readiness probe that makes a request to one of the services and one of the routes created by this workload every 10 seconds.
• 5 services, each one pointing to the TCP/8080 port of one of the nginx deployments.
• 2 edge routes pointing to the to first and second services respectively.
• 10 secrets containing a 2048-character random string.
• 10 config maps containing a 2048-character random string.
• 3 network policies:
  • deny-all traffic
  • allow traffic from client/nginx pods to server/nginx pods
  • allow traffic from openshift-ingress namespace (where routers are deployed by default) to the namespace

cluster-density-ms

Lightest version of this workload family, each iteration the following objects in each of the created namespaces:

• 1 image stream.
• 4 deployments with two pod replicas (pause) mounting 4 secrets, 4 config maps, and 1 downward API volume each.
• 2 services, each one pointing to the TCP/8080 and TCP/8443 ports of the first and second deployment respectively.
• 1 edge route pointing to the to first service.
• 20 secrets containing a 2048-character random string.
• 10 config maps containing a 2048-character random string.

Etcd density workloads

The etcd-density command groups four workload variants that stress the etcd database under different pressure patterns. Each variant targets a specific failure mode relevant to etcd sharding validation: event volume, crash-driven churn, storage quota exhaustion, and revision bloat.

All variants default to the etcd-density-metrics.yml metrics profile, which collects etcd DB size, WAL fsync latency, backend commit duration, raft proposal rates, event throughput, API server latency, and control-plane resource consumption.

event-storm

Fires a synthetic event storm in the background while deploying a victim workload. Measures how sustained event write pressure degrades pod scheduling and readiness latency.

The storm runs in namespace etcd-event-storm using a hook script that creates events concurrently before the victim workload starts. After the victim workload completes, the storm namespace is cleaned up.

Flag	Description	Default
--iterations	Victim workload iterations	100
--pod-ready-threshold	Pod ready timeout threshold	0 (disabled)
--deletion-strategy	Deletion strategy	default
--metrics-profile	Comma separated list of metrics profiles	etcd-density-metrics.yml
--event-iterations	Event creation iterations for the background storm	2000
--event-replicas	Event replicas per iteration	100
--storm-workers	Parallel workers for event creation	20
--storm-delay	Seconds to let the storm build pressure before victim	15
--pod-replicas	Pod replicas per victim deployment	3

kube-burner-ocp etcd-density event-storm --iterations=100 --event-iterations=2000 \
  --event-replicas=100 --storm-workers=20 --storm-delay=15 --pod-replicas=3 \
  --local-indexing

crashloop-flood

Creates pods that immediately exit with an error, causing continuous CrashLoopBackOff restarts. The sustained event churn from repeated container crashes pressures etcd event storage and API server watch caches.

After pod creation, the workload soaks for a configurable duration to let event pressure accumulate before collecting metrics.

Flag	Description	Default
--iterations	Number of namespaces to create crashlooping pods in	20
--pod-ready-threshold	Pod ready timeout threshold	0 (disabled)
--deletion-strategy	Deletion strategy	default
--metrics-profile	Comma separated list of metrics profiles	etcd-density-metrics.yml
--crashloop-replicas	Crashlooping pods per iteration	100
--soak-duration	Soak duration for event accumulation after pod creation	30m

kube-burner-ocp etcd-density crashloop-flood --iterations=20 \
  --crashloop-replicas=100 --soak-duration=30m \
  --local-indexing

db-quota-pressure

Fills etcd with large custom resource objects to stress storage quotas. Creates CRDs first, then fills namespaces with custom resource instances. Objects are intentionally not cleaned up (cleanup: false) to preserve the database size for analysis.

Database size calculation: iterations x kb-chunks x kb-size

iterations	kb-chunks	kb-size	Total DB size	iterations-per-namespace
6,551	8	100	~5GB	936
14,925	18	100	~27GB	415
11,194	16	100	~18GB	468
36,841	16	10	~6GB	4,605

Flag	Description	Default
--iterations	Number of object iterations for the db-quota-pressure job	6551
--iterations-per-namespace	Number of iterations per namespace	936
--metrics-profile	Comma separated list of metrics profiles	etcd-density-metrics.yml
--kb-chunks	Number of CRDs to create and object replicas per iteration	8
--kb-size	Size of each object in KB (10 or 100)	100

kube-burner-ocp etcd-density db-quota-pressure --iterations=6551 \
  --iterations-per-namespace=936 --kb-chunks=8 --kb-size=100 \
  --gc=false --gc-metrics=false \
  --local-indexing

annotation-churn

Creates ConfigMaps and then rapidly patches their annotations in multiple rounds, generating etcd revision bloat from high-frequency small writes. Measures compaction and defragmentation behavior under write-amplification pressure.

Flag	Description	Default
--iterations	Target creation iterations (one namespace each)	10
--deletion-strategy	Deletion strategy	default
--metrics-profile	Comma separated list of metrics profiles	etcd-density-metrics.yml
--patch-replicas	Target ConfigMaps per iteration	50
--patch-rounds	Number of patch rounds across all targets	10

kube-burner-ocp etcd-density annotation-churn --iterations=10 \
  --patch-replicas=50 --patch-rounds=10 \
  --local-indexing

!!! Note These workloads require OpenShift clusters with sufficiently large control plane nodes (m6a.4xlarge or r6a.4xlarge minimum) for meaningful pressure testing.

MicroShift

kube-burner-ocp detects MicroShift through cluster metadata and can run cluster-density-ms against clusters that expose the required Kubernetes APIs. On MicroShift, the wrapper skips OpenShift ClusterOperator health checks and uses the vanilla Kubernetes health checks.

Use an external Prometheus that scrapes MicroShift directly. OpenShift Prometheus discovery is not available on MicroShift, so metrics collection with the normal wrapper indexing flags requires --prometheus-url. Use --prometheus-token when that endpoint requires a bearer token.

The microshift-metrics.yml profile expects kubelet metrics with job="kubelet-microshift", cAdvisor metrics with job="kubelet-microshift-cadvisor", node exporter with job="node", CRI-O with job="crio", and named-process-exporter with job="process".

The microshiftComponentCPU and microshiftComponentMemoryRSS metrics group named-process-exporter process groups into MicroShift components, one series per component label: microshift, cni, csi, runtime, dns, ingress and service-ca. CPU is in percent of one core and memory is resident set size in bytes, like the other metrics in the profile.

Run a small workload with OpenSearch indexing:

kube-burner-ocp cluster-density-ms \
  --iterations=1 \
  --metrics-profile=microshift-metrics.yml \
  --prometheus-url=http://prometheus.example:9090 \
  --es-server=https://opensearch.example:9200 \
  --es-index=kube-burner \
  --alerting=false

The --metrics-endpoint flag remains available for advanced cases where the full Prometheus endpoint and indexer configuration are supplied in a separate file.

The index subcommand also honors --prometheus-url and --prometheus-token when scraping an existing MicroShift run. Like the workload subcommands, it resolves metrics profile names such as microshift-metrics.yml from the embedded profiles, so no --extract is needed; a local file with the same name takes precedence.

Summary metadata keeps the existing top-level OpenShift wrapper shape. Metrics documents keep metadata under their nested metadata field and include distribution, microshift, microshiftVersion, microshiftMajorVersion, k8sVersion, and totalNodes when those values are discovered.

cluster-density-ms renders optional OpenShift objects only when the APIs are served: imagestream.yml requires image.openshift.io, and route.yml requires route.openshift.io. cluster-density-v2 remains an OpenShift workload and still requires the full OpenShift image registry path.

Node density workloads

The workloads of this family create a single namespace with a set of pods, deployments, and services depending on the workload.

node-density

This workload is meant to fill with pause pods all the worker nodes from the cluster. It can be customized with the following flags. This workload is usually used to measure the Pod's ready latency KPI.

node-density-cni

It creates two deployments, a client/curl and a server/nxing, and 1 service backed by the previous server pods. The client application has configured an startup probe that makes requests to the previous service every second with a timeout of 600s.

Note: This workload calculates the number of iterations to create from the number of nodes and desired pods per node. In order to keep the test scalable and performant, chunks of 1000 iterations will by broken into separate namespaces, using the config variable iterationsPerNamespace.

node-density-cni with SRIOV VFs

It creates two deployments, a client/curl and a server/nxing, and 1 service backed by the previous server pods along with secondary network interfaces with SRIOV VFs. The client application has configured an startup probe that makes requests to the previous service every second with a timeout of 600s.

Pre-requisites for running with SRIOV VFs

• Cluster should be installed with DAY-2 SRIOV Operator
• If User want to run workload with Performance profile, It should be created.
• Below SRIOV-network should be created before running the workload

  ---
  apiVersion: sriovnetwork.openshift.io/v1
  kind: SriovNetwork
  metadata:
    name: sriov-net-node-density
    namespace: openshift-sriov-network-operator
  spec:
    ipam: |
      {
        "type": "whereabouts",
        "range": "172.20.0.0/16"
      }
    logLevel: info
    networkNamespace: node-density-cni-0
    resourceName: servervfs
    spoofChk: "off"
    trust: "on"
  

Note: This workload calculates the number of iterations to create from the number of nodes and desired pods per node. In order to keep the test scalable and performant, chunks of 1000 iterations will by broken into separate namespaces, using the config variable iterationsPerNamespace.

node-density-heavy

Creates two deployments, a postgresql database, and a simple client that performs periodic insert queries (configured through liveness and readiness probes) on the previous database and a service that is used by the client to reach the database.

Note: this workload calculates the number of iterations to create from the number of nodes and desired pods per node. In order to keep the test scalable and performant, chunks of 1000 iterations will by broken into separate namespaces, using the config variable iterationsPerNamespace.

node-scale

Leverages Kubemark to create a number of hollow nodes and allow to run experiments on simulated clusters. The primary use case is scalability testing, as simulated clusters can be much bigger than the real ones. The objective is to expose problems with the master components (API server, controller manager or scheduler) that appear only on bigger clusters (e.g. small memory leaks).

udn-density-l3-pods

For User-Defined Network (UDN) L3 segmentation testing. It creates two deployments, a client/curl and a server/nxing.

anp-density-pods

For AdminNetworkPolicy testing. It creates three deployments in each namespace, three namespaces as a tenant, it will create 1 BaselineAdminNetworkPolicy, 1 NodeSelector AdminNetworkPolicy, 7 PodSelector AdminNetworkPolicy, N - CIDR Selector AdminNetworkPolicy.

Network Policy workloads

Network policy scale testing tooling involved 2 components:

1. Template to include all network policy configuration options
2. Latency measurement through connection testing

A network policy defines the rules for ingress and egress traffic between pods in local and remote namespaces. These remote namespace addresses can be configured using a combination of namespace and pod selectors, CIDRs, ports, and port ranges. Given that network policies offer a wide variety of configuration options, we developed a unified template that incorporates all these configuration parameters. Users can specify the desired count for each option.

spec:
  podSelector:
    matchExpressions:
    - key: num
      operator: In
      values:
      - "1"
      - "2"
  ingress:
  - from:
    - namespaceSelector:
        matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: In
          values:
          - network-policy-perf-13
          - network-policy-perf-14
      podSelector:
       matchExpressions:
       - key: num
         operator: In
         values:
         - "1"
         - "2"
    ports:
    - port: 8080
      protocol: TCP

Scale Testing and Unique ACL Flows

In our scale tests, we aim to create between 10 to 100 network policies within a single namespace. The primary focus is on preventing duplicate configuration options, which ensures that each network policy generates unique Access Control List (ACL) flows. To achieve this, we carefully designed our templating approach based on the following considerations:

Round-Robin Assignment: We use a round-robin strategy to distribute

1. remote namespaces among ingress and egress rules across kube burner job iterations
2. remote namespaces among ingress and egress rules in the same kube burner job iteration

This ensures that we don’t overuse the same remote namespaces in a single iteration or among multiple iterations. For instance, if namespace-1 uses namespace-2 and namespace-3 as its remote namespaces, then namespace-2 will start using namespace-4 and namespace-5 as remote namespaces in the next iteration.

Unique Namespace and Pod Combinations: To avoid redundant flows, the templating system generates unique combinations of remote namespaces and pods for each network policy. Initially, we iterate through the list of remote namespaces, and once all remote namespaces are exhausted, we move on to iterate through the remote pods. This method ensures that every network policy within a namespace is assigned a distinct combination of remote namespaces and remote pods, avoiding duplicate pairs.

Templating Logic Our templating logic is implemented as follows:

// Iterate over the list of namespaces to configure network policies.
for namespace := namespaces {

  // Each network policy uses a combination of a remote namespace and a remote pod to allow traffic.
  for networkPolicy := networkPolicies {

    /*
    Iterate through the list of remote pods. Once all remote namespaces are exhausted,
    continue iterating through the remote pods to ensure unique namespace/pod combinations.
    */
    for i, remotePod := range remotePods {
        // Stop when we reach the maximum number of remote pods allowed.
        if i == num_remote_pods {
            break
        }

        // Iterate through the list of remote namespaces to pair with the remote pod.
        for idx, remoteNamespace := range remoteNamespaces {
            // Combine the remote namespace and pod into a unique pair for ACL configuration.
            combine := fmt.Sprintf("%s:%s", remoteNamespace, remotePod)

            // Stop iterating once we’ve exhausted the allowed number of remote namespaces.
            if idx == num_remote_namespace {
                break
            }
        }
    }
  }
}

CIDRs and Port Ranges We apply the same round-robin and unique combination logic to CIDRs and port ranges, ensuring that these options are not reused in network policies within the same namespace.

Connection Testing Support kube-burner measures network policy latency through connection testing. Currently, all pods are configured to listen on port 8080. As a result, client pods will send requests to port 8080 during testing.

Note: Egress rules should not be enabled for network policy latency measurement connection testing.

EgressIP workloads

This workload creates an egress IP for the client pods. SDN (OVN) will use egress IP for the traffic from client pods to external server instead of default node IP.

Each iteration creates the following objects in each of the created namespaces:

• 1 deployment with the configured number of client pod replicas. Client pod runs the quay.io/cloud-bulldozer/eipvalidator app which periodically sends http request to the configured "EXT_SERVER_HOST" server at an "DELAY_BETWEEN_REQ_SEC" interval with a request timeout of "REQ_TIMEOUT_SEC" seconds. Client pod then validates if the body of the response has configured "EGRESS_IPS". Once the client pod starts running and after receiving first successful response with configured "EGRESS_IPS", it sets "eip_startup_latency_total" prometheus metric.
• 1 EgressIP object. EgressIP object is cluster scoped. EgressIP object will have number of egress IP addresses which user specified through "addresses-per-iteration" cli option. kube-burner generates these addresses for the egressIP object from the egress IP list provided by kube-burner-ocp. OVN applies egressIPs to the pods in the current job iteration because of "namespaceSelector" and "podSelector" fields in the egressIP object.

Note: User has to manually create the external server or use the e2e-benchmarking(https://github.com/cloud-bulldozer/e2e-benchmarking/tree/master/workloads/kube-burner-ocp-wrapper#egressip) which deploys external server and runs the workload with required configuration.

Running 1 iteration with 1 egress IP address per iteration (or egressIP object).

kube-burner-ocp egressip --addresses-per-iteration=1 --iterations=1 --external-server-ip=10.0.34.43

With the command above, each namespace has one pod with a dedicated egress IP. OVN will use this dedicated egress IP for the http requests from client pod's to 10.0.34.43.

Web-burner workloads

This workload is meant to emulate some telco specific workloads. Before running web-burner-node-density or web-burner-cluster-density load the environment with web-burner-init first (without the garbage collection flag: --gc=false).

Pre-requisites:

• At least two worker nodes
• At least one of the worker nodes must have the node-role.kubernetes.io/worker-spk label

web-burner-init

• 35 (macvlan/sriov) networks for 35 lb namespace
• 35 lb-ns
  • 1 frr config map, 4 emulated lb pods on each namespace
• 35 app-ns
  • 1 emulated lb pod on each namespace for bfd session

web-burner-node-density

• 35 app-ns
  • 3 app pods and services on each namespace
• 35 normal-ns
  • 1 service with 60 normal pod endpoints on each namespace

web-burner-cluster-density

• 20 normal-ns
  • 30 configmaps, 38 secrets, 38 normal pods and services, 5 deployments with 2 replica pods on each namespace
• 35 served-ns
  • 3 app pods on each namespace
• 2 app-served-ns
  • 1 service(15 ports) with 84 pod endpoints, 1 service(15 ports) with 56 pod endpoints, 1 service(15 ports) with 25 pod endpoints
  • 3 service(15 ports each) with 24 pod endpoints, 3 service(15 ports each) with 14 pod endpoints
  • 6 service(15 ports each) with 12 pod endpoints, 6 service(15 ports each) with 10 pod endpoints, 6 service(15 ports each) with 9 pod endpoints
  • 12 service(15 ports each) with 8 pod endpoints, 12 service(15 ports each) with 6 pod endpoints, 12 service(15 ports each) with 5 pod endpoints
  • 29 service(15 ports each) with 4 pod endpoints, 29 service(15 ports each) with 6 pod endpoints

Core RDS workloads

The telco core reference design specification (RDS) describes OpenShift Container Platform clusters running on commodity hardware that can support large scale telco applications including control plane and some centralized data plane functions. It captures the recommended, tested, and supported configurations to get reliable and repeatable performance for clusters running the telco core profile.

Pre-requisites:

• A PerformanceProfile with isolated and reserved cores, 1G hugepages and and topologyPolicy=single-numa-node. Hugepages should be allocated in the first NUMA node (the one that would be used by DPDK deployments):

   hugepages:
   defaultHugepagesSize: 1G
   pages:
   - count: 160
     node: 0
     size: 1G
   - count: 6
     node: 1
     size: 1G
  

• MetalLB operator limiting speaker pods to specific nodes (approx. 10%, 12 in the case of 120 node iterations with the corresponding worker-metallb label):

  apiVersion: metallb.io/v1beta1
  kind: MetalLB
  metadata:
    name: metallb
    namespace: metallb-system
  spec:
    nodeSelector:
      node-role.kubernetes.io/worker-metallb: ""
    speakerTolerations:
    - key: "Example"
      operator: "Exists"
      effect: "NoExecute"
  

• SRIOV operator with its corresponding SriovNetworkNodePolicy

  Note : Please find the recommended SriovNetworkNodePolicy for pod churn at high %

  ---
  apiVersion: sriovnetwork.openshift.io/v1
  kind: SriovNetworkNodePolicy
  metadata:
    name: dpdk-vfs-policy
    namespace: openshift-sriov-network-operator
  spec:
    deviceType: vfio-pci
    isRdma: false
    nicSelector:
      pfNames:
      - ens1f1#0-19
    nodeSelector:
      node-role.kubernetes.io/worker-dpdk: ""
    numVfs: 60
    priority: 99
    resourceName: dpdkvfs
  ---
  apiVersion: sriovnetwork.openshift.io/v1
  kind: SriovNetworkNodePolicy
  metadata:
    name: sriov-vfs-policy-worker-dpdk
    namespace: openshift-sriov-network-operator
  spec:
    deviceType: netdevice
    isRdma: false
    nicSelector:
      pfNames:
      - ens1f1#20-59
    nodeSelector:
      node-role.kubernetes.io/worker-dpdk: ""
    numVfs: 60
    priority: 99
    resourceName: servervfs
  ---
  apiVersion: sriovnetwork.openshift.io/v1
  kind: SriovNetworkNodePolicy
  metadata:
    name: server-vfs-policy
    namespace: openshift-sriov-network-operator
  spec:
    deviceType: netdevice
    isRdma: false
    nicSelector:
      pfNames:
      - ens1f1
    nodeSelector:
      node-role.kubernetes.io/customcnf: "enabled"
    numVfs: 60
    priority: 50
    resourceName: servervfs
  ---
  apiVersion: sriovnetwork.openshift.io/v1
  kind: SriovNetworkNodePolicy
  metadata:
    name: server-vfs-policy-metallb-workers
    namespace: openshift-sriov-network-operator
  spec:
    deviceType: netdevice
    isRdma: false
    nicSelector:
      pfNames:
      - ens1f1
    nodeSelector:
      node-role.kubernetes.io/worker-metallb: ""
    numVfs: 60
    priority: 50
    resourceName: servervfs
  

• Some nodes (i.e.: 25% of them) with the worker-dpdk label to host the DPDK pods, i.e.:

  $ kubectl label node worker1 node-role.kubernetes.io/worker-dpdk=
  

Object count:

Iterations / nodes / namespaces	1	120
configmaps	30	3600
deployments_best_effort	25	3000
deployments_dpdk	2	240 (assuming 24 worker-dpdk nodes)
endpoints (210x service)	4200	504000
endpoints lb (90 x service)	90	10800
networkPolicy	3	360
namespaces	1	120
pods_best_effort (2 x deployment)	50	6000
pods_dpdk (1 x deployment)	2	240 (assuming 24 worker-dpdk nodes)
route	2	240
services	20	2400
services (lb)	1	120
secrets	42	5040

Input parameters specific to the workload:

Parameter	Description	Default value
dpdk-cores	Number of cores assigned for each DPDK pod (should fill all the isolated cores of one NUMA node)	2
performance-profile	Name of the performance profile implemented on the cluster	default
dpdk-scheduler	Scheduler used for the DPDK pods, e.g. topo-aware-scheduler for NUMA-aware scheduling	default-scheduler

Virt Workloads

This workload family is a focused on Virtualization creating different objects across the cluster.

The different variants are:

• virt-density
• virt-udn-density
• virt-capacity-benchmark
• virt-parallel
• virt-clone
• virt-clone-multi
• virt-ephemeral-restart
• virt-migration

Environment Requirements

Virtctl

The tests listed below verify that the VirtualMachine completed their boot using virtctl ssh. Therefore, virtctl must be installed and available in the PATH.

• virt-capacity-benchmark.
• virt-parallel
• virt-clone
• virt-clone-multi
• virt-ephemeral-restart
• virt-migration

See the Temporary SSH Keys for details on the SSH keys used for the test

Temporary SSH Keys

The test generated the SSH keys automatically. By default, it stores the pair in a temporary directory. Users may choose the store the key in a specified directory by setting --ssh-key-path

Virt Density

Similar to node-density, fills with VirtualMachines the worker nodes of the cluster (kubevirt/OpenShift Virtualization is required to run this workload). Meant to detect issues derived from spinning up high amounts VMs in a short amount of time and to track runningthe latencies of the different VM bootstrap stages.

Virt Density Udn

Similar to udn-density-pods scenario. Creates VMs, one Nginx server and several clients (the number depends on the vms-per-node variable) reaching it, on the same UDN per iteration. Each UDN-namespace has the same number of VMs, the number of clients deployed per UDN is computed as following: Nb of client per UDN = (Nb of worker * vms-per-node / Nb of UDN) -1 //-1 because the server is always deployed. This scenario is meant to test how many UDNs can be deployed in parallel and how it scales. It requires a version of OCP higher than 4.18, otherwise, UDN feature is not available.

Virt C-UDN Density

In order to be able to compare UserDefinedNetwork (UDN) and ClusterUserDefinedNetwork (CUDN) performance, this scenario uses the same logic than the previous one. It deploys VMs, one Nginx server and several clients into namespaces, the number of namespaces is the number of iterations. For each namespace, there is one C-UDN associated to it one to be close to the virt UDN density scenario.

Virt Capacity Benchmark

Test the capacity of Virtual Machines and Volumes supported by the cluster and a specific storage class.

Test Sequence

The test runs a workload in a loop without deleting previously created resources. By default it will continue until a failure occurs. Each loop is comprised of the following steps:

• Create VMs
• Resize the root and data volumes
• Restart the VMs
• Snapshot the VMs
• Migrate the VMs

Tested StorageClass

By default, the test will search for the StorageClass to use:

1. Use the default StorageClass for Virtualization annotated with storageclass.kubevirt.io/is-default-virt-class
2. If does not exist, use general default StorageClass annotated with storageclass.kubernetes.io/is-default-class
3. If does not exist, fail the test before starting

To use a different one, use --storage-class to provide a different name.

In addition, multiple StorageClasses can be used by passing a comma separated list names. The test will then choose a different StorageClass for each loop in round robin.

Please note that regardless to which StorageClass is used, it must:

• Support Volume Expansion: allowVolumeExpansion: true.
• Have a corresponding VolumeSnapshotClass using the same provisioner

Test Namespace

All VirtualMachines are created in the same namespace.

By default, the namespace is virt-capacity-benchmark. Set it by passing --namespace (or -n)

Test Size Parameters

Users may control the workload sizes by passing the following arguments:

• --max-iterations - Maximum number of iterations, or 0 (default) for infinite. In any case, the test will stop upon failure
• --vms - Number of VMs for each iteration (default 5)
• --data-volume-count - Number of data volumes for each VM (default 9)
• --min-vol-size - Set the minimal volume size supported by the storage class
• --min-vol-inc-size - Set the minimal volume size increment supported by the storage class

Skip test parts

Some storage classes have limitations requiring the test to skip some parts. All steps except VM creation can be skipped:

• --skip-resize-job - Skip volume resize job. Use when e.g. allowVolumeExpansion is false
• --skip-restart-job - Skip the VM restart job. Use when you want to test without VM restarts
• --skip-snapshot-job - Skip the VM snapshot job. Use when snapshots are not supported or not needed
• --skip-migration-job - Skip the migration job. Use when e.g. RWX accessMode is not supported

VM Image Source

By default, VMs are created directly from container disk images pulled from the registry. For testing storage cloning performance or reducing registry load, VMs can be cloned from a base image:

• --use-base-image - Create a base DataVolume from the container image, then clone all VMs from it (default: false)
• --use-snapshot - When using --use-base-image, clone from VolumeSnapshot instead of direct PVC clone (default: true)

When --use-base-image is enabled, the workflow creates a base DataVolume, optionally creates a VolumeSnapshot, then clones all VMs from the snapshot or PVC. This reduces registry pulls and enables storage cloning performance testing.

Cleanup

Since the test is expected to run until failure, it is designed to keep all allocated resources to allow investigating the failure. To cleanup all allocated resources once the test is done set --gc. Alternatively, run the test with only the --cleanup flag set to cleanup resources from past test runs

Virt Parallel

Test the parallel execution of virtualization workloads across increasing numbers of VMs. This workload progressively increases the VM count across iterations, performing a complete lifecycle of operations on each set of VMs before deletion.

Test Sequence

The test runs in a loop without preserving resources between iterations. Each iteration performs the following sequence:

• Create N VMs with multiple data volumes and verify network connectivity to all VMs
• Resize the root and data volumes of all VMs (optional, can be skipped with --skip-resize-job)
• Restart the VMs to apply volume changes (optional, can be skipped with --skip-restart-job)
• Snapshot all VMs (optional, can be skipped with --skip-snapshot-job)
• Randomly select a worker node and migrate all VMs running on that node (optional, can be skipped with --skip-migration-job)
• Delete all VMs and clean up resources
• Increment N by the configured amount (--increment) and repeat

Note: The only mandatory steps are VM creation and deletion. All other steps can be skipped based on storage class capabilities or test requirements.

Tested StorageClass

By default, the test will search for the StorageClass to use:

1. Use the default StorageClass for Virtualization annotated with storageclass.kubevirt.io/is-default-virt-class
2. If does not exist, use general default StorageClass annotated with storageclass.kubernetes.io/is-default-class
3. If does not exist, fail the test before starting

To use a different one, use --storage-class to provide a different name.

In addition, multiple StorageClasses can be used by passing a comma separated list names. The test will then choose a different StorageClass for each loop in round robin.

Please note that regardless to which StorageClass is used, it must:

• Support Volume Expansion: allowVolumeExpansion: true
• Have a corresponding VolumeSnapshotClass using the same provisioner

Test Namespace

All VirtualMachines are created in the same namespace.

By default, the namespace is virt-parallel. Set it by passing --namespace (or -n)

Test Size Parameters

Users may control the workload sizes by passing the following arguments:

• --max-iterations - Maximum number of iterations, or 0 (default) for infinite. In any case, the test will stop upon failure
• --initial-vms - Number of VMs to create in the first iteration (default 5)
• --increment - Number of additional VMs to add in each subsequent iteration (default 5)
• --data-volume-count - Number of data volumes per VM (default 9)
• --min-vol-size - Set the minimal volume size supported by the storage class
• --min-vol-inc-size - Set the minimal volume size increment supported by the storage class

Skip test parts

Some storage classes have limitations requiring the test to skip some parts. All steps except VM creation and deletion can be skipped:

• --skip-resize-job - Skip volume resize job. Use when e.g. allowVolumeExpansion is false
• --skip-restart-job - Skip the VM restart job. Use when you want to test without VM restarts
• --skip-snapshot-job - Skip the VM snapshot job. Use when snapshots are not supported or not needed
• --skip-migration-job - Skip the migration job. Use when e.g. RWX accessMode is not supported

VM Image Source

By default, VMs are created directly from container disk images pulled from the registry. For testing storage cloning performance or reducing registry load, VMs can be cloned from a base image:

• --use-base-image - Create a base DataVolume from the container image, then clone all VMs from it (default: false)
• --use-snapshot - When using --use-base-image, clone from VolumeSnapshot instead of direct PVC clone (default: true)

When --use-base-image is enabled, the workflow creates a base DataVolume, optionally creates a VolumeSnapshot, then clones all VMs from the snapshot or PVC. This reduces registry pulls and enables storage cloning performance testing.

Cleanup

Since the test is expected to run until failure, it is designed to keep all allocated resources to allow investigating the failure. To cleanup all allocated resources once the test is done set --gc. Alternatively, run the test with only the --cleanup flag set to cleanup resources from past test runs

Virt Clone

Test the capacity and performance of starting multiple virtual machines with a root disk as clones of a single volume. This test comes to mimic VDI sequence

Test Sequence

The test runs the following sequence:

1. Create a VirtualMachine in namespace A
2. Stop the VirtualMachine
3. Create a DataVolume in namespace B using the rootdisk of the VirtualMachine as the source
4. If the dataImportCronSourceFormat field of the StorageProfile status is set to snapshot, or --use-snapshot is set to true, create a VolumeSnapshot of the DataVolume
5. Create a DataSource, setting the source field to either the VolumeSnapshot (if was created) or the DataVolume
6. Create VirtualMachines in namespace B based in the DataSource

Tested StorageClass

By default, the test will use the default StorageClass. To use a different one, use --storage-class to provide a different name.

If --use-snapshot is explicitly set to true a corresponding VolumeSnapshotClass using the same provisioner must exist. Otherwise, the test will check the StorageProfile for the StorageClass and act accordingly.

Test Namespace

The test creates VirtualMachines in two namespaces: <baseName>-base and <baseName>-clones

By default, the baseName is virt-clone. Set it by passing --namespace (or -n)

Test Size Parameters

Users may control the workload sizes by passing the following arguments:

• --iteration - Number of iterations to run in step 6. Default is 1
• --iteration-clones - Number of VirtualMachines to create in each iteration of step 6. Default is 10
• --data-volume-count - Number of empty data volumes for each cloned VirtualMachine. Default is 0

Verification of cloned VirtualMachines creation

By default, the test waits for all VirtualMachines created in an iteration to reach the Ready condition. Users may configure the test to run differently.

• --verify-each-iteration - Whether the test should verify the cloned VirtualMachines on each iteration. Default true. If set to false verification will be postpone to the end of the job
• --job-iteration-delay - Time in Duration for the test to wait between iterations. Default is 0
• --verify-max-wait-time - Time in Duration to wait for VirtualMachines to become ready either for each iteration or at the end.

For example, to change the test to wait for a minute between iterations instead of the VirtualMachines to become Ready set: --verify-each-iteration=false --job-iteration-delay=1m

Volume Access Mode

By default, volumes are created with ReadWriteMany access mode as this is the recommended configuration for VirtualMachines. If not supported, the access mode may be changes by setting --access-mode. The supported values are RO, RWO and RWX.

Virt Clone Multi

Test high-scale VM cloning by deploying VMs across multiple namespaces with independent base images. This is a scalable version of the virt-clone workload, where each namespace has its own master image for cloning. The workload validates storage backend capacity, CDI (Containerized Data Importer) performance, and VM cloning efficiency at scale.

Key Differentiators from virt-clone

• Multi-namespace architecture: Each test namespace maintains its own master image (DataVolume/VolumeSnapshot/DataSource)
• Namespace isolation: Clones within a namespace are created from that namespace's master image, not a centralized source

Test Sequence

The test follows this workflow:

1. Create base VMs: Create N base VMs in a base namespace from a container disk
2. Stop base VMs: Stop all base VMs to ensure data consistency
3. Create test DataSources (per namespace, in parallel):
   • Create a DataVolume from the corresponding base VM's PVC
   • Optionally create a VolumeSnapshot of the DataVolume (if --use-snapshot=true)
   • Create a DataSource pointing to either the snapshot or the DataVolume
4. Clone VMs (per namespace, in batches):
   • Create VMs in iterations, cloning from the namespace's DataSource
   • Each VM optionally includes additional blank data volumes (controlled by --data-volume-count)

Tested StorageClass

By default, the test will search for the StorageClass to use:

1. Use the default StorageClass for Virtualization annotated with storageclass.kubevirt.io/is-default-virt-class
2. If does not exist, use general default StorageClass annotated with storageclass.kubernetes.io/is-default-class
3. If does not exist, fail the test before starting

To use a different one, use --storage-class to provide a different name.

If --use-snapshot is set to true (default), a corresponding VolumeSnapshotClass using the same provisioner must exist. If --use-snapshot=false, the test will clone directly from the PVC without creating snapshots.

Test Namespaces

Resources are distributed across multiple namespaces for scale testing.

By default, namespaces follow the pattern virt-clone-multi-0, virt-clone-multi-1, etc. Set the base name by passing --namespace (or -n)

Test Size Parameters

Users may control the workload scale by passing the following arguments:

• --namespaces - Number of test namespaces to create (default: 2)
• --iterations - Number of batches per namespace (default: 5)
• --vms-per-iteration - Number of VMs created per batch (default: 2)
• --data-volume-count - Number of additional blank data volumes per VM (default: 0)

Total VMs = namespaces × iterations × vms-per-iteration

Total PVCs = Total VMs × (1 + data-volume-count) (1 root PVC + N data volumes per VM)

Batching Control

• --job-iteration-delay - Delay between iterations within a namespace (default: 1m)

Namespaces are processed in parallel. Within each namespace, VMs are created in batches (iterations) with the specified delay between batches.

Volume Access Mode

By default, volumes are created with ReadWriteMany access mode. Since this workload does not perform migrations, ReadWriteOnce can be used for storage classes that don't support RWX.

The access mode can be changed by setting --access-mode. The supported values are RO, RWO and RWX.

Cleanup

To cleanup all allocated resources once the test is done set --gc. Alternatively, run the test with only the --cleanup flag set to cleanup resources from past test runs.

Virt Ephemeral Restart

Test the performance of restarting ephemeral VirtualMachines. Kubernetes native ephemeral volumes use local node storage. As a result, they cannot be used on large scale deployment. Instead, a restart is implemented by stopping the VirtualMachine, deleting the DataVolume backing its root volume and starting it.

Test Sequence

The test runs the following sequence:

1. Create a DataVolume using a container image as the source
2. If the dataImportCronSourceFormat field of the StorageProfile status is set to snapshot, or --use-snapshot is set to true, create a VolumeSnapshot of the DataVolume
3. Create a DataSource, setting the source field to either the VolumeSnapshot (if was created) or the DataVolume
4. Create VirtualMachines based in the DataSource
5. Stop all VirtualMachines
6. In batches, delete the DataVolume backing the root disk and start the VirtualMachines

Tested StorageClass

By default, the test will use the default StorageClass. To use a different one, use --storage-class to provide a different name.

If --use-snapshot is explicitly set to true a corresponding VolumeSnapshotClass using the same provisioner must exist. Otherwise, the test will check the StorageProfile for the StorageClass and act accordingly.

Test Namespace

All VirtualMachines are created in the same namespace.

By default, the namespace is virt-ephemeral-restart. Set it by passing --namespace (or -n)

Test Size Parameters

Users may control the workload sizes by passing the following arguments:

• --iteration-vms - Number of VirtualMachines to batch in each group in step 6
• --iteration - Number of batches to run in step 6

!!! Note

The total number of `VirtualMachines` created is `--iteration-vms` * `--iteration`

Volume Access Mode

By default, volumes are created with ReadWriteMany access mode as this is the recommended configuration for VirtualMachines. If not supported, the access mode may be changes by setting --access-mode. The supported values are RO, RWO and RWX.

Virt Migration

Test how the cluster and the storage backend handles mass migration of VMs

Test Sequence

The test runs the following sequence:

1. Create VirtualMachines with a node affinity rule that ties them to a specific worker node
2. Remove the node affinity rule to allow them to migrate
3. Call Migrate on all VirtualMachines

Tested StorageClass

If not --storage-class is not set, the test will:

1. Use the default StorageClass for Virtualization annotated with storageclass.kubevirt.io/is-default-virt-class
2. If does not exist, use general default StorageClass annotated with storageclass.kubernetes.io/is-default-class
3. If does not exist, fail the test before starting

Test Size Parameters

Users may control the workload sizes by passing the following arguments:

• --iterations - How many iterations to use when creating VirtualMachines in step 1
• --iteration-vms - How many VirtualMachines to create in each iteration in step 1
• --data-volume-count - Number of data volumes for each VM

!!! Note

The total number of `VirtualMachines` created is `--iterations` * `--iteration-vms`

Addition VirtualMachines load

The test can create additional VirtualMachines that will not be migrated to simulate load on the other nodes as well. By default, no additional VirtualMachines are created.

Set the following arguments to create load VirtualMachines:

• --load-iterations - Number of iterations to create load VMs
• --load-per-iteration - Number of VMs to create in each load VM iteration

!!! Note

The total number of load `VirtualMachines` created is `--load-vms-iterations` * `--iteration-load-vms`

Limiting migration requests load

By default, the test will be limited to 20 concurrent migrate request calls. Users can set a different value by passing --migration-qps.

!!! Note

This parameter limits the concurrent migration requests, not actual concurrent migrations

Initial Worker Node

The worker node on which all VMs are scheduled and migrated from can be set by passing the --worker-node parameter.

If not set, the test will randomly choose one

Test Namespace

All VirtualMachines are created in the same namespace.

By default, the namespace is virt-migation. Set it by passing --namespace (or -n)

DataVolume Clone

Test the capacity and performance of creating multiple data volumes that are clones of a single data volume

Test Sequence

The test runs the following sequence:

1. Create a DataVolume based on a container disk image
2. If the dataImportCronSourceFormat field of the StorageProfile status is set to snapshot, or --use-snapshot is set to true, create a VolumeSnapshot of the DataVolume
3. Create a DataSource, setting the source field to either the VolumeSnapshot (if was created) or the DataVolume
4. Create DataVolumes based on the DataSource. The creation process is divided into iterations

Tested StorageClass

By default, the test will use the default StorageClass. To use a different one, use --storage-class to provide a different name.

If --use-snapshot is explicitly set to true a corresponding VolumeSnapshotClass using the same provisioner must exist. Otherwise, the test will check the StorageProfile for the StorageClass and act accordingly.

Test Namespace

All the resources are created in the same namespace.

By default, the namespace is dv-clone. Set it by passing --namespace (or -n)

Test Container Disk Image

Users may set the container disk image used as a source to the base DataVolume using the --container-disk parameter. When setting --container-disk, make sure that it can fit to the default Volume Size 1Gi or set a new size by passing --datavolume-size.

If not passed, the test will use quay.io/yblum/tiny_image:latest for the the container disk image.

Test Size Parameters

Users may control the workload sizes by passing the following arguments:

• --iteration - Number of iterations to run in step 4
• --iteration-clones - Number of DataVolumes to create in each iteration of step 4

Volume Access Mode

By default, volumes are created with ReadWriteMany access mode as this is the recommended configuration for VirtualMachines. If not supported, the access mode may be changes by setting --access-mode. The supported values are RO, RWO and RWX.

EVPN Workload

This workload tests EVPN (Ethernet VPN) scenarios using Cluster User Defined Networks (CUDN). EVPN provides Layer 2 and Layer 3 VPN services over an IP/MPLS network, enabling efficient multi-tenant networking in OpenShift clusters.

Scenarios

The EVPN workload supports three scenarios, each designed to test different network traffic patterns:

East-West Scenario (--scenario east-west)

Tests pod-to-pod communication within the cluster using EVPN Layer 2 networks.

• Creates webserver deployments and services within each CUDN namespace
• Creates curl client deployments that communicate with the webservers
• Configures EVPN with Layer 2 topology and macVRF with routeTarget
• Traffic flows between pods within the same CUDN network

Use case: Validate internal cluster networking performance and EVPN Layer 2 overlay functionality.

North-South Scenario (--scenario north-south)

Tests pod-to-external communication using EVPN with both MAC and IP VRF.

• Creates only curl client deployments that communicate with an external webserver
• Configures EVPN with Layer 2 topology, macVRF and ipVRF
• Traffic flows from pods inside the cluster to an external endpoint
• Requires --external-webserver-ip and --external-webserver-port flags

Use case: Validate north-south traffic routing through EVPN with IP VRF for external connectivity.

North-South L3 Scenario (--scenario north-south-l3)

Tests pod-to-external communication using EVPN Layer 3 networks.

• Creates only curl client deployments that communicate with an external webserver
• Configures EVPN with Layer 3 topology and ipVRF only
• Traffic flows from pods inside the cluster to an external endpoint using Layer 3 routing
• Requires --external-webserver-ip and --external-webserver-port flags

Use case: Validate north-south traffic routing through EVPN Layer 3 networks for external connectivity.

CUDN Configuration

All scenarios use a single unified cudn.yml template that dynamically generates the appropriate CUDN configuration based on the --scenario flag:

Scenario	Network Topology	VRF Configuration	Use Case
east-west	Layer2	macVRF with routeTarget	Internal pod-to-pod traffic
north-south	Layer2	macVRF + ipVRF	External traffic via L2 overlay
north-south-l3	Layer3	ipVRF only	External traffic via L3 routing

The CUDN template uses Go template conditionals to generate scenario-specific configurations:

network:
{{ if eq .scenario "north-south-l3" }}
    topology: Layer3
    layer3:
      role: Primary
      subnets:
        - cidr: <generated-cidr>
          hostSubnet: 24
    evpn:
      vtep: evpn-vtep
      ipVRF:
        vni: <l3vni>
{{ else if eq .scenario "north-south" }}
    topology: Layer2
    layer2:
      role: Primary
      subnets:
        - <generated-cidr>
    evpn:
      vtep: evpn-vtep
      macVRF:
        vni: <l2vni>
      ipVRF:
        vni: <l3vni>
{{ else }}  # east-west (default)
    topology: Layer2
    layer2:
      role: Primary
      subnets:
        - <generated-cidr>
    evpn:
      vtep: evpn-vtep
      macVRF:
        vni: <l2vni>
        routeTarget: "65000:<l2vni>"
{{ end }}

Configuration Flags

Flag	Description	Default
--scenario	Test scenario: east-west, north-south, or north-south-l3	east-west
--iterations	Number of job iterations (namespaces)	10
--namespaces-per-cudn	Number of namespaces sharing the same CUDN	1
--external-webserver-ip	External webserver IP (required for north-south scenarios)	-
--external-webserver-port	External webserver port (required for north-south scenarios)	-
--connection-timeout	Timeout for external webserver reachability check	10s
--skip-reachability-check	Skip the external webserver reachability validation	false
--create-ext-frr-vrf	Setup external FRR VRF before workload and cleanup after completion	false
--l3vni-start	Starting L3 VNI number for EVPN configuration	100
--pod-ready-threshold	Pod ready timeout threshold	2m
--metrics-profile	Comma separated list of metrics profiles	metrics.yml

Usage Examples

East-West Scenario (Internal Traffic)

# Test internal pod-to-pod communication with 10 namespaces
kube-burner-ocp evpn --scenario east-west --iterations 10

North-South Scenario (External Traffic with L2+L3)

# Test external connectivity - requires external webserver. "--create-ext-frr-vrf" runs the external webserver on the local host and does other necessary configuration
kube-burner-ocp evpn --scenario north-south \
  --create-ext-frr-vrf \
  --external-webserver-ip 172.27.187.82 \
  --external-webserver-port 80 \
  --iterations 10

North-South L3 Scenario (External Traffic with L3 only).

# Test Layer 3 external connectivity. "--create-ext-frr-vrf" runs the external webserver on the local host and does other necessary configuration
kube-burner-ocp evpn --scenario north-south-l3 \
  --create-ext-frr-vrf \
  --external-webserver-ip 172.27.187.82 \
  --external-webserver-port 80 \
  --iterations 10

Workload Jobs

The EVPN workload executes the following jobs in sequence:

1. evpn-create-namespaces: Creates namespaces with required labels for CUDN
2. evpn-create-cudn: Creates ClusterUserDefinedNetwork resources using the unified cudn.yml template
3. evpn-route-advertisements: Creates RouteAdvertisement resources for the CUDNs
4. evpn-{scenario}: Creates the workload pods (webserver + curl for east-west, or curl only for north-south)

Template Files

The EVPN workload uses the following template files, optimized to minimize duplication:

Template	Description
evpn.yml	Main workload configuration with scenario-based conditionals
cudn.yml	Unified CUDN template supporting all three scenarios
curl-deployment.yml	Curl client deployment with configurable webserver target
webserver-deployment.yml	Webserver deployment (east-west only)
webserver-service.yml	Webserver service (east-west only)
ra.yml	RouteAdvertisement template
configmap.yml	ConfigMap for namespace initialization
setup_external_frr_vrf.sh	Script to setup external FRR VRF (used with --create-ext-frr-vrf)
cleanup_external_frr_vrf.sh	Script to cleanup external FRR VRF (used with --create-ext-frr-vrf)

External FRR VRF Setup

For north-south scenarios, you may need to set up an external FRR router with VRF configuration to simulate external connectivity. The --create-ext-frr-vrf flag automates this setup:

• Setup (setup_external_frr_vrf.sh): Runs before CUDN creation (evpn-create-cudn job)

  • Automatically detects the OCP node subnet (supports /8, /16, /24 and other CIDR masks)
  • Finds local IP on the same network as OCP nodes
  • Deploys FRR using the frr-k8s demo
  • Configures BGP EVPN with all cluster nodes
  • Creates VTEP with dynamically detected node subnet CIDR
  • Sets up VRFs (number matches JOB_ITERATIONS / NAMESPACES_PER_CUDN)
  • L3 VNI formula matches cudn.yml: VNI = l3vni_start + iteration (configurable via --l3vni-start)
  • Assigns the external webserver IP as anycast IP on each VRF
  • Starts an Nginx webserver on the host network

• Cleanup (cleanup_external_frr_vrf.sh): Runs after the last job (evpn-{scenario})

  • Stops the Nginx container
  • Removes FRR BGP VRF configurations
  • Deletes network interfaces (SVIs, VRFs)
  • Removes base bridge and VXLAN infrastructure
  • Deletes the VTEP resource
  • Cleans up frr-k8s directory

# Run north-south scenario with automatic FRR VRF setup
kube-burner-ocp evpn --scenario north-south \
  --external-webserver-ip 172.27.187.82 \
  --external-webserver-port 80 \
  --create-ext-frr-vrf

# With custom L3 VNI start
kube-burner-ocp evpn --scenario north-south \
  --external-webserver-ip 172.27.187.82 \
  --external-webserver-port 80 \
  --create-ext-frr-vrf \
  --l3vni-start 200

CUDN BGP Workload

This workload tests BGP route exchange import and export scenarios for the CUDNs.

Assumptions in this workload:

1. Kube burner must be running on a Linux host
2. Kube burner should be running on the host which is not used as a bastion host to deploy OCP cluster. a) Routes in the CUDN gateway router when we have the same host as the deployment host and kube burner host

sh-5.1# ovn-nbctl lr-route-list GR_cluster_udn_cudn.0_e34-h14-000-r650.rdu2.scalelab.redhat.com
IPv4 Routes
Route Table <main>:
              20.0.2.0/24             192.168.0.1 dst-ip rtoe-GR_cluster_udn_cudn.0_e34-h14-000-r650.rdu2.scalelab.redhat.com
                0.0.0.0/0               192.168.0.1 dst-ip rtoe-GR_cluster_udn_cudn.0_e34-h14-000-r650.rdu2.scalelab.redhat.com

b) Routes in the CUDN gateway router when the deployment host and the kube burner host is different

sh-5.1# ovn-nbctl lr-route-list GR_cluster_udn_cudn.0_e34-h14-000-r650.rdu2.scalelab.redhat.com
IPv4 Routes
Route Table <main>:
              20.0.2.0/24             192.168.0.145 dst-ip rtoe-GR_cluster_udn_cudn.0_e34-h14-000-r650.rdu2.scalelab.redhat.com
                0.0.0.0/0               192.168.0.1 dst-ip rtoe-GR_cluster_udn_cudn.0_e34-h14-000-r650.rdu2.scalelab.redhat.com

In case of same host i.e scenario a, ping reply will reach the kube burner even if the route "20.0.2.0/24" is not added, but using default route "0.0.0.0/0 192.168.0.1". Our purpose of the testing is to verify if the external route "20.0.2.0/24" is properly added or not in CUDN's gateway router. So we want the ping test to fail if this route is not correctly imported.

3. An external FRR will be running on the same host where the kube burner is running. a) Here kube burner is the generator of the external routes. External FRR imports these routes into the OCP cluster through internal FRR and OVN. b) Also when external FRR routes receive the routes from the OCP cluster, kube burner validates them.
4. External FRR will be created by the user and configured to pair up with OCP cluster's OVN internal FRR routers
5. External FRR additionally configured to advertise the host routes i.e

vtysh
configure terminal
router bgp 64512
redistribute static
redistribute connected
end
write

Unlike a UDN network, a CUDN network will be cluster scoped and can be used by multiple namespaces. However we restrict this to one namespace by default as our aim here is testing BGP route exchange.

This workload defines multiple jobs as per CUDN requirement. Some of the requirements:

1. Namespace with label "k8s.ovn.org/primary-user-defined-network" before the CUDN creation
2. OVN should create all the necessary resources for CUDN before a pod (or VM) is created on it. Currently we don't have a mechanism to detect if the OVN has created all CUDN's resources. So we are using separate jobs for CUDN and pods/VMs with jobPause. Workload defines one pod or VM per CUDN or, alternatively, --cidrs-per-cudn allows for multiple pods (or VMs) per subnet per CUDN.
3. RouteAdvertiments CRD selecting the CUDN. We use 1:1 RA:CUDN mapping.

As we want to measure BGP route exchange latency, this workload skips measurements for all the resources except RouteAdvertisements.

This workload has 2 BGP route exchange scenarios 1) route export scenaio 2) route import scenario When a RouteAdvertisement is created, it advertises the selected CUDN's subnet to outside cluster. However, an external route is imported to the CUDN's gateway router only when it is advertised by the routeAdvertisment. Hence RouteAdvertisemnt for CUDN is mandatory for both export and import scenarios.

Sequent of events during workload execution

1. Job1 creates namespaces
2. Job2 creates CUDNs
3. Job3 creates Pods (or VMs if --vm flag is used)
4. When Job4 execution starts, Kube burner calls start measurement. RouteAdvertisment Latency measurement code then a. Maintains a list of CUDN subnets and the pod addresses. b. Starts export scenario i) Main thread subscribes (through routeCh channel) to kernel's netlink sockets for route monitoring ii) Starts export workers, which read from the subscribed routeCh channel iii) Registers an informer for notifying router advertisement resource creation events
5. Kube burner creates RouteAdvertisments for CUDNs a. Kubernetes API notify the RouteAdvertisment resource to the listening kube burner b. OVN using the internal FRR advertises this route to the outside cluster (i.e to external FRR) c. External FRR adds the routes to the host. Kernel notifies the routes to the kube burner using netlink sockets
6. RouteAdvertisment Latency measurement code (watchers and export worker threads) then a. Records RouteAdvertisement name and creation timestamp when routeadvertisement resource is detected by the API b. For each route notified by the kernel, pings the corresponding CUDN's pod and records ping success timestamp
7. Kube burner calls stop measurements. RouteAdvertisment Latency measurement code then a. Waits for export scenario completion b. Starts import scenario i) Main thread creates interfaces, prepares IP addresses and pods to ping (which are needed for worker threads) ii) Starts import workers iii) Import workers add IP addresses on the interfaces iv) Kernel creates the linux route for the added IP address. Then external FRR router exports this to the cluster. v) OVN imports this route into the CUDN's gateway router. vi) Import worker pings the CUDN pod using the CUDN pod address as destination address and above added IP address as the source address. vii) Import worker records the ping success timestamp c. Waits for import scenario complettion
8. kube burner indexes all the latency measurements

Configuration Flags

Flag	Description	Default
--iterations	Number of job iterations (namespaces)	10
--namespaces-per-cudn	Number of namespaces sharing the same cluster UDN	1
--cidrs-per-cudn	Number of CIDRs per CUDN	1
--frr-external-ip	IP address of the external FRR router (required)	-
--vm	Deploy a VM for the test instead of a pod	false
--layer2	Use Layer2 topology for CUDNs instead of Layer3	false
--ra-connection-check	RA connection check mode (see below)	import,export
--metrics-profile	Comma separated list of metrics profiles to use	metrics.yml

RA Connection Check

By default, the workload runs all connectivity checks (accepted latency plus export and import ping tests). The --ra-connection-check flag controls which ping-based connectivity checks are executed:

Value	Accepted Latency	Export Ping	Import Ping
none	Yes	No	No
export	Yes	Yes	No
import	Yes	No	Yes
import,export (default)	Yes	Yes	Yes

• none: Only measures the time from RA creation to Accepted=True. No ping tests are performed. Export latency and netlink route latency are reported as -1.
• export: Measures accepted latency and validates that CUDN subnets are correctly advertised to the external host by pinging CUDN pods from outside the cluster.
• import: Measures accepted latency and validates that externally generated routes are correctly imported into CUDN gateway routers by pinging CUDN pods using newly created source addresses.
• import,export: Runs all checks (equivalent to the full workload behavior). Both orderings (export,import and import,export) are accepted, with or without spaces.

Usage Examples

Run both export and import ping tests (default):

kube-burner-ocp udn-bgp --iterations=10 --frr-external-ip=192.168.0.1

Run only export ping tests:

kube-burner-ocp udn-bgp --iterations=10 --frr-external-ip=192.168.0.1 --ra-connection-check=export

Measure only accepted latency:

kube-burner-ocp udn-bgp --iterations=10 --frr-external-ip=192.168.0.1 --ra-connection-check=none

Testing with VMs

The CUDN BGP workload supports testing with Virtual Machines (VMs) instead of pods. This allows you to measure BGP route advertisement latency for VMs running on CUDN networks, which is useful for validating virtualization workloads in OpenShift Virtualization environments.

Prerequisites

• OpenShift Virtualization (KubeVirt) must be installed and operational on the cluster
• The cluster must support CUDN networks (OCP 4.18+)
• External FRR router configured as described in the CUDN BGP Workload section

Usage

To run the workload with VMs instead of pods, use the --vm flag:

kube-burner-ocp udn-bgp --iterations=10 --vm

When the --vm flag is enabled:

• The workload creates VirtualMachine resources instead of Pod resources in Job3
• Each VM is configured to use the CUDN network through the namespace label k8s.ovn.org/primary-user-defined-network
• VMs are created with a Fedora-based container disk image and automatically start an HTTP server on port 8080
• The measurement code works seamlessly with VMs since KubeVirt creates VirtualMachineInstances (VMIs) which are pods under the hood

Differences from Pod-based Testing

• Boot Time: VMs take longer to boot compared to pods, which may affect the overall test duration
• Resource Usage: VMs consume more resources (memory, CPU) than lightweight pods
• Network Behavior: Network behavior should be identical since both pods and VMIs use the same CUDN network attachment mechanism

RouteAdvertisement Latency Metrics

RouteAdvertisement accepted latency is always measured. Export and import ping latencies are calculated only when enabled via --ra-connection-check. When a scenario is not enabled, its latency and netlinkRouteLatency fields are reported as -1, and import scenario metrics are not emitted at all. These latency metrics are in ms. It can be enabled with:

  measurements:
  - name: podLatency

The metrics collected are route advertisement latency timeseries (raLatencyMeasurement) and six documents holding a summary with different route latency quantiles of ping test and netlink route detection latency (raLatencyQuantilesMeasurement).

One document, such as the following, is indexed per each internal (through Routeadvertisment CRD) and external route (adding ip address on dummy interface) created by the workload:

[
  {
    "timestamp": "2025-04-15T08:41:10Z",
    "metricName": "raLatencyMeasurement",
    "uuid": "2c8a64a8-0409-4d17-8643-c28db8216821",
    "jobName": "udn-bgp-route-advertisements",
    "routeAdvertisementName": "ra-0",
    "metadata": {
      "ocpMajorVersion": "4.19",
      "ocpVersion": "4.19.0-ec.3"
    },
    "scenario": "ExportRoutes",
    "acceptedLatency": 120,
    "latency": 10031,
    "netlinkRouteLatency": 10026
  },
  {
    "timestamp": "2025-04-15T08:42:20.060393739Z",
    "metricName": "raLatencyMeasurement",
    "uuid": "2c8a64a8-0409-4d17-8643-c28db8216821",
    "jobName": "udn-bgp-route-advertisements",
    "routeAdvertisementName": "20.0.1.1/24",
    "metadata": {
      "ocpMajorVersion": "4.19",
      "ocpVersion": "4.19.0-ec.3"
    },
    "scenario": "ImportRoutes",
    "latency": 13
  }
]

OLMv1 Benchmark

To evaluate the performance of Operator Lifecycle Manager v1 (OLMv1), this benchmark initiates the creation of a series of ClusterCatalog and ClusterExtension custom resources. These resources serve as a representative workload to simulate realistic catalog and operator deployment operations within the cluster. Each ClusterExtension internally triggers the resolution and unpacking of an operator bundle through the configured catalog.

During the test execution, the system actively monitors and collects detailed resource usage metrics — specifically CPU and memory consumption — for the pods running in both the openshift-catalogd and openshift-operator-controller namespaces. These namespaces are critical components of the OLMv1 infrastructure, responsible for catalog resolution and operator deployment, respectively.

In addition to standard resource metrics, the performance test is also capable of gathering pprof profiling data, which provides in-depth runtime insights such as goroutine activity, heap allocations, and CPU profiles. This profiling capability is contingent on the user explicitly enabling pprof endpoints within the OLMv1 components prior to test execution. Once enabled, the test can automatically scrape and archive the profiling data for further analysis, facilitating root cause investigation and performance optimization of the OLMv1 stack.

Environment Requirements

OCP 4.18(OLMv1 GA) and above

Kueue Operator workloads

The workloads of this family are used to exercise the Kueue Operator by creating pods or jobs depending on the workload. In order to run these jobs, Kueue Operator and Kueue CR should be installed on the target cluster.

kueue-operator-jobs

This workload creates jobs in a single namespace that are handled by a single ClusterQueue with pre-defined CPU, memory and pod quotas. Key measurements are Kueue admission wait time, job start and completion latencies.

kueue-operator-jobs-shared

This workload creates jobs in multiple namespaces that are handled by 10 shared ClusterQueues with pre-defined pod quotas. Key measurements are Kueue admission wait time, job start and completion latencies.

kueue-operator-pods

This workload creates pods in a single namespace that are handled by a single ClusterQueue with pre-defined CPU, memory and pod quotas. Key measurements are Kueue admission wait time and pod ready latency.

MaaS Gateway Performance workload

This workload measures the latency and throughput overhead of the MaaS (Models-as-a-Service) AI Gateway on OpenShift. It runs A/B tests comparing direct-to-simulator performance (baseline) against gateway-routed performance across multiple AI providers, payload sizes, and concurrency levels.

The workload deploys a multi-provider LLM simulator (llm-d-inference-sim), registers it as 5 ExternalModel CRs with corresponding HTTPRoutes and Secrets, then runs GuideLLM benchmarks as native Kubernetes Jobs. The GuideLLM container image (ghcr.io/cloud-bulldozer/guidellm-results-parser) includes both the benchmark tool and a results parser that indexes directly to OpenSearch.

The job config uses Go template loops over providers, payload sizes, and concurrency levels — the full test matrix is generated dynamically from CLI flags with zero code duplication.

Architecture

kube-burner-ocp maas-gateway-perf
  Job 1: deploy-test-infra
    Simulator Deployment + Service
    5x ExternalModel CRs (openai, azure-openai, bedrock, anthropic, vertexai)
    5x HTTPRoutes (path-based + header-based routing)
    5x Secrets (dummy provider credentials)
  Jobs 2..N: baseline-{size}-{provider}-c{rate}
    GuideLLM K8s Job → direct to simulator (no gateway)
  Jobs N+1..2N: gateway-{size}-{provider}-c{rate}
    GuideLLM K8s Job → through MaaS gateway (BBR)
  Last job: cleanup

Flags

Flag	Description	Default
--gateway-host	Full gateway service URL	(required)
--simulator-host	Simulator service URL	(required)
--providers	Comma-separated provider model names	gpt-4o-openai,claude-sonnet-anthropic
--payload-sizes	Comma-separated sizes: small(32/64), medium(256/512), large(1024/1024), very-large(2048/2048)	small,medium
--concurrency-levels	Comma-separated concurrency levels	8,32,64,128,512
--job-deadline	Hard per-Job deadline (activeDeadlineSeconds); 0 derives it as ceil(samples/parallelism) × (warmup + benchmark-duration) × 3 + 5m	0 (derived)
--benchmark-duration	Duration of each benchmark run	90s
--warmup	Warmup period discarded from results	30s
--guidellm-image	GuideLLM container image with parser	ghcr.io/cloud-bulldozer/guidellm-results-parser:v0.0.5
--samples	Benchmark samples per Job (K8s Job completions)	3
--parallelism	K8s Job parallelism	1
--pause	Pause after each benchmark job completes	10s
--metrics-profile	Metrics profiles to use	maas-gateway-perf-metrics.yml

Test Matrix

Total GuideLLM Jobs = providers × payload_sizes × concurrency_levels × 2 (baseline + gateway)

With defaults (2 providers × 2 sizes × 5 levels × 2 modes) = 40 GuideLLM Jobs, each running --samples completions.

Available Providers

Provider	API Translation	Expected Overhead
gpt-4o-openai	Passthrough	Minimal
gpt-4o-azure	Path rewrite	Minimal
gpt-4o-bedrock	Passthrough	Minimal
claude-sonnet-anthropic	Full body translation (OpenAI → Anthropic Messages API)	Higher
claude-sonnet-vertex	Full body translation (OpenAI → Vertex AI)	Higher

Metrics Collected

The maas-gateway-perf-metrics.yml profile scrapes:

• Istio gateway request duration histograms (p50/p95/p99) by destination workload
• Authorino auth evaluation duration (p95) by authconfig and evaluator
• Container CPU/memory for gateway, payload-processing, simulator, and maas-api pods
• Limitador authorized/limited call rates
• Gateway request rates by response code

Usage Examples

Minimal run with required flags:

kube-burner-ocp maas-gateway-perf \
  --gateway-host=http://maas-default-gateway.openshift-ingress.svc.cluster.local:80 \
  --simulator-host=http://llm-d-inference-sim.maas-perf-test.svc.cluster.local:8000

Full 5-provider matrix with ES indexing:

kube-burner-ocp maas-gateway-perf \
  --gateway-host=http://maas-default-gateway.openshift-ingress.svc.cluster.local:80 \
  --simulator-host=http://llm-d-inference-sim.maas-perf-test.svc.cluster.local:8000 \
  --providers=gpt-4o-openai,gpt-4o-azure,gpt-4o-bedrock,claude-sonnet-anthropic,claude-sonnet-vertex \
  --payload-sizes=small,medium,large,very-large \
  --concurrency-levels=2,4,8,16,32,64,128,256,512,1024 \
  --es-server=https://opensearch.example.com:9200 \
  --es-index=maas-gateway-perf

Quick smoke test:

kube-burner-ocp maas-gateway-perf \
  --gateway-host=http://maas-default-gateway.openshift-ingress.svc.cluster.local:80 \
  --simulator-host=http://llm-d-inference-sim.maas-perf-test.svc.cluster.local:8000 \
  --providers=gpt-4o-openai \
  --payload-sizes=small \
  --concurrency-levels=8,64 \
  --benchmark-duration=30s --warmup=10s --samples=1

Extract and customize the config:

kube-burner-ocp maas-gateway-perf --extract
ls maas-gateway-perf/
# external-model.yml  guidellm-job.yml  httproute.yml  maas-gateway-perf.yml
# secret.yml  simulator-deployment.yml  simulator-service.yml
vi maas-gateway-perf/maas-gateway-perf.yml  # Modify job config
kube-burner-ocp maas-gateway-perf \
  --gateway-host=... --simulator-host=...   # Run with modified config

KPIs

Metric	Source	Description
Request latency (mean/p50/p95/p99)	GuideLLM → OpenSearch	End-to-end request latency per provider/payload/concurrency
Requests per second	GuideLLM → OpenSearch	Throughput under load
TTFT	GuideLLM → OpenSearch	Time to first token
ITL	GuideLLM → OpenSearch	Inter-token latency
Gateway overhead	Derived (gateway − baseline)	Net latency added by the MaaS stack
Istio gateway latency	Prometheus → kube-burner	Gateway-level request duration
Authorino auth duration	Prometheus → kube-burner	Authentication evaluation time
Pod CPU/memory	Prometheus → kube-burner	Resource consumption of MaaS components

Berserker-load workload

The berserker-load workload deploys DaemonSets whose pods are evenly distributed across cluster nodes. Each pod runs multiple containers that produce four types of load: process spawning, file activity, listening network endpoints, and network connections. This makes it useful for stress-testing monitoring and security tooling that observe pod and cluster behavior.

Architecture

The workload creates namespaced iterations, each containing:

1. ConfigMaps: Configuration files that define the berserker workload parameters (endpoint distribution, network connection rates, process/file activity rates). The ConfigMap template is customizable via the BERSERKER_CONFIGMAP_TEMPLATE environment variable.

2. DaemonSets: Deployed across cluster nodes, each running containers that generate:

   • Process and file activity: Spawns processes and performs file operations at configurable rates
   • Endpoint load: Creates listening network endpoints with a configurable port distribution
   • Connection load: Generates network connections with configurable arrival/departure rates and connection counts

   The container definitions are customizable via the BERSERKER_CONTAINERS_FILE environment variable.

3. Services: ClusterIP services exposing HTTP (80) and HTTPS (443) ports, backed by the DaemonSet pods. The Service template is customizable via the BERSERKER_SERVICE_TEMPLATE environment variable.

Configuration Flags

• --job-iterations: Number of job iterations to create (default: 5)
• --job-pause: Duration to pause after creating resources (default: 0)
• --max-wait-timeout: Maximum time to wait for created objects to be ready (default: 12m)
• --pod-ready-threshold: Pod ready timeout threshold (default: 5m)
• --deletion-strategy: GC deletion mode (default: gvr)
• --daemonset-replicas: Number of berserker DaemonSets to create per namespace (default: 6)
• --service-replicas: Number of services per namespace (default: 6)
• --churn-cycles: Number of churn cycles, 0 = infinite when churn-duration > 0 (default: 0)
• --churn-duration: Churn duration, 0 disables churn (default: 0)
• --churn-delay: Time to wait between each churn (default: 10m)
• --churn-percent: Percentage of job iterations to churn each round (default: 80)
• --churn-mode: Churn mode: namespaces or objects (default: namespaces)

Build-Farm workload

The build-farm workload simulates a realistic build farm environment where build jobs are continuously created and churned. This workload is designed to stress-test the control plane by creating controllers that watch jobs and pods, while simultaneously creating and churning build jobs across multiple namespaces.

Architecture

The workload consists of two main components:

1. Build Farm Controller: Deployed in the build-farm-controllers namespace this controller simulate build farm operators that watch and manage build jobs.

2. Build Jobs: Created in build-farm-tenant namespaces with:

   • Service accounts, roles and role bindings for RBAC
   • Secrets (1.5x iterations per namespace)
   • ConfigMaps (0.5x iterations per namespace)
   • Small build jobs (default 80% of total) - lighter weight builds
   • Large build jobs (default 20% of total) - heavier builds
   • Job distribution is configurable via --small-job-percent flag
   • Support metadata updates and label distribution for watchers sharding

Churn Behavior

By default, churning is enabled to simulate realistic build farm behavior where jobs are continuously created, completed and cleaned up. The churn configuration allows you to:

• Define the number of churn cycles (default: 5)
• Set the percentage of jobs to churn each round (default: 60%)
• Configure delays between churn cycles (default: 10 minutes)

Configuration Flags

The build-farm workload supports customization through command-line flags:

Controller configuration flags:

• --num-controllers: Number of controller replicas (default: 4)
• --num-threads: Number of threads per controller (default: 4)
• --enable-job-watcher: Enable job watcher (default: true)
• --enable-pod-watcher: Enable pod watcher (default: true)
• --enable-jobs-listing: Enable periodic jobs listing (default: true)
• --enable-secrets-listing: Enable periodic secrets listing (default: false)
• --watcher-restart-interval: Watcher restart interval (default: 600s)
• --sleep-before-restart: Sleep duration before restarting watchers (default: 60s)
• --secrets-list-interval: Secrets list interval (default: 5s)
• --jobs-list-interval: Jobs list interval (default: 5s)
• --namespace-filter-regex: Namespace filter regex. Allows filtering namespaces for list operations

Build job configuration flags:

• --metadata-iterations: Number of metadata update iterations per job (default: 2)
• --metadata-iterations-delay: Delay between metadata iterations (default: 5s)
• --num-watchers: Number of watchers for label distribution (default: 32)
• --build-image: Container image for build simulation (default: "quay.io/prometheus/busybox")
• --small-job-percent: Percentage of small build jobs (0-100, large jobs get remainder) (default: 80)

Use Cases

This workload is particularly useful for:

• Simulating realistic build farm behavior with Jobs churn
• Measuring the impact of periodic list operations on the API server memory
• Testing metadata updates impact on etcd db size with frequent changes

Batch Churn workload

This workload creates kubelet watch pressure via volume-mounted secrets and configmaps. Kubelet's WatchingSecretManager / WatchingConfigMapManager opens one independent ListAndWatch goroutine per unique secret/configmap referenced by pods on a node. Per-pod unique objects multiply the watch count linearly with pod count; shared common objects add watch payload pressure.

The workload operates in two modes:

Batch-churn mode (default)

Each iteration creates the following objects in a new namespace:

• 1 common secret (7 files x 10KiB, matching real customer patterns).
• 2 common configmaps (256KiB each), shared across all pods.
• 2 unique secrets per pod (10 KV pairs, 24-char values) -- creates per-pod watch pressure.
• 2 unique configmaps per pod (same structure).
• 240 single-replica deployments, each mounting all common objects + its own unique objects.

Watch count per node = pods_on_node x (UNIQUE_SECRETS + UNIQUE_CMS) + COMMON_SECRETS + COMMON_CMS.

Churning is enabled by default (5 cycles, 100% namespace replacement per cycle), continuously destroying and recreating namespaces to stress the control plane.

An optional chaos hook can be triggered during load via CHAOS_ACTION: rollout (API server rollout), kill-apiserver, kill-node, or kill-etcd (force leader election).

Incremental load

Set INCREMENTAL_STEP_SIZE to enable kube-burner's native incrementalLoad. Namespaces are created in linear steps (adding stepSize namespaces each step) up to --iterations, with INCREMENTAL_STEP_DELAY between steps. This creates gradually increasing watch pressure so you can observe API server thread counts at each load level. Incremental load and churn are mutually exclusive -- when incremental load is active, churn is skipped.

Watcher-spam mode

Enabled by setting --set WATCHER_MODE=true. Opens thousands of LIST+WATCH streams directly from the kube-burner process against the API server -- no pods, no kubelet, no scheduling. Each watcher is a long-lived TLS connection. A small set of configmaps/secrets is created as watch targets.

Default watcher counts: 5000 secret watchers, 5000 configmap watchers, 2000 each for nodes, endpoints, service accounts, events, and pods (20k total).

Configuration flags

Flag / Variable	Description	Default
--iterations	Number of batch namespaces	10
--churn-cycles	Churn cycles to execute	5
--churn-duration	Per-cycle churn duration	5m
--churn-delay	Delay between churn cycles	30s
--churn-percent	Percent of namespaces churned per cycle	100
--set DEPLOYMENT_COUNT=<n>	Pods (single-replica deployments) per namespace	240
--set UNIQUE_SECRETS=<n>	Unique secrets per pod	2
--set UNIQUE_CMS=<n>	Unique configmaps per pod	2
--set UNIQUE_KV=<n>	Key-value pairs per unique secret/configmap	10
--set UNIQUE_KV_LEN=<n>	Char length of each unique value	24
--set COMMON_SECRETS=<n>	Shared secrets per namespace	1
--set COMMON_SECRET_FILES=<n>	Files per common secret	7
--set COMMON_SECRET_FILE_SIZE=<n>	Bytes per file in common secret	10240
--set COMMON_CMS=<n>	Shared configmaps per namespace	2
--set COMMON_CM_SIZE=<n>	Bytes per common configmap	262144
--set ENV_VARS=<n>	Large env vars per pod (inflates pod spec)	6
--set ENV_VAR_SIZE=<n>	Bytes per env var	900
--set POD_LABELS=<n>	Random labels per pod	10
--set POD_ANNOTATIONS=<n>	Random annotations per pod	10
--set INCREMENTAL_STEP_SIZE=<n>	Namespaces to add per step (0=disabled)

Usage examples

# Default load -- 10 namespaces x 240 pods with unique + common mounts
kube-burner-ocp init -c config.yml --iterations 10 --churn-cycles 5

# Incremental load -- add 2 namespaces per step up to 10
kube-burner-ocp init -c config.yml --iterations 10 \
  --set INCREMENTAL_STEP_SIZE=2 --set INCREMENTAL_STEP_DELAY=5m

# Kill the etcd leader node 60s after full load (5 rounds)
kube-burner-ocp init -c config.yml --iterations 10 --churn-cycles 5 \
  --set CHAOS_ACTION=kill-node --set CHAOS_DELAY=60 --set CHAOS_CYCLES=5

# Watcher-spam mode -- 20k direct watches + API server rollout
kube-burner-ocp init -c config.yml --iterations 1 \
  --set WATCHER_MODE=true \
  --set CHAOS_ACTION=rollout --set CHAOS_DELAY=60

Custom Workload: Bring your own workload

To kickstart kube-burner-ocp with a custom workload, init becomes your go-to command. This command is equipped with flags that enable to seamlessly integrate and run your personalized workloads. Here's a breakdown of the flags accepted by the init command:

$ kube-burner-ocp init --help
Runs custom workload

Usage:
  kube-burner-ocp init [flags]

Flags:
      --churn-cycles int               Churn cycles to execute
      --churn-delay duration           Time to wait between each churn (default 2m0s)
      --churn-duration duration        Churn duration (default 5m0s)
      --churn-mode string              Either namespaces, to churn entire namespaces or objects, to churn individual objects (default "namespaces")
      --churn-percent int              Percentage of job iterations that kube-burner will churn each round (default 10)
  -c, --config string                  Config file path or url
      --deletion-strategy string       GC deletion mode, default deletes entire namespaces and gvr deletes objects within namespaces before deleting the parent namespace (default "default")
  -h, --help                           help for init
      --iterations int                 Job iterations. Mutually exclusive with '--pods-per-node'
      --iterations-per-namespace int   Iterations per namespace (default 1)
      --namespaced-iterations          Namespaced iterations (default true)
      --pods-per-node int              Pods per node. Mutually exclusive with '--iterations'
      --service-latency                Enable service latency measurement

Creating a custom workload for kube-burner-ocp is a seamless process, and you have the flexibility to craft it according to your specific needs. Below is a template to guide you through the customization of your workload:

---
indexers:
  - esServers: ["{{.ES_SERVER}}"]
    insecureSkipVerify: true
    defaultIndex: {{.ES_INDEX}}
    type: opensearch
global:
  gc: {{.GC}}
  gcMetrics: {{.GC_METRICS}}
  measurements:
    - name: <metric_name>
      thresholds:
        - <threshold_key>: <threshold_value>

jobs:
  - name: <job_name>
    namespace: <namespace_name>
    jobIterations: <number of iterations>
    qps: {{.QPS}}     # Both QPS and BURST can be specified through the CLI
    burst: {{.BURST}}
    namespacedIterations: <bool>
    podWait: <bool>
    waitWhenFinished: <bool>
    preLoadImages: <bool>
    namespaceLabels:
      <namespaceLabels_key>: <namespaceLabels_value>
    objects:

      - objectTemplate: <template_config>
        replicas: <replica_int>
        inputVars:
          <inputVar1>:<inputVar1_value>

You can start from scratch or explore pre-built workloads in the /config folder, offering a variety of examples used by kube-burner-ocp. Dive into the details of each section in the template to tailor the workload precisely to your requirements. Experiment, iterate, and discover the optimal configuration for your workload to seamlessly integrate with kube-burner-ocp.

Index

Just like the regular kube-burner, kube-burner-ocp also has an indexing functionality which is exposed as index subcommand.

$ kube-burner-ocp index --help
If no other indexer is specified, local indexer is used by default

Usage:
  kube-burner-ocp index [flags]

Flags:
  -m, --metrics-profile string     Metrics profile file (default "metrics.yml")
      --metrics-directory string   Directory to dump the metrics files in, when using default local indexing (default "collected-metrics")
  -s, --step duration              Prometheus step size (default 30s)
      --start int                  Epoch start time
      --end int                    Epoch end time
  -j, --job-name string            Indexing job name (default "kube-burner-ocp-indexing")
      --user-metadata string       User provided metadata file, in YAML format
  -h, --help                       help for index

Metrics-profile type

By specifying --profile-type, kube-burner can use two different metrics profiles when scraping metrics from prometheus. By default is configured with both, meaning that it will use the regular metrics profiles bound to the workload in question and the reporting metrics profile.

When using the regular profiles (metrics-aggregated or metrics), kube-burner scrapes and indexes metrics timeseries.

The reporting profile is very useful to reduce the number of documents sent to the configured indexer. Thanks to the combination of aggregations and instant queries for prometheus metrics, and 4 summaries for latency measurements, only a few documents will be indexed per benchmark. This flag makes possible to specify one or both of these profiles indistinctly.

Grafana Dashboards

Grafana dashboards are available to visualize kube-burner metrics and performance data. See the dashboards documentation for details on how to import and use the dashboards with their corresponding metrics profiles.

Customizing workloads

It is possible to customize any of the above workload configurations by extracting, updating, and finally running it:

$ kube-burner-ocp node-density --extract
$ ls
alerts.yml  metrics.yml  node-density.yml  pod.yml  metrics-report.yml
$ vi node-density.yml                               # Perform modifications accordingly
$ kube-burner-ocp node-density --pods-per-node=100  # Run workload