Runtime Zero

Enabling Pod Monitoring on VKS Clusters via the VCF Operations API

2026-08-20T00:00:00.000Z

Automatic observability of VKS Clusters in VCF Operations

Part of the VKS addon catalog are Prometheus and Telegraf. In the past, to enable the metrics collection (and shipping to VCF Operations) those had to be installed manually. An excellent guide on how to do this can be found here: Monitoring VKS Clusters in VCF Operations.

With the introduction of VCF 9.1 the platform is using the new addon management framework to automate the installation of the required packages similar to how Cluster Autoscaler installation is handled. For all new VKS clusters, this is enabled by default - no extra steps necessary. Here is an example of the annotations that get added by the system:

If for some reason you don't want that, you can set the annotation to disabled. The official documentation provides more detail on the behaviour: Monitoring VKS Clusters Using VCF Operations.

Motivation

By default data is aggregated at the node/cluster level and namespace, deployment and replicaset level as well, as can be seen in this example:

Actual pod and container metrics are not enabled by default. This makes sense - pod/container data is often short-lived due to their ephemeral nature. But sometimes this level of detail is necessary. VCF Operations 9.1 lets you toggle Pod Monitoring for a VKS (vSphere Kubernetes Service) cluster from the UI, once that cluster has been added to the system.

Since this is a manual UI task, I wanted to find out the API way to do this, in preparation for further automation down the line - like toggle this setting right from within VCF Automation.

VCF Operations exposes a REST API endpoint called suite-api - this is the same, officially supported integration API documented at developer.broadcom.com/xapis/vcf-operations-api, and it's also self-hosted on every appliance at:

https://<vcf-ops-host>/suite-api/docs/rest/index.html
https://<vcf-ops-host>/suite-api/doc/swagger-ui.html

The built-in Swagger UI makes it especially easy to try out the API and start experimenting. What is not spelled out anywhere in the docs is which field controls pod monitoring for a VKS cluster. That part had to be reverse-engineered from the live resource model.

Step 1 - Authenticate

Every call needs a bearer token from /api/auth/token/acquire:

curl -sk -X POST https://<vcf-ops-host>/suite-api/api/auth/token/acquire \
  -H "Content-Type: application/json" -H "Accept: application/json" \
  -d '{"username":"<username>","password":"<password>"}'

Response:

{"token":"978f65d0-...::f3b9b83c-...", "validity":..., "expiresAt":"...", "roles":[]}

Send it back as a header on every subsequent request:

Authorization: OpsToken <token>

Step 2 - Find the adapter kind that owns Kubernetes objects

VCF Operations models everything as adapter kinds -> resource kinds -> resources. To find where VKS clusters live, list adapter kinds and look for one whose resource kinds mention Kubernetes objects:

curl -sk https://<host>/suite-api/api/adapterkinds \
  -H "Authorization: OpsToken $TOKEN" -H "Accept: application/json"

The adapter kind SupervisorAdapter ("vSphere Supervisor") turned up with resource kinds including GuestCluster, SupervisorCluster, KubernetesPod, KubernetesNamespace, etc. GuestCluster is what the UI labels VKS Cluster - confirmed by describing it:

curl -sk https://<host>/suite-api/api/adapterkinds/SupervisorAdapter/resourcekinds/GuestCluster \
  -H "Authorization: OpsToken $TOKEN" -H "Accept: application/json"

POD_CONTAINER_MONITORING immediately stands out - that's the field the "Enable Pod Monitoring" toggle in the UI is writing to.

{
  "key": "GuestCluster",
  "name": "VKS Cluster",
  "adapterKind": "SupervisorAdapter",
  "resourceIdentifierTypes": [
    {"name": "GUEST_CLUSTER_UUID", "dataType": "STRING", "isPartOfUniqueness": true},
    {"name": "MONITORING_ENABLED", "dataType": "STRING", "isPartOfUniqueness": false},
    {"name": "POD_CONTAINER_MONITORING", "dataType": "STRING", "isPartOfUniqueness": false},
    {"name": "SUPERVISOR_UUID", "dataType": "STRING", "isPartOfUniqueness": true},
    {"name": "VC_UUID", "dataType": "STRING", "isPartOfUniqueness": true}
  ]
}

Listing actual GuestCluster resources across several VKS clusters confirmed it: clusters with pod monitoring enabled in the UI had POD_CONTAINER_MONITORING = "true", and the one cluster with it turned off had "false".

curl -sk "https://<host>/suite-api/api/resources?resourceKind=GuestCluster&adapterKind=SupervisorAdapter&name=metrics-test" \
  -H "Authorization: OpsToken $TOKEN" -H "Accept: application/json"

Here is a response for one cluster (with some data trimmed out that is noise for our purposes):

{
  "resourceList": [
    {
      "resourceKey": {
        "name": "metrics-test",
        "adapterKindKey": "SupervisorAdapter",
        "resourceKindKey": "GuestCluster",
        "resourceIdentifiers": [
          {
            "identifierType": {"name": "GUEST_CLUSTER_UUID", "dataType": "STRING", "isPartOfUniqueness": true},
            "value": "b6cc361b-a0d3-4757-8d77-7a7977be9f75"
          },
          {
            "identifierType": {"name": "MONITORING_ENABLED", "dataType": "STRING", "isPartOfUniqueness": false},
            "value": "true"
          },
          {
            "identifierType": {"name": "POD_CONTAINER_MONITORING", "dataType": "STRING", "isPartOfUniqueness": false},
            "value": "false"
          },
          {
            "identifierType": {"name": "SUPERVISOR_UUID", "dataType": "STRING", "isPartOfUniqueness": true},
            "value": "6e227f64-7211-48c5-b201-1e25ce19678e"
          },
          {
            "identifierType": {"name": "VC_UUID", "dataType": "STRING", "isPartOfUniqueness": true},
            "value": "f7839017-1f73-4122-b27c-3ed8450729e6"
          }
        ]
      },
      "identifier": "38ff122b-8776-4d70-8b3c-fac80c9825c3",
      "links": [
        {"href": "/suite-api/api/resources/38ff122b-8776-4d70-8b3c-fac80c9825c3", "rel": "SELF", "name": "linkToSelf"}
      ]
    }
  ]
}

Worth noting is the identifier (top-level, 38ff122b-8776-4d70-8b3c-fac80c9825c3) - the ID needed for the next step.

Step 3 - Flip it

The write path is PUT /suite-api/api/resources. The body is the full resource, with the identifier's value changed, and the resource's own ID passed in a field literally called identifier:

# file: update_resource.json
{
  "identifier": "<resource-id>",
  "resourceKey": {
    "name": "<cluster-name>",
    "adapterKindKey": "SupervisorAdapter",
    "resourceKindKey": "GuestCluster",
    "resourceIdentifiers": [
      {"identifierType": {"name": "GUEST_CLUSTER_UUID", "dataType": "STRING", "isPartOfUniqueness": true}, "value": "..."},
      {"identifierType": {"name": "MONITORING_ENABLED", "dataType": "STRING", "isPartOfUniqueness": false}, "value": "true"},
      {"identifierType": {"name": "POD_CONTAINER_MONITORING", "dataType": "STRING", "isPartOfUniqueness": false}, "value": "true"},
      {"identifierType": {"name": "SUPERVISOR_UUID", "dataType": "STRING", "isPartOfUniqueness": true}, "value": "..."},
      {"identifierType": {"name": "VC_UUID", "dataType": "STRING", "isPartOfUniqueness": true}, "value": "..."}
    ]
  }
}

curl -sk -X PUT "https://<host>/suite-api/api/resources" \
  -H "Authorization: OpsToken $TOKEN" -H "Content-Type: application/json" -H "Accept: application/json" \
  -d @update_resource.json

A follow-up GET on the resource or check in the UI confirmed the change has been applied successfully, with pod and container details now showing up in VCF Operations:

Putting it into a Python script

Broadcom put together an official Python SDK for VCF: vcf-sdk, a meta-package that pulls in per-product client libraries - vmware-vcenter, vcf-nsx, vcf-operations, etc. The one we need is vcf-operations:

Since I am not much of a programmer, I utilized the SDK and AI to help me out with putting this together. The code can be found here: Python client

Install and usage:

python3 -m venv .venv && source .venv/bin/activate
pip install vcf-operations==9.1.0.0

# list clusters and current state
python3 set_pod_monitoring.py --host ops-a.site-a.vcf.lab --username admin --list

# enable on a named cluster
python3 set_pod_monitoring.py --host ops-a.site-a.vcf.lab --username admin --cluster bookstore-app --enable

# no --cluster given -> interactive picker
python3 set_pod_monitoring.py --host ops-a.site-a.vcf.lab --username admin --enable

The password is prompted for interactively when --password is omitted. For unattended runs, set VCF_OPS_PASSWORD instead of passing it on the command line.

I ran this end-to-end (list -> enable -> verify -> disable) against a live VCF Operations 9.1 appliance:

$ python3 set_pod_monitoring.py --host ops-a.site-a.vcf.lab --username admin --list
NAME                     POD MONITORING
metrics-test             disabled
kubernetes-cluster-93hv  disabled
$ python3 set_pod_monitoring.py --host ops-a.site-a.vcf.lab --username admin --cluster metrics-test --enable
metrics-test: pod monitoring enabled
$ python3 set_pod_monitoring.py --host ops-a.site-a.vcf.lab --username admin --list
NAME                     POD MONITORING
metrics-test             enabled
kubernetes-cluster-93hv  disabled
$ python3 set_pod_monitoring.py --host ops-a.site-a.vcf.lab --username admin --cluster metrics-test --cluster kubernetes-cluster-93hv --disable
metrics-test: pod monitoring disabled
kubernetes-cluster-93hv: already disabled, skipping

Wrapping up

Pod and container metrics carry a cost - more data points, more storage, more noise - which is why the VCF Operations leaves them off by default. Being able to flip that on and off through the API, rather than only by clicking through the UI, makes it practical to enable the extra detail exactly when it's needed - chasing down a noisy workload or an OOM-killed pod - and turn it back off once the investigation is done, instead of leaving it running indefinitely out of convenience.

Next up: I want to do this in VCF Automation, so pod monitoring can be toggled automatically as part of a cluster's day-2 workflow or deployment.

Configuring Cluster Autoscaler Scan Interval on VKS

2026-08-01T00:00:00.000Z

Update Aug 6 2026: verified the setting survives a cluster and addon upgrade — see Testing upgrade workflows.

Cluster Autoscaler in VMware vSphere Kubernetes Service (VKS)

Cluster Autoscaler is provided as one of the available and supported addons in VKS. This has been the case for many years and there have been meaningful improvements in how Cluster Autoscaler is implemented and its capabilities. Two of my favorites are "scale-to-zero" and the automatic installation provided by VKS during cluster deployment. VKS also takes care of upgrading Cluster Autoscaler to match the cluster version - pretty neat.

Configuring Cluster Autoscaler Behaviour

One of my clients asked me if it is possible to influence how Cluster Autoscaler behaves, specifically how quickly scaling happens. The answer was not easily found and is not officially documented (yet).

First let's look at what is documented. Cluster Autoscaler provides various configuration parameters to influence its mode of operation. A helpful resource can be found here: Cluster Autoscaler FAQ. The parameter to influence scale up speed is determined by the --scan-interval setting, which should default to 10s if not specified.

The current state of what can be configured is covered in the VMware Cloud Foundation Consumption documentation. VKS is using Carvel tools for packaging and managing addons, which stores the configuration in a Kubernetes secret. The available parameters are:

arguments:
  ignoreDaemonsetsUtilization: true
  maxNodeProvisionTime: 15m
  maxNodesTotal: 0
  metricsPort: 8085
  scaleDownDelayAfterAdd: 10m
  scaleDownDelayAfterDelete: 10s
  scaleDownDelayAfterFailure: 3m
  scaleDownUnneededTime: 10m
clusterConfig:
  clusterName: "CLUSTER_NAME"
  clusterNamespace: "VSPHERE_NAMESPACE"
paused: false

Notably the scan-interval is missing from the arguments list, but can still be specified using an undocumented extraArguments section like so:

arguments:
  ignoreDaemonsetsUtilization: true
  maxNodeProvisionTime: 15m
  maxNodesTotal: 0
  metricsPort: 8085
  scaleDownDelayAfterAdd: 10m
  scaleDownDelayAfterDelete: 10s
  scaleDownDelayAfterFailure: 3m
  scaleDownUnneededTime: 10m
  extraArguments:
  - scan-interval=60s
clusterConfig:
  clusterName: "CLUSTER_NAME"
  clusterNamespace: "VSPHERE_NAMESPACE"
paused: false

Time to try this out and verify the result.

Verification

I am using VCF Automation to get a cluster up and running quickly with autoscaling already enabled. This particular cluster is running vSphere Kubernetes Release (VKr) version 1.35.5, with a single control plane and autoscaler number of nodes set to 0-3.

Logging into the cluster and checking the default behaviour, we can confirm that the default interval is indeed 10s. This can be seen in the timing of the log messages and also the message itself:

$ export KUBECONFIG=Downloads/kubernetes-cluster-746q-kubeconfig.yaml
$ kubectl logs cluster-autoscaler-7cb8bb799c-8ctfv -n kube-system -f | grep triggered
I0730 14:26:07.622322       1 trigger.go:98] Autoscaler loop triggered by a 10s timer
I0730 14:26:19.638775       1 trigger.go:98] Autoscaler loop triggered by a 10s timer

In order to change this we can apply the configuration from before, which overrides the existing secret containing the current configuration. For reference the complete definition looks like this:

# file: autoscaler-values.yaml
apiVersion: v1
kind: Secret
metadata:
  name: kubernetes-cluster-746q-cluster-autoscaler-values
  namespace: vmware-system-tkg
stringData:
  values.yaml: |
    ---
    vksAddonManagement: true
    priorityClassName: ""
    clusterConfig:
      clusterName: "kubernetes-cluster-746q"
      clusterNamespace: "development-east-a-nlg6r"
    arguments:
      metricsPort: 8085
      maxNodesTotal: 0
      scaleDownDelayAfterAdd: "10m"
      scaleDownDelayAfterDelete: "10s"
      scaleDownDelayAfterFailure: "3m"
      scaleDownUnneededTime: "10m"
      maxNodeProvisionTime: "15m"
      ignoreDaemonsetsUtilization: true
      startupTaint: "node.cluster.x-k8s.io/uninitialized"
      extraArguments:
      - scan-interval=60s
    paused: false
    resources:
      requests:
        cpu: "25m"
        memory: "50Mi"

$ kubectl apply -f autoscaler-values.yaml
secret/kubernetes-cluster-746q-cluster-autoscaler-values configured
$ kubectl get pods -n kube-system | grep auto
cluster-autoscaler-76f9ff5574-vssmc                           1/1     Running   0               31s
$ kubectl logs cluster-autoscaler-76f9ff5574-vssmc -n kube-system -f | grep triggered
I0730 14:56:00.215137       1 trigger.go:98] Autoscaler loop triggered by a 1m0s timer
I0730 14:57:02.246144       1 trigger.go:98] Autoscaler loop triggered by a 1m0s timer

And indeed the change got picked up! This confirms that the scan-interval parameter can be set via the extraArguments field.

Choosing a value

I used 60s here because a longer interval is easy to confirm in the logs, not because it's a recommendation. The parameter works in both directions: a shorter interval makes Cluster Autoscaler re-evaluate more often — faster reaction to pending pods, at the cost of more API calls and CPU on the autoscaler pod. If your goal is quicker scale-up, go below the 10s default, not above it. Keep in mind scan-interval is only one factor in how fast a node actually arrives — maxNodeProvisionTime and the underlying VM provisioning time usually dominate. VCF 9.1 has you covered though, because of the new "Fast Deploy" capability - dropping provisioning time significantly.

Caveats

extraArguments is undocumented at this time. It works, but it isn't in the VCF documentation.
Applying the change restarts the autoscaler, as the deployment needs to get reconciled after the secret has been changed. That means scaling will not be available for a short period of time.
If the addon is managed by VKS, which upgrades Cluster Autoscaler along with the cluster, read the update section on where to properly adjust the addon values so an upgrade does not reset things.
There are two more ways to install addons in VKS. Either through plain kubectl or vcf CLI addon management. If you use one of the two, then applying the extraArguments section either survives the upgrade or gets reapplied as part of the upgrade workflow. This is well documented in the respective upgrade sections here: VMware Cloud Foundation Consumption

Update: Testing upgrade workflows

When using the automatic installation for Cluster Autoscaler the configuration source for the addon lives outside the cluster in the vSphere Namespace where the cluster is instantiated. It is captured in an AddonConfig that gets created before the cluster is installed and follows a deterministic naming scheme: <CLUSTER-NAME>-cluster-autoscaler. So let's try the upgrade from the beginning:

First a fresh cluster is needed - again using version 1.35.5 and VCF Automation to provide it:

Find the AddonConfig after logging into the vSphere Namespace:

$ vcf context use
? Select a context sv:development-east-a-nlg6r
[ok] Token is still active. Skipped the token refresh for context "sv:development-east-a-nlg6r"
[i] Successfully activated context 'sv:development-east-a-nlg6r' (Type: kubernetes) 
[i] Fetching recommended plugins for active context 'sv:development-east-a-nlg6r'...
$ kubectl get AddonConfig | grep autoscaler 
NAME                                            ADDONCONFIGDEFINITION                                                   CLUSTER                   READY   AGE
kubernetes-cluster-2qh5-cluster-autoscaler      cluster-autoscaler.vks.vmware.com.1.35.0---vmware.3-vks.1               kubernetes-cluster-2qh5   True    103s

Adding the extraArguments section to the AddonConfig:

$ kubectl edit  addonconfig kubernetes-cluster-2qh5-cluster-autoscaler
addonconfig.addons.kubernetes.vmware.com/kubernetes-cluster-2qh5-cluster-autoscaler edited
$ kubectl get  addonconfig kubernetes-cluster-2qh5-cluster-autoscaler -o yaml | grep -B 8 scan
spec:
  addonConfigDefinitionRef:
    name: cluster-autoscaler.vks.vmware.com.1.35.0---vmware.3-vks.1
    namespace: vmware-system-vks-public
  clusterName: kubernetes-cluster-2qh5
  values:
    arguments:
      extraArguments:
      - scan-interval=60s

And checking the change is live in the cluster:

$ export KUBECONFIG=Downloads/kubernetes-cluster-2qh5-kubeconfig.yaml 
$ kubectl get nodes
NAME                                                              STATUS   ROLES           AGE     VERSION
kubernetes-cluster-2qh5-bsgqs-bxwnq                               Ready    control-plane   5m12s   v1.35.5+vmware.1
kubernetes-cluster-2qh5-kubernetes-cluster-2qh5-np-hxss-trrxtvj   Ready    <none>          3m18s   v1.35.5+vmware.1
$ kubectl get pods -n kube-system | grep scale
cluster-autoscaler-6c78d798cf-5bls6                           1/1     Running   0          69s
$ kubectl logs cluster-autoscaler-6c78d798cf-5bls6 -n kube-system | grep timer
I0806 07:56:34.223878       1 trigger.go:98] Autoscaler loop triggered by a 1m0s timer

Nice - scan-interval got changed to our desired 60s value. Now the cluster is upgraded to version 1.36.1 - again via the comfort of VCF Automation, where this is a one-click operation:

Confirm Cluster Autoscaler got upgraded along with the VKr:

$ kubectl get AddonConfig
NAME                                            ADDONCONFIGDEFINITION                                                   CLUSTER                   READY   AGE
kubernetes-cluster-2qh5-cluster-autoscaler      cluster-autoscaler.vks.vmware.com.1.36.0---vmware.1-vks.1               kubernetes-cluster-2qh5   True    
$ kubectl get  addonconfig kubernetes-cluster-2qh5-cluster-autoscaler -o yaml | grep -B 8 scan
spec:
  addonConfigDefinitionRef:
    name: cluster-autoscaler.vks.vmware.com.1.36.0---vmware.1-vks.1
    namespace: vmware-system-vks-public
  clusterName: kubernetes-cluster-2qh5
  values:
    arguments:
      extraArguments:
      - scan-interval=60s

The AddonConfig got updated to reflect the new version - the extraArguments section is still in place. Now also confirm in the cluster itself:

$ export KUBECONFIG=Downloads/kubernetes-cluster-2qh5-kubeconfig.yaml 
$ kubectl get nodes
NAME                                                              STATUS   ROLES           AGE     VERSION
kubernetes-cluster-2qh5-bsgqs-cnlz9                               Ready    control-plane   5m53s   v1.36.1+vmware.4
kubernetes-cluster-2qh5-kubernetes-cluster-2qh5-np-hxss-trgmnkg   Ready    <none>          2m5s    v1.36.1+vmware.4
$ kubectl get pkgi -A | grep scale
NAMESPACE            NAME                                                 PACKAGE NAME                                  PACKAGE VERSION                 DESCRIPTION                                                            AGE   PAUSED
vmware-system-tkg    kubernetes-cluster-2qh5-cluster-autoscaler           cluster-autoscaler.kubernetes.vmware.com      1.36.0+vmware.1-vks.1           Reconcile succeeded                                                    15m   
$ kubectl get pods -n kube-system | grep scale
cluster-autoscaler-775dc8747c-4d882                           1/1     Running   0          6m54s
$ kubectl logs cluster-autoscaler-775dc8747c-4d882 -n kube-system | grep timer
I0806 08:06:04.780859       1 trigger.go:98] Autoscaler loop triggered by a 1m0s timer
I0806 08:07:06.814368       1 trigger.go:98] Autoscaler loop triggered by a 1m0s timer
I0806 08:08:08.432838       1 trigger.go:98] Autoscaler loop triggered by a 1m0s timer

The cluster got upgraded correctly, Cluster Autoscaler version got bumbed to match the Kubernetes version and the custom configuration is still in place.

In the Making

2026-07-01T00:00:00.000Z

Welcome to Runtime Zero

Runtime Zero is a technical blog with field notes on VMware technology, Kubernetes, industry trends and other things - written for the people who run this stuff and sometimes for the ones in charge of the budget.

Opinions are my own.

It is a clean and highly opinionated space. I highly value privacy, so this site runs without trackers, popups, CDNs or ads.

The Site Manifesto

# file: manifesto.yaml
site: "runtimezero.eu"
status: "Active & Under Construction"

maintainer:
  name: "Christian Schausberger"
  current_title: "Solutions Architect at Broadcom"
  experience: "20+ years"
  opinions: "unfiltered, my own"

architecture:
  stack: ["Eleventy", "Markdown", "Clarity"]
  bloat_index: "0.0%"
  trackers_installed: 0
  annoying_popups: 0
  newsletter_walls: 0
  cookies_set: "1 (light or dark mode preference)"

intended_audience:
  - "Everyone who wants to understand how (but is too lazy to read the docs)"
  - "Everyone who wants to understand why (but is too lazy to read the docs)"
  - "The brave souls managing the infrastructure budget"