How to optimize the Kubernetes client memory usage in a large-scale cluster?

"Large-scale" here refers not only to many nodes, but to many resources — like Jobs or ServiceMonitors.

Background

When extending Kubernetes cluster functionality, we often use k8s.io/client-go to interact with the Kubernetes API server, List/Watch resources, and do something. For example, kube-state-metrics Lists/Watches Pods, Deployments, Jobs, DaemonSets, etc., generates Prometheus metrics from resource data and exposes them for Prometheus to scrape. Prometheus Operator Lists/Watches ServiceMonitors, PodMonitors, etc., to update the Prometheus configuration file for scraping metrics.

k8s.io/client-go uses SharedIndexInformer to cache cluster resources in memory, reducing requests to the API server and easing its CPU/memory pressure. The catch: SharedIndexInformer first Lists all resources (e.g., all Jobs or all ServiceMonitors). If the cluster has a large number of resources, caching everything in memory can cause client memory usage to spike, risking OOM¹ if limits are misconfigured or node resources are tight.

Mitigation Strategies

As client developers, we should optimize code to reduce memory usage. Here are several approaches I've gathered from reading Kubernetes component source code.

1. WithTransform

WithTransform trims objects by removing unused fields, like ManagedFields.

func WithTransform(transform cache.TransformFunc) SharedInformerOption

WithTransform places a TransformFunc in the Delta FIFO queue. Before objects are added to the queue (on List/Watch add/update/delete), they are trimmed — unused fields are stripped out, never cached in memory.

client-go-controller-interaction

Image source: https://github.com/kubernetes/sample-controller/blob/v0.36.0/docs/images/client-go-controller-interaction.jpeg

For example, kube-controller-manager uses WithTransform to strip .metadata.managedFields from API server responses²:

trim := func(obj interface{}) (interface{}, error) {
	if accessor, err := meta.Accessor(obj); err == nil {
		if accessor.GetManagedFields() != nil {
			accessor.SetManagedFields(nil)
		}
	}
	return obj, nil
}
sharedInformers := informers.NewSharedInformerFactoryWithOptions(client, ResyncPeriod(s)(), informers.WithTransform(trim))

Implementation: https://github.com/kubernetes/kubernetes/pull/118455

2. Set ListOptions.ResourceVersion to "0"

When strong consistency isn't required, set ResourceVersion to "0" to read from the API server's watch cache. This improves performance and reduces etcd load, at the cost of potentially stale data.

As of Kubernetes 1.36, omitting ResourceVersion (or setting it to "") also reads from the watch cache but guarantees consistency.

Example from kube-state-metrics³:

if i.useAPIServerCache {
	options.ResourceVersion = "0"
}

Implementation: https://github.com/kubernetes/kube-state-metrics/pull/1548

More on Resource versions: Resource versions

3. WatchListCompression Feature Gate 4

Enables gzip compression for WatchList requests. Enabled by default in Kubernetes 1.37+.

4. WatchListClient Feature Gate 5

Enables the WatchList client on the client side to stream List responses instead of fetching in chunks. Enabled by default in Kubernetes v1.32 and v1.34+⁶.

genericfeatures.WatchList: {
	{Version: version.MustParse("1.27"), Default: false, PreRelease: featuregate.Alpha},
	{Version: version.MustParse("1.32"), Default: true, PreRelease: featuregate.Beta},
	{Version: version.MustParse("1.33"), Default: false, PreRelease: featuregate.Beta},
	{Version: version.MustParse("1.34"), Default: true, PreRelease: featuregate.Beta},
},

kube-apiserver and kube-controller-manager implementation: https://github.com/kubernetes/kubernetes/pull/132704

5. Set ListOptions.Limit

Configure Limit on List calls to cap the maximum number of objects returned per response. Note: if the WatchListClient feature gate is enabled, the Reflector in client-go will only make Watch calls.

kube-state-metrics implementation: https://github.com/kubernetes/kube-state-metrics/pull/2626

6. Set a Memory Soft Limit 7

From Go's perspective, you can set a memory soft limit to tune GC frequency and release memory.

Three ways:

Set the GOMEMLIMIT environment variable
Use runtime/debug.SetMemoryLimit at runtime
Use the github.com/KimMachineGun/automemlimit package to auto-detect cgroup memory limits and auto-set GOMEMLIMIT

kube-state-metrics auto-detection implementation: https://github.com/kubernetes/kube-state-metrics/pull/2447

Community Efforts

The Kubernetes community continues to optimize — including consistent reads from cache, streaming List responses, and server-side sharded List/Watch.

Summary

There are now many ways to reduce Kubernetes client memory usage — configuring ListOptions fields, feature gates, transform functions, and memory limits. The Kubernetes community continues to improve SharedIndexInformer and List/Watch calls.