Reduce the amount of stored ReplicaSet data

[swiatekm]

What does this PR do?

Use a transform function to drop all ReplicaSet data in the kubernetes provider, except for the owner references, which we need to find the Deployment name.

Unfortunately, the autodiscovery library doesn't let us pass the transform function in, so I had to copy their implementation and make the change myself. The plan is to upstream this change later.

Why is it important?

In clusters with a lot of Deployments, we end up storing a lot of ReplicaSets in the local cache of each agent, resulting in significant unnecessary memory consumption.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

There isn't an easy way. You can start a local K8s cluster, create a lot of Deployments, deploy Agent using the Helm Chart here, and check the memory consumption.

Related issues

• Partially resolves Kubernetes metadata overwhelms memory limits in the Agent process #4729
• Closes Agent and beats store too much ReplicaSet data in K8s #5623

[mergify]

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[swiatekm]

Note: I'd like to verify that this actually improves the situation in a real cluster. I'm working with our SRE team to do this. Until then, I'm keeping this PR as Draft and not adding unit tests or changelog entries.

[pkoutsovasilis]

I think that if you don't include ResourceVersion: old.GetResourceVersion() here then you effectively make the opts.IsUpdated check inside UpdateFunc of the informer to be always false

[pkoutsovasilis]

Observations:

1. I like the introduction of the transform func and I left a comment on it
2. I am wondering the following; since it seems we want only the Metadata why aren't we using something like a metadata shared informer?! e.g. snippet for replicasets

   metadataClient, err := metadata.NewForConfig(config) // "k8s.io/client-go/metadata"
   if err != nil {
     panic(err.Error())
   }
   
   // Define the resource to watch metadata for
   gvr := schema.GroupVersionResource{
       Group:    "apps",
   Version:  "v1",
   Resource: "replicasets",
   }
   
   resyncPeriod := 10 * time.Second
   
   indexers := cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}
   
   // Create the SharedIndexInformer using the provided code snippet
   informer := cache.NewSharedIndexInformer(
       &cache.ListWatch{
           ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {
               return metadataClient.Resource(gvr).List(ctx, options)
           },
           WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {
               return metadataClient.Resource(gvr).Watch(ctx, options)
           },
       },
       &metav1.PartialObjectMetadata{},
       resyncPeriod,
       indexers,
   )
   

   after a quick local-testing because PartialObjectMetadata struct is way smaller than ReplicaSet struct used in this snippet the allocations to unmarshall the former (happening before calling transform func) are way less than the ones required for the latter and this is evident in List and Stream funcs of the informer.

[elastic-sonarqube]

Quality Gate failed

Failed conditions
9.8% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube

[swiatekm]

Note: This code is going to move to the autodiscovery library, because we need to make a similar change in beats as well. I'm going to keep it here for now, and rebase once that's done. See #5580 for more details.

[swiatekm]

Closing in favor of elastic/elastic-agent-autodiscover#111. I'm going to recreate this PR after releasing a new version of autodiscover and updating go.mod in agent.