Race condition between secret reconciler and object reference index

[backjo]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

We recently noticed similar errors to #4672 where our secrets were getting "Not Found" errors despite very much existing and being referenced by KongConsumer via credentials. After checking our CRDs as mentioned in that issue, we added some custom logging to understand why our secrets were not getting populated into the cache. The custom logging revealed that the controller for Secrets was evaluating reference checks before the controller for KongConsumer had reconciled the consumers and populated the references, resulting in 0 secrets being reconciled.

Expected Behavior

All initial resources should be loaded before calculating references.

Steps To Reproduce

No response

Kong Ingress Controller version

3.0.0, but appears to be happening in 2.11.* as well

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.9", GitCommit:"a1a87a0a2bcd605820920c6b0e618a8ab7d117d4", GitTreeState:"clean", BuildDate:"2023-04-12T12:16:51Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26+", GitVersion:"v1.26.10-eks-4f4795d", GitCommit:"164dfb62db432c0b28a1fced3956256af68533b6", GitTreeState:"clean", BuildDate:"2023-10-20T23:21:27Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}

Anything else?

Similer to #4672

[randmonkey]

I checked the code, When the KongConsumer gets reconciled, the controller will retrieve the referred secrets in its credentials and fetch the secrets in cluster then add them to cache.
This satisfies the eventual consistency, where the KongConsumer and Secret used in its credentials are all added to the cache after KongConsumer gets reconciled.
What other unexpected behaviors did you found other than the "Not Found" logs?

[backjo]

Hey @randmonkey - the main unexpected behavior we observe is consistent with the credentials being loaded - namely that our ACL plugin is blocking requests for ~5-10 minutes after startup due to the credential presented not being found.

[backjo]

Going to close this until I can provide more debug info

[backjo]

hi @randmonkey - did some further digging here and the behavior I generally can reproduce is:

1. Controller starts up and reconciles KongConsumer objects. Secrets have not yet been loaded, so it requeues the reconcile operation.
2. Controller writes initial configuration to the Kong Admin API - without information from Secret objects. For us, this means that it doesn't write the relevant "groups" information from credentials for the ACL plugin to function.
3. KongConsumer objects are reconciled on the requeue, which loads the credentials successfully.
4. Controller writes a second configuration to the Kong Admin API with updated groups.

The second update to the admin api happens a few seconds later from the logs we put in place, but for the few seconds between the first and second writes, we have an effectively 'broken' configuration.

[backjo]

This seems fixable by #2249 - maybe the best intermediate solve here is to allow InitCacheSyncDuration to be configurable instead of just hard coded to 5s

[backjo]

Ah - I think there was a regression to the #2249 fix here. In #4101, InitCacheSyncDuration started being passed in when the synchronizer is created, but InitCacheSyncDuration is not being initialized to anything while previously DefaultCacheSyncWaitDuration in synchronizer.go was being initialized to 5 seconds