New Kong instances fail to boostrap configuration with KongPlugin invalid configuration

[edsonmarquezani]

Summary

Kong Ingress controller version
0.7.1

Kong or Kong Enterprise version
Kong (open-source)

Kubernetes version

Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.10", GitCommit:"575467a0eaf3ca1f20eb86215b3bde40a5ae617a", GitTreeState:"clean", BuildDate:"2019-12-11T12:32:32Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

Environment

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Container Linux by CoreOS stable (2303.4.0)
• Kernel (e.g. uname -a): Linux ip-10-35-37-57.ec2.internal 4.19.95-coreos Status of controller? #1 SMP Thu Feb 6 12:30:43 -00 2020 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux
• Install tools: KOPS

What happened

After setting an unknown plugin config option in a KongPlugin (one in use), new Kong pods won't get ANY configuration at bootstrap, returning only 404. The more new pods you have, the greater your error rate. An eventual recycling of all pods would lead to a complete unavailability of APIs.

Expected behavior

Kong should be configured preperly, according to resources set on cluster, despite the existence of a broken/incorrect KongPlugin configuration.

Steps To Reproduce

1. Create a KongPlugin resource with valid configuration, and attach it to an Ingress;
2. Add an arbitrary invalid option to that KongPugin as (config.foo=bar);
3. Ingress controller will start logging as below:

ingress-kong-7444fb49-4595l ingress-controller posting new config to /config: 400 Bad Request {"fields":{"plugins":[null,null,{"config":{"foo":"unknown field"}},{"config":{"foo":"unknown field"}}]},"name":"invalid declarative configuration","code":14,"message":"declarative config is invalid: {plugins={[4]={config={foo=\"unknown field\"}},[3]={config={foo=\"unknown field\"}}}}"}

4. Delete pods
   kubectl delete pods -l app=ingress-kong -n kong

5. Kong will initialize and no configuration will be set by the Ingress Controller to it.

6. An eventual fix in the plugin configuration (added in step 2) will bring everything up again.

[hbagdi]

I understand that this is a pain-point and we have to fix it. Our current solution against this problem is to use Admission Controller to guard against erroneous behaviors. You can use the following guide to set it up: https:/Kong/kubernetes-ingress-controller/blob/master/docs/deployment/admission-webhook.md

[edsonmarquezani]

Oh... I thought the default deployment files were already setting this up and not even verified. Ok, then, since this prevents incorrect configs from being created, we can go ahead with Kong Ingress until the controller itself is more resilient to these kind of errors. Thanks for pointing this up!

[edsonmarquezani]

@hbagdi I set up the validatingWebhook properly and it worked as expected. Thanks for the advice. Should I keep this issue open? Is there another one for this "problem" already?