Using flannel as CNI no longer works

[benmoss]

What happened:
After installing Flannel as my CNI, pods fail to start with the error Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "40a61cb16293c414497736143f53da331ef0dca2236e223f3057bd930d51c1c6": failed to find plugin "flannel" in path [/opt/cni/bin]

What you expected to happen:
Pods work out of the box as before!

How to reproduce it (as minimally and precisely as possible):
Using 0.7.0, create a cluster with networking.disableDefaultCNI: true. Install the Flannel CNI yaml. See that CoreDNS pods in kube-system fail to start.

Anything else we need to know?:
It works in 0.6.1, it looks like 281a20c is what broke it.

Normally Flannel assumes that the flannel CNI binary is included on the host, since it is part of the kubernetes-cni package and the default https:/containernetworking/plugins repo.

[BenTheElder]

this is working as intended, your deployment manifests for flannel should install the requisite CNI binaries at the correct version instead of assuming the host has them

most CNI install daemonsets include a step that copies these binaries out from a container image.

[BenTheElder]

disableDefaultCNI should be more clearly marked as a "here be dragons" option, we don't test any external CNI currently, at the moment that's a bit out of scope, but the escape hatch is provided to make it easier for power users to do this anyhow.

regardless of that though, the flannel manifests should really be handling this, as weave, calico, etc. do. even if these binaries are present they may be at the wrong version.

[benmoss]

Yeah, it seems like flannel is the odd one out that has its CNI plugin included in the containernetworking/plugins repo, and so doesn't bother to install it in its deployment manifest. I'll see if I can fix it upstream.

[BenTheElder]

Thanks! I forgot to add, more importantly this is going to break on other hosts, when 281a20c was filed we'd just had an upstream discussion in kubernetes about how the plugins are re-packaged in the kubernetes debs / RPMs...

The conclusion of the discussion was a decision to stop packaging them in the future in favor of only the plugins directly used by kubelet being included in the kubelet package and letting the user install CNI as needed instead of packaging all the upstream plugins in a special debian depended on by kubeadm.

I'm not sure when this will land, but when it does flannel will need this for a lot of kubeadm based installs.

[benmoss]

This looks like a reasonable start to the paper trail you seem to be talking about https://groups.google.com/forum/#!topic/kubernetes-sig-release/yhf7hAqJEN0/discussion 😸

[BenTheElder]

yes, that! had to run to a meeting and didn't have that handy 😅 there was also some discussion in slack I think, but that's the official paper trail.

…

On Tue, Feb 18, 2020 at 12:38 PM Ben Moss ***@***.***> wrote: This looks like a reasonable start to the paper trail you seem to be talking about https://groups.google.com/forum/#!topic/kubernetes-sig-release/yhf7hAqJEN0/discussion 😸 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1340?email_source=notifications&email_token=AAHADKZ7PEKBE4CG3TM3GYDRDRBMPA5CNFSM4KXLZUB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMETZKI#issuecomment-587807913>, or unsubscribe <https:/notifications/unsubscribe-auth/AAHADK2X3VLDGJFEJCVCDRDRDRBMPANCNFSM4KXLZUBQ> .

[BenTheElder]

If you have any luck upstream, this should also likely go in https:/coreos/flannel/pull/1229/files

We had fun with xtables.lock and kind CI flakiness before doing a similar fix in kindnetd.

[benmoss]

I haven't had a lot of luck with the maintainers of flannel, it seems to have been abandoned except for critical fixes. We can close this issue since it's not exactly a kind bug.

[BenTheElder]

Ack, that's too bad 😞