Pod creation fails when containerd is used as runtime with Win-overlay CNI plugin

[selansen]

This is regarding making containerd (1.6.0.beta) as default runtime on Windows node (K8s1.23). We are using docker runtime currently and all works well. We use https:/containernetworking/plugins as our CNI plugin.
Containerd & Kubelet services are up and running. The Windows node is in a Ready state. When we try to create POD on Windows node, we are getting below error in containerd logs (failed to setup network for sandbox in RunPodSandbox() ). Looks like we need to update in CNI plugin code to make it compatible with contained.

Error logs :
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:09.701293100Z" level=trace msg="Version with client side version "0.1.0""
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:09.701293100Z" level=trace msg="Version returns &VersionResponse{Version:0.1.0,RuntimeName:containerd,RuntimeVersion:1.17.5,RuntimeApiVersion:v1,}"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:09.935305000Z" level=info msg="RunPodsandbox for &PodSandboxMetadata{Name:win-webserver-5db7f85d96-bsss7,Uid:3b12e32d-3c91-4979-a591-7f36a3b5a768,Namespace:default,Attempt:0,}"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:09.935863300Z" level=debug msg="Sandbox config &PodSandboxConfig{Metadata:&PodSandboxMetadata{Name:win-webserver-5db7f85d96-bsss7,Uid:3b12e32d-3c91-4979-a591-7f36a3b5a768,Namespace:default,Attempt:0,},Hostname:win-webserver-5db7f85d96-bsss7,LogDirectory:\var\log\pods\default_win-webserver-5db7f85d96-bsss7_3b12e32d-3c91-4979-a591-7f36a3b5a768,DnsConfig:&DNSConfig{Servers:[172.30.0.10],Searches:[default.svc.cluster.local svc.cluster.local cluster.local],Options:[ndots:5],},PortMappings:[]*PortMapping{},Labels:map[string]string{app: win-webserver,io.kubernetes.pod.name: win-webserver-5db7f85d96-bsss7,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: 3b12e32d-3c91-4979-a591-7f36a3b5a768,pod-template-hash: 5db7f85d96,},Annotations:map[string]string{kubernetes.io/config.seen: 2022-02-09T00:16:09.318808300Z,kubernetes.io/config.source: api,},Linux:&LinuxPodSandboxConfig{CgroupParent:,SecurityContext:&LinuxSandboxSecurityContext{NamespaceOptions:&NamespaceOption{Network:POD,Pid:CONTAINER,Ipc:POD,TargetId:,},SelinuxOptions:nil,RunAsUser:nil,ReadonlyRootfs:false,SupplementalGroups:[],Privileged:false,SeccompProfilePath:runtime/default,RunAsGroup:nil,Seccomp:&SecurityProfile{ProfileType:RuntimeDefault,LocalhostRef:,},Apparmor:nil,},Sysctls:map[string]string{},Overhead:nil,Resources:nil,},Windows:&WindowsPodSandboxConfig{SecurityContext:&WindowsSandboxSecurityContext{RunAsUsername:,CredentialSpec:,HostProcess:false,},},}"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:09.935962400Z" level=debug msg="generated id for sandbox name "win-webserver-5db7f85d96-bsss7_default_3b12e32d-3c91-4979-a591-7f36a3b5a768_0"" podsandboxid=bd0be36249a626c0d546eed7d8e909e92494a1870993696c7f1cf787891f307d
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:09.935962400Z" level=debug msg="use OCI runtime {Type:io.containerd.runhcs.v1 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0}" podsandboxid=bd0be36249a626c0d546eed7d8e909e92494a1870993696c7f1cf787891f307d
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:09.935962400Z" level=debug msg="hcn::HostComputeNamespace::Create id="
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:09.935962400Z" level=debug msg="hcn::HostComputeNamespace::Create JSON: {"SchemaVersion":{"Major":0,"Minor":0}}"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:09.964833400Z" level=debug msg="begin cni setup" podsandboxid=bd0be36249a626c0d546eed7d8e909e92494a1870993696c7f1cf787891f307d
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:10.089218500Z" level=trace msg="ListPodSandbox with filter nil"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:10.089218500Z" level=trace msg="ListPodSandbox returns pod sandboxes []"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:10.089780000Z" level=trace msg="ListContainers with filter &ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},}"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:10.089780000Z" level=trace msg="ListContainers with filter &ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},} returns containers []"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:10.360180200Z" level=debug msg="hcn::HostComputeNamespace::Delete id=4eb59999-002b-4883-8942-ee11af89e2e2"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:10.369711300Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:win-webserver-5db7f85d96-bsss7,Uid:3b12e32d-3c91-4979-a591-7f36a3b5a768,Namespace:default,Attempt:0,} failed, error" error="failed to setup network for sandbox "bd0be36249a626c0d546eed7d8e909e92494a1870993696c7f1cf787891f307d": plugin type="win-overlay" name="OVNKubernetesHybridOverlayNetwork" failed (add): error while AddHnsEndpoint(bd0be36249a626c0d546eed7d8e909e92494a1870993696c7f1cf787891f307d_OVNKubernetesHybridOverlayNetwork,39F25BE4-FC25-4E46-BC15-3BCD5D8FD97C,bd0be36249a626c0d546eed7d8e909e92494a1870993696c7f1cf787891f307d): failed to find HNSEndpoint bd0be36249a626c0d546eed7d8e909e92494a1870993696c7f1cf787891f307d_OVNKubernetesHybridOverlayNetwork: Endpoint bd0be36249a626c0d546eed7d8e909e92494a1870993696c7f1cf787891f307d_OVNKubernetesHybridOverlayNetwork not found"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:10.706178300Z" level=trace msg="ListPodSandbox with filter &PodSandboxFilter{Id:,State:&PodSandboxStateValue{State:SANDBOX_READY,},LabelSelector:map[string]string{},}"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:10.706178300Z" level=trace msg="ListPodSandbox returns pod sandboxes []"
ip-10-0-131-233.ec2.internal time="2022-02-09T00:16:10.707076000Z" level=trace msg="ListContainers with filter &ContainerFilter{Id:,State:&ContainerStateValue{State:CONTAINER_RUNNING,},PodSandboxId:,LabelSelector:map[string]string{},}"

[mansikulkarni96]

CNI plugin used here is: win-overlay

[mansikulkarni96]

/assign @selansen

[YanzhaoLi]

@selansen
I think win-bridge and win-overlay are specific for dockerd as runtime. For containerd on windows, I know sdnbridge works.

[selansen]

We are trying to move to containerd and we use win-overlay for our Windows solution. So our goal is to make win-overlay work with containerd as we use an overlay network.