path to CNI bridge by default

[tonistiigi]

BuildKit supports CNI networking for containers, but in the release container images the default is to use host network (assumption is that container image itself already runs under docker bridge).

Afaik this was done for two reasons:

• there was a significant performance overhead for setting up the networking
• it was unknown how stable the implementation was

Thanks to #3107 I think these concerns have mainly gone away.

I also opened #3860 . The network resource monitoring only works if all containers are on an isolated network.

I propose migration to use CNI bridge by default and add the helper binaries into the container image.

I think for v0.12 we could include the binaries and add a simple way to opt-in (something that could be set in docker buildx create for example).

If everything goes smoothly then switch from opt-in to opt-out in release after.

@aaronlehmann @AkihiroSuda

[thaJeztah]

Will this only apply to buildkit running standalone, or would this (in future) mean we need to package additional binaries / dependencies with docker engine?

[tonistiigi]

@thaJeztah That's up to the engine if it wants to switch from libnetwork bridge to CNI bridge. This change is only for the buildkitd image default.