[Bug]: Using Github Actions Runner Controller: No 'run' handler configured for type 'container'

[frizzr]

Bug

Current behavior

We are running Garden in an instance of Github Enterprise using Github Actions, and we serve up runners from an AWS EKS cluster using Github Actions Runner Controller (ARC.) This means the runner runs in a Pod in that EKS cluster.

From within that runner pod container (which does have its parent container's /var/run/docker.sock mapped into it and is able to run containers inside the environment even without Garden) while trying to do a garden run of run type container, we always encounter this error:

No 'run' handler configured for Run type 'container' in environment 'ci'. Are you missing a provider configuration?

The docs state the container provider is automatically included, but we have tried it both ways without specifying any providers and explicitly specifying the container provider in the root project.garden.yml

The container we are running Garden in has full internet access and we have successfully been able to download and/or update Garden either inline during the build and also installing garden beforehand to our base runner image.

We are not running using the Garden dashboard, so we do not login to Garden, but we understand this should not be an issue.

As a base test to eliminate our runner image environment, we have manually run our base runner image outside of our ARC runner on a local Docker running on a laptop. We cannot reproduce the error in a local environment and garden runs successfully, so there is something specific to running within ARC that causes this error.

Expected behavior

A successful run of Garden within a K8s pod in an AWS EKS cluster running ARC.

Reproducible example

Of course this is kind of hard to provide given it requires the use of ARC in the context of AWS EKS to reproduce the issue.

We realize there is something unique about the environment we are running Garden in that causes this error, but the Garden error logs are not particularly helpful in troubleshooting the issue even at the lowest log level.

Still, below is a subset of our build repository using Garden with source files removed but including Github Action workflows along with a copy of the output of the Github Actions workflow (including the full set of environment variables printed out) in the build-debug-outputs directory:

garden-debug

More than anything, we are simply looking for a way to debug this issue and some help pointing us in the right direction. We are able to pause the CI/CD process and we can shell into the running build job container in the EKS cluster if need be, but we would like some direction on things to change in that environment before we go further. Hopefully it will be something simpler than that.

Workaround

None.

Suggested solution(s)

None.

Additional context

See above.

Your environment

• OS: Ubuntu (which is what ARC uses for its base image)
• How I'm running Kubernetes: Within AWS EKS (but not running Garden directly using our EKS cluster.)

garden version: 0.13.41

[10ko]

Hi @frizzr, thanks for reporting this! We are taking a look.

[twelvemo]

Hi @frizzr, thanks for the repo with the reproducible example, that was super helpful.
Unfortunately what you are trying to do - make garden run a container locally to run something in that like creating some artifacts - does not work and is not the intended behavior. The Run action of type container will run the command you specify in a container running in a Kubernetes cluster (can be local or remote and this is where the missing provider error message is stemming from). Garden does not natively support running containers locally and is more geared to running them in Kubernetes. However there are a bunch of questions we had regarding your setup and there might be solutions to accommodate your use case.

1. For the maven-build: should the result of this build be a container image or a java binary? If the further is the case you could try a Build action of type container which will accomplish that. If you want to build the binary but in it's own container, we'd have to work with an Build action of type exec see here. You'd have to specify the docker command as part of the spec.command.
2. For the helm-build you could also use a Build action of type exec to prepare the chart. This can also either be done in the ci runner container or if needed with a docker command in spec.command.

Does that make sense? Please feel free to ask any follow-up questions that come to mind!