[FEATURE]: timeouts for cleanup commands

[salotz]

Feature Request

Background / Motivation

I need to run cleanup namespace in a workflow and I'm hitting the hard-coded timeouts for this as it can take a while. When this runs in CI this becomes a problem as it then causes CI to fail as the task exits as failed.

What should the user be able to do?

Specify a timeout when running cleanup ... commands with garden.

Suggested Implementation(s)

garden cleanup --timeout=1000 [...]

How important is this feature for you/your team?

Somewhere between:

🥀 Crucial, Garden is unusable for us without it

🌵 Not having this feature makes using Garden painful

As I don't really know a workaround.

[eysi09]

Hi @salotz

Thanks for flagging this. Your suggestion makes sense and we'll add support for this.

[eysi09]

Actually, it might make more sense to just bump the default timeout.

Do you have a rough feeling for what a sensible value could be for your use case?

[salotz]

@eysi09 Usually increasing timeouts to 15min does it for me, but I don't have a good intuition about which operations actually take so long. Especially when cleaning up namespaces I find sometimes k8s gets stuck.

[eysi09]

Ok gotcha. The current default timeout is 300 seconds so that would be quite a large bump.

In that case I think the quickest win would be to set this is an environment variable. The cleanup command can technically delete things beyond K8s namespaces so the --timeout flag wouldn't really be well defined and you might need different values for different "kinds" of namespaces in the future.

Do you think the env var approach would work for your use case?

Something like:

GARDEN_K8S_DELETE_NS_TIMEOUT_MS=900 garden cleanup namespace

(A little verbose but the quickest win AFIACT since this blocking you)

[salotz]

That should work yes.

[salotz]

I do have a workaround implemented in CI to just use kubectl for deleting the namespace which I can set timeouts for btw.