k8s_grace_period not respected

[mterwill]

Expected Behavior

tilt ci fails after k8s_grace_period is reached.

Current Behavior

tilt ci reproducibly hangs for 5 minutes.

Steps to Reproduce

Tiltfile:

ci_settings(k8s_grace_period='10s')
k8s_yaml('job.yaml')

job.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: always-fails
spec:
  template:
    spec:
      containers:
      - name: always-fails
        image: busybox
        command:
          - sh
          - -c
          - echo "$(date -u) hardcoded failure" && exit 1
      restartPolicy: Never
  backoffLimit: 0

Output:

[matt.terwilliger@D94GKM7XNP tilt]$ date -u; k delete job always-fails; time tilt ci; date -u
Tue Jan 10 21:50:12 UTC 2023
job.batch "always-fails" deleted
Tilt started on http://localhost:10350/
v0.31.1, built 2023-01-09

Initial Build
Loading Tiltfile at: /private/tmp/tilt/Tiltfile
Successfully loaded Tiltfile (5.987042ms)
 always-fails │
 always-fails │ Initial Build
 always-fails │ STEP 1/1 — Deploying
 always-fails │      Applying YAML to cluster
 always-fails │      Objects applied to cluster:
 always-fails │        → always-fails:job
 always-fails │
 always-fails │      Step 1 - 0.49s (Deploying)
 always-fails │      DONE IN: 0.49s
 always-fails │
 always-fails │
 always-fails │ Tracking new pod rollout (always-fails-qshj2):
 always-fails │      ┊ Scheduled       - <1s
 always-fails │      ┊ Initialized     - <1s
 always-fails │      ┃ Not Ready       - (ContainersNotReady): containers with unready status: [always-fails]
 always-fails │ [event: pod ws-matt-terwilliger/always-fails-qshj2] Pulling image "busybox"
 always-fails │ [event: pod ws-matt-terwilliger/always-fails-qshj2] Successfully pulled image "busybox" in 216.417472ms
 always-fails │ [event: job ws-matt-terwilliger/always-fails] Job has reached the specified backoff limit
 always-fails │ Tue Jan 10 21:50:16 UTC 2023 hardcoded failure
Error: exceeded grace period: Pod "always-fails-qshj2" failed
tilt ci  1.09s user 0.45s system 0% cpu 5:02.20 total
Tue Jan 10 21:55:15 UTC 2023

About Your Use Case

Using tilt ci to run a series of smoke tests in CI.

[mterwill]

Thanks 😄