cgroups: make sure cgroup still exists after task restart

[shoenig]

This PR modifies raw_exec and exec to ensure the cgroup for a task
they are driving still exists during a task restart. These drivers
have the same bug but with different root cause.

For raw_exec, we were removing the cgroup in 2 places - the cpuset
manager, and in the unix containment implementation (the thing that
uses freezer cgroup to clean house). During a task restart, the
containment would remove the cgroup, and when the task runner hooks
went to start again would block on waiting for the cgroup to exist,
which will never happen, because it gets created by the cpuset manager
which only runs as an alloc pre-start hook. The fix here is to simply
not delete the cgroup in the containment implementation; killing the
PIDs is enough. The removal happens in the cpuset manager later anyway.

For exec, it's the same idea, except DestroyTask is called on task
failure, which in turn calls into libcontainer, which in turn deletes
the cgroup. In this case we do not have control over the deletion of
the cgroup, so instead we hack the cgroup back into life after the
call to DestroyTask.

All of this only applies to cgroups v2.

Fixes #12863

No CL because cgroupsv2 hasn't shipped yet.

[shoenig]

repro example

job "restarts" {
  datacenters = ["dc1"]
  type = "service"

  group "g1" {
    restart {
      attempts = 5
      mode = "delay"
      delay = "1s"
      interval = "5s"
    }

    task "t1-raw_exec" {
      driver = "raw_exec"
      config {
	command = "/usr/bin/bash"
	args = ["-c", "sleep 10"]
      }
    }

    task "t2-exec" {
      driver = "exec"
      config {
	command = "/usr/bin/bash"
	args = ["-c", "sleep 10"]
      }
    }    
  }
}

with fix:

Task "t1-raw_exec" is "running"
Task Resources
CPU        Memory          Disk     Addresses
0/100 MHz  42 MiB/300 MiB  300 MiB  

Task Events:
Started At     = 2022-05-04T19:01:57Z
Finished At    = N/A
Total Restarts = 5
Last Restart   = 2022-05-04T14:01:57-05:00

Recent Events:
Time                       Type        Description
2022-05-04T14:01:57-05:00  Started     Task started by client
2022-05-04T14:01:57-05:00  Restarting  Task restarting in 1.144459209s
2022-05-04T14:01:57-05:00  Terminated  Exit Code: 0
2022-05-04T14:01:47-05:00  Started     Task started by client
2022-05-04T14:01:47-05:00  Restarting  Task restarting in 1.08660356s
2022-05-04T14:01:47-05:00  Terminated  Exit Code: 0
2022-05-04T14:01:37-05:00  Started     Task started by client
2022-05-04T14:01:37-05:00  Restarting  Task restarting in 1.23187761s
2022-05-04T14:01:37-05:00  Terminated  Exit Code: 0
2022-05-04T14:01:27-05:00  Started     Task started by client

Task "t2-exec" is "running"
Task Resources
CPU        Memory           Disk     Addresses
0/100 MHz  224 KiB/300 MiB  300 MiB  

Task Events:
Started At     = 2022-05-04T19:01:50Z
Finished At    = N/A
Total Restarts = 4
Last Restart   = 2022-05-04T14:01:49-05:00

Recent Events:
Time                       Type        Description
2022-05-04T14:01:50-05:00  Started     Task started by client
2022-05-04T14:01:49-05:00  Restarting  Task restarting in 1.08660356s
2022-05-04T14:01:49-05:00  Terminated  Exit Code: 0
2022-05-04T14:01:39-05:00  Started     Task started by client
2022-05-04T14:01:39-05:00  Restarting  Task restarting in 1.23187761s
2022-05-04T14:01:39-05:00  Terminated  Exit Code: 0
2022-05-04T14:01:29-05:00  Started     Task started by client
2022-05-04T14:01:29-05:00  Restarting  Task restarting in 1.042526215s
2022-05-04T14:01:29-05:00  Terminated  Exit Code: 0
2022-05-04T14:01:19-05:00  Started     Task started by client

[shoenig]

Running relevant tests locally on a cgroups v2 machine

➜ mount -l | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)

test-cgutil.log
test-exec.log
test-raw_exec.log
test-taskrunner.log

[shoenig]

Will follow up with CI changes to run on ubuntu-22.04 in GHA
actions/runner-images#5490

[tgross]

LGTM

Does the most recent commit cover #12877 as well do you think or is that going to need to be a separate investigation?

[shoenig]

#12877

Let's keep it open; so far I haven't been able to reproduce but my laptop is on an older kernel

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.