feat: also gracefully shutdown on SIGTERM

[BugenZhao]

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

...because Kubernetes will send SIGTERM when killing a pod.

As described in #17662, this can empower more seamless scaling-in in Kubernetes deployments. (correct me if I'm wrong)

After this PR, I suppose we can get rid of the extra manual step of risingwave ctl meta unregister-worker when scaling-in, as described in the doc.

Checklist

• I have written necessary rustdoc comments
• I have added necessary unit tests and integration tests
• All checks passed in ./risedev check (or alias, ./risedev c)

Documentation

• My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

The step 1 of manual unregistration of the compute node when scaling-in (doc) is not necessary any more. Users can directly apply the new yaml file to trigger a graceful scale-in.

[BugenZhao]

We don't support Windows, so directly use unix-only features here.

[BugenZhao]

• feat: also gracefully shutdown on SIGTERM #17802 👈
• feat: compute node unregisters from meta for graceful shutdown #17662
• refactor: graceful shutdown in standalone mode #17633
• fix(risedev): fixes and improvements for risedev-dev running risingwave commands #17586
• refactor: graceful shutdown on meta node & unify election path #17608
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @BugenZhao and the rest of your teammates on Graphite

[stdrc]

👍

[BugenZhao]

Tested with risingwave-operator and it works like a charm. 🥰

compute:

2024-07-30T09:10:05.276715082Z  INFO risingwave_rt: received SIGTERM, shutting down...
2024-07-30T09:10:05.283413439Z  INFO risingwave_rpc_client::meta_client: successfully unregistered from meta service worker_id=2004
2024-07-30T09:10:05.283480937Z  WARN risingwave_stream::task::barrier_manager: shutdown with running actors, scaling or migration will be triggered
2024-07-30T09:10:05.283581726Z  INFO risingwave_stream::task::barrier_manager: waiting for meta service to close control stream...
2024-07-30T09:10:05.285297803Z  INFO risingwave_rpc_client::meta_client: Heartbeat loop is stopped
2024-07-30T09:10:05.285344052Z  INFO risingwave_common::telemetry::report: Telemetry exit

meta:

2024-07-30T09:10:05.283791678Z  WARN risingwave_meta::barrier::info: node with running actors is deleted node_id=2004 node=Some(WorkerNode { id: 2004, r#type: ComputeNode, host: Some(HostAddress { host: "risingwave-compute-2.risingwave-compute", port: 5688 }), state: Running, property: Some(Property { is_streaming: true, is_serving: true, is_unschedulable: false }), transactional_id: Some(4), resource: None, started_at: None, parallelism: 1 }) actors={2007, 2008}
2024-07-30T09:10:05.283967173Z  INFO failure_recovery{error=worker node 2004 is shutting down prev_epoch=6887532391759872}: risingwave_meta::barrier::recovery: recovery start!
2024-07-30T09:10:05.287488493Z  INFO risingwave_meta::hummock::manager::worker: Released hummock context 2004
2024-07-30T09:10:05.288955702Z  INFO failure_recovery{error=worker node 2004 is shutting down prev_epoch=6887532391759872}:recovery_attempt: risingwave_meta::manager::catalog::fragment: cleaning dirty downstream merge nodes for table sink
2024-07-30T09:10:05.293052214Z  INFO failure_recovery{error=worker node 2004 is shutting down prev_epoch=6887532391759872}:recovery_attempt: risingwave_meta::barrier::recovery: recovering mview progress
2024-07-30T09:10:05.293080297Z  INFO failure_recovery{error=worker node 2004 is shutting down prev_epoch=6887532391759872}:recovery_attempt: risingwave_meta::barrier::recovery: recovered mview progress
2024-07-30T09:10:05.304094784Z  INFO failure_recovery{error=worker node 2004 is shutting down prev_epoch=6887532391759872}:recovery_attempt: risingwave_meta::barrier::recovery: offline rescheduling for job 2001 in recovery is done

[fuyufjh]

cc. @arkbriar