Add health check probe for k8s upgrade containers. #15223

lixiaoyuner · 2023-05-26T03:31:31Z

Why I did it

After k8s upgrade a container, k8s can only know the container is running, don't know the service's status inside container. So we need a probe inside container, k8s will call the probe to check whether the container is really ready.

Work item tracking

Microsoft ADO (number only):
22453004

How I did it

Add a health check probe inside config engine container, the probe will check whether the start service exit normally or not if the start service exists and call the python script to do container self-related specific checks if the script is there. The hook script should be implemented by feature owner if it's needed.

more details: design doc

How to verify it

Check path /usr/bin/readiness_probe.sh inside container.

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

20220531.28

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

losha228

LGTM

qiluo-msft · 2023-06-22T07:04:41Z

src/sonic-ctrmgrd/ctrmgr/readiness_probe.sh

+
+#### exit code contract, k8s only cares zero or not none-zero, but we want to use none-zero code to indicate different error
+# 0: readiness
+# 1: python script crach exit code


crach

typo #Closed

qiluo-msft · 2023-06-22T07:06:08Z

src/sonic-ctrmgrd/ctrmgr/readiness_probe.sh

+# if the start service exists, check if it exits normally
+# if the start service doesn't exist normally, exit with code 2
+pre_check_service_name="start"
+supervisorctl status |awk '{print $1}' |grep -w $pre_check_service_name > /dev/null


supervisorctl status

You can use one command

supervisorctl status start ``` #Closed

Only do "supervisorctl status start", We can't do judgement by exit code, because start not existing and some failed state exit codes are the same. If only do "supervisorctl status start", need to judge by the outputs "start: ERROR (no such process)", "start EXITED Jun 21 05:28 PM". I do checking whether start exists in advance, I think code logic is more easy to understand here.

This is an example

root@sonic:/# supervisorctl status start start EXITED Jul 04 12:38 AM root@sonic:/# supervisorctl status dependent-startup EXITED Jul 04 12:38 AM lldp-syncd RUNNING pid 26, uptime 0:03:54 lldpd RUNNING pid 20, uptime 0:03:57 lldpmgrd RUNNING pid 30, uptime 0:03:52 rsyslogd RUNNING pid 11, uptime 0:04:02 start EXITED Jul 04 12:38 AM supervisor-proc-exit-listener RUNNING pid 10, uptime 0:04:04 waitfor_lldp_ready EXITED Jul 04 12:38 AM

qiluo-msft · 2023-06-22T07:24:15Z

src/sonic-ctrmgrd/ctrmgr/readiness_probe.sh

+# check if the post_check_script exists
+# if the post_check_script exists, run it
+# if the post_check_script exits with non-zero code, exit with the code
+post_check_script="/usr/bin/readiness_probe.py"


/usr/bin/readiness_probe.py

Do not assume python3.
How about /usr/bin/readiness_probe_hook.

if [ -x $post_check_script ]; then $post_check_script

#Closed

qiluo-msft · 2023-06-22T07:24:47Z

src/sonic-ctrmgrd/ctrmgr/readiness_probe.sh

+# check if the start service exists
+# if the start service exists, check if it exits normally
+# if the start service doesn't exist normally, exit with code 2
+pre_check_service_name="start"


start

Will you check all the critical processes? #Closed

The critical processes unexpected event will be handled by the supervisord exit-listener for now, the listener will kill the container, I don't think we need to check them here. Is this correct?

lguohan · 2023-06-30T16:08:05Z

where is public design doc for such health check probe?

lixiaoyuner · 2023-07-03T07:18:14Z

where is public design doc for such health check probe?

We have a OneNote page, I put the link into this PR related ADO discussion before. ADO number: 22453004.
I also put the design doc to SONiC repo

#### Why I did it After k8s upgrade a container, k8s can only know the container is running, don't know the service's status inside container. So we need a probe inside container, k8s will call the probe to check whether the container is really ready. ##### Work item tracking - Microsoft ADO **(number only)**: 22453004 #### How I did it Add a health check probe inside config engine container, the probe will check whether the start service exit normally or not if the start service exists and call the python script to do container self-related specific checks if the script is there. The python script should be implemented by feature owner if it's needed. more details: [design doc](https:/sonic-net/SONiC/blob/master/doc/kubernetes/health-check.md) #### How to verify it Check path /usr/bin/readiness_probe.sh inside container. #### Which release branch to backport (provide reason below if selected) - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [x] 202205 - [x] 202211 #### Tested branch (Please provide the tested image version) - [x] 20220531.28

mssonicbld · 2023-07-13T16:35:17Z

Cherry-pick PR to 202205: #15823

#### Why I did it After k8s upgrade a container, k8s can only know the container is running, don't know the service's status inside container. So we need a probe inside container, k8s will call the probe to check whether the container is really ready. ##### Work item tracking - Microsoft ADO **(number only)**: 22453004 #### How I did it Add a health check probe inside config engine container, the probe will check whether the start service exit normally or not if the start service exists and call the python script to do container self-related specific checks if the script is there. The python script should be implemented by feature owner if it's needed. more details: [design doc](https:/sonic-net/SONiC/blob/master/doc/kubernetes/health-check.md) #### How to verify it Check path /usr/bin/readiness_probe.sh inside container. #### Which release branch to backport (provide reason below if selected) - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [x] 202205 - [x] 202211 #### Tested branch (Please provide the tested image version) - [x] 20220531.28

mssonicbld · 2023-07-13T16:35:22Z

Cherry-pick PR to 202211: #15824

#### Why I did it After k8s upgrade a container, k8s can only know the container is running, don't know the service's status inside container. So we need a probe inside container, k8s will call the probe to check whether the container is really ready. ##### Work item tracking - Microsoft ADO **(number only)**: 22453004 #### How I did it Add a health check probe inside config engine container, the probe will check whether the start service exit normally or not if the start service exists and call the python script to do container self-related specific checks if the script is there. The python script should be implemented by feature owner if it's needed. more details: [design doc](https:/sonic-net/SONiC/blob/master/doc/kubernetes/health-check.md) #### How to verify it Check path /usr/bin/readiness_probe.sh inside container. #### Which release branch to backport (provide reason below if selected) - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [x] 202205 - [x] 202211 #### Tested branch (Please provide the tested image version) - [x] 20220531.28

mssonicbld · 2023-07-17T15:22:05Z

Cherry-pick PR to 202305: #15867

#### Why I did it After k8s upgrade a container, k8s can only know the container is running, don't know the service's status inside container. So we need a probe inside container, k8s will call the probe to check whether the container is really ready. ##### Work item tracking - Microsoft ADO **(number only)**: 22453004 #### How I did it Add a health check probe inside config engine container, the probe will check whether the start service exit normally or not if the start service exists and call the python script to do container self-related specific checks if the script is there. The python script should be implemented by feature owner if it's needed. more details: [design doc](https:/sonic-net/SONiC/blob/master/doc/kubernetes/health-check.md) #### How to verify it Check path /usr/bin/readiness_probe.sh inside container. #### Which release branch to backport (provide reason below if selected) - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [x] 202205 - [x] 202211 #### Tested branch (Please provide the tested image version) - [x] 20220531.28 Co-authored-by: lixiaoyuner <[email protected]>

#### Why I did it After k8s upgrade a container, k8s can only know the container is running, don't know the service's status inside container. So we need a probe inside container, k8s will call the probe to check whether the container is really ready. ##### Work item tracking - Microsoft ADO **(number only)**: 22453004 #### How I did it Add a health check probe inside config engine container, the probe will check whether the start service exit normally or not if the start service exists and call the python script to do container self-related specific checks if the script is there. The python script should be implemented by feature owner if it's needed. more details: [design doc](https:/sonic-net/SONiC/blob/master/doc/kubernetes/health-check.md) #### How to verify it Check path /usr/bin/readiness_probe.sh inside container. #### Which release branch to backport (provide reason below if selected) - [ ] 201811 - [ ] 201911 - [ ] 202006 - [ ] 202012 - [ ] 202106 - [ ] 202111 - [x] 202205 - [x] 202211 #### Tested branch (Please provide the tested image version) - [x] 20220531.28

…sonic-buildimage into internal Fix conflict for rsyslog. Skip partial DNS unit test in internal branch after confirmed with Gang. Related work items: sonic-net#113, sonic-net#131, sonic-net#132, sonic-net#134, sonic-net#321, sonic-net#331, sonic-net#381, sonic-net#382, sonic-net#2525, sonic-net#2676, sonic-net#2698, sonic-net#2737, sonic-net#2789, sonic-net#2839, sonic-net#2845, sonic-net#2850, sonic-net#2882, sonic-net#2885, sonic-net#2887, sonic-net#2890, sonic-net#2895, sonic-net#13338, sonic-net#14105, sonic-net#15142, sonic-net#15223, sonic-net#15456, sonic-net#15487, sonic-net#15520, sonic-net#15726, sonic-net#15727, sonic-net#15758, sonic-net#15764, sonic-net#15765, sonic-net#15772, sonic-net#15779, sonic-net#15782, sonic-net#15785, sonic-net#15797, sonic-net#15798, sonic-net#15810, sonic-net#15811, sonic-net#15821

Add health check probe for k8s upgrade containers

d819b27

lixiaoyuner requested review from qiluo-msft, xumia and lguohan as code owners May 26, 2023 03:31

lixiaoyuner added 2 commits May 26, 2023 13:29

Update

4b53a12

Update comments

a74b408

losha228 approved these changes May 26, 2023

View reviewed changes

qiluo-msft reviewed Jun 22, 2023

View reviewed changes

Fix typo issue

28ae09b

lixiaoyuner and others added 3 commits July 3, 2023 18:52

Update the hook script to an executable file

30b4c06

Update check start service logic

353c0c0

Merge branch 'master' into add-health-check-probe

95008f7

qiluo-msft previously approved these changes Jul 7, 2023

View reviewed changes

Merge branch 'master' into add-health-check-probe

f914042

lixiaoyuner dismissed qiluo-msft’s stale review via f914042 July 7, 2023 09:52

qiluo-msft approved these changes Jul 7, 2023

View reviewed changes

qiluo-msft merged commit c470b7d into sonic-net:master Jul 11, 2023
17 checks passed

lixiaoyuner added Request for 202205 Branch Request for 202211 Branch Request for 202305 Branch labels Jul 11, 2023

yxieca added Approved for 202205 Branch Approved for 202211 Branch labels Jul 13, 2023

mssonicbld added the Created PR to 202205 Branch label Jul 13, 2023

mssonicbld mentioned this pull request Jul 13, 2023

[action] [PR:15223] Add health check probe for k8s upgrade containers. #15823

Merged

10 tasks

mssonicbld added the Created PR to 202211 Branch label Jul 13, 2023

mssonicbld mentioned this pull request Jul 13, 2023

[action] [PR:15223] Add health check probe for k8s upgrade containers. #15824

Merged

10 tasks

mssonicbld added Included in 202205 Branch and removed Approved for 202205 Branch Created PR to 202205 Branch labels Jul 13, 2023

mssonicbld added Included in 202211 Branch and removed Approved for 202211 Branch Created PR to 202211 Branch labels Jul 14, 2023

StormLiangMS added the Approved for 202305 Branch label Jul 17, 2023

mssonicbld added the Created PR to 202305 Branch label Jul 17, 2023

mssonicbld mentioned this pull request Jul 17, 2023

[action] [PR:15223] Add health check probe for k8s upgrade containers. #15867

Merged

10 tasks

mssonicbld added Included in 202305 Branch and removed Created PR to 202305 Branch labels Jul 19, 2023

lixiaoyuner deleted the add-health-check-probe branch February 7, 2024 06:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add health check probe for k8s upgrade containers. #15223

Add health check probe for k8s upgrade containers. #15223

lixiaoyuner commented May 26, 2023 •

edited

Loading

losha228 left a comment

qiluo-msft Jun 22, 2023 •

edited

Loading

lixiaoyuner Jun 22, 2023

qiluo-msft Jun 22, 2023 •

edited

Loading

lixiaoyuner Jun 22, 2023

qiluo-msft Jul 4, 2023

lixiaoyuner Jul 4, 2023

qiluo-msft Jun 22, 2023 •

edited

Loading

lixiaoyuner Jun 22, 2023 •

edited

Loading

qiluo-msft Jun 22, 2023 •

edited

Loading

lixiaoyuner Jun 22, 2023

lguohan commented Jun 30, 2023

lixiaoyuner commented Jul 3, 2023 •

edited

Loading

mssonicbld commented Jul 13, 2023

mssonicbld commented Jul 13, 2023

mssonicbld commented Jul 17, 2023

Add health check probe for k8s upgrade containers. #15223

Add health check probe for k8s upgrade containers. #15223

Conversation

lixiaoyuner commented May 26, 2023 • edited Loading

Why I did it

Work item tracking

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Tested branch (Please provide the tested image version)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

losha228 left a comment

Choose a reason for hiding this comment

qiluo-msft Jun 22, 2023 • edited Loading

Choose a reason for hiding this comment

lixiaoyuner Jun 22, 2023

Choose a reason for hiding this comment

qiluo-msft Jun 22, 2023 • edited Loading

Choose a reason for hiding this comment

lixiaoyuner Jun 22, 2023

Choose a reason for hiding this comment

qiluo-msft Jul 4, 2023

Choose a reason for hiding this comment

lixiaoyuner Jul 4, 2023

Choose a reason for hiding this comment

qiluo-msft Jun 22, 2023 • edited Loading

Choose a reason for hiding this comment

lixiaoyuner Jun 22, 2023 • edited Loading

Choose a reason for hiding this comment

qiluo-msft Jun 22, 2023 • edited Loading

Choose a reason for hiding this comment

lixiaoyuner Jun 22, 2023

Choose a reason for hiding this comment

lguohan commented Jun 30, 2023

lixiaoyuner commented Jul 3, 2023 • edited Loading

mssonicbld commented Jul 13, 2023

mssonicbld commented Jul 13, 2023

mssonicbld commented Jul 17, 2023

lixiaoyuner commented May 26, 2023 •

edited

Loading

qiluo-msft Jun 22, 2023 •

edited

Loading

qiluo-msft Jun 22, 2023 •

edited

Loading

qiluo-msft Jun 22, 2023 •

edited

Loading

lixiaoyuner Jun 22, 2023 •

edited

Loading

qiluo-msft Jun 22, 2023 •

edited

Loading

lixiaoyuner commented Jul 3, 2023 •

edited

Loading