k3s-server creates improper etcd member name

[brandond]

K3s tracking issue for:

• rke2-server creates improper etcd member name rancher/rke2#5482

Appears to be a race condition caused by running etcd-snapshot commands while the server is initially starting up. If there are no etcd files, the snapshot command would bail out immediately, but if some of the files exist, the command would create the etcd/name file with an empty hostname, before the server can get around to doing it with the hostname set.

This does not appear to have any serious negative impact on the cluster, other than the status condition not being set properly due to it failing to find any members whose name contains the node name. This could also be handled better, since the etcd member name is available in annotation and we don't need to rely on hostname matching.

[ShylajaDevadiga]

Reproduced issue on rke2 v1.26.14+rke2r1 and rancher v2.7.11

ubuntu@shyr26-pool1-64db900c-222bq:~$ rke2 -v
rke2 version v1.26.14+rke2r1 (93518400c62043756be7e321480bcd5f19f42bbf)
go version go1.21.7 X:boringcrypto
ubuntu@shyr26-pool1-64db900c-222bq:~$ sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/rke2/server/tls/etcd/server-client.crt --key /var/lib/rancher/rke2/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt member list -w table
+------------------+---------+-----------+---------------------------+---------------------------+
|        ID        | STATUS  |   NAME    |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+-----------+---------------------------+---------------------------+
|  9448c44cf1aee45 | started | -3e0b0d3a | https://172.31.12.18:2380 | https://172.31.12.18:2379 |
| d54e1ad5efacd6e0 | started | -d80fae6b |  https://172.31.1.17:2380 |  https://172.31.1.17:2379 |
| d5cd023dd1503387 | started | -6fe57d4c | https://172.31.3.236:2380 | https://172.31.3.236:2379 |
+------------------+---------+-----------+---------------------------+---------------------------+
ubuntu@shyr26-pool1-64db900c-222bq:~$ 

Issue is not reproducible in k3s

ubuntu@shyk26-pool1-9efa54f4-dlv6c:~$ k3s -v
k3s version v1.26.14+k3s1 (c7e6922a)
go version go1.21.7
ubuntu@shyk26-pool1-9efa54f4-dlv6c:~$ sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt member list
54722fc798b4f59, started, shyk26-pool1-9efa54f4-qnqwj-0eaa1c2b, https://172.31.3.161:2380, https://172.31.3.161:2379
3ecbd4674c04ed16, started, shyk26-pool1-9efa54f4-dlv6c-187b7a20, https://172.31.12.53:2380, https://172.31.12.53:2379
e9b49b3917a51398, started, shyk26-pool1-9efa54f4-pwhp4-258c779b, https://172.31.0.94:2380, https://172.31.0.94:2379

Validated there is no regression in k3s using latest rc on supported versions

ubuntu@shyk3s26-pool1-3c4176c7-g7gwd:~$ k3s -v
k3s version v1.26.15-rc1+k3s1 (13297236)
go version go1.21.8
ubuntu@shyk3s26-pool1-3c4176c7-g7gwd:~$ sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt member list -w table
+------------------+---------+----------------------------------------+---------------------------+---------------------------+
|        ID        | STATUS  |                  NAME                  |        PEER ADDRS         |       CLIENT ADDRS        |
+------------------+---------+----------------------------------------+---------------------------+---------------------------+
| 5c520a59005e97d8 | started | shyk3s26-pool1-3c4176c7-g7gwd-c3d79eaa | https://172.31.9.203:2380 | https://172.31.9.203:2379 |
| 741442bac7165caf | started | shyk3s26-pool1-3c4176c7-ndcd4-64cd59f8 | https://172.31.15.85:2380 | https://172.31.15.85:2379 |
| a0130e947f251373 | started | shyk3s26-pool1-3c4176c7-mmbm4-5a643ec2 |  https://172.31.7.51:2380 |  https://172.31.7.51:2379 |
+------------------+---------+----------------------------------------+---------------------------+---------------------------+
ubuntu@shyk3s26-pool1-3c4176c7-g7gwd:~$ 

ubuntu@shyk3s27-pool1-bef584fb-928lq:~$ k3s -v
k3s version v1.27.12-rc1+k3s1 (78ad5756)
go version go1.21.8
ubuntu@shyk3s27-pool1-bef584fb-928lq:~$ sudo ETCDCTL_API=3 etcdctl --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt member list -w table
+------------------+---------+----------------------------------------+----------------------------+----------------------------+
|        ID        | STATUS  |                  NAME                  |         PEER ADDRS         |        CLIENT ADDRS        |
+------------------+---------+----------------------------------------+----------------------------+----------------------------+
| 4039854d3de5adf2 | started | shyk3s27-pool1-bef584fb-q4td9-306d2998 | https://172.31.10.253:2380 | https://172.31.10.253:2379 |
| a23a6aa1439b4994 | started | shyk3s27-pool1-bef584fb-928lq-092581e5 |  https://172.31.3.110:2380 |  https://172.31.3.110:2379 |
| fc56a40a9a3164cd | started | shyk3s27-pool1-bef584fb-skjr9-e0d136ab |    https://172.31.0.4:2380 |    https://172.31.0.4:2379 |
+------------------+---------+----------------------------------------+----------------------------+----------------------------+
ubuntu@shyk3s27-pool1-bef584fb-928lq:~$