etcd revision occurs Inconsistent

[didihongsheng]

etcd version : 3.5.0
k8s version : 1.20

we are in production environment, the k8s log shown below, it shows the revision has a problem :

so we check endpoints status in etcd cluster, as shown below:

etcd raft index seems healthy, but revision is inconsistent, one node is too far behind. I know the revision is maintained in mvcc , but in normal case , if etcd raft index is heathy in cluster ,revision should not occurs like this, I don't known the reason.

[ahrtr]

Any reason why you setup etcd cluster of 3.5.0 for K8s 1.20? Note that K8s 1.20 is supposed to work with etcd 3.4.13, see go.mod#L447. There is a similar issue 13547.

Can you always see this issue when running K8s 1.12 together with etcd 3.5.0?

[serathius]

@didihongsheng One idea is that you run Etcd instances with slightly different apply logic. This could be caused by incorrectly doing cluster upgrade.

Could you specify if this cluster was created in version v3.5.0 or it was upgraded? What versions you run before this? Did you have any problems during upgrade?

[didihongsheng]

an you always see this is

this issue only appear once when the k8s's control plane crash ，then I remove the problematic member and delete its datadir ,and member add it back

[didihongsheng]

the version before 3.5 is 3.3，we update 3.3 to 3.4, then 3.4 to 3.5, I didn't meet with problems .

[serathius]

Data corruption issue was found in v3.5.[0-2] release. Please upgrade to v3.5.3.