[Bug] Unable to smoothly upgrade from 0.2.2 to 0.2.6

[cserwen]

1. Question:

• Open batch push, upgrade from 0.2.2 to 0.2.6 version
• Operate the slave node, master node, slave node in sequence, and the last slave node will not be able to synchronize

2. Reason

• 0.2.2 return to master is endIndex, and count of response is 0, so the watermark of slave will be less 1 than real value.

  dledger/src/main/java/io/openmessaging/storage/dledger/DLedgerEntryPusher.java

  Line 601 in d207481

  updatePeerWaterMark(x.getTerm(), peerId, x.getIndex() + x.getCount() - 1);

• when last slave upgrade, follow code may will be run
  

  dledger/src/main/java/io/openmessaging/storage/dledger/DLedgerEntryPusher.java

  Line 672 in d207481

  doCheckBatchAppendResponse();

• But in doCheckBatchAppendResponse, peerWaterMark is less 1 than real. so peerWaterMark + 1 will be endIndex and it is not stored in batchPendingMap. So pair will always be null and pusher will always execute the check logic.

private void doCheckBatchAppendResponse() throws Exception {
    long peerWaterMark = getPeerWaterMark(term, peerId);
    Pair pair = batchPendingMap.get(peerWaterMark + 1);
    if (pair != null && System.currentTimeMillis() - (long) pair.getKey() > dLedgerConfig.getMaxPushTimeOutMs()) {
        long firstIndex = peerWaterMark + 1;
        long lastIndex = firstIndex + (int) pair.getValue() - 1;
        logger.warn("[Push-{}]Retry to push entry from {} to {}", peerId, firstIndex, lastIndex);
        batchAppendEntryRequest.clear();
        for (long i = firstIndex; i <= lastIndex; i++) {
            DLedgerEntry entry = dLedgerStore.get(i);
            batchAppendEntryRequest.addEntry(entry);
        }
        sendBatchAppendEntryRequest();
    }
}

[RongtongJin]

Good catch