Volume snapshot schedule shows error after successful retry

[lukegriffith]

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

What happened: Failed volume snapshot was retried, but the status on the volumesnapshotschedule still indicates an error.

# status item on the volume snapshot schedule
      - creationTimestamp: "2021-04-09T00:01:09Z"
        finishTimestamp: "2021-04-09T00:02:48Z"
        name: REDACT-APPLICATION-weekly-cloud-daily-2021-04-09-000109
        status: Error

# the completed volume snapshot 
NAME                                                   PVC               STATUS    CREATED               COMPLETED             TYPE
REDACT-APPLICATION-weekly-cloud-daily-2021-04-09-000109   REDACT-APPLICATION   Ready     09 Apr 21 01:01 BST   09 Apr 21 01:02 BST   cloud

What you expected to happen:
The volume snapshot schedule to be correctly updated with completed.

How to reproduce it (as minimally and precisely as possible): Unsure how to reproduce the snapshot failure right now

This go error seems to be the cause of the initial failure;

E0409 00:02:12.212477       1 goroutinemap.go:150] Operation for "createREDACTED/REDACTED-APPLICATION-weekly-cloud-daily-2021-04-09-000109-992c7132-7f48-4955-83c3-418bc8f6c3eeREDACTED-APPLICATION" failed. No retries permitted until 2021-04-09 00:02:12.712449701 +0000 UTC m=+2214460.338826427 (durationBeforeRetry 500ms). Error: "Failed to take snapshot of the volume pvc-f74e5996-46f0-479e-a97b-7c3842a1087e: \"rpc error: code = Internal desc = Failed to create backup: Other cloudsnaps Requests pending, try little later\""

Anything else we need to know?: We're using portworx

Environment:

• Kubernetes version (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.12-eks-7684af", GitCommit:"7684af4ac41370dd109ac13817023cb8063e3d45", GitTreeState:"clean", BuildDate:"2020-10-20T22:57:40Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: AWS
• OS (e.g. from /etc/os-release): Amazon Linux 2
• Kernel (e.g. uname -a):

Linux ip-ADDRESS.eu-central-1.compute.internal 5.4.58-32.125.amzn2.x86_64 #1 SMP Thu Sep 3 19:06:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:
• Others:

[adityadani]

@lukegriffith This looks like a reporting bug from stork. We are planning to take this fix in the next release of stork.
Thank you for reporting.

[andrewh1978]

STOR-377