Restore using Restic into a deployment that been scaled down and PV/PVCs deleted hangs

[yashbhutwala]

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)

PV restore with restic hangs InProgress when scaling down deployment:

backup create
scale down postgres deployment
delete pvc, pv
restore create --from-backup
scale up postgres deployment

But works with:

backup create
delete deployment + delete pvc, pv
restore create --from-backup

What did you expect to happen:
Scaling down deployment should still result in a successful restore.

[yashbhutwala]

Perhaps due to the controller wanting to put the annotation:
velero.io/backup-name: yb-backup-0 velero.io/restore-name: yb-backup-0-20190927125644

Before the backup, I only patch the pod to have the annotation, I scale down the deployment, but the deployment does not have annotation. Velero controller wants to patch the deployment to add the backup and restore annotation and it can't ...that's where I suspect the bug is...if this helps in the analysis.

[yashbhutwala]

If it's the above case, it should be made explicit in the docs to annotate the deployment instead: https://velero.io/docs/v1.1.0/restic/

although I do see: This annotation can also be provided in a pod template spec if you use a controller to manage your pods.

[nrb]

As discussed in slack, velero.io/backup-name isn't the same annotation that's used to identify restic volumes. velero.io/backup-volumes is the annotation used for this. The same annotation can be placed directly on a pod or within a pod template spec in an object like a Deployment, DaemonSet, or StatefulSet.

To explain some of the things happening here, if the deployment is scaled to 0 and exists in the cluster, Velero won't actually try to restore that resource, as it already exists. If Velero restored the requested pods, the reconciliation loops that manage Deployments would delete the running pods because the scale is set to be 0, so Kubernetes will ensure that's the case. Until the pods are created and remain running, the current logic around restic restores won't work.

This is a consequence of how we implement restic support, and I think the current best approach will be to document this behavior.

[dsu-igeek]

The behavior is going to fail, but we shouldn't hang. Leaving as a bug.