NoCredentialProviders: no valid providers in chain.

[Rohmilchkaese]

What steps did you take and what happened:
No matter how I pass my S3 Creds to Velero - it keeps failing.

What did you expect to happen:
Being able to backup.

The following information will help us better understand what's going on:
I'm installing via HELM via ArgoCD:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: velero
spec:
  destination:
    namespace: velero
    server: 'https://kubernetes.default.svc'
  source:
    repoURL: 'https://vmware-tanzu.github.io/helm-charts'
    targetRevision: 4.4.0
    chart: velero
    helm:
      values: |-
        initContainers:
          - name: velero-plugin-for-aws
            image: velero/velero-plugin-for-aws:v1.7.0
            imagePullPolicy: IfNotPresent
            volumeMounts:
              - mountPath: /target
                name: plugins
        configuration:
          backupStorageLocation: 
            # name is the name of the backup storage location where backups should be stored. If a name is not provided,
            # a backup storage location will be created with the name "default". Optional.
          - name: exoscale
            # provider is the name for the backup storage location provider.
            provider: aws
            # bucket is the name of the bucket to store backups in. Required.
            bucket: cevin-open200-io
            # prefix is the directory under which all Velero data should be stored within the bucket. Optional.
            prefix: velero
            # default indicates this location is the default backup storage location. Optional.
            default: true
            # accessMode determines if velero can write to this backup storage location. Optional.
            # default to ReadWrite, ReadOnly is used during migrations and restores.
            accessMode: ReadWrite
            credential:
              # name of the secret used by this backupStorageLocation.
              name: velero-s3
              # name of key that contains the secret data to be used.
              key: exoscale
            # Additional provider-specific configuration. See link above
            # for details of required/optional fields for your provider.
            config: 
              region: AT-VIE-1
              s3ForcePathStyle: false
              publicUrl: https://sos-at-vie-1.exo.io
              profile: exoscale
              enableSharedConfig: true
            #  s3Url:
            #  kmsKeyId:
            #  resourceGroup:
            #  The ID of the subscription containing the storage account, if different from the cluster’s subscription. (Azure only)
            #  subscriptionId:
            #  storageAccount:
            #  Name of the GCP service account to use for this backup storage location. Specify the
            #  service account here if you want to use workload identity instead of providing the key file.(GCP only)
            #  serviceAccount:
            #  Option to skip certificate validation or not if insecureSkipTLSVerify is set to be true, the client side should set the
            #  flag. For Velero client Command like velero backup describe, velero backup logs needs to add the flag --insecure-skip-tls-verify
            #  insecureSkipTLSVerify:
          volumeSnapshotLocation:
          - name: exoscale
            provider: aws
            config:
              region: AT-VIE-1
              profile: exoscale

        # Info about the secret to be used by the Velero deployment, which
        # should contain credentials for the cloud provider IAM account you've
        # set up for Velero.
        credentials:
          # Whether a secret should be used. Set to false if, for examples:
          # - using kube2iam or kiam to provide AWS IAM credentials instead of providing the key file. (AWS only)
          # - using workload identity instead of providing the key file. (GCP only)
          useSecret: true
          # Name of the secret to create if `useSecret` is true and `existingSecret` is empty
          #name:
          # Name of a pre-existing secret (if any) in the Velero namespace
          # that should be used to get IAM account credentials. Optional.
          existingSecret: velero-s3
          # Data to be stored in the Velero secret, if `useSecret` is true and `existingSecret` is empty.
          # As of the current Velero release, Velero only uses one secret key/value at a time.
          # The key must be named `cloud`, and the value corresponds to the entire content of your IAM credentials file.
          # Note that the format will be different for different providers, please check their documentation.
          # Here is a list of documentation for plugins maintained by the Velero team:
          # [AWS] https:/vmware-tanzu/velero-plugin-for-aws/blob/main/README.md
          # [GCP] https:/vmware-tanzu/velero-plugin-for-gcp/blob/main/README.md
          # [Azure] https:/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/README.md
          #secretContents: {}
          #  cloud: |
          #    [default]
          #    aws_access_key_id=<REDACTED>
          #    aws_secret_access_key=<REDACTED>
          # additional key/value pairs to be used as environment variables such as "DIGITALOCEAN_TOKEN: <your-key>". Values will be stored in the secret.
          #extraEnvVars: {}
          # Name of a pre-existing secret (if any) in the Velero namespace
          # that will be used to load environment variables into velero and node-agent.
          # Secret should be in format - https://kubernetes.io/docs/concepts/configuration/secret/#use-case-as-container-environment-variables
          #extraSecretRef: ""
  project: base
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

This is my secret (velero-s3) in the velero namespace of course:

Name:         velero-s3
Namespace:    velero
Labels:       <none>
Annotations:  reflector.v1.k8s.emberstack.com/auto-reflects: True
              reflector.v1.k8s.emberstack.com/reflected-at: "2023-08-03T14:55:20.5517199+00:00"
              reflector.v1.k8s.emberstack.com/reflected-version: 557401
              reflector.v1.k8s.emberstack.com/reflects: default/velero-s3

Type:  StringData

Data
====
cloud:  112 bytes

Decoded Secret:

cloud: |
  [exoscale]
  aws_access_key_id=XX
  aws_secret_access_key=XX

Logs from Pod:

velero time="2023-08-09T13:50:24Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"exoscale\" is unavailable: unable to get credentials: unable to g
et key for secret: \"velero-s3\" secret is missing data for key \"exoscale\")" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"
velero time="2023-08-09T13:50:54Z" level=error msg="Error getting backup store for this location" backupLocation=velero/exoscale controller=backup-sync error="unable to get credentials: unable to get key for se
cret: \"velero-s3\" secret is missing data for key \"exoscale\"" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/util/kube/secrets.go:47" error.function=github.com/vmware-tanzu/velero/pkg/util/kube.GetSe
cretKey logSource="pkg/controller/backup_sync_controller.go:100"
velero time="2023-08-09T13:51:24Z" level=error msg="Error getting a backup store" backup-storage-location=velero/exoscale controller=backup-storage-location error="unable to get credentials: unable to get key f
or secret: \"velero-s3\" secret is missing data for key \"exoscale\"" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/util/kube/secrets.go:47" error.function=github.com/vmware-tanzu/velero/pkg/util/kube.
GetSecretKey logSource="pkg/controller/backup_storage_location_controller.go:148"
velero time="2023-08-09T13:51:24Z" level=info msg="BackupStorageLocation is invalid, marking as unavailable" backup-storage-location=velero/exoscale controller=backup-storage-location logSource="pkg/controller/
backup_storage_location_controller.go:131"
velero time="2023-08-09T13:51:24Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"exoscale\" is unavailable: unable to get credentials: unable to g
et key for secret: \"velero-s3\" secret is missing data for key \"exoscale\")" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:192"

Anything else you would like to add:

Environment:

• Velero version (use velero version): 1.11.1
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version): 1.26.6
• Kubernetes installer & version:
• Cloud provider or hardware configuration: -> 8 Cores / 16 GB Ram for each Worker
• OS (e.g. from /etc/os-release): Debian 12

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

The S3 Creds are working - tested them of course.
Otherwise I might be missing something, I'm super grateful, if you let me know ;-)

One more thing to mention:
My secret comes from external secrets operator:

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: velero-s3
spec:
  refreshInterval: "3600s"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: velero-s3
    template:
      engineVersion: v2
      type: Opaque
      data:
        cloud: |
          [exoscale]
          aws_access_key_id={{ .aws_access_key_id }}
          aws_secret_access_key={{ .aws_secret_access_key }}
      templateFrom:
        - target: Annotations
          literal: |
            reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
            reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
            reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "velero"
  data:
  - secretKey: aws_access_key_id
    remoteRef:
      key: PATH
      property: exoscale_access_key
  - secretKey: aws_secret_access_key
    remoteRef:
      key: PATH
      property: exoscale_secret_key

I tried various variations of the secret. I had many times it was complaining about not being able to find the "exoscale" key, which I think is not the problem anymore.

Its basically a repost of:
#6601.

Sorry, but I would love to get that fixed and I dont see the problem to be honest.

[Rohmilchkaese]

Thanks solved myself