Velero AWS plugin: S3 compatibility broken with Qumulo in recent versions

[Mise]

Hello,

We are attempting to integrate Velero and Velero plugin for AWS with Qumulo storage, which the vendor claims is fully compatible with AWS S3.

However, it seems that the requests made by the Velero AWS plugin to our storage include parameters that it does not recognize. For example, some requests fail due to the inclusion of the x-id parameter, for example:

GET https://storage.domain.tld:443/bucketname/backups/test-backup-qumulo/test-backup-qumulo-volumeinfo.json.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=blablaaws4_request&X-Amz-Date=20241015T093024Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&x-id=GetObject&X-Amz-Signature=blabla"

Other requests, such as PutObject requests, fail similarly. The storage vendor asserts that x-id parameter is not part of the S3 specification, and I could not find any information about it in the S3 API reference.

We tried Velero versions 1.13.2 and 1.14.1 and AWS plugin versions 1.9.0 and 1.10, which exhibit the same behavior, and a combination of older Velero v1.12.4 with AWS plugin v1.8.2, which work correctly.

Could you please advise why newer Velero and plugin versions are including the x-id parameter in their S3 requests, and whether there is a way to configure Velero to omit this parameter when communicating with our S3 storage?

Thank you!

[kaovilai]

dupe of #8265

[kaovilai]

bump to aws-sdk-go-v2 forced addition of some headers and drop usage of older "legacy" (as aws determined) apis.

Everyone who claims S3 compatible are only via testing/replicating functionality, which can get out of date and eventually incompatible with what aws uses overtime.

[kaovilai]

"x-id" is an expected part of requests made by aws-sdk-go-v2
https:/aws/aws-sdk-go-v2/blob/04e7aca073a0a7ed479aa37cad88a1cf58a979a1/service/s3/internal/customizations/presign_test.go#L35-L41

[blackpiglet]

/assign @kaovilai

[kzakhark]

"x-id" is an expected part of requests made by aws-sdk-go-v2 https:/aws/aws-sdk-go-v2/blob/04e7aca073a0a7ed479aa37cad88a1cf58a979a1/service/s3/internal/customizations/presign_test.go#L35-L41

Is there any indication of why it is an expected part, as there's no mention of this parameter in the API reference?