[velero-plugin-for-aws:v1.9.1] Brakes S3 compatible external storage location - oracle cloud issue

[leftyb]

Steps followed:
Upgrade to :
velero-plugin-for-aws:v1.9.1
Velero chart version : version: "6.7.0"
Velero version: appVersion: 1.13.2

Previous versions:
velero-plugin-for-aws:v1.8.2
Velero chart version : version: "5.2.2"
Velero version: appVersion: 1.12.3

What did you expect to happen:
Velero Backups properly stored to external S3 compatible storage location.
Before upgrade was working properly.

From velero logs:
When using checksumAlgorithm: "" Following error logs:
operation error S3: PutObject, https response error StatusCode: 501, RequestID: , HostID: , api error NotImplemented: The s3 command you requested is not implemented."

When NOT setting checksumAlgorithm so by default Velero BackupStorageLocation setting to CRC32
Getting following errors from velero logs (tested all available algorithms and got same error) :
operation error S3: PutObject, https response error StatusCode: 400, RequestID: , HostID: , api error InvalidArgument: x-amz-content-sha256 must be UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256 value."

[sseago]

What s3 provider is this? We've seen issues with IBM Cloud which, in some configurations, won't work with the latest version, but I don't think we got the "not implemented" error message there. It could be that whatever s3 driver is being used needs to be upgraded to a newer version with aws sdk v2 support.

[DiouKnox]

i'm using --plugins velero/velero-plugin-for-aws:latest and having the same problem, if anyone knows how to solve it please let me know

[kaovilai]

Please specify your s3 compatible provider that hits this issue. It's been shown that not all s3 compatible has feature parity with AWS S3, which meant additional configuration may need to be done to disable/enable certain flags to make it work again.

[DiouKnox]

oh sorry, oracle cloud.
the setup used to work just fine, but i'm not sure when it stopped.

[kaovilai]

You may want to try debug logging like here and send the logs over to the respective provider support channels for implementation.

In the meantime, you could try use velero aws plugins prior to v1.9

[reasonerjt]

@DiouKnox
Since v1.9.0 we bumped up to aws-sdk-go-v2 following the suggestion by AWS.
If you set checksumAlgorithm: "" and still see the failure, it's probably a compatibility issue on oracle cloud, please check with oracle cloud engineer.

I'll keep this one open but it will likely be closed as a "won't fix" after we hear back from you.

[DiouKnox]

oracle has confirmed that it is a compatibility issue, downgrading velero version has solved the issue. thankyou for the replies.

[reasonerjt]

Thanks, I'm resovling this one.

[leftyb]

Hi all,
I apologise for the late response.
The S3 backend used is ontap S3 .
https://docs.netapp.com/us-en/ontap/s3-config/ontap-s3-supported-actions-reference.html#bucket-operations

Before the upgrade to Velero version: appVersion: 1.13.* Backend was working properly and could store backups.
After the upgrade, stopped working.
Please provide any info if any.
Thank you.