Add support for `X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER`

crimsonfez commented

2024-05-21 06:52:45 +00:00

When I configured HTTPS via Caddy I get this error when velero tries to upload a backup.

Response: error 400 Bad Request, Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0

When I use HTTP it does not have this issue even through Caddy.

Here is the log before showing what request failed

PUT /404sys-ord/backups/test5/velero-backup.json?x-id=PutObject

When I configured HTTPS via Caddy I get this error when velero tries to upload a backup. `Response: error 400 Bad Request, Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0` When I use HTTP it does not have this issue even through Caddy. Here is the log before showing what request failed `PUT /404sys-ord/backups/test5/velero-backup.json?x-id=PutObject`

crimsonfez commented

2024-05-21 07:10:44 +00:00

Author

I believe the problem here is related to X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER

I believe the problem here is related to `X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER`

crimsonfez commented

2024-05-22 07:32:05 +00:00

Author

Turns out the aws library that Velero uses will use this type of content signing only with TLS connections, which is why this is not an issue on HTTP.

👍 1

crimsonfez changed title from ~~Invalid content sha256 hash when using HTTPS~~ to Add support for X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER

2024-05-22 07:33:21 +00:00

tabongobong commented

2024-05-24 10:56:17 +00:00

Turns out the aws library that Velero uses will use this type of content signing only with TLS connections, which is why this is not an issue on HTTP.

This seems to circumvent the issue on the velero side. Same problem seems to also be present in minio https://github.com/vmware-tanzu/velero/issues/7696#issuecomment-2078241888

> Turns out the aws library that Velero uses will use this type of content signing only with TLS connections, which is why this is not an issue on HTTP. This seems to circumvent the issue on the velero side. Same problem seems to also be present in minio https://github.com/vmware-tanzu/velero/issues/7696#issuecomment-2078241888

lx commented

2024-05-24 16:38:51 +00:00

Owner

I believe the problem here is related to X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER

Indeed, it's not implemented currently.

> I believe the problem here is related to X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER Indeed, it's not implemented currently.

lx added the

scope

s3-api

label 2024-05-24 16:39:00 +00:00

quentin added the

kind

wrong-behavior

label 2024-08-07 10:32:06 +00:00

quentin added the

action

need-funding

label 2024-08-07 10:37:20 +00:00

quentin commented

2025-01-17 14:20:33 +00:00

Owner

Here is a quick way to reproduce the issue:

export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx
export AWS_DEFAULT_REGION=xxx
export AWS_ENDPOINT_URL=xxx

python -m venv .venv
source .venv/bin/activate
pip install boto3

cat > test.py <<EOF
import boto3
client = boto3.client('s3')
client.put_object(Body=b'hello world', Bucket='quentin.dufour.io', Key='test.s3.txt')
EOF

python test.py

And you'll have this error:

botocore.errorfactory.InvalidRequest: An error occurred (InvalidRequest) when calling the PutObject operation: Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0

Here are the generated headers:

headers={
  'User-Agent': b'Boto3/1.36.1 md/Botocore#1.36.1 ua/2.0 os/linux#6.6.69 md/arch#x86_64 lang/python#3.12.7 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.36.1', 
  'Expect': b'100-continue', 
  'Transfer-Encoding': b'chunked', 
  'Content-Encoding': b'aws-chunked', 
  'X-Amz-Trailer': b'x-amz-checksum-crc32', 
  'X-Amz-Decoded-Content-Length': b'11', 
  'x-amz-sdk-checksum-algorithm': b'CRC32', 
  'X-Amz-Date': b'20250117T141200Z', 
  'X-Amz-Content-SHA256': b'STREAMING-UNSIGNED-PAYLOAD-TRAILER', 
  'Authorization': b'AWS4-HMAC-SHA256 Credential=<REDACTED>/20250117/garage/s3/aws4_request, SignedHeaders=content-encoding;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer, Signature=<REDACTED>',
 'amz-sdk-invocation-id': b'f507119e-964f-47fb-a9d9-567978875026', 
  'amz-sdk-request': b'attempt=1'
}

Here are the relevant links on Amazon new data integrity policy:

Here are some people encountering the issue:

https://github.com/matrix-org/synapse-s3-storage-provider/issues/121

Here is a quick way to reproduce the issue: ```bash export AWS_ACCESS_KEY_ID=xxx export AWS_SECRET_ACCESS_KEY=xxx export AWS_DEFAULT_REGION=xxx export AWS_ENDPOINT_URL=xxx python -m venv .venv source .venv/bin/activate pip install boto3 cat > test.py <<EOF import boto3 client = boto3.client('s3') client.put_object(Body=b'hello world', Bucket='quentin.dufour.io', Key='test.s3.txt') EOF python test.py ``` And you'll have this error: ``` botocore.errorfactory.InvalidRequest: An error occurred (InvalidRequest) when calling the PutObject operation: Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0 ``` Here are the generated headers: ``` headers={ 'User-Agent': b'Boto3/1.36.1 md/Botocore#1.36.1 ua/2.0 os/linux#6.6.69 md/arch#x86_64 lang/python#3.12.7 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.36.1', 'Expect': b'100-continue', 'Transfer-Encoding': b'chunked', 'Content-Encoding': b'aws-chunked', 'X-Amz-Trailer': b'x-amz-checksum-crc32', 'X-Amz-Decoded-Content-Length': b'11', 'x-amz-sdk-checksum-algorithm': b'CRC32', 'X-Amz-Date': b'20250117T141200Z', 'X-Amz-Content-SHA256': b'STREAMING-UNSIGNED-PAYLOAD-TRAILER', 'Authorization': b'AWS4-HMAC-SHA256 Credential=<REDACTED>/20250117/garage/s3/aws4_request, SignedHeaders=content-encoding;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer, Signature=<REDACTED>', 'amz-sdk-invocation-id': b'f507119e-964f-47fb-a9d9-567978875026', 'amz-sdk-request': b'attempt=1' } ``` Here are the relevant links on Amazon new data integrity policy: - https://github.com/boto/boto3/issues/4400 - https://github.com/boto/boto3/issues/4392 - https://docs.aws.amazon.com/sdkref/latest/guide/feature-dataintegrity.html - https://aws.amazon.com/fr/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/ Here are some people encountering the issue: - https://github.com/matrix-org/synapse-s3-storage-provider/issues/121

quentin commented

2025-01-17 14:28:50 +00:00

Owner

From minio source code:

// http Header "x-amz-content-sha256" == "UNSIGNED-PAYLOAD" indicates that the
// client did not calculate sha256 of the payload.
const unsignedPayload = "UNSIGNED-PAYLOAD"

// http Header "x-amz-content-sha256" == "STREAMING-UNSIGNED-PAYLOAD-TRAILER" indicates that the
// client did not calculate sha256 of the payload and there is a trailer.
const unsignedPayloadTrailer = "STREAMING-UNSIGNED-PAYLOAD-TRAILER"

https://github.com/minio/minio/blob/master/cmd/signature-v4-utils.go#L37-L43

From minio source code: ```go // http Header "x-amz-content-sha256" == "UNSIGNED-PAYLOAD" indicates that the // client did not calculate sha256 of the payload. const unsignedPayload = "UNSIGNED-PAYLOAD" // http Header "x-amz-content-sha256" == "STREAMING-UNSIGNED-PAYLOAD-TRAILER" indicates that the // client did not calculate sha256 of the payload and there is a trailer. const unsignedPayloadTrailer = "STREAMING-UNSIGNED-PAYLOAD-TRAILER" ``` https://github.com/minio/minio/blob/master/cmd/signature-v4-utils.go#L37-L43

quentin commented

2025-01-17 14:43:41 +00:00

Owner

The list of all possible values for x-amz-content-sha256:

Header value	Description
Actual payload checksum value	This value is the actual checksum of your object and is only possible when you are uploading the data in a single chunk.
UNSIGNED-PAYLOAD	Use this when you are uploading the object as a single unsigned chunk.
STREAMING-UNSIGNED-PAYLOAD-TRAILER	Use this when sending an unsigned payload over multiple chunks. In this case you also have a trailing header after the chunk is uploaded.
STREAMING-AWS4-HMAC-SHA256-PAYLOAD	Use this when sending a payload over multiple chunks, and the chunks are signed using AWS4-HMAC-SHA256. This produces a SigV4 signature.
STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER	Use this when sending a payload over multiple chunks, and the chunks are signed using AWS4-HMAC-SHA256. This produces a SigV4 signature. In addition, the digest for the chunks is included as a trailing header.
STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD	Use this when sending a payload over multiple chunks, and the chunks are signed using AWS4-ECDSA-P256-SHA256. This produces a SigV4A signature.
STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER	Use this when sending a payload over multiple chunks, and the chunks are signed using AWS4-ECDSA-P256-SHA256. This produces a SigV4A signature. In addition, the digest for the chunks is included as a trailing header.

https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

The list of all possible values for `x-amz-content-sha256`: | Header value | Description | |-|-| | Actual payload checksum value | This value is the actual checksum of your object and is only possible when you are uploading the data in a single chunk. | | UNSIGNED-PAYLOAD | Use this when you are uploading the object as a single unsigned chunk. | STREAMING-UNSIGNED-PAYLOAD-TRAILER | Use this when sending an unsigned payload over multiple chunks. In this case you also have a trailing header after the chunk is uploaded. | STREAMING-AWS4-HMAC-SHA256-PAYLOAD | Use this when sending a payload over multiple chunks, and the chunks are signed using AWS4-HMAC-SHA256. This produces a SigV4 signature. | STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER | Use this when sending a payload over multiple chunks, and the chunks are signed using AWS4-HMAC-SHA256. This produces a SigV4 signature. In addition, the digest for the chunks is included as a trailing header. | STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD | Use this when sending a payload over multiple chunks, and the chunks are signed using AWS4-ECDSA-P256-SHA256. This produces a SigV4A signature. | STREAMING-AWS4-ECDSA-P256-SHA256-PAYLOAD-TRAILER | Use this when sending a payload over multiple chunks, and the chunks are signed using AWS4-ECDSA-P256-SHA256. This produces a SigV4A signature. In addition, the digest for the chunks is included as a trailing header. https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

baptiste commented

2025-01-20 11:21:24 +00:00

Owner

The context may not be clear: Amazon recently updated all the official S3 clients to make "Data Integrity Protections for Amazon S3" mandatory, without any consideration for third-party S3 services that don't implement it.

This effectively breaks access to Garage (and several others third-part S3 implementations) from many S3 clients.

A workaround is to downgrade to a previous version of affected S3 clients.

Some references:

The context may not be clear: Amazon recently updated all the official S3 clients to make "Data Integrity Protections for Amazon S3" mandatory, without any consideration for third-party S3 services that don't implement it. This effectively breaks access to Garage (and several others third-part S3 implementations) from many S3 clients. A workaround is to downgrade to a previous version of affected S3 clients. Some references: * https://github.com/boto/boto3/issues/4392 * https://status.digitalocean.com/incidents/zbrpd3j7hrrd

lx added this to the v1.1 milestone 2025-02-15 15:28:25 +00:00

lx added the

prio

critical

label 2025-02-15 15:29:25 +00:00

lx referenced this issue from a pull request that will close it,

2025-02-16 17:40:43 +00:00

implement STREAMING-*-PAYLOAD-TRAILER #960

lx closed this issue

2025-02-19 09:59:33 +00:00

lx referenced this issue

2025-02-20 18:27:49 +00:00

WIP: Garage v1.1.0 #968

Rows
Columns

Add support for X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER #824

Add support for `X-Amz-Content-Sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER` #824