Metadata keys are returned with weird casing (instead of all-lowercase)

zotan commented

2024-07-25 23:38:42 +00:00

I'm getting a Invalid Signature response when sending a CopyObject request from boto3 (specifically, when running this zulip migration: 7c07ea86ee/zerver/migrations/0509_fix_emoji_metadata.py (L77-L82)).

Trace logs:

2024-07-25T23:33:30.252896Z  INFO garage_api::generic_server: 2a01:4f8:2191:2f8c::1 (via [::1]:57922) PUT /2/emoji/images/1.png
2024-07-25T23:33:30.252902Z DEBUG garage_api::generic_server: Request { method: PUT, uri: /2/emoji/images/1.png, version: HTTP/1.1, headers: {"host": "iceshrimp-chat-avatars.s3.iceshrimp.dev", "user-agent": "Boto3/1.34.149 md/Botocore#1.34.149 ua/2.0 os/linux#6.9.7-arch1-1 md/arch#x86_64 lang/python#3.12.4 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.149 Resource", "content-length": "0", "accept-encoding": "identity", "amz-sdk-invocation-id": "c9831324-1306-41c9-b89e-60b44f0b0001", "amz-sdk-request": "attempt=1", "authorization": "AWS4-HMAC-SHA256 Credential=GKa83e13c5a504c16ddf718bf3/20240725/garage/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-meta-realm_id;x-amz-meta-user_profile_id;x-amz-metadata-directive, Signature=aec915b7abe242253fe86228de6cb1d377654a5f420b971f3ba56216a383e352", "content-type": "image/png", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-copy-source": "iceshrimp-chat-avatars/2/emoji/images/1.png", "x-amz-date": "20240725T233330Z", "x-amz-meta-realm_id": "2", "x-amz-meta-user_profile_id": "8", "x-amz-metadata-directive": "REPLACE", "x-forwarded-for": "2a01:4f8:2191:2f8c::1", "x-forwarded-host": "iceshrimp-chat-avatars.s3.iceshrimp.dev", "x-forwarded-proto": "https"}, body: Body(Empty) }
2024-07-25T23:33:30.252916Z DEBUG garage_api::generic_server: Endpoint: CopyObject
2024-07-25T23:33:30.252935Z TRACE garage_api::signature::payload: canonical request:
PUT
/2/emoji/images/1.png
content-type:image/png
host:iceshrimp-chat-avatars.s3.iceshrimp.dev
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-copy-source:iceshrimp-chat-avatars/2/emoji/images/1.png
x-amz-date:20240725T233330Z
x-amz-meta-realm_id:2
x-amz-meta-user_profile_id:8
x-amz-metadata-directive:REPLACE
content-type;host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-meta-realm_id;x-amz-meta-user_profile_id;x-amz-metadata-directive
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-07-25T23:33:30.252937Z TRACE garage_api::signature::payload: string to sign:
AWS4-HMAC-SHA256
20240725T233330Z
20240725/garage/s3/aws4_request
08f003341fe4032116e666a6b215ea22536731a43bcd89444ce5b6de7225896c
2024-07-25T23:33:30.252969Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature

I'm getting a `Invalid Signature` response when sending a `CopyObject` request from boto3 (specifically, when running this zulip migration: https://github.com/zulip/zulip/blob/7c07ea86ee59581685ce189def18edc3e7294e11/zerver/migrations/0509_fix_emoji_metadata.py#L77-L82). Trace logs: ``` 2024-07-25T23:33:30.252896Z INFO garage_api::generic_server: 2a01:4f8:2191:2f8c::1 (via [::1]:57922) PUT /2/emoji/images/1.png 2024-07-25T23:33:30.252902Z DEBUG garage_api::generic_server: Request { method: PUT, uri: /2/emoji/images/1.png, version: HTTP/1.1, headers: {"host": "iceshrimp-chat-avatars.s3.iceshrimp.dev", "user-agent": "Boto3/1.34.149 md/Botocore#1.34.149 ua/2.0 os/linux#6.9.7-arch1-1 md/arch#x86_64 lang/python#3.12.4 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.149 Resource", "content-length": "0", "accept-encoding": "identity", "amz-sdk-invocation-id": "c9831324-1306-41c9-b89e-60b44f0b0001", "amz-sdk-request": "attempt=1", "authorization": "AWS4-HMAC-SHA256 Credential=GKa83e13c5a504c16ddf718bf3/20240725/garage/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-meta-realm_id;x-amz-meta-user_profile_id;x-amz-metadata-directive, Signature=aec915b7abe242253fe86228de6cb1d377654a5f420b971f3ba56216a383e352", "content-type": "image/png", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-copy-source": "iceshrimp-chat-avatars/2/emoji/images/1.png", "x-amz-date": "20240725T233330Z", "x-amz-meta-realm_id": "2", "x-amz-meta-user_profile_id": "8", "x-amz-metadata-directive": "REPLACE", "x-forwarded-for": "2a01:4f8:2191:2f8c::1", "x-forwarded-host": "iceshrimp-chat-avatars.s3.iceshrimp.dev", "x-forwarded-proto": "https"}, body: Body(Empty) } 2024-07-25T23:33:30.252916Z DEBUG garage_api::generic_server: Endpoint: CopyObject 2024-07-25T23:33:30.252935Z TRACE garage_api::signature::payload: canonical request: PUT /2/emoji/images/1.png content-type:image/png host:iceshrimp-chat-avatars.s3.iceshrimp.dev x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-copy-source:iceshrimp-chat-avatars/2/emoji/images/1.png x-amz-date:20240725T233330Z x-amz-meta-realm_id:2 x-amz-meta-user_profile_id:8 x-amz-metadata-directive:REPLACE content-type;host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-meta-realm_id;x-amz-meta-user_profile_id;x-amz-metadata-directive e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-07-25T23:33:30.252937Z TRACE garage_api::signature::payload: string to sign: AWS4-HMAC-SHA256 20240725T233330Z 20240725/garage/s3/aws4_request 08f003341fe4032116e666a6b215ea22536731a43bcd89444ce5b6de7225896c 2024-07-25T23:33:30.252969Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature ```

zotan commented

2024-07-26 17:43:54 +00:00

Author

Update: traced this down to the following difference in the hashed canonical request:
boto3 signed it as

x-amz-meta-realm_id:2,2
x-amz-meta-user_profile_id:8,8

This comes from there being duplicate keys in the metadata collection, one with an uppercase first letter, and one with a lowercase one.

From what I can tell, garage seems to be capitalizing the first letter of metadata keys, as zulip is sending them all-lowercase (and is expecting to receive them as such). AWS says that these should be case-insensitive, and all-lowercase: https://docs.aws.amazon.com/codeguru/detector-library/java/s3-object-user-metadata-key-case-sensitivity/

Update: traced this down to the following difference in the hashed canonical request: boto3 signed it as ``` x-amz-meta-realm_id:2,2 x-amz-meta-user_profile_id:8,8 ``` This comes from there being duplicate keys in the metadata collection, one with an uppercase first letter, and one with a lowercase one. From what I can tell, garage seems to be capitalizing the first letter of metadata keys, as zulip is sending them all-lowercase (and is expecting to receive them as such). AWS says that these should be case-insensitive, and all-lowercase: https://docs.aws.amazon.com/codeguru/detector-library/java/s3-object-user-metadata-key-case-sensitivity/

zotan changed title from ~~Invalid signature on CopyObject request sent from boto3~~ to Metadata keys are returned with weird casing (instead of all-lowercase)

2024-07-26 17:58:09 +00:00

zotan commented

2024-07-26 18:10:36 +00:00

Author

I think the ideal solution would be to return metadata keys all-lowercase, or at least preserve the casing that's sent. The current implementation seems to break things.

quentin commented

2024-08-06 07:41:34 +00:00

Owner

Custom object headers are sent here during a GET/HEAD:
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/get.rs#L66-L83

Model to store them is defined here:
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/model/s3/object_table.rs#L267-L274

They are persisted here durig a PUT:
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/put.rs#L72-L75

More especially, the persisted HeaderList is computed from the HTTP framework headers in the get_headers() function: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/put.rs#L621-L629

IMO this one is a good candidate to put a to_lowercase call.

Custom object headers are sent here during a GET/HEAD: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/get.rs#L66-L83 Model to store them is defined here: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/model/s3/object_table.rs#L267-L274 They are persisted here durig a PUT: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/put.rs#L72-L75 More especially, the persisted `HeaderList` is computed from the HTTP framework headers in the `get_headers()` function: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/put.rs#L621-L629 IMO this one is a good candidate to put a `to_lowercase` call.

quentin added the

labels 2024-08-06 07:41:51 +00:00

quentin added this to the v1.1 milestone 2024-08-06 07:41:54 +00:00

quentin added the

kind

wrong-behavior

label 2024-08-07 09:36:26 +00:00

lx closed this issue

2025-01-27 18:31:12 +00:00

lx reopened this issue

2025-01-27 18:31:16 +00:00

lx commented

2025-01-27 18:32:50 +00:00

Owner

I think the ideal solution would be to return metadata keys all-lowercase, or at least preserve the casing that's sent. The current implementation seems to break things.

It's not clear to me what part of the code is adding an uppercase letter when the header was sent in all lowercase, but I agree that adding a to_lowercase call where @quentin said makes us closer to the behavior of AWS

> I think the ideal solution would be to return metadata keys all-lowercase, or at least preserve the casing that's sent. The current implementation seems to break things. It's not clear to me what part of the code is adding an uppercase letter when the header was sent in all lowercase, but I agree that adding a `to_lowercase` call where @quentin said makes us closer to the behavior of AWS

lx referenced this issue from a commit

2025-01-27 18:58:29 +00:00

s3 api: make x-amz-meta-* headers lowercase (fix #844)

lx referenced this issue from a pull request that will close it,

2025-01-27 18:58:39 +00:00

s3 api: make x-amz-meta-* headers lowercase (fix #844) #938

lx closed this issue

2025-01-27 19:32:20 +00:00

lx referenced this issue from a commit

2025-01-27 19:32:21 +00:00

Merge pull request 's3 api: make x-amz-meta-* headers lowercase (fix #844)' (#938) from fix-844 into main

lx referenced this issue

2025-02-20 18:21:00 +00:00

WIP: Garage v1.1.0 #968

Rows
Columns

Metadata keys are returned with weird casing (instead of all-lowercase) #844