Metadata keys are returned with weird casing (instead of all-lowercase) #844

New issue

Open

opened 2024-07-25 23:38:42 +00:00 by zotan · 3 comments

zotan commented

2024-07-25 23:38:42 +00:00

I'm getting a Invalid Signature response when sending a CopyObject request from boto3 (specifically, when running this zulip migration: 7c07ea86ee/zerver/migrations/0509_fix_emoji_metadata.py (L77-L82)).

Trace logs:

2024-07-25T23:33:30.252896Z  INFO garage_api::generic_server: 2a01:4f8:2191:2f8c::1 (via [::1]:57922) PUT /2/emoji/images/1.png
2024-07-25T23:33:30.252902Z DEBUG garage_api::generic_server: Request { method: PUT, uri: /2/emoji/images/1.png, version: HTTP/1.1, headers: {"host": "iceshrimp-chat-avatars.s3.iceshrimp.dev", "user-agent": "Boto3/1.34.149 md/Botocore#1.34.149 ua/2.0 os/linux#6.9.7-arch1-1 md/arch#x86_64 lang/python#3.12.4 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.149 Resource", "content-length": "0", "accept-encoding": "identity", "amz-sdk-invocation-id": "c9831324-1306-41c9-b89e-60b44f0b0001", "amz-sdk-request": "attempt=1", "authorization": "AWS4-HMAC-SHA256 Credential=GKa83e13c5a504c16ddf718bf3/20240725/garage/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-meta-realm_id;x-amz-meta-user_profile_id;x-amz-metadata-directive, Signature=aec915b7abe242253fe86228de6cb1d377654a5f420b971f3ba56216a383e352", "content-type": "image/png", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-copy-source": "iceshrimp-chat-avatars/2/emoji/images/1.png", "x-amz-date": "20240725T233330Z", "x-amz-meta-realm_id": "2", "x-amz-meta-user_profile_id": "8", "x-amz-metadata-directive": "REPLACE", "x-forwarded-for": "2a01:4f8:2191:2f8c::1", "x-forwarded-host": "iceshrimp-chat-avatars.s3.iceshrimp.dev", "x-forwarded-proto": "https"}, body: Body(Empty) }
2024-07-25T23:33:30.252916Z DEBUG garage_api::generic_server: Endpoint: CopyObject
2024-07-25T23:33:30.252935Z TRACE garage_api::signature::payload: canonical request:
PUT
/2/emoji/images/1.png
content-type:image/png
host:iceshrimp-chat-avatars.s3.iceshrimp.dev
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-copy-source:iceshrimp-chat-avatars/2/emoji/images/1.png
x-amz-date:20240725T233330Z
x-amz-meta-realm_id:2
x-amz-meta-user_profile_id:8
x-amz-metadata-directive:REPLACE
content-type;host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-meta-realm_id;x-amz-meta-user_profile_id;x-amz-metadata-directive
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-07-25T23:33:30.252937Z TRACE garage_api::signature::payload: string to sign:
AWS4-HMAC-SHA256
20240725T233330Z
20240725/garage/s3/aws4_request
08f003341fe4032116e666a6b215ea22536731a43bcd89444ce5b6de7225896c
2024-07-25T23:33:30.252969Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature

I'm getting a `Invalid Signature` response when sending a `CopyObject` request from boto3 (specifically, when running this zulip migration: https://github.com/zulip/zulip/blob/7c07ea86ee59581685ce189def18edc3e7294e11/zerver/migrations/0509_fix_emoji_metadata.py#L77-L82). Trace logs: ``` 2024-07-25T23:33:30.252896Z INFO garage_api::generic_server: 2a01:4f8:2191:2f8c::1 (via [::1]:57922) PUT /2/emoji/images/1.png 2024-07-25T23:33:30.252902Z DEBUG garage_api::generic_server: Request { method: PUT, uri: /2/emoji/images/1.png, version: HTTP/1.1, headers: {"host": "iceshrimp-chat-avatars.s3.iceshrimp.dev", "user-agent": "Boto3/1.34.149 md/Botocore#1.34.149 ua/2.0 os/linux#6.9.7-arch1-1 md/arch#x86_64 lang/python#3.12.4 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.149 Resource", "content-length": "0", "accept-encoding": "identity", "amz-sdk-invocation-id": "c9831324-1306-41c9-b89e-60b44f0b0001", "amz-sdk-request": "attempt=1", "authorization": "AWS4-HMAC-SHA256 Credential=GKa83e13c5a504c16ddf718bf3/20240725/garage/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-meta-realm_id;x-amz-meta-user_profile_id;x-amz-metadata-directive, Signature=aec915b7abe242253fe86228de6cb1d377654a5f420b971f3ba56216a383e352", "content-type": "image/png", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-copy-source": "iceshrimp-chat-avatars/2/emoji/images/1.png", "x-amz-date": "20240725T233330Z", "x-amz-meta-realm_id": "2", "x-amz-meta-user_profile_id": "8", "x-amz-metadata-directive": "REPLACE", "x-forwarded-for": "2a01:4f8:2191:2f8c::1", "x-forwarded-host": "iceshrimp-chat-avatars.s3.iceshrimp.dev", "x-forwarded-proto": "https"}, body: Body(Empty) } 2024-07-25T23:33:30.252916Z DEBUG garage_api::generic_server: Endpoint: CopyObject 2024-07-25T23:33:30.252935Z TRACE garage_api::signature::payload: canonical request: PUT /2/emoji/images/1.png content-type:image/png host:iceshrimp-chat-avatars.s3.iceshrimp.dev x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-copy-source:iceshrimp-chat-avatars/2/emoji/images/1.png x-amz-date:20240725T233330Z x-amz-meta-realm_id:2 x-amz-meta-user_profile_id:8 x-amz-metadata-directive:REPLACE content-type;host;x-amz-content-sha256;x-amz-copy-source;x-amz-date;x-amz-meta-realm_id;x-amz-meta-user_profile_id;x-amz-metadata-directive e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2024-07-25T23:33:30.252937Z TRACE garage_api::signature::payload: string to sign: AWS4-HMAC-SHA256 20240725T233330Z 20240725/garage/s3/aws4_request 08f003341fe4032116e666a6b215ea22536731a43bcd89444ce5b6de7225896c 2024-07-25T23:33:30.252969Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature ```

zotan commented

2024-07-26 17:43:54 +00:00

Author

Update: traced this down to the following difference in the hashed canonical request:
boto3 signed it as

x-amz-meta-realm_id:2,2
x-amz-meta-user_profile_id:8,8

This comes from there being duplicate keys in the metadata collection, one with an uppercase first letter, and one with a lowercase one.

From what I can tell, garage seems to be capitalizing the first letter of metadata keys, as zulip is sending them all-lowercase (and is expecting to receive them as such). AWS says that these should be case-insensitive, and all-lowercase: https://docs.aws.amazon.com/codeguru/detector-library/java/s3-object-user-metadata-key-case-sensitivity/

Update: traced this down to the following difference in the hashed canonical request: boto3 signed it as ``` x-amz-meta-realm_id:2,2 x-amz-meta-user_profile_id:8,8 ``` This comes from there being duplicate keys in the metadata collection, one with an uppercase first letter, and one with a lowercase one. From what I can tell, garage seems to be capitalizing the first letter of metadata keys, as zulip is sending them all-lowercase (and is expecting to receive them as such). AWS says that these should be case-insensitive, and all-lowercase: https://docs.aws.amazon.com/codeguru/detector-library/java/s3-object-user-metadata-key-case-sensitivity/

zotan changed title from ~~Invalid signature on CopyObject request sent from boto3~~ to Metadata keys are returned with weird casing (instead of all-lowercase)

2024-07-26 17:58:09 +00:00

zotan commented

2024-07-26 18:10:36 +00:00

Author

I think the ideal solution would be to return metadata keys all-lowercase, or at least preserve the casing that's sent. The current implementation seems to break things.

quentin commented

2024-08-06 07:41:34 +00:00

Owner

Custom object headers are sent here during a GET/HEAD:
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/get.rs#L66-L83

Model to store them is defined here:
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/model/s3/object_table.rs#L267-L274

They are persisted here durig a PUT:
https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/put.rs#L72-L75

More especially, the persisted HeaderList is computed from the HTTP framework headers in the get_headers() function: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/put.rs#L621-L629

IMO this one is a good candidate to put a to_lowercase call.

Custom object headers are sent here during a GET/HEAD: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/get.rs#L66-L83 Model to store them is defined here: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/model/s3/object_table.rs#L267-L274 They are persisted here durig a PUT: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/put.rs#L72-L75 More especially, the persisted `HeaderList` is computed from the HTTP framework headers in the `get_headers()` function: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3/put.rs#L621-L629 IMO this one is a good candidate to put a `to_lowercase` call.