Deuxfleurs/garage

Fork 75

Can't perform PutObject operation #738

New issue

Closed

opened 2024-02-29 11:49:38 +00:00 by ghashy · 4 comments

ghashy commented

2024-02-29 11:49:38 +00:00

Thank you very much for garage! Previously, I was using minio with aws-sdk-s3 in Rust. Now, I am attempting to transition to garage, but my previous code that was working no longer functions correctly. In this code snippet, I am trying to upload a file to an S3 bucket named mustore-data with the key received/user1:886773de-cc58-4d2b-81bb-b4db05349817:avatar.png:

        let _put_response = self
            .client
            .put_object()
            .bucket(&self.settings.bucket_name)
            .key(key.as_ref())
            .content_encoding(mediatype.to_string())
            .body(
                ByteStream::try_from(SdkBody::from(bytes))
                    .context("Failed to create ByteStream from bytes")?,
            )
            .send()
            .await
            .context("Failed to upload file to the object storage")?;

However, the output I received indicates an error:

2024-02-29T11:40:32.826738Z  INFO garage_api::generic_server: [::ffff:192.168.215.0]:53406 HEAD /
2024-02-29T11:40:52.472320Z  INFO garage_api::generic_server: [::ffff:192.168.215.0]:53406 PUT /received/user1%3A886773de-cc58-4d2b-81bb-b4db05349817%3Aavatar.png?x-id=PutObject
2024-02-29T11:40:52.472908Z  INFO garage_api::generic_server: Response: error 404 Not Found, Bucket not found: received

The issue is that received/ is being interpreted as a bucket name instead of a virtual directory as intended.
Also, if I use ':' instead of '/', with key 'received:user1:524bb5c8-c585-4123-b36a-1257d826325b:avatar.png', I got:

2024-02-29T11:46:35.611013Z  INFO garage_api::generic_server: [::ffff:192.168.215.0]:53469 HEAD /
2024-02-29T11:46:46.581825Z  INFO garage_api::generic_server: [::ffff:192.168.215.0]:53469 PUT /received%3Auser1%3A524bb5c8-c585-4123-b36a-1257d826325b%3Aavatar.png?x-id=PutObject
2024-02-29T11:46:46.582313Z  INFO garage_api::generic_server: Response: error 400 Bad Request, Bad request: Invalid create bucket XML query

Please, how would I fix that?

Thank you very much for garage! Previously, I was using minio with `aws-sdk-s3` in Rust. Now, I am attempting to transition to garage, but my previous code that was working no longer functions correctly. In this code snippet, I am trying to upload a file to an S3 bucket named `mustore-data` with the key `received/user1:886773de-cc58-4d2b-81bb-b4db05349817:avatar.png`: ```rust let _put_response = self .client .put_object() .bucket(&self.settings.bucket_name) .key(key.as_ref()) .content_encoding(mediatype.to_string()) .body( ByteStream::try_from(SdkBody::from(bytes)) .context("Failed to create ByteStream from bytes")?, ) .send() .await .context("Failed to upload file to the object storage")?; ``` However, the output I received indicates an error: ``` 2024-02-29T11:40:32.826738Z INFO garage_api::generic_server: [::ffff:192.168.215.0]:53406 HEAD / 2024-02-29T11:40:52.472320Z INFO garage_api::generic_server: [::ffff:192.168.215.0]:53406 PUT /received/user1%3A886773de-cc58-4d2b-81bb-b4db05349817%3Aavatar.png?x-id=PutObject 2024-02-29T11:40:52.472908Z INFO garage_api::generic_server: Response: error 404 Not Found, Bucket not found: received ``` The issue is that `received/` is being interpreted as a bucket name instead of a virtual directory as intended. Also, if I use ':' instead of '/', with key 'received:user1:524bb5c8-c585-4123-b36a-1257d826325b:avatar.png', I got: ``` 2024-02-29T11:46:35.611013Z INFO garage_api::generic_server: [::ffff:192.168.215.0]:53469 HEAD / 2024-02-29T11:46:46.581825Z INFO garage_api::generic_server: [::ffff:192.168.215.0]:53469 PUT /received%3Auser1%3A524bb5c8-c585-4123-b36a-1257d826325b%3Aavatar.png?x-id=PutObject 2024-02-29T11:46:46.582313Z INFO garage_api::generic_server: Response: error 400 Bad Request, Bad request: Invalid create bucket XML query ``` Please, how would I fix that?

lx commented

2024-02-29 11:57:55 +00:00

Owner

There are two style of URLs in AWS S3: path-style and dns-style. Here, Garage is expecting a path-style URL, where the fist component is the name of the bucket, but your client is sending a dns-style request, where the bucket name is supposed to be part of the host name. Configure your client to use path style (the option is variously called use_path_style or force_path_style) and it should work.

👍 1

ghashy commented

2024-02-29 12:51:53 +00:00

Author

@lx, That works, thanks you very much!

ghashy closed this issue

2024-02-29 12:51:54 +00:00

Mako commented

2024-05-13 09:22:08 +00:00

There are two style of URLs in AWS S3: path-style and dns-style. Here, Garage is expecting a path-style URL, where the fist component is the name of the bucket, but your client is sending a dns-style request, where the bucket name is supposed to be part of the host name. Configure your client to use path style (the option is variously called use_path_style or force_path_style) and it should work.

Hi!
Looks like I have similar issue but force_path_style(true) does not help:

ServiceError { source: Unhandled(Unhandled { source: ErrorMetadata { code: Some("InvalidRequest"), message: Some("Bad request: Invalid create bucket XML query"), extras: None },
 meta: ErrorMetadata { code: Some("InvalidRequest"), 
 message: Some("Bad request: Invalid create bucket XML query"), extras: None } }),
  raw: Response { status: StatusCode(400),
  headers: Headers { headers: {"content-type": HeaderValue { _private: H0("application/xml") },
  "content-length": HeaderValue { _private: H0("204") }, 
  "date": HeaderValue { _private: H0("Mon, 13 May 2024 08:00:43 GMT") }} }, 
  body: SdkBody { inner: Once(Some(b"<?xml version=\"1.0\" encoding=\"UTF-8\"?><Error>
  <Code>InvalidRequest</Code>
  <Message>Bad request: Invalid create bucket XML query</Message>
  <Resource>/test_file_XhHytm</Resource>
  <Region>garage</Region>
  </Error>")), retryable: true }, 
  extensions: Extensions { extensions_02x: Extensions, extensions_1x: Extensions } } } 
   name: test_file_XhHytm input: 2048

INFO garage_api::generic_server: [::ffff:127.0.0.1]:54170 PUT /test_file_wh0xzw?x-id=PutObject
INFO garage_api::generic_server: Response: error 400 Bad Request, Bad request: Invalid create bucket XML query

Any ideas?

> There are two style of URLs in AWS S3: path-style and dns-style. Here, Garage is expecting a path-style URL, where the fist component is the name of the bucket, but your client is sending a dns-style request, where the bucket name is supposed to be part of the host name. Configure your client to use path style (the option is variously called use_path_style or force_path_style) and it should work. Hi! Looks like I have similar issue but force_path_style(true) does not help: ``` ServiceError { source: Unhandled(Unhandled { source: ErrorMetadata { code: Some("InvalidRequest"), message: Some("Bad request: Invalid create bucket XML query"), extras: None }, meta: ErrorMetadata { code: Some("InvalidRequest"), message: Some("Bad request: Invalid create bucket XML query"), extras: None } }), raw: Response { status: StatusCode(400), headers: Headers { headers: {"content-type": HeaderValue { _private: H0("application/xml") }, "content-length": HeaderValue { _private: H0("204") }, "date": HeaderValue { _private: H0("Mon, 13 May 2024 08:00:43 GMT") }} }, body: SdkBody { inner: Once(Some(b"<?xml version=\"1.0\" encoding=\"UTF-8\"?><Error> <Code>InvalidRequest</Code> <Message>Bad request: Invalid create bucket XML query</Message> <Resource>/test_file_XhHytm</Resource> <Region>garage</Region> </Error>")), retryable: true }, extensions: Extensions { extensions_02x: Extensions, extensions_1x: Extensions } } } name: test_file_XhHytm input: 2048 ``` ``` INFO garage_api::generic_server: [::ffff:127.0.0.1]:54170 PUT /test_file_wh0xzw?x-id=PutObject INFO garage_api::generic_server: Response: error 400 Bad Request, Bad request: Invalid create bucket XML query ``` Any ideas?

lx commented

2024-05-24 16:52:43 +00:00

Owner

@Mako Your problem seems to be on the CreateBucket endpoint, so it would be better to open a different issue to discuss it.

To answer your question, it looks like the SDK you are using is adding a parameter in the CreateBucket request that Garage does not support. We would need to dump the XML that your client is sending to Garage to see what it contains and why Garage considers it invalid, and then either take the additionnal parameters into account (if applicable), or more likely, just ignore it.

@Mako Your problem seems to be on the CreateBucket endpoint, so it would be better to open a different issue to discuss it. To answer your question, it looks like the SDK you are using is adding a parameter in the CreateBucket request that Garage does not support. We would need to dump the XML that your client is sending to Garage to see what it contains and why Garage considers it invalid, and then either take the additionnal parameters into account (if applicable), or more likely, just ignore it.

No milestone

No project

No assignees

3 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#738

No description provided.