Implement `HeadBucket`

Gusted commented

2023-08-26 23:19:13 +00:00

Would it be possible to implement the HeadBucket API? This is useful for determining whether a bucket exists or not and whether the authenticated user has access to it. It can also be used by software to determine whether the bucket should be created or not.

Thanks,
Gusted

Would it be possible to implement the [`HeadBucket`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadBucket.html) API? This is useful for determining whether a bucket exists or not and whether the authenticated user has access to it. It can also be used by software to determine whether the bucket should be created or not. Thanks, Gusted

lx commented

2023-08-27 01:46:25 +00:00

Owner

HeadBucket is supposed to be implemented, what issues are you having with it ?

Gusted commented

2023-08-27 10:21:24 +00:00

Author

An user of Forgejo (using Garage as S3 storage) reported that 400 Bad Request was returned on a function call from BucketExists after a321a4c2fc. From what I gather from the documentation it meant that the function isn't implemented, but I don't have much experience with S3 so it was premature on my part to say it wasn't implemented. If you say it's implemented I will go back and see what else it could've caused this.

An user of Forgejo (using Garage as S3 storage) reported that `400 Bad Request` was returned on a function call from [`BucketExists`](https://github.com/minio/minio-go/blob/a750ce3dc18eec7e4bfb484872527920256b46ef/api-stat.go#L29) after https://github.com/go-gitea/gitea/commit/a321a4c2fce9eb6c72f4dda0ee9928e56e9ec223. From what I gather from the documentation it meant that the function isn't implemented, but I don't have much experience with S3 so it was premature on my part to say it wasn't implemented. If you say it's implemented I will go back and see what else it could've caused this.

earl-warren commented

2023-08-28 13:56:04 +00:00

Here is a reproducer for this issue. It may simply be a misconfiguration but it not trivial to figure out for me because I do not have any experience (yet) with garage.

Launch garage from this script
mc ls --recursive testgarage/

tail /tmp/forgejo-upgrades/garage*.log

2023-08-28T13:39:44.923948Z  INFO garage_api::generic_server: 127.0.0.1:60900 GET /
2023-08-28T13:39:44.924026Z  INFO garage_api::generic_server: Response: error 400 Bad Request, Authorization header malformed, unexpected scope: 20230828/us-east-1/s3/aws4_request
2023-08-28T13:39:45.036012Z  INFO garage_api::generic_server: 127.0.0.1:60900 GET /

The same error shows when trying to connect from Forgejo.

Here is a reproducer for this issue. It may simply be a misconfiguration but it not trivial to figure out for me because I do not have any experience (yet) with garage. * Launch garage [from this script](https://codeberg.org/earl-warren/forgejo/src/branch/wip-s3-upgrade/.forgejo/upgrades/test-upgrade.sh#L162-L204) * `mc ls --recursive testgarage/` * `tail /tmp/forgejo-upgrades/garage*.log` ``` 2023-08-28T13:39:44.923948Z INFO garage_api::generic_server: 127.0.0.1:60900 GET / 2023-08-28T13:39:44.924026Z INFO garage_api::generic_server: Response: error 400 Bad Request, Authorization header malformed, unexpected scope: 20230828/us-east-1/s3/aws4_request 2023-08-28T13:39:45.036012Z INFO garage_api::generic_server: 127.0.0.1:60900 GET / ``` The same error shows when trying to connect from Forgejo.

lx commented

2023-08-28 14:59:34 +00:00

Owner

The error seems to indicate a problem with the signature of the request, not with the HeadBucket endpoint per se. Can you ensure that your S3 client is configured to use the correct region? This is often a cause of this kind of issues.

Also, if you have time, please post the logs with RUST_LOG=garage_api=debug

The error seems to indicate a problem with the signature of the request, not with the HeadBucket endpoint per se. Can you ensure that your S3 client is configured to use the correct region? This is often a cause of this kind of issues. Also, if you have time, please post the logs with `RUST_LOG=garage_api=debug`

👍 1

earl-warren commented

2023-08-28 15:10:51 +00:00

I would like to let garage know that 20230828/us-east-1/s3/aws4_request is a valid scope. A pointer on how to do that would be much appreciated.

I would like to let garage know that `20230828/us-east-1/s3/aws4_request` is a valid scope. A pointer on how to do that would be much appreciated.

lx commented

2023-08-28 15:11:59 +00:00

Owner

If you have set s3_region = "us-east-1" in your config file then it is a valid scope, otherwise it is not (but there is a redirection mechanism in the error message that clients can use to select the correct region instead, maybe that's just not implemented).

If you have set `s3_region = "us-east-1"` in your config file then it is a valid scope, otherwise it is not (but there is a redirection mechanism in the error message that clients can use to select the correct region instead, maybe that's just not implemented).

👀 1

trinity-1686a commented

2023-08-28 15:12:23 +00:00

Owner

I think some applications do a HeadBucket with us-east-1 (the original AWS region) so they can learn the real region from the x-amz-bucket-region: header in the response, and not require it as a configuration variable. RIght now, Garage has no support for that I think

I think some applications do a `HeadBucket` with `us-east-1` (the original AWS region) so they can learn the real region from the `x-amz-bucket-region:` header in the response, and not require it as a configuration variable. RIght now, Garage has no support for that I think

earl-warren commented

2023-08-28 15:13:16 +00:00

The logs with debug:

2023-08-28T15:12:50.827687Z  INFO garage_api::generic_server: 127.0.0.1:40918 HEAD /forgejo/
2023-08-28T15:12:50.827700Z DEBUG garage_api::generic_server: Request { method: HEAD, uri: /forgejo/, version: HTTP/1.1, headers: {"host": "minio:9000", "user-agent": "MinIO (linux; amd64) minio-go/v7.0.61", "authorization": "AWS4-HMAC-SHA256 Credential=123456/20230828/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d1e448c98a7ed65a1702aee284da1de7db80d9b291394ae9d042c1b843ad6313", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20230828T151250Z"}, body: Body(Empty) }
2023-08-28T15:12:50.827725Z DEBUG garage_api::generic_server: Endpoint: HeadBucket
2023-08-28T15:12:50.827782Z  INFO garage_api::generic_server: Response: error 400 Bad Request, Authorization header malformed, unexpected scope: 20230828/us-east-1/s3/aws4_request

The logs with debug: ``` 2023-08-28T15:12:50.827687Z INFO garage_api::generic_server: 127.0.0.1:40918 HEAD /forgejo/ 2023-08-28T15:12:50.827700Z DEBUG garage_api::generic_server: Request { method: HEAD, uri: /forgejo/, version: HTTP/1.1, headers: {"host": "minio:9000", "user-agent": "MinIO (linux; amd64) minio-go/v7.0.61", "authorization": "AWS4-HMAC-SHA256 Credential=123456/20230828/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=d1e448c98a7ed65a1702aee284da1de7db80d9b291394ae9d042c1b843ad6313", "x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "x-amz-date": "20230828T151250Z"}, body: Body(Empty) } 2023-08-28T15:12:50.827725Z DEBUG garage_api::generic_server: Endpoint: HeadBucket 2023-08-28T15:12:50.827782Z INFO garage_api::generic_server: Response: error 400 Bad Request, Authorization header malformed, unexpected scope: 20230828/us-east-1/s3/aws4_request ```

earl-warren commented

2023-08-28 15:13:49 +00:00

I'll try to switch to MD5

earl-warren commented

2023-08-28 15:20:34 +00:00

But MD5 is only relevant when content is PUT/GET, my bad.

earl-warren commented

2023-08-31 10:32:31 +00:00

If you have set s3_region = "us-east-1" in your config file then it is a valid scope, otherwise it is not

That fixed the problem.

(but there is a redirection mechanism in the error message that clients can use to select the correct region instead, maybe that's just not implemented).

Now the real fix is client side, not garage side. I'll check the content of the error message and look for the redirection. Is there documentation somewhere about that?

I think this issue can be closed now, thanks a lot for your help. And I'm personally very happy that garage is using Fogejo.

> If you have set `s3_region = "us-east-1"` in your config file then it is a valid scope, otherwise it is not That fixed the problem. > (but there is a redirection mechanism in the error message that clients can use to select the correct region instead, maybe that's just not implemented). Now the real fix is client side, not garage side. I'll check the content of the error message and look for the redirection. Is there documentation somewhere about that? I think this issue can be closed now, thanks a lot for your help. And I'm personally very happy that garage is using Fogejo.

lx commented

2023-08-31 10:41:18 +00:00

Owner

I can't seem to find any official documentation for this, but it is attested that S3 sometimes returns a <Region> tag in the XML body of all its error responses, including AuthorizationHeaderMalformed. This region corresponds to the region of the server. There is an example here with an official response from AWS S3. Garage returns this tag in all of its error responses with value equal to the region that is set in the config file.

I can't seem to find any official documentation for this, but it is attested that S3 sometimes returns a `<Region>` tag in the XML body of all its error responses, including `AuthorizationHeaderMalformed`. This region corresponds to the region of the server. There is an example [here](https://stackoverflow.com/questions/48269677/authorizationheadermalformed-when-trying-to-upload-object-to-s3-via-js-sdk) with an official response from AWS S3. Garage returns this tag in all of its error responses with value equal to the region that is set in the config file.

👍 1

lx closed this issue

2023-08-31 10:42:03 +00:00

Implement HeadBucket #618

Implement `HeadBucket` #618