Can't perform PutObject operation with current aws crates #822

New issue

Open

opened 2024-05-16 07:33:25 +00:00 by Mako · 1 comment

Mako commented

2024-05-16 07:33:25 +00:00

Hi! Thank you for your great garage!

garage v1.0.0 [features: k2v, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs]

With these old crates put works:

aws-config = "0.51"
aws-sdk-s3 = "0.21"
aws-smithy-async = "0.51"

But with current crates:

aws-config = { version = "1.4", features = ["behavior-version-latest"] }
aws-sdk-s3 = { version = "1.29", features = ["rt-tokio"] }

it has the error Received an unknown query parameter: 'x-id'

let config = aws_config::defaults(BehaviorVersion::latest())
    .endpoint_url("http://127.0.0.1:3900/")
    .load()
    .await;

let s3_config = aws_sdk_s3::config::Builder::from(&config)
    .force_path_style(true)
    .build();

let s3 = aws_sdk_s3::Client::from_conf(s3_config);

garage_api::generic_server: Request { method: PUT, uri: /test_file_fTK9Ti?x-id=PutObject, version: HTTP/1.1, headers: {"content-type": "application/octet-stream", "content-length": "2048", "user-agent": "aws-sdk-rust/1.2.1 os/linux lang/rust/1.77.2", "x-amz-user-agent": "aws-sdk-rust/1.2.1 api/s3/1.28.0 os/linux lang/rust/1.77.2", "x-amz-date": "20240515T095349Z", "authorization": "AWS4-HMAC-SHA256 Credential=GKc0ed675144fd7ed5bbd79bc6/20240515/garage/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=71be79aadcaafba5bee1e33b0a4d4793e9bc5cfdbb3cbeb14ecb5af83383d39a", "x-amz-content-sha256": "10fc3c51a152e90e5b90319b601d92ccf37290ef53c35ff92507687d8a911a08", "amz-sdk-request": "attempt=1; max=1", "amz-sdk-invocation-id": "0a668aea-c053-41f2-a996-14f73b91edb0", "host": "127.0.0.1:3900"}, body: Body(Streaming) }
2024-05-15T09:53:49.027402Z DEBUG garage_api::s3::router: Received an unknown query parameter: 'x-id'
2024-05-15T09:53:49.027412Z DEBUG garage_api::generic_server: Endpoint: CreateBucket
2024-05-15T09:53:49.027806Z  INFO garage_api::generic_server: Response: error 400 Bad Request, Bad request: Invalid create bucket XML query

I asked at smithy-rs and got the answer:
This seems like an issue with the third-party tool

Hi! Thank you for your great garage! garage v1.0.0 [features: k2v, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs] With these old crates put works: ``` aws-config = "0.51" aws-sdk-s3 = "0.21" aws-smithy-async = "0.51" ``` But with current crates: ``` aws-config = { version = "1.4", features = ["behavior-version-latest"] } aws-sdk-s3 = { version = "1.29", features = ["rt-tokio"] } ``` it has the error **` Received an unknown query parameter: 'x-id'`** ``` let config = aws_config::defaults(BehaviorVersion::latest()) .endpoint_url("http://127.0.0.1:3900/") .load() .await; let s3_config = aws_sdk_s3::config::Builder::from(&config) .force_path_style(true) .build(); let s3 = aws_sdk_s3::Client::from_conf(s3_config); ``` ``` garage_api::generic_server: Request { method: PUT, uri: /test_file_fTK9Ti?x-id=PutObject, version: HTTP/1.1, headers: {"content-type": "application/octet-stream", "content-length": "2048", "user-agent": "aws-sdk-rust/1.2.1 os/linux lang/rust/1.77.2", "x-amz-user-agent": "aws-sdk-rust/1.2.1 api/s3/1.28.0 os/linux lang/rust/1.77.2", "x-amz-date": "20240515T095349Z", "authorization": "AWS4-HMAC-SHA256 Credential=GKc0ed675144fd7ed5bbd79bc6/20240515/garage/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-user-agent, Signature=71be79aadcaafba5bee1e33b0a4d4793e9bc5cfdbb3cbeb14ecb5af83383d39a", "x-amz-content-sha256": "10fc3c51a152e90e5b90319b601d92ccf37290ef53c35ff92507687d8a911a08", "amz-sdk-request": "attempt=1; max=1", "amz-sdk-invocation-id": "0a668aea-c053-41f2-a996-14f73b91edb0", "host": "127.0.0.1:3900"}, body: Body(Streaming) } 2024-05-15T09:53:49.027402Z DEBUG garage_api::s3::router: Received an unknown query parameter: 'x-id' 2024-05-15T09:53:49.027412Z DEBUG garage_api::generic_server: Endpoint: CreateBucket 2024-05-15T09:53:49.027806Z INFO garage_api::generic_server: Response: error 400 Bad Request, Bad request: Invalid create bucket XML query ``` I asked at [smithy-rs](https://github.com/smithy-lang/smithy-rs/issues/3640) and got the answer: **This seems like an issue with the third-party tool**

lx added the

Bug

S3 Compatibility

labels 2024-05-24 16:39:38 +00:00

lx commented

2024-05-24 16:41:48 +00:00

Owner

Garage should just ignore the x-id query parameter when it is present as it does not use it to disambiguate endpoints

Related: - <https://stackoverflow.com/questions/76680025/why-does-aws-sdk-v2-add-x-id-query-param-to-s3-getobject-requests> - <https://github.com/smithy-lang/smithy-rs/issues/1012> - <https://github.com/smithy-lang/smithy/issues/1030> Garage should just ignore the x-id query parameter when it is present as it does not use it to disambiguate endpoints

lx added the

Newcomer

label 2024-05-24 16:42:17 +00:00

lx added this to the v1.1 milestone 2024-05-24 17:10:01 +00:00