Minio : Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0 #205

New Issue

Closed

opened 2022-01-25 17:18:43 +00:00 by KillianKemps · 3 comments

KillianKemps commented

2022-01-25 17:18:43 +00:00

Hello, I'm trying Garage locally running with Docker and I'm having issues to do a copy operation with the Minio client.

I followed the quick start guide, and I also adapated the Multi-Node Deployment guide to run it with Docker Compose.

How to reproduce

$ echo 1 > a.txt
$ mcli cp a.txt my-test-garage/data-test-bucket
mcli: <ERROR> Failed to copy `a.txt`. Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0

What to expect

The same behaviour as with aws-cli

$ aws --endpoint-url http://127.0.0.1:3900 s3 cp a.txt s3://data-test-bucket
upload: ./a.txt to s3://data-test-bucket/a.txt                  
$ aws --endpoint-url http://127.0.0.1:3900 s3 ls data-test-bucket
2022-01-25 18:15:57          2 a.txt

Context

garage.toml:

metadata_dir = "/tmp/garage/meta"
data_dir = "/tmp/garage/data"

replication_mode = "none"

rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "1799bccfd7411eddcf9ebd316bc1f5287ad12a68094e1c6ac6abde7e6feae1ec"

bootstrap_peers = []

[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
root_domain = ".s3.garage"

[s3_web]
bind_addr = "[::]:3902"
root_domain = ".web.garage"
index = "index.html"

docker-compose.yml:

version: '3'
services:
  garaged:
    image: dxflrs/amd64_garage:v0.5.1
    network_mode: 'host'
    volumes:
        - ./garage.toml:/etc/garage.toml
        - /tmp/garage/meta:/tmp/garage/meta
        - /tmp/garage/data:/tmp/garage/data

Minio client version : mcli version RELEASE.2021-12-10T00-14-28Z
Garage version : garage 0.5.1

Hello, I'm trying Garage locally running with Docker and I'm having issues to do a copy operation with the Minio client. I followed the [quick start guide](https://garagehq.deuxfleurs.fr/quick_start/index.html), and I also adapated the [Multi-Node Deployment guide](https://garagehq.deuxfleurs.fr/cookbook/real_world.html?highlight=docker#get-a-docker-image) to run it with Docker Compose. ## How to reproduce ``` $ echo 1 > a.txt $ mcli cp a.txt my-test-garage/data-test-bucket mcli: <ERROR> Failed to copy `a.txt`. Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0 ``` ## What to expect The same behaviour as with aws-cli ``` $ aws --endpoint-url http://127.0.0.1:3900 s3 cp a.txt s3://data-test-bucket upload: ./a.txt to s3://data-test-bucket/a.txt $ aws --endpoint-url http://127.0.0.1:3900 s3 ls data-test-bucket 2022-01-25 18:15:57 2 a.txt ``` ## Context `garage.toml`: ``` metadata_dir = "/tmp/garage/meta" data_dir = "/tmp/garage/data" replication_mode = "none" rpc_bind_addr = "[::]:3901" rpc_public_addr = "127.0.0.1:3901" rpc_secret = "1799bccfd7411eddcf9ebd316bc1f5287ad12a68094e1c6ac6abde7e6feae1ec" bootstrap_peers = [] [s3_api] s3_region = "garage" api_bind_addr = "[::]:3900" root_domain = ".s3.garage" [s3_web] bind_addr = "[::]:3902" root_domain = ".web.garage" index = "index.html" ``` `docker-compose.yml`: ``` version: '3' services: garaged: image: dxflrs/amd64_garage:v0.5.1 network_mode: 'host' volumes: - ./garage.toml:/etc/garage.toml - /tmp/garage/meta:/tmp/garage/meta - /tmp/garage/data:/tmp/garage/data ``` Minio client version : `mcli version RELEASE.2021-12-10T00-14-28Z` Garage version : `garage 0.5.1`

KillianKemps changed title from ~~Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0~~ to Minio : Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0

2022-01-25 17:19:45 +00:00

trinity-1686a commented

2022-01-25 18:16:26 +00:00

Owner

I believe this S is the start of STREAMING-AWS4-HMAC-SHA256-PAYLOAD (if you could confirm this by looking in Garage logs or by using Wireshark it would be amazing).

This mode of payload verification is used my minio when sending an object over http (but not https). Support for it was added in #156. It is already available on main, and will be on next release.

There is already a release candidate for it that you can test : dxflrs/amd64_garage:v0.6.0-rc1. Note that there is a manual migration to do if you plan to keep the same data from 0.5 to 0.6, which is documented here, or you can delete garage meta and data to start fresh.

I believe this `S` is the start of `STREAMING-AWS4-HMAC-SHA256-PAYLOAD` (if you could confirm this by looking in Garage logs or by using Wireshark it would be amazing). This mode of payload verification is used my minio when sending an object over http (but not https). Support for it was added in #156. It is already available on `main`, and will be on next release. There is already a release candidate for it that you can test : `dxflrs/amd64_garage:v0.6.0-rc1`. Note that there is a manual migration to do if you plan to keep the same data from 0.5 to 0.6, which [is documented here](https://garagehq.deuxfleurs.fr/working_documents/migration_06.html), or you can delete garage meta and data to start fresh.

KillianKemps commented

2022-01-26 09:02:15 +00:00

Author

Here are Garage logs when doing a mcli cp b.txt my-test-garage/data-test-bucket:

garage_demo-garaged-1  |  INFO  garage_api::api_server  > [::ffff:127.0.0.1]:45688 HEAD /data-test-bucket/
garage_demo-garaged-1  |  INFO  garage_api::api_server  > [::ffff:127.0.0.1]:45688 HEAD /data-test-bucket/
garage_demo-garaged-1  |  INFO  garage_api::api_server  > [::ffff:127.0.0.1]:45688 GET /data-test-bucket/?object-lock=
garage_demo-garaged-1  |  WARN  garage_api::api_server  > Response: error 501 Not Implemented, Unimplemented action: GetObjectLockConfiguration
garage_demo-garaged-1  |  INFO  garage_api::api_server  > [::ffff:127.0.0.1]:45688 HEAD /data-test-bucket/
garage_demo-garaged-1  |  INFO  garage_api::api_server  > [::ffff:127.0.0.1]:45688 PUT /data-test-bucket/b.txt
garage_demo-garaged-1  |  INFO  garage_api::api_server  > Response: error 400 Bad Request, Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0

While searching on Wireshark I found this which seems to confirm your idea :

Hypertext Transfer Protocol
    PUT /data-test-bucket/b.txt HTTP/1.1\r\n
    Host: 127.0.0.1:3900\r\n
    User-Agent: MinIO (linux; amd64) minio-go/v7.0.16 mcli/RELEASE.2021-12-10T00-14-28Z\r\n
    Content-Length: 177\r\n
     [truncated]Authorization: AWS4-HMAC-SHA256 Credential=GK734a31068a86550c25b22e01/20220126/garage/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=b5292e6bc66485a8ac95531fa90a8953ab
    Content-Type: text/plain\r\n
    X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD\r\n
    X-Amz-Date: 20220126T085343Z\r\n
    X-Amz-Decoded-Content-Length: 5\r\n
    \r\n
    [Full request URI: http://127.0.0.1:3900/data-test-bucket/b.txt]
    [HTTP request 5/5]
    [Prev request in frame: 84]
    [Response in frame: 94]
    File Data: 177 bytes
Line-based text data: text/plain (5 lines)

I will try with Garage v0.6.0-rc1

Here are Garage logs when doing a `mcli cp b.txt my-test-garage/data-test-bucket`: ``` garage_demo-garaged-1 | INFO garage_api::api_server > [::ffff:127.0.0.1]:45688 HEAD /data-test-bucket/ garage_demo-garaged-1 | INFO garage_api::api_server > [::ffff:127.0.0.1]:45688 HEAD /data-test-bucket/ garage_demo-garaged-1 | INFO garage_api::api_server > [::ffff:127.0.0.1]:45688 GET /data-test-bucket/?object-lock= garage_demo-garaged-1 | WARN garage_api::api_server > Response: error 501 Not Implemented, Unimplemented action: GetObjectLockConfiguration garage_demo-garaged-1 | INFO garage_api::api_server > [::ffff:127.0.0.1]:45688 HEAD /data-test-bucket/ garage_demo-garaged-1 | INFO garage_api::api_server > [::ffff:127.0.0.1]:45688 PUT /data-test-bucket/b.txt garage_demo-garaged-1 | INFO garage_api::api_server > Response: error 400 Bad Request, Bad request: Invalid content sha256 hash: Invalid character 'S' at position 0 ``` While searching on Wireshark I found this which seems to confirm your idea : ``` Hypertext Transfer Protocol PUT /data-test-bucket/b.txt HTTP/1.1\r\n Host: 127.0.0.1:3900\r\n User-Agent: MinIO (linux; amd64) minio-go/v7.0.16 mcli/RELEASE.2021-12-10T00-14-28Z\r\n Content-Length: 177\r\n [truncated]Authorization: AWS4-HMAC-SHA256 Credential=GK734a31068a86550c25b22e01/20220126/garage/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=b5292e6bc66485a8ac95531fa90a8953ab Content-Type: text/plain\r\n X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD\r\n X-Amz-Date: 20220126T085343Z\r\n X-Amz-Decoded-Content-Length: 5\r\n \r\n [Full request URI: http://127.0.0.1:3900/data-test-bucket/b.txt] [HTTP request 5/5] [Prev request in frame: 84] [Response in frame: 94] File Data: 177 bytes Line-based text data: text/plain (5 lines) ``` I will try with Garage v0.6.0-rc1

KillianKemps commented

2022-01-26 09:08:12 +00:00

Author

I tried with Garage v0.6.0-rc1 and now mcli cp b.txt my-test-garage/data-test-bucket works!

I tried with Garage v0.6.0-rc1 and now `mcli cp b.txt my-test-garage/data-test-bucket ` works!

KillianKemps closed this issue

2022-01-26 09:08:37 +00:00