Unable to delete content with Minio Client

quentin commented

2021-04-27 19:31:25 +00:00

Owner

How to reproduce

$ echo 1 > a.txt
$ mc cp a.txt garage/quentin.bibliotheque
a.txt:                    2 B / 2 B
[=====================================================] 3 B/s 0s
$ mc ls garage/quentin.bibliotheque/a.txt
[2021-04-27 21:32:13 CEST]     2B a.txt
$ mc rm garage/quentin.bibliotheque/a.txt

What we get

The file is not deleted and we have the following error:

mc: <DEBUG> HEAD /quentin.bibliotheque/a.txt HTTP/1.1
Host: garage.deuxfleurs.fr
User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-03-12T03-36-59Z
Authorization: AWS4-HMAC-SHA256 Credential=GK6608fa7254a2886250289850/20210427/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210427T193604Z

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 2
Accept-Ranges: bytes
Content-Type: text/plain
Date: Tue, 27 Apr 2021 19:36:04 GMT
Etag: "b026324c6904b2a9cb4b88d6d61c81d1"
Last-Modified: Tue, 27 Apr 2021 19:36:02 GMT
Vary: Accept-Encoding

mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2021-06-19 08:06:08 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Organization: Digital Signature Trust Co.
mc: <DEBUG>  >> Expires: 2021-09-29 19:21:40 +0000 UTC
mc: <DEBUG> Response Time:  273.661555ms

mc: <DEBUG> HEAD /quentin.bibliotheque/a.txt HTTP/1.1
Host: garage.deuxfleurs.fr
User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-03-12T03-36-59Z
Authorization: AWS4-HMAC-SHA256 Credential=GK6608fa7254a2886250289850/20210427/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210427T193605Z

mc: <DEBUG> HTTP/1.1 200 OK
Content-Length: 2
Accept-Ranges: bytes
Content-Type: text/plain
Date: Tue, 27 Apr 2021 19:36:04 GMT
Etag: "b026324c6904b2a9cb4b88d6d61c81d1"
Last-Modified: Tue, 27 Apr 2021 19:36:02 GMT
Vary: Accept-Encoding

mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2021-06-19 08:06:08 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Organization: Digital Signature Trust Co.
mc: <DEBUG>  >> Expires: 2021-09-29 19:21:40 +0000 UTC
mc: <DEBUG> Response Time:  79.694345ms

Removing `garage/quentin.bibliotheque/a.txt`.
mc: <DEBUG> POST /quentin.bibliotheque/?delete= HTTP/1.1
Host: garage.deuxfleurs.fr
User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-03-12T03-36-59Z
Content-Length: 69
Authorization: AWS4-HMAC-SHA256 Credential=GK6608fa7254a2886250289850/20210427/garage/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
Content-Md5: x0FQOH+s1UFedrTctTzz5A==
X-Amz-Content-Sha256: bbaae5250e862e69f6ba91002037700c16c65240ff629f388834880f3444f9e7
X-Amz-Date: 20210427T193605Z
Accept-Encoding: gzip

mc: <DEBUG> HTTP/1.1 400 Bad Request
Content-Length: 38
Content-Type: text/plain; charset=utf-8
Date: Tue, 27 Apr 2021 19:36:05 GMT
Vary: Accept-Encoding

Bad request: Invalid delete XML query
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Country: US
mc: <DEBUG>  >> Organization: Let's Encrypt
mc: <DEBUG>  >> Expires: 2021-06-19 08:06:08 +0000 UTC
mc: <DEBUG> TLS Certificate found: 
mc: <DEBUG>  >> Organization: Digital Signature Trust Co.
mc: <DEBUG>  >> Expires: 2021-09-29 19:21:40 +0000 UTC
mc: <DEBUG> Response Time:  108.126619ms

mc: <ERROR> Failed to remove `garage/quentin.bibliotheque/a.txt`. 400 Bad Request
 (1) rm-main.go:316 cmd.removeSingle(..) Tags: [garage/quentin.bibliotheque/a.txt]
 (0) client-s3.go:1172 cmd.(*S3Client).Remove.func1(..)
 Release-Tag:RELEASE.2021-03-12T03-36-59Z | Commit:501b791408ba | Host:lheureduthe | OS:linux | Arch:amd64 | Lang:go1.16.1 | Mem:3.3 MB/76 MB | Heap:3.3 MB/66 MB

What we expect

The file is deleted without error similarly to the AWS client:

$ awsdf s3 --debug  rm s3://quentin.bibliotheque/a.txt
[...]
2021-04-27 21:39:26,245 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - CanonicalRequest:
DELETE
/quentin.bibliotheque/a.txt

host:garage.deuxfleurs.fr
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20210427T193926Z

host;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
[...]
delete: s3://quentin.bibliotheque/a.txt

Investigation

It seems there is 2 ways to delete an object on S3:

Using the DELETE verb on the element (implemented in garage)
Using POST on the prefix with the ?delete= parameter (probably not implemented in garage)

What should we do?

I don't know yet, I just want to track this issue for now :P

## How to reproduce ``` $ echo 1 > a.txt $ mc cp a.txt garage/quentin.bibliotheque a.txt: 2 B / 2 B [=====================================================] 3 B/s 0s $ mc ls garage/quentin.bibliotheque/a.txt [2021-04-27 21:32:13 CEST] 2B a.txt $ mc rm garage/quentin.bibliotheque/a.txt ``` ## What we get The file is not deleted and we have the following error: ``` mc: <DEBUG> HEAD /quentin.bibliotheque/a.txt HTTP/1.1 Host: garage.deuxfleurs.fr User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-03-12T03-36-59Z Authorization: AWS4-HMAC-SHA256 Credential=GK6608fa7254a2886250289850/20210427/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20210427T193604Z mc: <DEBUG> HTTP/1.1 200 OK Content-Length: 2 Accept-Ranges: bytes Content-Type: text/plain Date: Tue, 27 Apr 2021 19:36:04 GMT Etag: "b026324c6904b2a9cb4b88d6d61c81d1" Last-Modified: Tue, 27 Apr 2021 19:36:02 GMT Vary: Accept-Encoding mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: US mc: <DEBUG> >> Organization: Let's Encrypt mc: <DEBUG> >> Expires: 2021-06-19 08:06:08 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Organization: Digital Signature Trust Co. mc: <DEBUG> >> Expires: 2021-09-29 19:21:40 +0000 UTC mc: <DEBUG> Response Time: 273.661555ms mc: <DEBUG> HEAD /quentin.bibliotheque/a.txt HTTP/1.1 Host: garage.deuxfleurs.fr User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-03-12T03-36-59Z Authorization: AWS4-HMAC-SHA256 Credential=GK6608fa7254a2886250289850/20210427/garage/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 X-Amz-Date: 20210427T193605Z mc: <DEBUG> HTTP/1.1 200 OK Content-Length: 2 Accept-Ranges: bytes Content-Type: text/plain Date: Tue, 27 Apr 2021 19:36:04 GMT Etag: "b026324c6904b2a9cb4b88d6d61c81d1" Last-Modified: Tue, 27 Apr 2021 19:36:02 GMT Vary: Accept-Encoding mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: US mc: <DEBUG> >> Organization: Let's Encrypt mc: <DEBUG> >> Expires: 2021-06-19 08:06:08 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Organization: Digital Signature Trust Co. mc: <DEBUG> >> Expires: 2021-09-29 19:21:40 +0000 UTC mc: <DEBUG> Response Time: 79.694345ms Removing `garage/quentin.bibliotheque/a.txt`. mc: <DEBUG> POST /quentin.bibliotheque/?delete= HTTP/1.1 Host: garage.deuxfleurs.fr User-Agent: MinIO (linux; amd64) minio-go/v7.0.11 mc/RELEASE.2021-03-12T03-36-59Z Content-Length: 69 Authorization: AWS4-HMAC-SHA256 Credential=GK6608fa7254a2886250289850/20210427/garage/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED** Content-Md5: x0FQOH+s1UFedrTctTzz5A== X-Amz-Content-Sha256: bbaae5250e862e69f6ba91002037700c16c65240ff629f388834880f3444f9e7 X-Amz-Date: 20210427T193605Z Accept-Encoding: gzip mc: <DEBUG> HTTP/1.1 400 Bad Request Content-Length: 38 Content-Type: text/plain; charset=utf-8 Date: Tue, 27 Apr 2021 19:36:05 GMT Vary: Accept-Encoding Bad request: Invalid delete XML query mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Country: US mc: <DEBUG> >> Organization: Let's Encrypt mc: <DEBUG> >> Expires: 2021-06-19 08:06:08 +0000 UTC mc: <DEBUG> TLS Certificate found: mc: <DEBUG> >> Organization: Digital Signature Trust Co. mc: <DEBUG> >> Expires: 2021-09-29 19:21:40 +0000 UTC mc: <DEBUG> Response Time: 108.126619ms mc: <ERROR> Failed to remove `garage/quentin.bibliotheque/a.txt`. 400 Bad Request (1) rm-main.go:316 cmd.removeSingle(..) Tags: [garage/quentin.bibliotheque/a.txt] (0) client-s3.go:1172 cmd.(*S3Client).Remove.func1(..) Release-Tag:RELEASE.2021-03-12T03-36-59Z | Commit:501b791408ba | Host:lheureduthe | OS:linux | Arch:amd64 | Lang:go1.16.1 | Mem:3.3 MB/76 MB | Heap:3.3 MB/66 MB ``` ## What we expect The file is deleted without error similarly to the AWS client: ``` $ awsdf s3 --debug rm s3://quentin.bibliotheque/a.txt [...] 2021-04-27 21:39:26,245 - ThreadPoolExecutor-0_0 - botocore.auth - DEBUG - CanonicalRequest: DELETE /quentin.bibliotheque/a.txt host:garage.deuxfleurs.fr x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20210427T193926Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 [...] delete: s3://quentin.bibliotheque/a.txt ``` ## Investigation It seems there is 2 ways to delete an object on S3: - Using the `DELETE` verb on the element (implemented in garage) - Using `POST` on the prefix with the `?delete=` parameter (probably not implemented in garage) ## What should we do? I don't know yet, I just want to track this issue for now :P

quentin added the

Bug

label 2021-04-27 19:37:02 +00:00

lx commented

2021-04-27 20:48:00 +00:00

Owner

This looks like a DeleteObjects call, which is implemented but has a bug:

Bad request: Invalid delete XML query

This line indicates that the code in Garage that parses the delete query isn't working correctly (probably).

How to debug this: enable all logs and if necessary add a trace statement in the DeleteObjects handler code. Then check the parsing logic and find where it is flawed.

This looks like a [DeleteObjects call](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html), which is implemented but has a bug: ``` Bad request: Invalid delete XML query ``` This line indicates that the code in Garage that parses the delete query isn't working correctly (probably). How to debug this: enable all logs and if necessary add a trace statement in the DeleteObjects handler code. Then check the parsing logic and find where it is flawed.

👍 1