`content-length` on `HEAD` object #805

New Issue

Closed

opened 2024-04-11 19:19:07 +00:00 by sebadob · 3 comments

sebadob commented

2024-04-11 19:19:07 +00:00

Hey,

I just stumbled about a difference to the behavior of Minio, which I have been using until now, on HEAD requests for an object.
When I do a HEAD, Minio sets the content-length header to the actual file / object size in bytes, while Garage seems to set the value to the length of the actual HEAD request itself (I guess).
I used HEAD to fetch information about file sizes of my object on S3. This logic breaks of course when Garage is returning a different value.

I am not sure which is the correct implementation. I am currently writing a minimal, fast s3 bucket crud client with connection pooling and being optimized to serve lots of files frequently. The currently existing Rust implementations have all their flaws unfortunately.
I just stumbled about it because some of my test cases started failing when I switched to Garage.

I am currently running

Garage version: git:v0.9.3-36-g95eb8808 [features: lmdb, sqlite, kubernetes-discovery, metrics, bundled-libs]

Edit:

I just created a new container image for v1.0.0 and tested with it, which returns the same results.

Hey, I just stumbled about a difference to the behavior of Minio, which I have been using until now, on `HEAD` requests for an object. When I do a `HEAD`, Minio sets the `content-length` header to the actual file / object size in bytes, while Garage seems to set the value to the length of the actual `HEAD` request itself (I guess). I used `HEAD` to fetch information about file sizes of my object on S3. This logic breaks of course when Garage is returning a different value. I am not sure which is the correct implementation. I am currently writing a minimal, fast s3 bucket crud client with connection pooling and being optimized to serve lots of files frequently. The currently existing Rust implementations have all their flaws unfortunately. I just stumbled about it because some of my test cases started failing when I switched to Garage. I am currently running ``` Garage version: git:v0.9.3-36-g95eb8808 [features: lmdb, sqlite, kubernetes-discovery, metrics, bundled-libs] ``` Edit: I just created a new container image for v1.0.0 and tested with it, which returns the same results.

lx commented

2024-04-11 20:47:30 +00:00

Owner

Does this work correctly on older versions such as 0.9.0 and 0.8.6?

The code should return the size of the object as saved in the object's metadata, bit maybe that information is wrong, although that would be strange as it should cause asserts or errors at other places

Does this work correctly on older versions such as 0.9.0 and 0.8.6? The code should return the size of the object as saved in the object's metadata, bit maybe that information is wrong, although that would be strange as it should cause asserts or errors at other places

sebadob commented

2024-04-12 10:26:00 +00:00

Author

Does this work correctly on older versions such as 0.9.0 and 0.8.6?

I can do some tests later. So far I tested only 0.9.3 and the stable v1.0.0.

I did add some integration tests for the library and when Minio returns exactly 128_000 in the content_length for my exactly 128kB file, I got something like 221 back from Garage.

I will report back when I tested against 0.9.0 or 0.8.6. I just have to build container images for this, because I am using it inside K8s and I did not find a way to interact with the cluster easily without having access to the garage binary inside the container. Maybe I missed something there, but I basically build my own images with almalinux:minimal to then be able to alias garage='kubectl -n garage exec -it garage-0 -- ./garage'.

But I will do tests and give feedback.
I could also extract the raw answer from Garage from the test case.

> Does this work correctly on older versions such as 0.9.0 and 0.8.6? I can do some tests later. So far I tested only 0.9.3 and the stable v1.0.0. I did add some integration tests for the library and when Minio returns exactly `128_000` in the `content_length` for my exactly 128kB file, I got something like 221 back from Garage. I will report back when I tested against 0.9.0 or 0.8.6. I just have to build container images for this, because I am using it inside K8s and I did not find a way to interact with the cluster easily without having access to the `garage` binary inside the container. Maybe I missed something there, but I basically build my own images with [almalinux:minimal](https://hub.docker.com/_/almalinux) to then be able to `alias garage='kubectl -n garage exec -it garage-0 -- ./garage'`. But I will do tests and give feedback. I could also extract the raw answer from Garage from the test case.

sebadob commented

2024-04-12 12:19:00 +00:00

Author

I was actually able to solve it.

I digged deeper into the problem and it was partly my fault as well.
I did not check the return code, only extracted headers and saw the content-length not being correct.

What I actually got back from Garage was an HTTP 400 with an empty body, body the following headers:

{"content-length": "214", "content-type": "application/xml", "date": "Fri, 12 Apr 2024 12:06:30 GMT"}

The reason for this was actually pretty tricky. I only found it because I found problems with DeleteObject and GetObjectRange as well, which returned the following error:

<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>Bad request: signed header `content-length` is not present</Message><Resource>/test/test_data_128000</Resource><Region>home</Region></Error>

The content-length was present. I always added it to the signed headers by default, even when there was no content and then content-length was actually 0. This worked fine with Minio, but Garage returned an error that the header would not be there, even when it was. I don't know the reason for this.

As soon as I added excluded the content-length: 0 for:

DeleteObject
GetObjectRange
HeadObject

the problems all went away.

There is a difference in API behavior to Minio though, which I am using as a reference here, but this was partly my fault, sorry about the issue in that case.

I know that a few currently existing rust s3 crates add the content-length: 0 as well, because I re-used parts of the signature algorithm from existing crates, but with the exclusion it works for Minio and Garage at the same time, so I will just leave it out for compatibility.
I don't know if you want to investigate this difference further, but I guess the issue can be closed.

Edit:

There is just one additional thing I noticed in this case.
Even when I add content-length: 0 while doing a GetObjectRange, I receive an HTTP 400 but actually with the body I would only expect from an HTTP 206.

I was actually able to solve it. I digged deeper into the problem and it was partly my fault as well. I did not check the return code, only extracted headers and saw the `content-length` not being correct. What I actually got back from Garage was an `HTTP 400` with an empty body, body the following headers: ```json {"content-length": "214", "content-type": "application/xml", "date": "Fri, 12 Apr 2024 12:06:30 GMT"} ``` The reason for this was actually pretty tricky. I only found it because I found problems with `DeleteObject` and `GetObjectRange` as well, which returned the following error: ```xml <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidRequest</Code><Message>Bad request: signed header `content-length` is not present</Message><Resource>/test/test_data_128000</Resource><Region>home</Region></Error> ``` The `content-length` was present. I always added it to the signed headers by default, even when there was no content and then `content-length` was actually `0`. This worked fine with Minio, but Garage returned an error that the header would not be there, even when it was. I don't know the reason for this. As soon as I added excluded the `content-length: 0` for: - `DeleteObject` - `GetObjectRange` - `HeadObject` the problems all went away. There is a difference in API behavior to Minio though, which I am using as a reference here, but this was partly my fault, sorry about the issue in that case. I know that a few currently existing rust s3 crates add the `content-length: 0` as well, because I re-used parts of the signature algorithm from existing crates, but with the exclusion it works for Minio and Garage at the same time, so I will just leave it out for compatibility. I don't know if you want to investigate this difference further, but I guess the issue can be closed. Edit: There is just one additional thing I noticed in this case. Even when I add `content-length: 0` while doing a `GetObjectRange`, I receive an `HTTP 400` but actually with the body I would only expect from an `HTTP 206`.