Multi-char delimiters, was: CocroachDB backup failes with 400 bad request #692
Labels
No labels
action
check-aws
action
discussion-needed
action
for-external-contributors
action
for-newcomers
action
more-info-needed
action
need-funding
action
triage-required
kind
correctness
kind
ideas
kind
improvement
kind
performance
kind
testing
kind
usability
kind
wrong-behavior
prio
critical
prio
low
scope
admin-api
scope
background-healing
scope
build
scope
documentation
scope
k8s
scope
layout
scope
metadata
scope
ops
scope
rpc
scope
s3-api
scope
security
scope
telemetry
No project
No assignees
2 participants
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: Deuxfleurs/garage#692
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Hello,
I have switched from ceph to garageDB because I like the architecture and simplicity of administration. Thank you for writing this piece of software.
I have now connected a CockroachDB cluster and try to setup backups. This gives me an error and I see the following in the logs:
This returns the following error in the cockroachDB console:
I'm not very familiar with the S3 API iteself and hope to find some help here.
Kind regards,
Daan
It looks like CockroachDB is making irregular usage of the S3 API : it is sending a multi-character value for the
delimiter
parameter of the ListObjects API call. According to the S3 documentation, the delimiter is supposed to be a single character, and Garage is implemented to support only single-character delimiters. However since CockroachDB sends multi-character delimiters, it must mean that they are supported by AWS S3, so this is arguably a compatibility bug in Garage.In order, we need to:
Confirm that multi-character delimiters work on AWS S3, and determine their exact behavior
Implement the same thing in Garage
Hi @lx,
Thank you so much for your very fast reply and insights. I have searched on this multi-character delimiter and it is described in this cockroachDB issue: https://github.com/cockroachdb/cockroach/issues/100123
It seems that the S3 documentation is not clear (https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html):
<Delimiter>string</Delimiter>
I don't know the exact behavior or also how I can test it. But it seems that the above issue and cockroachDB validated that AWS S3 is actually accepting strings a delimiter.
If I can assist in any way, please let me know.
I was wondering if there might be a "gotcha" in S3, for example: multi-character delimiters being accepted but only the last character actually being taken into account. To make sure that the full delimiter string is actually used, we would ideally need to create a test bucket on AWS and insert some objects with various names and paths, and then check the result of ListObjects with multi-char delimiters. For instance, if we have two objects called
a/b/c/d
anda/c/b/e
, listing the entire bucket with delimiter/
would give us justa/
as a result in CommonPrefixes, but with a delimiter ofb/
we would have two results in CommonPrefixes:a/b/
anda/c/b/
. If you'd like to make these tests against AWS S3 and/or Minio, that would be of great help. You can use thes3api
subcommand ofawscli
v2 to directly run requests against an S3 server from the command line (documentation)Anyways, I'm tagging this for v1.0 since once we get the behavior confirmed, this shouldn't be too hard to implement (I think we're already supporting multi-byte delimiters for unicode characters, so supporting arbitrary-length strings won't be a problem)
Hi @lx,
I have tried to make the requests to AWS. I first created the object structure:
And then I have done your requests:
It thus seems that your understanding of how it should work is correct. I hope this helps.
Kind regards,
Daan
Thank you for testing, I'll keep you updated on the implementation.
CocroachDB backup failes with 400 bad requestto Multi-char delimiters, was: CocroachDB backup failes with 400 bad requestlx referenced this issue2024-03-01 14:14:56 +00:00