KokaKiwi/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues Pull requests Projects Releases Wiki Activity
garage/doc/book/src/reference_manual/s3_compatibility.md
Quentin b4592a00fe Implement ListMultipartUploads (#171) 
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-01-12 19:04:55 +01:00

2.8 KiB
Raw Blame History

S3 Compatibility status

Global S3 features

Implemented:

  • path-style URLs (garage.tld/bucket/key)
  • vhost-style URLs (bucket.garage.tld/key)
  • putting and getting objects in buckets
  • multipart uploads
  • listing objects
  • access control on a per-key-per-bucket basis

Not implemented:

Endpoint implementation

All APIs that are not mentionned are not implemented and will return a 400 bad request.

Endpoint Status
AbortMultipartUpload Implemented
CompleteMultipartUpload Implemented
CopyObject Implemented
CreateBucket Implemented
CreateMultipartUpload Implemented
DeleteBucket Implemented
DeleteBucketWebsite Implemented
DeleteObject Implemented
DeleteObjects Implemented
GetBucketLocation Implemented
GetBucketVersioning Stub (see below)
GetBucketWebsite Unsupported
GetObject Implemented
HeadBucket Implemented
HeadObject Implemented
ListBuckets Implemented
ListObjects Implemented, bugs? (see below)
ListObjectsV2 Implemented
ListMultipartUpload Implemented
ListParts Missing
PutObject Implemented
PutBucketWebsite Partially implemented (see below)
UploadPart Implemented

  • GetBucketVersioning: Stub implementation (Garage does not yet support versionning so this always returns "versionning not enabled").

  • ListObjects: Implemented, but there isn't a very good specification of what encoding-type=url covers so there might be some encoding bugs. In our implementation the url-encoded fields are in the same in ListObjects as they are in ListObjectsV2.

  • PutBucketWebsite: Implemented, but only store if website is enabled, not more complexe informations.

  • GetBucketWebsite: Not implemented yet, will be when PubBucketWebsite store more informations.