KokaKiwi
/
garage
forked from Deuxfleurs/garage
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
603 Commits 13 Branches 1 Tag 80 MiB
Commit Graph

603 Commits

Author SHA1 Message Date
Jill 11a1f3f6cf
garage_api(fixup): Verify Content-SHA256 header for multipart upload only when needed. 2022-01-14 14:09:18 +01:00
Jill b2eda2c13e
garage_api(fixup): Fix stream early ending edge-case 2022-01-13 16:45:23 +01:00
Jill e7e8ce73e3
garage_api(fixup): Fix unlimited buffering 2022-01-13 16:23:55 +01:00
Jill 847b5ad407
garage_api(fixup): Fix unexpected EOF error 2022-01-13 16:23:54 +01:00
Jill 561f614b6e
ci: Use HTTP endpoint for minio 2022-01-13 16:23:54 +01:00
Jill bf08e62ce0
garage_api(fixup): Handle interrupted stream (unittest included) 2022-01-13 16:23:54 +01:00
Jill 2b034b7c4e
garage_api(fixup): Fixups from reviews 2022-01-13 16:23:52 +01:00
Jill f0cb931a45
nix(garage_api): Add missing signature submodule 2022-01-13 16:14:56 +01:00
Jill 006e5cc231
garage_api: Validate signature for chunked PUT payload 2022-01-13 16:14:56 +01:00
Jill f1bfc939aa
garage_api: Handle chunked PUT payload 2022-01-13 16:14:55 +01:00
Jill 732b4d0b63
garage_api: Refactor BodyChunker for stream-composition 2022-01-13 16:14:53 +01:00
Alex f7349f4005
Add quotes in returned etags 2022-01-13 14:03:33 +01:00
Alex 1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures 2022-01-13 14:03:30 +01:00
Alex 6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
Alex 3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187) 2022-01-13 13:56:55 +01:00
Quentin b4592a00fe Implement ListMultipartUploads (#171) 
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
 2022-01-12 19:04:55 +01:00
Alex 9cb2e9e57c
Add documentation to migrate to 0.6 2022-01-10 14:42:58 +01:00
Alex 3586c7257c
Clarify error message 2022-01-10 14:10:04 +01:00
Alex 17ea28a438
Fix trivial bug in CLI 2022-01-10 12:38:33 +01:00
Alex 8f39360f22
Update documentation 2022-01-07 17:14:37 +01:00
Alex 7ee11f0eb6
Fix unit tests 2022-01-05 17:34:48 +01:00
Alex 168a90dfb5
Fix some error codes 2022-01-05 17:07:36 +01:00
Alex fb1e31add0
Small CLI changes 2022-01-05 16:28:46 +01:00
Alex 135858d067
Implement DeleteBucket 2022-01-05 16:28:19 +01:00
Alex 8395030e48
Implement CreateBucket 2022-01-05 15:56:48 +01:00
Alex 9431090b1e
Implement key allow|deny --create-bucket 2022-01-05 15:12:59 +01:00
Alex 677ab60cc1
Small changes in key model and refactoring 2022-01-04 18:59:17 +01:00
Alex df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates 2022-01-04 12:53:14 +01:00
Alex 1bcd6fabbd
New buckets for 0.6.0: small changes 
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
 2022-01-04 12:52:47 +01:00
Alex ba7f268b99
Rename and change query filters 2022-01-04 12:52:46 +01:00
Alex de37658b94
Hopefully fix Nix build 2022-01-04 12:52:46 +01:00
Alex e59c23a69d
Refactor logic for setting/unsetting aliases 2022-01-04 12:52:46 +01:00
Alex 2140cd7205
Remove website redirects 2022-01-04 12:52:46 +01:00
Alex beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
Alex d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
Alex c7d5c73244
Add must_use to some CRDT functions 2022-01-04 12:47:28 +01:00
Alex b76d0580a0
Fix forgotten flag 2022-01-04 12:47:28 +01:00
Alex 87121dce9d
New buckets for 0.6.0: documentation and build files 2022-01-04 12:47:06 +01:00
Alex b1cfd16913
New buckets for 0.6.0: small fixes, including: 
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
 2022-01-04 12:46:41 +01:00
Alex 5db600e231
More complete output to bucket info and key info 2022-01-04 12:46:41 +01:00
Alex 4d30e62db4
New buckets for 0.6.0: migration code and build files 2022-01-04 12:46:13 +01:00
Alex 0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
Alex 53f71b3a57
Implement bucket alias and bucket unalias 2022-01-04 12:45:51 +01:00
Alex 5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
Alex 8f6026de5e
Make table name a const in trait 2021-12-15 15:39:10 +01:00
trinity-1686a 945b75dbf1 update s3 compatibility list (#177) 
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#177
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 15:05:54 +01:00
trinity-1686a ca7b438f3f less strict ListBuckets (#178) 
fix #175

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#178
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 15:05:36 +01:00
trinity-1686a 1eb972b1ac Add compression using zstd (#173) 
fix #27

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#173
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 11:26:43 +01:00
trinity-1686a 60d4459926 BucketWebsite (#174) 
fix #77

this does not store anything but a on/off switch for website, and does not implement GetBucketWebsite as it would require storing more. GetBucketWebsite should be pretty easy to implement once data is stored though.

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#174
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 10:41:39 +01:00
Quentin 3b3a1f275f
Add a second plot 2021-12-13 11:58:03 +01:00