Commit graph

1038 commits

Author SHA1 Message Date
b4592a00fe Implement ListMultipartUploads (#171)
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-01-12 19:04:55 +01:00
9cb2e9e57c
Add documentation to migrate to 0.6 2022-01-10 14:42:58 +01:00
3586c7257c
Clarify error message 2022-01-10 14:10:04 +01:00
17ea28a438
Fix trivial bug in CLI 2022-01-10 12:38:33 +01:00
8f39360f22
Update documentation 2022-01-07 17:14:37 +01:00
7ee11f0eb6
Fix unit tests 2022-01-05 17:34:48 +01:00
168a90dfb5
Fix some error codes 2022-01-05 17:07:36 +01:00
fb1e31add0
Small CLI changes 2022-01-05 16:28:46 +01:00
135858d067
Implement DeleteBucket 2022-01-05 16:28:19 +01:00
8395030e48
Implement CreateBucket 2022-01-05 15:56:48 +01:00
9431090b1e
Implement key allow|deny --create-bucket 2022-01-05 15:12:59 +01:00
677ab60cc1
Small changes in key model and refactoring 2022-01-04 18:59:17 +01:00
df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates 2022-01-04 12:53:14 +01:00
1bcd6fabbd
New buckets for 0.6.0: small changes
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
2022-01-04 12:52:47 +01:00
ba7f268b99
Rename and change query filters 2022-01-04 12:52:46 +01:00
de37658b94
Hopefully fix Nix build 2022-01-04 12:52:46 +01:00
e59c23a69d
Refactor logic for setting/unsetting aliases 2022-01-04 12:52:46 +01:00
2140cd7205
Remove website redirects 2022-01-04 12:52:46 +01:00
beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
c7d5c73244
Add must_use to some CRDT functions 2022-01-04 12:47:28 +01:00
b76d0580a0
Fix forgotten flag 2022-01-04 12:47:28 +01:00
87121dce9d
New buckets for 0.6.0: documentation and build files 2022-01-04 12:47:06 +01:00
b1cfd16913
New buckets for 0.6.0: small fixes, including:
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
2022-01-04 12:46:41 +01:00
5db600e231
More complete output to bucket info and key info 2022-01-04 12:46:41 +01:00
4d30e62db4
New buckets for 0.6.0: migration code and build files 2022-01-04 12:46:13 +01:00
0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
53f71b3a57
Implement bucket alias and bucket unalias 2022-01-04 12:45:51 +01:00
5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
8f6026de5e
Make table name a const in trait 2021-12-15 15:39:10 +01:00
945b75dbf1 update s3 compatibility list (#177)
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#177
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 15:05:54 +01:00
ca7b438f3f less strict ListBuckets (#178)
fix #175

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#178
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 15:05:36 +01:00
1eb972b1ac Add compression using zstd (#173)
fix #27

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#173
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 11:26:43 +01:00
60d4459926 BucketWebsite (#174)
fix #77

this does not store anything but a on/off switch for website, and does not implement GetBucketWebsite as it would require storing more. GetBucketWebsite should be pretty easy to implement once data is stored though.

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#174
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 10:41:39 +01:00
3b3a1f275f
Add a second plot 2021-12-13 11:58:03 +01:00
dba9af2968
Update benchmark 2021-12-09 18:42:45 +01:00
e9358054ac
Typos and dead links 2021-12-08 14:40:14 +01:00
f9e5520ffb
Add a benchmark page with a first benchmark 2021-12-08 11:30:07 +01:00
4b369347c0
S3 compatibility target 2021-12-06 17:18:45 +01:00
224c89ad6e
Reorganize and improve documentation 2021-12-06 16:33:01 +01:00
7c2037ba87
WIP front page garage 2021-12-06 15:33:43 +01:00
c4ac8835d3 add proper request router for s3 api (#163)
fix #161

Current request router was organically grown, and is getting messier and messier with each addition.
This router cover exaustively existing API endpoints (with exceptions listed in [#161(comment)](Deuxfleurs/garage#161 (comment)) either because new and old api endpoint can't feasabily be differentied, or it's more lambda than s3).

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#163
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-06 15:17:47 +01:00
ccce75bc25
Remove TODO and genkeys.sh 2021-12-06 13:15:50 +01:00
7f26ed55cd Improved handling of HTTP ranges
- correct HTTP code when range syntax is invalid (fix #140)
- when multiple ranges are given, simply ignore and send whole file

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#157
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-11-29 11:52:42 +01:00
8811bb08e6
In ListBuckets, hide entry if no perms 2021-11-22 12:10:28 +01:00
85b2e4ca29
Start socat only once
Fixes #124
2021-11-17 10:59:32 +01:00
c94406f428
Improve how node roles are assigned in Garage
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
2021-11-16 16:05:53 +01:00
53888995bd update doc and comments 2021-11-16 15:41:41 +01:00
f0893b904d update cargo.nix 2021-11-16 15:41:41 +01:00
396fe4c702 clippy 2021-11-16 15:41:41 +01:00