Commit graph

1043 commits

Author SHA1 Message Date
3ea8ca1b9e
Implement GetBucketWebsite 2022-01-13 14:23:52 +01:00
f7349f4005
Add quotes in returned etags
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-13 14:03:33 +01:00
1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures 2022-01-13 14:03:30 +01:00
6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187) 2022-01-13 13:56:55 +01:00
b4592a00fe Implement ListMultipartUploads (#171)
All checks were successful
continuous-integration/drone/push Build is passing
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: #171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-01-12 19:04:55 +01:00
9cb2e9e57c
Add documentation to migrate to 0.6
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
2022-01-10 14:42:58 +01:00
3586c7257c
Clarify error message
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-10 14:10:04 +01:00
17ea28a438
Fix trivial bug in CLI
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is passing
2022-01-10 12:38:33 +01:00
8f39360f22
Update documentation
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-01-07 17:14:37 +01:00
7ee11f0eb6
Fix unit tests
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-05 17:34:48 +01:00
168a90dfb5
Fix some error codes
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-01-05 17:07:36 +01:00
fb1e31add0
Small CLI changes
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-01-05 16:28:46 +01:00
135858d067
Implement DeleteBucket 2022-01-05 16:28:19 +01:00
8395030e48
Implement CreateBucket
Some checks failed
continuous-integration/drone/push Build is failing
2022-01-05 15:56:48 +01:00
9431090b1e
Implement key allow|deny --create-bucket
Some checks failed
continuous-integration/drone/push Build is failing
2022-01-05 15:12:59 +01:00
677ab60cc1
Small changes in key model and refactoring
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-04 18:59:17 +01:00
df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-04 12:53:14 +01:00
1bcd6fabbd
New buckets for 0.6.0: small changes
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
2022-01-04 12:52:47 +01:00
ba7f268b99
Rename and change query filters 2022-01-04 12:52:46 +01:00
de37658b94
Hopefully fix Nix build 2022-01-04 12:52:46 +01:00
e59c23a69d
Refactor logic for setting/unsetting aliases 2022-01-04 12:52:46 +01:00
2140cd7205
Remove website redirects 2022-01-04 12:52:46 +01:00
beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
c7d5c73244
Add must_use to some CRDT functions 2022-01-04 12:47:28 +01:00
b76d0580a0
Fix forgotten flag 2022-01-04 12:47:28 +01:00
87121dce9d
New buckets for 0.6.0: documentation and build files 2022-01-04 12:47:06 +01:00
b1cfd16913
New buckets for 0.6.0: small fixes, including:
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
2022-01-04 12:46:41 +01:00
5db600e231
More complete output to bucket info and key info 2022-01-04 12:46:41 +01:00
4d30e62db4
New buckets for 0.6.0: migration code and build files 2022-01-04 12:46:13 +01:00
0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
53f71b3a57
Implement bucket alias and bucket unalias 2022-01-04 12:45:51 +01:00
5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
8f6026de5e
Make table name a const in trait 2021-12-15 15:39:10 +01:00
945b75dbf1 update s3 compatibility list (#177)
All checks were successful
continuous-integration/drone/push Build is passing
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #177
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 15:05:54 +01:00
ca7b438f3f less strict ListBuckets (#178)
All checks were successful
continuous-integration/drone/push Build is passing
fix #175

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #178
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 15:05:36 +01:00
1eb972b1ac Add compression using zstd (#173)
All checks were successful
continuous-integration/drone/push Build is passing
fix #27

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #173
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 11:26:43 +01:00
60d4459926 BucketWebsite (#174)
All checks were successful
continuous-integration/drone/push Build is passing
fix #77

this does not store anything but a on/off switch for website, and does not implement GetBucketWebsite as it would require storing more. GetBucketWebsite should be pretty easy to implement once data is stored though.

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #174
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 10:41:39 +01:00
3b3a1f275f
Add a second plot
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-12-13 11:58:03 +01:00
dba9af2968
Update benchmark
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-12-09 18:42:45 +01:00
e9358054ac
Typos and dead links
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-12-08 14:40:14 +01:00
f9e5520ffb
Add a benchmark page with a first benchmark
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-12-08 11:30:07 +01:00
4b369347c0
S3 compatibility target
Some checks reported errors
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone Build encountered an error
2021-12-06 17:18:45 +01:00
224c89ad6e
Reorganize and improve documentation
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-12-06 16:33:01 +01:00
7c2037ba87
WIP front page garage 2021-12-06 15:33:43 +01:00
c4ac8835d3 add proper request router for s3 api (#163)
All checks were successful
continuous-integration/drone/push Build is passing
fix #161

Current request router was organically grown, and is getting messier and messier with each addition.
This router cover exaustively existing API endpoints (with exceptions listed in [#161(comment)](#161 (comment)) either because new and old api endpoint can't feasabily be differentied, or it's more lambda than s3).

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #163
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-06 15:17:47 +01:00
ccce75bc25
Remove TODO and genkeys.sh
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-12-06 13:15:50 +01:00
7f26ed55cd Improved handling of HTTP ranges
Some checks failed
continuous-integration/drone/push Build is failing
- correct HTTP code when range syntax is invalid (fix #140)
- when multiple ranges are given, simply ignore and send whole file

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #157
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-11-29 11:52:42 +01:00
8811bb08e6
In ListBuckets, hide entry if no perms
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-11-22 12:10:28 +01:00