Commit graph

1705 commits

Author SHA1 Message Date
820924534a use clamp instead of min(max())
All checks were successful
continuous-integration/drone/push Build is passing
2022-01-24 11:56:59 +01:00
94f0e7c135 Test ListParts endpoint with awscli
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-21 10:42:35 +01:00
440374524b Implement ListParts 2022-01-21 10:42:30 +01:00
fe003d6fbc Add ListPartsResult structure 2022-01-20 16:38:55 +01:00
e55fa38c99 Add date verification to presigned urls (#196)
All checks were successful
continuous-integration/drone/push Build is passing
fix #96
fix #162 by returning Forbidden instead Bad Request

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-01-18 12:22:31 +01:00
178e35f868 refactor s3_router and api_server to make unused Endpoint parameters more obvious
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-17 15:50:24 +01:00
7c049f1c94 Fix extreme value to be less extreme so that integration test works on 32bits
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
2022-01-17 12:56:29 +01:00
fdcddbe168 Fix Multipart Upload with WinSCP (#164) (#193)
All checks were successful
continuous-integration/drone/push Build is passing
Closes #164.

Reviewed-on: #193
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 11:18:40 +01:00
b45dcc1925 Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156)
All checks were successful
continuous-integration/drone/push Build is passing
Closes #64.

Reviewed-on: #156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 10:55:31 +01:00
60c0033c8b
Update documentation
Some checks reported errors
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build encountered an error
2022-01-13 14:25:22 +01:00
d4dd2e2640
Make use of website config, return error document on error 2022-01-13 14:25:19 +01:00
9eb211948e
Allow setting index document and error document on the CLI 2022-01-13 14:25:19 +01:00
3ea8ca1b9e
Implement GetBucketWebsite 2022-01-13 14:23:52 +01:00
f7349f4005
Add quotes in returned etags
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-13 14:03:33 +01:00
1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures 2022-01-13 14:03:30 +01:00
6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187) 2022-01-13 13:56:55 +01:00
b4592a00fe Implement ListMultipartUploads (#171)
All checks were successful
continuous-integration/drone/push Build is passing
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: #171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-01-12 19:04:55 +01:00
9cb2e9e57c
Add documentation to migrate to 0.6
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
2022-01-10 14:42:58 +01:00
3586c7257c
Clarify error message
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-10 14:10:04 +01:00
17ea28a438
Fix trivial bug in CLI
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is passing
2022-01-10 12:38:33 +01:00
8f39360f22
Update documentation
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-01-07 17:14:37 +01:00
7ee11f0eb6
Fix unit tests
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-05 17:34:48 +01:00
168a90dfb5
Fix some error codes
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-01-05 17:07:36 +01:00
fb1e31add0
Small CLI changes
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-01-05 16:28:46 +01:00
135858d067
Implement DeleteBucket 2022-01-05 16:28:19 +01:00
8395030e48
Implement CreateBucket
Some checks failed
continuous-integration/drone/push Build is failing
2022-01-05 15:56:48 +01:00
9431090b1e
Implement key allow|deny --create-bucket
Some checks failed
continuous-integration/drone/push Build is failing
2022-01-05 15:12:59 +01:00
677ab60cc1
Small changes in key model and refactoring
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-04 18:59:17 +01:00
df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-04 12:53:14 +01:00
1bcd6fabbd
New buckets for 0.6.0: small changes
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
2022-01-04 12:52:47 +01:00
ba7f268b99
Rename and change query filters 2022-01-04 12:52:46 +01:00
de37658b94
Hopefully fix Nix build 2022-01-04 12:52:46 +01:00
e59c23a69d
Refactor logic for setting/unsetting aliases 2022-01-04 12:52:46 +01:00
2140cd7205
Remove website redirects 2022-01-04 12:52:46 +01:00
beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
c7d5c73244
Add must_use to some CRDT functions 2022-01-04 12:47:28 +01:00
b76d0580a0
Fix forgotten flag 2022-01-04 12:47:28 +01:00
87121dce9d
New buckets for 0.6.0: documentation and build files 2022-01-04 12:47:06 +01:00
b1cfd16913
New buckets for 0.6.0: small fixes, including:
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
2022-01-04 12:46:41 +01:00
5db600e231
More complete output to bucket info and key info 2022-01-04 12:46:41 +01:00
4d30e62db4
New buckets for 0.6.0: migration code and build files 2022-01-04 12:46:13 +01:00
0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
53f71b3a57
Implement bucket alias and bucket unalias 2022-01-04 12:45:51 +01:00
5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
8f6026de5e
Make table name a const in trait 2021-12-15 15:39:10 +01:00
945b75dbf1 update s3 compatibility list (#177)
All checks were successful
continuous-integration/drone/push Build is passing
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #177
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 15:05:54 +01:00
ca7b438f3f less strict ListBuckets (#178)
All checks were successful
continuous-integration/drone/push Build is passing
fix #175

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #178
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 15:05:36 +01:00
1eb972b1ac Add compression using zstd (#173)
All checks were successful
continuous-integration/drone/push Build is passing
fix #27

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #173
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 11:26:43 +01:00