Commit graph

423 commits

Author SHA1 Message Date
3baa841d6f
tests: Fix garage integration test 2022-02-10 17:55:49 +01:00
dd407e7041
tests: Add garage integration tests (base) 2022-02-10 17:55:49 +01:00
af261e1789 Fix a bug when a migration is followed by a rebalance
Nodes would stabilize on different encoding formats for the values,
some having the pre-migration format and some having the post-migration
format. This would be reflected in the Merkle trees never converging
and thus having an infinite resync loop.
2022-02-10 17:38:27 +01:00
ae2f32baf1
Hide deleted key in bucket info (fix #211) 2022-02-02 17:12:48 +01:00
c99f55c420
Add restriction on part ordering in CompleteMultipartUpload 2022-01-25 12:45:00 +01:00
acdf893362
Fix partnumber 2022-01-25 12:25:23 +01:00
338b1b83ee
Implement part_number for GetObject 2022-01-24 21:04:42 +01:00
6dab836f3a
Multipart improvements
- support part_number for HeadObject
- add checks in complete_multipart_upload
2022-01-24 21:04:40 +01:00
513a6b15f9
Handle OPTIONS on website endpoint 2022-01-24 12:32:28 +01:00
ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
2022-01-24 11:58:00 +01:00
820924534a use clamp instead of min(max()) 2022-01-24 11:56:59 +01:00
440374524b Implement ListParts 2022-01-21 10:42:30 +01:00
fe003d6fbc Add ListPartsResult structure 2022-01-20 16:38:55 +01:00
e55fa38c99 Add date verification to presigned urls (#196)
fix #96
fix #162 by returning Forbidden instead Bad Request

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-01-18 12:22:31 +01:00
178e35f868 refactor s3_router and api_server to make unused Endpoint parameters more obvious 2022-01-17 15:50:24 +01:00
fdcddbe168 Fix Multipart Upload with WinSCP (#164) (#193)
Closes #164.

Reviewed-on: Deuxfleurs/garage#193
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 11:18:40 +01:00
b45dcc1925 Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156)
Closes #64.

Reviewed-on: Deuxfleurs/garage#156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 10:55:31 +01:00
d4dd2e2640
Make use of website config, return error document on error 2022-01-13 14:25:19 +01:00
9eb211948e
Allow setting index document and error document on the CLI 2022-01-13 14:25:19 +01:00
3ea8ca1b9e
Implement GetBucketWebsite 2022-01-13 14:23:52 +01:00
f7349f4005
Add quotes in returned etags 2022-01-13 14:03:33 +01:00
1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures 2022-01-13 14:03:30 +01:00
6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187) 2022-01-13 13:56:55 +01:00
b4592a00fe Implement ListMultipartUploads (#171)
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-01-12 19:04:55 +01:00
3586c7257c
Clarify error message 2022-01-10 14:10:04 +01:00
17ea28a438
Fix trivial bug in CLI 2022-01-10 12:38:33 +01:00
7ee11f0eb6
Fix unit tests 2022-01-05 17:34:48 +01:00
168a90dfb5
Fix some error codes 2022-01-05 17:07:36 +01:00
fb1e31add0
Small CLI changes 2022-01-05 16:28:46 +01:00
135858d067
Implement DeleteBucket 2022-01-05 16:28:19 +01:00
8395030e48
Implement CreateBucket 2022-01-05 15:56:48 +01:00
9431090b1e
Implement key allow|deny --create-bucket 2022-01-05 15:12:59 +01:00
677ab60cc1
Small changes in key model and refactoring 2022-01-04 18:59:17 +01:00
df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates 2022-01-04 12:53:14 +01:00
1bcd6fabbd
New buckets for 0.6.0: small changes
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
2022-01-04 12:52:47 +01:00
ba7f268b99
Rename and change query filters 2022-01-04 12:52:46 +01:00
e59c23a69d
Refactor logic for setting/unsetting aliases 2022-01-04 12:52:46 +01:00
2140cd7205
Remove website redirects 2022-01-04 12:52:46 +01:00
beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
c7d5c73244
Add must_use to some CRDT functions 2022-01-04 12:47:28 +01:00
b76d0580a0
Fix forgotten flag 2022-01-04 12:47:28 +01:00
87121dce9d
New buckets for 0.6.0: documentation and build files 2022-01-04 12:47:06 +01:00
b1cfd16913
New buckets for 0.6.0: small fixes, including:
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
2022-01-04 12:46:41 +01:00
5db600e231
More complete output to bucket info and key info 2022-01-04 12:46:41 +01:00
4d30e62db4
New buckets for 0.6.0: migration code and build files 2022-01-04 12:46:13 +01:00
0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
53f71b3a57
Implement bucket alias and bucket unalias 2022-01-04 12:45:51 +01:00
5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00