Alex
8a5bbc3b0b
More permissive OPTIONS on S3 API
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone Build is passing
2022-03-01 11:15:16 +01:00
Alex
97f245f218
Add tracing output to signature calculation
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-02-28 12:22:39 +01:00
Alex
8129a98291
Process CORS earlier in pipeline
2022-02-28 12:22:39 +01:00
trinity-1686a
f6f8b7f1ad
Support for PostObject ( #222 )
...
continuous-integration/drone/push Build is passing
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html )
- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data
Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #222
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-02-21 23:02:30 +01:00
Jill
84613e66a2
garage(tests): Remove RNG stuff
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-02-11 10:50:55 +01:00
Jill
c8b30ebc79
garage(tests): Remove superfluous test
2022-02-10 17:55:51 +01:00
Jill
d7decda3f4
garage(tests): Add random suffix to created buckets.
2022-02-10 17:55:50 +01:00
Jill
cd13ea461b
garage(tests): Add some unsafe-usage doc and tweaks
2022-02-10 17:55:50 +01:00
Quentin
5d19f3d2d7
Add integration tests to Drone
2022-02-10 17:55:50 +01:00
Jill
3baa841d6f
tests: Fix garage integration test
2022-02-10 17:55:49 +01:00
Jill
dd407e7041
tests: Add garage integration tests (base)
2022-02-10 17:55:49 +01:00
Alex
af261e1789
Fix a bug when a migration is followed by a rebalance
...
continuous-integration/drone/push Build is passing
Nodes would stabilize on different encoding formats for the values,
some having the pre-migration format and some having the post-migration
format. This would be reflected in the Merkle trees never converging
and thus having an infinite resync loop.
2022-02-10 17:38:27 +01:00
Alex
ae2f32baf1
Hide deleted key in bucket info ( fix #211 )
continuous-integration/drone/push Build is failing
2022-02-02 17:12:48 +01:00
Alex
c99f55c420
Add restriction on part ordering in CompleteMultipartUpload
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone Build is passing
2022-01-25 12:45:00 +01:00
Alex
acdf893362
Fix partnumber
2022-01-25 12:25:23 +01:00
Alex
338b1b83ee
Implement part_number for GetObject
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-24 21:04:42 +01:00
Alex
6dab836f3a
Multipart improvements
...
- support part_number for HeadObject
- add checks in complete_multipart_upload
2022-01-24 21:04:40 +01:00
Alex
513a6b15f9
Handle OPTIONS on website endpoint
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-24 12:32:28 +01:00
Alex
ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general
...
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
2022-01-24 11:58:00 +01:00
Trinity Pointard
820924534a
use clamp instead of min(max())
continuous-integration/drone/push Build is passing
2022-01-24 11:56:59 +01:00
Quentin
440374524b
Implement ListParts
2022-01-21 10:42:30 +01:00
Quentin
fe003d6fbc
Add ListPartsResult structure
2022-01-20 16:38:55 +01:00
trinity-1686a
e55fa38c99
Add date verification to presigned urls ( #196 )
...
continuous-integration/drone/push Build is passing
fix #96
fix #162 by returning Forbidden instead Bad Request
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-01-18 12:22:31 +01:00
trinity-1686a
178e35f868
refactor s3_router and api_server to make unused Endpoint parameters more obvious
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-17 15:50:24 +01:00
Jill
fdcddbe168
Fix Multipart Upload with WinSCP ( #164 ) ( #193 )
...
continuous-integration/drone/push Build is passing
Closes #164 .
Reviewed-on: #193
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 11:18:40 +01:00
Jill
b45dcc1925
Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD ( #64 ) ( #156 )
...
continuous-integration/drone/push Build is passing
Closes #64 .
Reviewed-on: #156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 10:55:31 +01:00
Alex
d4dd2e2640
Make use of website config, return error document on error
2022-01-13 14:25:19 +01:00
Alex
9eb211948e
Allow setting index document and error document on the CLI
2022-01-13 14:25:19 +01:00
Alex
3ea8ca1b9e
Implement GetBucketWebsite
2022-01-13 14:23:52 +01:00
Alex
f7349f4005
Add quotes in returned etags
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-13 14:03:33 +01:00
Alex
1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures
2022-01-13 14:03:30 +01:00
Alex
6617a72220
Implement UploadPartCopy
2022-01-13 13:58:47 +01:00
Alex
3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy ( fix #187 )
2022-01-13 13:56:55 +01:00
Quentin
b4592a00fe
Implement ListMultipartUploads ( #171 )
...
continuous-integration/drone/push Build is passing
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.
It took me some times as I wanted to propose the following things:
- Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156 ) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476 ).
- Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
- An IO closure to make unit tests possibles.
- Unit tests, to track regressions and document how to interact with the code
- Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.
Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.
Some points that we might want to discuss:
- ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
- Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
- Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).
Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: #171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-01-12 19:04:55 +01:00
Alex
3586c7257c
Clarify error message
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-10 14:10:04 +01:00
Alex
17ea28a438
Fix trivial bug in CLI
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is passing
2022-01-10 12:38:33 +01:00
Alex
7ee11f0eb6
Fix unit tests
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-05 17:34:48 +01:00
Alex
168a90dfb5
Fix some error codes
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-01-05 17:07:36 +01:00
Alex
fb1e31add0
Small CLI changes
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-01-05 16:28:46 +01:00
Alex
135858d067
Implement DeleteBucket
2022-01-05 16:28:19 +01:00
Alex
8395030e48
Implement CreateBucket
continuous-integration/drone/push Build is failing
2022-01-05 15:56:48 +01:00
Alex
9431090b1e
Implement key allow|deny --create-bucket
continuous-integration/drone/push Build is failing
2022-01-05 15:12:59 +01:00
Alex
677ab60cc1
Small changes in key model and refactoring
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-04 18:59:17 +01:00
Alex
df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-04 12:53:14 +01:00
Alex
1bcd6fabbd
New buckets for 0.6.0: small changes
...
- Fix bucket delete
- fix merge of bucket creation date
- Replace deletable with option in aliases
Rationale: if two aliases point to conflicting bucket, resolving
by making an arbitrary choice risks making data accessible when it
shouldn't be. We'd rather resolve to deleting the alias until
someone puts it back.
2022-01-04 12:52:47 +01:00
Alex
ba7f268b99
Rename and change query filters
2022-01-04 12:52:46 +01:00
Alex
e59c23a69d
Refactor logic for setting/unsetting aliases
2022-01-04 12:52:46 +01:00
Alex
2140cd7205
Remove website redirects
2022-01-04 12:52:46 +01:00
Alex
beeef4758e
Some movement of helper code and refactoring of error handling
2022-01-04 12:52:46 +01:00
Alex
d8ab5bdc3e
New buckets for 0.6.0: fix model and migration
2022-01-04 12:47:28 +01:00