KokaKiwi/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues Pull requests Projects Releases Wiki Activity
701 commits 13 branches 1 tag 80 MiB
Commit graph

138 commits

Author SHA1 Message Date
Alex 2814d41842
Allow garage layout assign to assign to several nodes at once 2022-03-24 15:27:13 +01:00
Alex 413ab0eaed
Small change to partition assignation algorithm 
This change helps ensure that nodes for each partition are spread
over all datacenters, a property that wasn't ensured previously
when going from a 2 DC deployment to a 3 DC deployment
 2022-03-24 15:27:10 +01:00
Alex c1d9854d2c
Move block manager to separate module 2022-03-23 10:25:15 +01:00
trinity-1686a 8565f7dc31 cleanup 2022-03-23 10:22:37 +01:00
trinity-1686a 8db6b84559 add test for create bucket and put website with streaming signature 2022-03-23 10:22:37 +01:00
trinity-1686a 1eb7fdb08f add test framework for arbitraty S3 requests 
and implement some basic test with it
 2022-03-23 10:22:36 +01:00
Alex db46cdef79
Update netapp to v0.4.1 2022-03-15 17:09:57 +01:00
Alex ba6b56ae68
Fix some new clippy lints 2022-03-14 12:27:49 +01:00
Alex 9b2b531f4d
Make admin server optional 2022-03-14 10:54:25 +01:00
Alex 203e8d2c34
Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
Alex dc8d0496cc
Refactoring: rename config files, make modifications less invasive 2022-03-14 10:53:51 +01:00
Alex 2a5609b292
Add metrics to API endpoint 2022-03-14 10:53:36 +01:00
Alex 818daa5c78
Refactor how durations are measured 2022-03-14 10:53:35 +01:00
Alex bb04d94fa9
Update to Netapp 0.4 which supports distributed tracing 2022-03-14 10:52:30 +01:00
Alex 8c2fb0c066
Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
Alex 2cab84b1fe
Add many metrics in table/ and rpc/ 2022-03-14 10:51:50 +01:00
Maximilien R 1e2cf26373
Implement basic metrics in table 2022-03-14 10:51:17 +01:00
mricher e349af13a7
Update dependencies and add admin module with metrics 
- Global dependencies updated in Cargo.lock
- New module created in src/admin to host:
  - the (future) admin REST API
  - the metric collection
- add configuration block

No metrics implemented yet
 2022-03-14 10:51:12 +01:00
Quentin c00b2c9948 Functional tests for admin commands 2022-03-07 17:32:07 +01:00
Quentin 8df1e186de Functional tests for website endpoints 2022-03-07 17:32:07 +01:00
Quentin 2ef60b8417 Functional test for multipart endpoints 2022-03-07 17:32:07 +01:00
Quentin 1e639ec67c Functional test for ListMultipartUploads 2022-03-07 17:32:07 +01:00
Quentin cfea1e0315 Functional tests for bucket endpoints 2022-03-07 17:32:02 +01:00
Quentin 05eb79929e Functional tests for object operations 2022-03-07 17:05:10 +01:00
Quentin 0f4e0e8bb9 Move ListObjects tests to Rust 2022-03-07 17:05:10 +01:00
Jill 84613e66a2
garage(tests): Remove RNG stuff 2022-02-11 10:50:55 +01:00
Jill c8b30ebc79
garage(tests): Remove superfluous test 2022-02-10 17:55:51 +01:00
Jill d7decda3f4
garage(tests): Add random suffix to created buckets. 2022-02-10 17:55:50 +01:00
Jill cd13ea461b
garage(tests): Add some unsafe-usage doc and tweaks 2022-02-10 17:55:50 +01:00
Quentin 5d19f3d2d7
Add integration tests to Drone 2022-02-10 17:55:50 +01:00
Jill 3baa841d6f
tests: Fix garage integration test 2022-02-10 17:55:49 +01:00
Jill dd407e7041
tests: Add garage integration tests (base) 2022-02-10 17:55:49 +01:00
Alex ae2f32baf1
Hide deleted key in bucket info (fix #211) 2022-02-02 17:12:48 +01:00
Alex 9eb211948e
Allow setting index document and error document on the CLI 2022-01-13 14:25:19 +01:00
Quentin b4592a00fe Implement ListMultipartUploads (#171) 
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
 2022-01-12 19:04:55 +01:00
Alex 17ea28a438
Fix trivial bug in CLI 2022-01-10 12:38:33 +01:00
Alex fb1e31add0
Small CLI changes 2022-01-05 16:28:46 +01:00
Alex 8395030e48
Implement CreateBucket 2022-01-05 15:56:48 +01:00
Alex 9431090b1e
Implement key allow|deny --create-bucket 2022-01-05 15:12:59 +01:00
Alex 677ab60cc1
Small changes in key model and refactoring 2022-01-04 18:59:17 +01:00
Alex df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates 2022-01-04 12:53:14 +01:00
Alex 1bcd6fabbd
New buckets for 0.6.0: small changes 
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
 2022-01-04 12:52:47 +01:00
Alex e59c23a69d
Refactor logic for setting/unsetting aliases 2022-01-04 12:52:46 +01:00
Alex 2140cd7205
Remove website redirects 2022-01-04 12:52:46 +01:00
Alex beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
Alex d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
Alex b1cfd16913
New buckets for 0.6.0: small fixes, including: 
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
 2022-01-04 12:46:41 +01:00
Alex 5db600e231
More complete output to bucket info and key info 2022-01-04 12:46:41 +01:00
Alex 4d30e62db4
New buckets for 0.6.0: migration code and build files 2022-01-04 12:46:13 +01:00
Alex 0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00