KokaKiwi/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues Pull requests Projects Releases Wiki Activity
701 commits 13 branches 1 tag 80 MiB
Commit graph

118 commits

Author SHA1 Message Date
Alex 077dd1cde9
Clippy 2022-03-23 10:25:39 +01:00
Alex e480aaf338
Make background tranquility a configurable parameter 2022-03-23 10:25:19 +01:00
Alex c3982a90b6
Move DataBlock out of manager.rs 2022-03-23 10:25:19 +01:00
Alex c1d9854d2c
Move block manager to separate module 2022-03-23 10:25:15 +01:00
Alex db46cdef79
Update netapp to v0.4.1 2022-03-15 17:09:57 +01:00
Alex ba6b56ae68
Fix some new clippy lints 2022-03-14 12:27:49 +01:00
Alex 0af314b295
Add comment for fsync 2022-03-14 11:54:00 +01:00
Alex d78bf379fb
Fix resync queue to not drop items 2022-03-14 11:51:37 +01:00
Alex f7e6f4616f
Spawn a single resync worker 2022-03-14 11:51:37 +01:00
Alex dc5ec4ecf9
Add appropriate fsync() calls in write_block 
to ensure that data is persisted properly
 2022-03-14 11:51:32 +01:00
Alex fe62d01b7e
Implement exponential backoff for resync retries 2022-03-14 11:41:20 +01:00
Alex 2377a92f6b
Add wrapper over sled tree to count items (used for big queues) 2022-03-14 10:54:25 +01:00
Alex 203e8d2c34
Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
Alex 2a5609b292
Add metrics to API endpoint 2022-03-14 10:53:36 +01:00
Alex 818daa5c78
Refactor how durations are measured 2022-03-14 10:53:35 +01:00
Alex bb04d94fa9
Update to Netapp 0.4 which supports distributed tracing 2022-03-14 10:52:30 +01:00
Alex 8c2fb0c066
Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
Alex 2cab84b1fe
Add many metrics in table/ and rpc/ 2022-03-14 10:51:50 +01:00
Alex ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general 
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
 2022-01-24 11:58:00 +01:00
trinity-1686a e55fa38c99 Add date verification to presigned urls (#196) 
fix #96
fix #162 by returning Forbidden instead Bad Request

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-01-18 12:22:31 +01:00
Alex 6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
Quentin b4592a00fe Implement ListMultipartUploads (#171) 
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
 2022-01-12 19:04:55 +01:00
Alex 8395030e48
Implement CreateBucket 2022-01-05 15:56:48 +01:00
Alex 677ab60cc1
Small changes in key model and refactoring 2022-01-04 18:59:17 +01:00
Alex df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates 2022-01-04 12:53:14 +01:00
Alex 1bcd6fabbd
New buckets for 0.6.0: small changes 
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
 2022-01-04 12:52:47 +01:00
Alex ba7f268b99
Rename and change query filters 2022-01-04 12:52:46 +01:00
Alex e59c23a69d
Refactor logic for setting/unsetting aliases 2022-01-04 12:52:46 +01:00
Alex 2140cd7205
Remove website redirects 2022-01-04 12:52:46 +01:00
Alex beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
Alex d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
Alex b1cfd16913
New buckets for 0.6.0: small fixes, including: 
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
 2022-01-04 12:46:41 +01:00
Alex 4d30e62db4
New buckets for 0.6.0: migration code and build files 2022-01-04 12:46:13 +01:00
Alex 0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
Alex 53f71b3a57
Implement bucket alias and bucket unalias 2022-01-04 12:45:51 +01:00
Alex 5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
Alex 8f6026de5e
Make table name a const in trait 2021-12-15 15:39:10 +01:00
trinity-1686a 1eb972b1ac Add compression using zstd (#173) 
fix #27

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#173
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 11:26:43 +01:00
Alex c94406f428
Improve how node roles are assigned in Garage 
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
 2021-11-16 16:05:53 +01:00
Alex ad7ab31411
Implement GC delay for table data 2021-11-08 15:47:47 +01:00
Alex 74a7a550eb
Safety: never voluntarily delete block in 10min interval after RC reaches zero 2021-11-08 15:47:47 +01:00
Alex 2090a6187f
Add tranquilizer mechanism to improve on token bucket mechanism 2021-11-04 13:26:59 +01:00
Alex 69b89fb46d
Fix race in block resync 2021-10-27 12:01:12 +02:00
Alex 6b47c294f5
Refactoring on repair commands 2021-10-27 11:14:55 +02:00
Trinity Pointard 28c015d9ff
add cli parameter to verify local bloc integrity 
reuse code for listing local blocks
add disk i/o speed limit on integrity check
 2021-10-27 10:31:03 +02:00
Alex 43e13a501d
Use published netapp crate instead of git repo 2021-10-26 10:36:57 +02:00
Alex ada7899b24
Fix clippy lints (fix #121) 2021-10-26 10:20:05 +02:00
Alex df8a4068d9
Refactor block manager code, and hopefully fix deadlock 2021-10-25 14:21:51 +02:00
Alex de4276202a
Improve CLI, adapt tests, update documentation 2021-10-25 14:21:48 +02:00
Alex 1b450c4b49
Improvements to CLI and various fixes for netapp version 
Discovery via consul, persist peer list to file
 2021-10-22 16:55:24 +02:00