KokaKiwi/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues Pull requests Projects Releases Wiki Activity
701 commits 13 branches 1 tag 80 MiB
Commit graph

82 commits

Author SHA1 Message Date
Alex 9d0ed78887 Add feature flag for Kubernetes discovery 2022-03-24 16:57:43 +01:00
Alex e480aaf338
Make background tranquility a configurable parameter 2022-03-23 10:25:19 +01:00
Alex db46cdef79
Update netapp to v0.4.1 2022-03-15 17:09:57 +01:00
Alex fe62d01b7e
Implement exponential backoff for resync retries 2022-03-14 11:41:20 +01:00
Alex 9b2b531f4d
Make admin server optional 2022-03-14 10:54:25 +01:00
Alex 2377a92f6b
Add wrapper over sled tree to count items (used for big queues) 2022-03-14 10:54:25 +01:00
Alex 203e8d2c34
Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
Alex dc8d0496cc
Refactoring: rename config files, make modifications less invasive 2022-03-14 10:53:51 +01:00
Alex d9a35359bf
Add metrics to web endpoint 2022-03-14 10:53:50 +01:00
Alex 2a5609b292
Add metrics to API endpoint 2022-03-14 10:53:36 +01:00
Alex 818daa5c78
Refactor how durations are measured 2022-03-14 10:53:35 +01:00
Alex 55d4471599
Remove ... at end of hex IDs 2022-03-14 10:52:31 +01:00
Alex bb04d94fa9
Update to Netapp 0.4 which supports distributed tracing 2022-03-14 10:52:30 +01:00
Alex 8c2fb0c066
Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
mricher e349af13a7
Update dependencies and add admin module with metrics 
- Global dependencies updated in Cargo.lock
- New module created in src/admin to host:
  - the (future) admin REST API
  - the metric collection
- add configuration block

No metrics implemented yet
 2022-03-14 10:51:12 +01:00
Max Audron 9d44127245
add support for kubernetes service discovery 
This commit adds support to discover garage instances running in
kubernetes.

Once enabled by setting `kubernetes_namespace` and
`kubernetes_service_name` garage will create a Custom Resources
`garagenodes.deuxfleurs.fr` with nodes public key as the resource name.
and IP and Port information as spec in the namespace configured by
`kubernetes_namespace`.

For discovering nodes the resources are filtered with the optionally set
`kubernetes_service_name` which sets a label
`garage.deuxfleurs.fr/service` on the resources.

This allows to separate multiple garage deployments in a single
namespace.

the `kubernetes_skip_crd` variable allows to disable the creation of the
CRD by garage itself. The user must deploy this manually.
 2022-03-12 13:05:52 +01:00
Alex ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general 
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
 2022-01-24 11:58:00 +01:00
Alex d4dd2e2640
Make use of website config, return error document on error 2022-01-13 14:25:19 +01:00
Alex 1bcd6fabbd
New buckets for 0.6.0: small changes 
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
 2022-01-04 12:52:47 +01:00
Alex beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
Alex d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
Alex c7d5c73244
Add must_use to some CRDT functions 2022-01-04 12:47:28 +01:00
Alex 87121dce9d
New buckets for 0.6.0: documentation and build files 2022-01-04 12:47:06 +01:00
Alex b1cfd16913
New buckets for 0.6.0: small fixes, including: 
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
 2022-01-04 12:46:41 +01:00
Alex 0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
Alex 5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
trinity-1686a 1eb972b1ac Add compression using zstd (#173) 
fix #27

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#173
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-12-15 11:26:43 +01:00
Alex c94406f428
Improve how node roles are assigned in Garage 
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
 2021-11-16 16:05:53 +01:00
Trinity Pointard 9c58ec28d3 add support for vhost-style s3 bucket 2021-11-16 15:41:41 +01:00
Trinity Pointard 9d7535c3f5 allow missing bootstrap_peers in garage.toml 2021-11-05 16:36:25 +01:00
Alex 2090a6187f
Add tranquilizer mechanism to improve on token bucket mechanism 2021-11-04 13:26:59 +01:00
Alex 6f13d083ab
Add semaphore to limit RAM used by buffered outgoing requests 2021-11-03 18:02:57 +01:00
Alex 6b47c294f5
Refactoring on repair commands 2021-10-27 11:14:55 +02:00
Alex 43e13a501d
Use published netapp crate instead of git repo 2021-10-26 10:36:57 +02:00
Alex de4276202a
Improve CLI, adapt tests, update documentation 2021-10-25 14:21:48 +02:00
Alex 1b450c4b49
Improvements to CLI and various fixes for netapp version 
Discovery via consul, persist peer list to file
 2021-10-22 16:55:24 +02:00
Alex 4067797d01
First port of Garage to Netapp 2021-10-22 15:55:18 +02:00
Alex b9127dd6f8
Prepare for v0.3.0 and add migration path from v0.2.1.x 2021-05-28 15:29:58 +02:00
Alex b490ebc7f6
Many improvements on ring/replication and its configuration: 
- Explicit "replication_mode" configuration parameters that takes
  either "none", "2" or "3" as values, instead of letting user configure
  replication factor themselves. These are presets whose corresponding
  replication/quorum values can be found in replication/mode.rs

- Explicit support for single-node and two-node deployments
  (number of nodes must be at least "replication_mode", with "none"
  we can have only one node)

- Ring is now stored much more compactly with 256*8 + n*32 bytes,
  instead of 256*32 bytes

- Support for gateway-only nodes that do not store data
  (these nodes still need a metadata_directory to store the list
  of bucket and keys since those are stored on all nodes; it also
  technically needs a data_directory to start but it will stay
  empty unless we have bugs)
 2021-05-28 14:07:36 +02:00
Alex 6ccffc3162
Improved XML serialization 
- Use quick_xml and serde for all XML response returned by the S3 API.
- Include tests for all structs used to generate XML
- Remove old manual XML escaping function which was unsafe
 2021-05-06 22:37:15 +02:00
Trinity Pointard e4b9e4e24d
rename types to CamelCase 2021-05-03 22:15:09 +02:00
Trinity Pointard f05bb111c2
fix clippy warnings on util and rpc 2021-05-03 22:11:41 +02:00
Alex 575726358c
Tune Sled configuration 
- Make sled cache size and flush interval configurable
- Set less agressive default values:
  - cache size 128MB instead of 1GB
  - Flush interval 2 seconds instead of .5 seconds
 2021-05-03 17:27:43 +02:00
Trinity Pointard 74373aebcf
make most requested changes 2021-04-27 16:47:08 +02:00
Trinity Pointard f871689571
run cargo fmt on util and make missing doc warning 2021-04-27 16:37:10 +02:00
Trinity Pointard f9bd2d8fb7
document util crate 2021-04-27 16:37:10 +02:00
Alex 64b91c2645
Keep old data 2021-04-27 16:37:09 +02:00
Alex 9ced9f78dc
Improve bootstraping: do it regularly; persist peer list 2021-04-27 16:37:08 +02:00
Alex f859d15062 update to v0.2.1 2021-03-19 13:39:18 +01:00
Alex fd8f4caa81 Support old CPUs 2021-03-19 12:19:40 +01:00