Steam/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues Pull requests Projects Releases Wiki Activity
729 commits 14 branches 0 tags 70 MiB
Commit graph

86 commits

Author SHA1 Message Date
Alex Auvolat
617f28bfa4
Correct small formatting issue 2022-05-05 14:21:57 +02:00
Mendes
948ff93cf1 Corrected the warnings and errors issued by cargo clippy 2022-05-01 16:05:39 +02:00
Alex Auvolat
2aeaddd5e2
Apply cargo fmt 2022-05-01 09:57:05 +02:00
Alex Auvolat
c1d1646c4d
Change the way new layout assignations are computed. 
The function now computes an optimal assignation (with respect to partition size) that minimizes the distance to the former assignation, using flow algorithms.

This commit was written by Mendes Oulamara <mendes.oulamara@pm.me>
 2022-05-01 09:54:19 +02:00
Alex Auvolat
94f1e48fff Update to netapp 0.4.2 (a tiny fix) 2022-04-07 11:50:03 +02:00
Alex Auvolat
9d0ed78887 Add feature flag for Kubernetes discovery 2022-03-24 16:57:43 +01:00
Alex Auvolat
509d256c58
Make layout optimization work in relative terms 2022-03-24 15:27:14 +01:00
Alex Auvolat
7e0e2ffda2
Slight change and add comment to layout assignation algo 2022-03-24 15:27:13 +01:00
Alex Auvolat
413ab0eaed
Small change to partition assignation algorithm 
This change helps ensure that nodes for each partition are spread
over all datacenters, a property that wasn't ensured previously
when going from a 2 DC deployment to a 3 DC deployment
 2022-03-24 15:27:10 +01:00
Alex Auvolat
db46cdef79
Update netapp to v0.4.1 2022-03-15 17:09:57 +01:00
Alex Auvolat
ba6b56ae68
Fix some new clippy lints 2022-03-14 12:27:49 +01:00
Alex Auvolat
2377a92f6b
Add wrapper over sled tree to count items (used for big queues) 2022-03-14 10:54:25 +01:00
Alex Auvolat
203e8d2c34
Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
Alex Auvolat
f869ca625d
Add spans to table calls, change span names in RPC 2022-03-14 10:54:12 +01:00
Alex Auvolat
0cc31ee169
add missing netapp telemetry feature 2022-03-14 10:54:11 +01:00
Alex Auvolat
dc8d0496cc
Refactoring: rename config files, make modifications less invasive 2022-03-14 10:53:51 +01:00
Alex Auvolat
2a5609b292
Add metrics to API endpoint 2022-03-14 10:53:36 +01:00
Alex Auvolat
818daa5c78
Refactor how durations are measured 2022-03-14 10:53:35 +01:00
Alex Auvolat
bb04d94fa9
Update to Netapp 0.4 which supports distributed tracing 2022-03-14 10:52:30 +01:00
Alex Auvolat
8c2fb0c066
Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
Alex Auvolat
2cab84b1fe
Add many metrics in table/ and rpc/ 2022-03-14 10:51:50 +01:00
Max Audron
9d44127245
add support for kubernetes service discovery 
This commit adds support to discover garage instances running in
kubernetes.

Once enabled by setting `kubernetes_namespace` and
`kubernetes_service_name` garage will create a Custom Resources
`garagenodes.deuxfleurs.fr` with nodes public key as the resource name.
and IP and Port information as spec in the namespace configured by
`kubernetes_namespace`.

For discovering nodes the resources are filtered with the optionally set
`kubernetes_service_name` which sets a label
`garage.deuxfleurs.fr/service` on the resources.

This allows to separate multiple garage deployments in a single
namespace.

the `kubernetes_skip_crd` variable allows to disable the creation of the
CRD by garage itself. The user must deploy this manually.
 2022-03-12 13:05:52 +01:00
Alex Auvolat
beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
Alex Auvolat
5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
Alex Auvolat
c94406f428
Improve how node roles are assigned in Garage 
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
 2021-11-16 16:05:53 +01:00
Alex Auvolat
e8811f7c9d
Request strategy: don't launch all 3 requests if not needed 2021-11-04 16:19:27 +01:00
Alex Auvolat
6f13d083ab
Add semaphore to limit RAM used by buffered outgoing requests 2021-11-03 18:02:57 +01:00
Alex Auvolat
8c4f418fe8
Fix peer list persistence: do not forget previous peers 2021-11-03 17:34:44 +01:00
Alex Auvolat
43e13a501d
Use published netapp crate instead of git repo 2021-10-26 10:36:57 +02:00
Alex Auvolat
ada7899b24
Fix clippy lints (fix #121) 2021-10-26 10:20:05 +02:00
Alex Auvolat
de4276202a
Improve CLI, adapt tests, update documentation 2021-10-25 14:21:48 +02:00
Alex Auvolat
1b450c4b49
Improvements to CLI and various fixes for netapp version 
Discovery via consul, persist peer list to file
 2021-10-22 16:55:24 +02:00
Alex Auvolat
4067797d01
First port of Garage to Netapp 2021-10-22 15:55:18 +02:00
Alex Auvolat
fa394dcd27
Support pkcs8 private keys (allowing for ed25519 to be used for rpc) 2021-07-06 11:16:01 +02:00
trinity-1686a
30a7dee920 exit when inconsistent level of replication is detected (#92) 
fix #88

Authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#92
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2021-06-02 13:30:39 +02:00
Trinity Pointard
289521886b make most changes suggested during install-party 2021-05-29 21:37:49 +02:00
Alex Auvolat
b9127dd6f8
Prepare for v0.3.0 and add migration path from v0.2.1.x 2021-05-28 15:29:58 +02:00
Alex Auvolat
ddb2b29bfd
Rename datacenters into zones (doc not yet updated) 2021-05-28 14:07:36 +02:00
Alex Auvolat
b490ebc7f6
Many improvements on ring/replication and its configuration: 
- Explicit "replication_mode" configuration parameters that takes
  either "none", "2" or "3" as values, instead of letting user configure
  replication factor themselves. These are presets whose corresponding
  replication/quorum values can be found in replication/mode.rs

- Explicit support for single-node and two-node deployments
  (number of nodes must be at least "replication_mode", with "none"
  we can have only one node)

- Ring is now stored much more compactly with 256*8 + n*32 bytes,
  instead of 256*32 bytes

- Support for gateway-only nodes that do not store data
  (these nodes still need a metadata_directory to store the list
  of bucket and keys since those are stored on all nodes; it also
  technically needs a data_directory to start but it will stay
  empty unless we have bugs)
 2021-05-28 14:07:36 +02:00
Trinity Pointard
e4b9e4e24d
rename types to CamelCase 2021-05-03 22:15:09 +02:00
Trinity Pointard
f05bb111c2
fix clippy warnings on util and rpc 2021-05-03 22:11:41 +02:00
Trinity Pointard
2812a027ea
change some more comments and revert changes on TableSchema 2021-04-27 16:49:07 +02:00
Trinity Pointard
74373aebcf
make most requested changes 2021-04-27 16:47:08 +02:00
Alex Auvolat
16300bbd89
remove useless comment 2021-04-27 16:44:01 +02:00
Trinity Pointard
f871689571
run cargo fmt on util and make missing doc warning 2021-04-27 16:37:10 +02:00
Trinity Pointard
8e0524ae15
document rpc crate 2021-04-27 16:37:10 +02:00
Alex Auvolat
6b2b400292
small simplify 2021-04-27 16:37:09 +02:00
Alex Auvolat
8c33d565d6
Merge discovery loop with consul 2021-04-27 16:37:09 +02:00
Alex Auvolat
948e44a3f6
cargo fmt 2021-04-27 16:37:09 +02:00
Alex Auvolat
3e2e38c830
Print stats 2021-04-27 16:37:09 +02:00