add support for kubernetes service discovery #262

audron · 2022-03-06T14:19:10Z

audron commented

2022-03-06 14:19:10 +00:00

This commit adds support to discover garage instances running in
kubernetes.

Once enabled by setting kubernetes_namespace and
kubernetes_service_name garage will create a Custom Resources
garagenodes.deuxfleurs.fr with nodes public key as the resource name.
and IP and Port information as spec in the namespace configured by
kubernetes_namespace.

For discovering nodes the resources are filtered with the optionally set
kubernetes_service_name which sets a label
garage.deuxfleurs.fr/service on the resources.

This allows to separate multiple garage deployments in a single
namespace.

the kubernetes_skip_crd variable allows to disable the creation of the
CRD by garage itself. The user must deploy this manually.

The new dependencies on kube and pnet i had to pin on an older version
because you are currently using rust 1.55 for building, which is too old
to support rust edition 2021, and these projects got bumped to edition
2021 in newer versions. They can be updated once the rust version was updated.

This commit adds support to discover garage instances running in kubernetes. Once enabled by setting `kubernetes_namespace` and `kubernetes_service_name` garage will create a Custom Resources `garagenodes.deuxfleurs.fr` with nodes public key as the resource name. and IP and Port information as spec in the namespace configured by `kubernetes_namespace`. For discovering nodes the resources are filtered with the optionally set `kubernetes_service_name` which sets a label `garage.deuxfleurs.fr/service` on the resources. This allows to separate multiple garage deployments in a single namespace. the `kubernetes_skip_crd` variable allows to disable the creation of the CRD by garage itself. The user must deploy this manually. The new dependencies on `kube` and `pnet` i had to pin on an older version because you are currently using rust 1.55 for building, which is too old to support rust edition 2021, and these projects got bumped to edition 2021 in newer versions. They can be updated once the rust version was updated.

👍 3

lx requested changes 2022-03-07 10:24:47 +00:00

lx left a comment

Thanks for the great work! Remember to run cargo fmt to reformat your code according to standard style guidelines, and to check that cargo clippy does not return any warning. Will merge when this + the two little comments below are addressed.

Thanks for the great work! Remember to run `cargo fmt` to reformat your code according to standard style guidelines, and to check that `cargo clippy` does not return any warning. Will merge when this + the two little comments below are addressed.

src/rpc/system.rs Outdated

					
				@ -298,0 +326,4 @@

									.ips

									.first()

									.unwrap()

									.ip();

lx commented

2022-03-07 10:21:48 +00:00

This should be put in a separate utility function to be also used in the function that advertises to Consul, it would be really helpful!

audron marked this conversation as resolved

src/rpc/system.rs Outdated

					
				@ -298,0 +328,4 @@

									.unwrap()

									.ip();

								SocketAddr::new(default_ip, 3901)

lx commented

2022-03-07 10:20:59 +00:00

The port isn't always 3901, it should be derived from the .port() of the RPC bind addr that is set in the configuration file.

The port isn't always 3901, it should be derived from the `.port()` of the RPC bind addr that is set in the configuration file.

audron marked this conversation as resolved

trinity-1686a reviewed 2022-03-07 15:07:46 +00:00

src/rpc/Cargo.toml Outdated

					
				@ -33,0 +33,4 @@

				# rust edition 2021

				kube = { version = "0.62", features = ["runtime", "derive"] }

				k8s-openapi = { version = "0.13", features = ["v1_22"] }

				schemars = "0.8"

trinity-1686a commented

2022-03-07 15:07:46 +00:00

could you add the following line please?

openssl = { version = "0.10", features = ["vendored"] }

Garage tries to be cross-compilation friendly, and by default kube seems to use system openssl, which poses problem. Adding this feature flag makes compilation a bit slower as it requires compiling openssl, but makes cross-compilation possible.

could you add the following line please? ```toml openssl = { version = "0.10", features = ["vendored"] } ``` Garage tries to be cross-compilation friendly, and by default kube seems to use system openssl, which poses problem. Adding this feature flag makes compilation a bit slower as it requires compiling openssl, but makes cross-compilation possible.

audron commented

2022-03-12 11:40:10 +00:00

kube also has rustls support, but i couldn't make it work for whatever reason rn.
I added your suggestion.

kube also has rustls support, but i couldn't make it work for whatever reason rn. I added your suggestion.

audron marked this conversation as resolved