Deuxfleurs/garage

Fork 73

Kubernetes? #241

New issue

Closed

opened 2022-02-11 01:42:52 +00:00 by schmitch · 7 comments

schmitch commented

2022-02-11 01:42:52 +00:00

hello,
I'm trying out garage and tried to make it work inside kubernetes and can't figure out my head around it, if it really works there?

most of the time kubernetes works by dns, because in k8s all ip addresses are volatile.

it looks like it's build with ip addresses in mind, is there any way to use k8s at all?

hello, I'm trying out garage and tried to make it work inside kubernetes and can't figure out my head around it, if it really works there? most of the time kubernetes works by dns, because in k8s all ip addresses are volatile. it looks like it's build with ip addresses in mind, is there any way to use k8s at all?

quentin commented

2022-02-11 14:42:15 +00:00

Owner

Hi schmitch,

At Deuxfleurs, we are not using Kubernetes but Nomad, that has way less abstractions on the network. So for sure, I don't know what would be the idiomatic way to integrate Garage with Kubernetes.

At least, if you have a Consul server on your network, you can configure your Garage nodes to register themselves in Consul and they will be able to discover their other peers.

You may also try to expose your Garage's instance ports on the server IP address (NodePort?). And then, in the bootstrap_peer section, you could put the IP address of one of your instance to trigger the autodiscovery.

Hopefully, someone more knowledgeable in Kubernetes may be able to help you in a better way. I will try to ping the people I know that operate K8S clusters.

Hi schmitch, At Deuxfleurs, we are not using Kubernetes but Nomad, that has way less abstractions on the network. So for sure, I don't know what would be the idiomatic way to integrate Garage with Kubernetes. At least, if you have a Consul server on your network, you can configure your Garage nodes to register themselves in Consul and they will be able to discover their other peers. You may also try to expose your Garage's instance ports on the server IP address (NodePort?). And then, in the bootstrap_peer section, you could put the IP address of one of your instance to trigger the autodiscovery. Hopefully, someone more knowledgeable in Kubernetes may be able to help you in a better way. I will try to ping the people I know that operate K8S clusters.

trinity-1686a commented

2022-02-11 23:59:18 +00:00

Owner

Garage supports connecting to a node using something like <hex blob>@my-domain.tld:1234. You can use that in bootstrap_peers in config, or when doing garage node connect <peer-id>.
Internally, Garage will resolve this domain as soon as it learns about it (at each restart for bootstrap_peers, when you run the command for garage node connect).
I believe if a peer changes IP (keeping the same domain or not, but keeping the same public key), and if it manages to connect to any other peer, it should be able to learn IPs of all other peers and connect to them, allowing them to learn it's new IP in return.

Garage supports connecting to a node using something like `<hex blob>@my-domain.tld:1234`. You can use that in bootstrap_peers in config, or when doing `garage node connect <peer-id>`. Internally, Garage will resolve this domain as soon as it learns about it (at each restart for bootstrap_peers, when you run the command for `garage node connect`). I believe if a peer changes IP (keeping the same domain or not, but keeping the same public key), and if it manages to connect to any other peer, it should be able to learn IPs of all other peers and connect to them, allowing them to learn it's new IP in return.

quentin added the

scope

documentation

label 2022-02-23 11:17:24 +00:00

withinboredom commented

2022-02-27 18:51:36 +00:00

Contributor

I'm trying out garage and tried to make it work inside kubernetes and can't figure out my head around it, if it really works there?

This seems like something that would live outside of k8s so if your cluster goes away you don't lose your data.

> I'm trying out garage and tried to make it work inside kubernetes and can't figure out my head around it, if it really works there? This seems like something that would live outside of k8s so if your cluster goes away you don't lose your data.

schmitch commented

2022-03-01 08:54:11 +00:00

Author

ok so from what I understand, if I correctly set bootstrap_peers to a valid DNS name and run garage node connect after a reboot it should work?

I think the biggest problem when it comes to garage than, that some processes are also hard to automate. Kubernetes is mostly a declarative system. I think for garage to work, somebody would need to write a wrapper, so that basically garage can be put into a Statefulset, which would be impossible at the moment (especially since there is no way of specifying a public key upfront.

the things looks really hard to me.
basically at the moment I think something like this would need to be automated by some wrapper inside k8s:

start garage server
wait for at least other 3 nodes to come online (the wrapper can queue k8s for that if there are other nodes inside the statefulset, i.e. k8s will apply a static dns name for each node)
generate a config with bootstrap_peers (i.e. public_key is not known upfront) and upload it maybe into a config map or something
get the config on all nodes
restart garage server on all nodes with the new config
call garage connect

thus this means the wrapper would probably also need to use k8s to have some kind of leader election.

btw. it's basically impossible to fill out rpc_public_addr upfront or keep STABLE ip addresses in k8s (some networks allow that, but most won't for a reason).

(correct me if I am wrong in the initial bootstrap process tough!)

when I'm rebooting a single node I guess I need to do the same, expect that I already have a valid config, thus I can skip step 2, 3, 4, 5 (as long as at least 1 other node is online)

(maybe I will try to look into it and make a simple wrapper to check if that works)

ok so from what I understand, if I correctly set `bootstrap_peers` to a valid DNS name and run `garage node connect` after a reboot it should work? I think the biggest problem when it comes to `garage` than, that some processes are also hard to automate. Kubernetes is mostly a declarative system. I think for garage to work, somebody would need to write a wrapper, so that basically `garage` can be put into a Statefulset, which would be impossible at the moment (especially since there is no way of specifying a public key upfront. the things looks really hard to me. basically at the moment I think something like this would need to be automated by some wrapper inside k8s: 1. start `garage server` 2. wait for at least other 3 nodes to come online (the wrapper can queue k8s for that if there are other nodes inside the statefulset, i.e. k8s will apply a static dns name for each node) 3. generate a config with bootstrap_peers (i.e. public_key is not known upfront) and upload it maybe into a config map or something 4. get the config on all nodes 5. restart `garage server` on all nodes with the new config 6. call `garage connect` thus this means the wrapper would probably also need to use k8s to have some kind of leader election. btw. it's basically impossible to fill out `rpc_public_addr` upfront or keep STABLE ip addresses in k8s (some networks allow that, but most won't for a reason). (correct me if I am wrong in the initial bootstrap process tough!) when I'm rebooting a single node I guess I need to do the same, expect that I already have a valid config, thus I can skip step 2, 3, 4, 5 (as long as at least 1 other node is online) (maybe I will try to look into it and make a simple wrapper to check if that works)

lx commented

2022-03-01 09:43:48 +00:00

Owner

ok so from what I understand, if I correctly set bootstrap_peers to a valid DNS name and run garage node connect after a reboot it should work?

You also have to put the public key of a node, something like this:

bootstrap_peers = [
	"ec79480e0ce52ae26fd00c9da684e4fa56658d9c64cdcecb094e936de0bfe71f@my.bootstrap.peer.dns.name:3901",
]

But maybe in your case something you could try is autodiscovery with Consul (I think Consul can probably run in kubernetes?).

btw. it's basically impossible to fill out rpc_public_addr upfront or keep STABLE ip addresses in k8s (some networks allow that, but most won't for a reason).

Yes it's true that's an issue. As said @withinboredom, Garage should probably be set up manually outside of your Kubernetes cluster. What we do at Deuxfleurs is run it in a Nomad cluster but using host networking, so that IP addresses of Garage nodes are the IP addresses of the machines they are running on (which don't change).

So basically I see two relatively easy solutions in your use case:

Run Garage outside of Kubernetes
Run Consul and Garage in Kubernetes, find a way for nodes to know their rpc_public_addr when they start (even if it's a new IP each time that's not a problem, just fill in the configuration dynamically), and let Garage nodes advertise themselves in the Consul catalog.

We could probably implement other autodiscovery mechanisms (maybe using the Kubernetes equivalent of the Consul catalog? I don't really know what that would be). Basically the following things would be required:

When they start, Nodes have a way of registering their public key in the catalog
There is a way to associate public keys of nodes with their IP addresses
- either the node sends the IP when it registers its public key (what we do with Consul, but it requires to be able to set rpc_public_addr)
- or there is a way to know what container is making the registration API call, and this allows us to determine its IP address)
When nodes try to discover other nodes in the network, they can read the catalog and get a list of public key + IP address pairs, one for each node registered.

If you have information on how to do this with Kubernetes, we could probably work towards a prototype.

> ok so from what I understand, if I correctly set bootstrap_peers to a valid DNS name and run garage node connect after a reboot it should work? You also have to put the public key of a node, something like this: ``` bootstrap_peers = [ "ec79480e0ce52ae26fd00c9da684e4fa56658d9c64cdcecb094e936de0bfe71f@my.bootstrap.peer.dns.name:3901", ] ``` But maybe in your case something you could try is autodiscovery with Consul (I think Consul can probably run in kubernetes?). > btw. it's basically impossible to fill out rpc_public_addr upfront or keep STABLE ip addresses in k8s (some networks allow that, but most won't for a reason). Yes it's true that's an issue. As said @withinboredom, Garage should probably be set up manually outside of your Kubernetes cluster. What we do at Deuxfleurs is run it in a Nomad cluster but using host networking, so that IP addresses of Garage nodes are the IP addresses of the machines they are running on (which don't change). So basically I see two relatively easy solutions in your use case: 1. Run Garage outside of Kubernetes 2. Run Consul and Garage in Kubernetes, find a way for nodes to know their `rpc_public_addr` when they start (even if it's a new IP each time that's not a problem, just fill in the configuration dynamically), and let Garage nodes advertise themselves in the Consul catalog. We could probably implement other autodiscovery mechanisms (maybe using the Kubernetes equivalent of the Consul catalog? I don't really know what that would be). Basically the following things would be required: 1. When they start, Nodes have a way of registering their public key in the catalog 2. There is a way to associate public keys of nodes with their IP addresses - either the node sends the IP when it registers its public key (what we do with Consul, but it requires to be able to set `rpc_public_addr`) - or there is a way to know what container is making the registration API call, and this allows us to determine its IP address) 3. When nodes try to discover other nodes in the network, they can read the catalog and get a list of public key + IP address pairs, one for each node registered. If you have information on how to do this with Kubernetes, we could probably work towards a prototype.

quentin commented

2022-03-01 10:21:50 +00:00

Owner

I searched for "Kubernetes Autodiscovery" as this is this property we want, and we could integrate it in Garage, but it does not seem to be a common idiom.

K8S often uses etcd, and we could use it as a discovery service, similarly to Consul, but again, it seems that we would break K8S' abstraction by doing so.

I also found a documentation page about Cassandra + StatefulSet. As you mentioned, the difference is that we need to pre-generate a key.

This key is part of Garage's security features and we do not want to drop it. We need to understand how people are managing cluster applications on K8S that use TLS client auth between nodes of the cluster, as we will be able to translate their pattern to Garage.

I know that K8S has some Custom Resources and/or Custom Resource Definitions but it seems to be targeted to "human management" so not designed for programmatic access. Do you know if K8S defines some ways for an app to push some data about its states?

Edit: we might need to write some codes so Garage get a ServiceAccount then we might ask Garage to register some Secrets / ConfigMaps. We may take inspiration from Akka, a java library to ease distributed/clustered application devlopment, they have a K8S section in their doc.

I searched for "Kubernetes Autodiscovery" as this is this property we want, and we could integrate it in Garage, but it does not seem to be a common idiom. K8S often uses etcd, and we could use it as a discovery service, similarly to Consul, but again, it seems that we would break K8S' abstraction by doing so. I also found a documentation page about [Cassandra + StatefulSet](https://kubernetes.io/docs/tutorials/stateful-application/cassandra/). As you mentioned, the difference is that we need to pre-generate a key. This key is part of Garage's security features and we do not want to drop it. We need to understand how people are managing cluster applications on K8S that use TLS client auth between nodes of the cluster, as we will be able to translate their pattern to Garage. I know that K8S has some [Custom Resources](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) and/or [Custom Resource Definitions](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/) but it seems to be targeted to "human management" so not designed for programmatic access. Do you know if K8S defines some ways for an app to push some data about its states? --- Edit: we might need to write some codes so Garage get a [ServiceAccount](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) then we might ask Garage to register some [Secrets](https://kubernetes.io/fr/docs/concepts/configuration/secret/) / [ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/). We may take inspiration from Akka, a java library to ease distributed/clustered application devlopment, they have a [K8S section in their doc](https://doc.akka.io/docs/akka-management/current/discovery/kubernetes.html).

schmitch commented

2022-03-01 15:52:29 +00:00

Author

btw. another good example for autodiscovery via k8s is patroni from zalando to bootstrap postgresql: https://github.com/zalando/patroni/blob/master/docs/kubernetes.rst

however their implementation might not be straight forward: https://github.com/zalando/patroni/blob/master/patroni/dcs/kubernetes.py
basically they created what somebody would need to do, to run garage on k8s at the moment. A wrapper "script" that register everything needed for service discovery and they put everything onto an endpoint, the same thing would be a little bit more work for garage since it probably needs an endpoint for each garage search and query them via k8s labels.

but kubernetes autodiscovery is something that basically everybody makes a little bit different, depending on the need.

Edit:

I will look into it if I might have a good solution also based on something somebody written inside element:

https://matrix.to/#/!BCLqSdZmZvTrJncZYQ:zinz.dev/$DZ7hOBZAPi7GfjTiybwH6U1hqK0e3a5Q2FVrdUK6V68?via=zinz.dev&via=matrix.org&via=deuxfleurs.fr

btw. another good example for autodiscovery via k8s is patroni from zalando to bootstrap postgresql: https://github.com/zalando/patroni/blob/master/docs/kubernetes.rst however their implementation might not be straight forward: https://github.com/zalando/patroni/blob/master/patroni/dcs/kubernetes.py basically they created what somebody would need to do, to run `garage` on k8s at the moment. A wrapper "script" that register everything needed for service discovery and they put everything onto an endpoint, the same thing would be a little bit more work for garage since it probably needs an endpoint for each garage search and query them via k8s labels. but kubernetes autodiscovery is something that basically everybody makes a little bit different, depending on the need. Edit: I will look into it if I might have a good solution also based on something somebody written inside element: https://matrix.to/#/!BCLqSdZmZvTrJncZYQ:zinz.dev/$DZ7hOBZAPi7GfjTiybwH6U1hqK0e3a5Q2FVrdUK6V68?via=zinz.dev&via=matrix.org&via=deuxfleurs.fr

quentin referenced this issue from a pull request that will close it,

2022-03-07 16:49:54 +00:00

add support for kubernetes service discovery #262