Declarative bucket and key configuration #450

New issue

Open

opened 2022-12-14 08:00:19 +00:00 by pinpox · 3 comments

pinpox commented

2022-12-14 08:00:19 +00:00

This would make migration to a new machine easier and also guarantee a certain a state of the server to always be present.

From a user's perspective you would define everything once and have the complete
"schema" with all your buckets, keys and so on in a file that you can backup and use to replicate the setup on a new server.

It would be great if there was a way to specify all buckets, keys, layouts and other configuration declaratively, i.e. as a configuration file instead of having to use the cli to setup everything after installation. The setup would probably have to be inmutable in this case. This would make migration to a new machine easier and also guarantee a certain a state of the server to always be present. From a user's perspective you would define everything once and have the complete "schema" with all your buckets, keys and so on in a file that you can backup and use to replicate the setup on a new server.

👍 2

lx added the

kind

improvement

label 2022-12-14 09:50:52 +00:00

jpds commented

2023-01-02 14:35:02 +00:00

Contributor

You can try using the Terraform provider at:

You can try using the Terraform provider at: - https://registry.terraform.io/providers/prologin/garage/ - https://gitlab.com/prologin/tech/infra/terraform/terraform-provider-garage

kfirfer commented

2023-12-16 17:06:39 +00:00

relating to:
#678

but im seeking how to configure layout , but i want to make it 100% automate so the problem the ID's i cant know beforehand

relating to: https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/678 but im seeking how to configure layout , but i want to make it 100% automate so the problem the ID's i cant know beforehand

max commented

2024-05-24 17:15:57 +00:00

For reference, I wrote a little thing to achieve this with Nix, Consul, and a bunch of shell scripts.

https://github.com/privatevoid-net/depot/blob/master/cluster/services/storage/garage-options.nix

This allows you to declaratively define the layout:
https://github.com/privatevoid-net/depot/blob/master/cluster/services/storage/garage-layout.nix
As well as buckets, keys and permissions:
https://github.com/privatevoid-net/depot/blob/master/cluster/services/monitoring/default.nix#L77-L86

The layout is configured by hostname, so public keys don't need to be known beforehand. There is no boostrapping step (except for node discovery, which Garage can already do automatically with Consul).

It's a bit hacky (e.g. shouldn't have funny characters in bucket names, bucket aliases aren'y supported, hostname matching may be flawed), but it works pretty well as it is already. Maybe it can be used as a reference for how to implement this in Garage directly. Alternatively, having machine-readable CLI output would make it easier to create third-party declarative config tools.

For reference, I wrote a little thing to achieve this with Nix, Consul, and a bunch of shell scripts. https://github.com/privatevoid-net/depot/blob/master/cluster/services/storage/garage-options.nix This allows you to declaratively define the layout: https://github.com/privatevoid-net/depot/blob/master/cluster/services/storage/garage-layout.nix As well as buckets, keys and permissions: https://github.com/privatevoid-net/depot/blob/master/cluster/services/monitoring/default.nix#L77-L86 The layout is configured by hostname, so public keys don't need to be known beforehand. There is no boostrapping step (except for node discovery, which Garage can already do automatically with Consul). It's a bit hacky (e.g. shouldn't have funny characters in bucket names, bucket aliases aren'y supported, hostname matching may be flawed), but it works pretty well as it is already. Maybe it can be used as a reference for how to implement this in Garage directly. Alternatively, having machine-readable CLI output would make it easier to create third-party declarative config tools.

❤️ 1