Garage on k3s with # ssds < # nodes #913

New issue

Closed

opened 2024-12-25 19:46:58 +00:00 by magsol · 4 comments

magsol commented

2024-12-25 19:46:58 +00:00

Hi all, I was recommended garage by the Mastodon team as a potential storage option for my k3s cluster (6x Raspberry Pi 4B+). I'm looking to move off a single NFS (ie, single point of failure) to an SSD-backed physical storage medium for my kubernetes cluster. Right now, I have 2x 2TB SSDs, each one plugged directly into two of the six Raspberry Pis in the cluster.

My question: can garage be configured to work with this topology: 6 nodes, but only 2 of which have available storage?

Thanks in advance.

maximilien commented

2024-12-26 03:45:27 +00:00

Owner

Hello,

Thanks for considering Garage.
A garage cluster can have "gateway" nodes that will not store data. But usually three data nodes is what we advice, and you seem to only have two. Are those six raspberries hosted in a single geographic location?

Hello, Thanks for considering Garage. A garage cluster can have "gateway" nodes that will not store data. But usually three data nodes is what we advice, and you seem to only have two. Are those six raspberries hosted in a single geographic location?

magsol commented

2024-12-26 20:30:32 +00:00

Author

Yes, it’s a homelab setup so all 6 SBCs are stacked on a small rack in my home office. I could potentially get another disk or two if 3 replicates is a highly recommended configuration; I had been planning to set the replication factor to 2 in my case but I’ll happily follow your guidelines.

But it sounds like there is plenty of allowance in the design of garage for nodes without attached storage?

magsol commented

2025-01-06 18:51:17 +00:00

Author

If not, or even if so but not recommended, what would a recommended configuration for this particular setup be (e.g., simply add a third SSD? something more sophisticated? or maybe Garage isn't the best solution, check out longhorn)?

Is this an acceptable configuration for Garage (running on a k3s kubernetes cluster of 6 Raspberry Pi 4Bs, only 2 of which have attached storage, so Garage would be configured with a replication factor of 2 instead of 3 and with the four non-storage nodes in "gateway" mode)? If not, or even if so but not recommended, what would a recommended configuration for this particular setup be (e.g., simply add a third SSD? something more sophisticated? or maybe Garage isn't the best solution, check out longhorn)?

maximilien commented

2025-01-06 21:43:27 +00:00

Owner

Replication set at 2 and 4 gateway would work, although it would not be ideal because if you want strong consistency, you'll need both nodes to be up, hence the recommandation to have at least 3 replicas, so you can take one node down without affecting reads or write consistency. This setup is definitely not where garage shine the most (it was built to be geodistributed), but there is nothing wrong with it.