diff --git a/_posts/2021-09-23-quick-tour-of-garage.md b/_posts/2021-09-23-quick-tour-of-garage.md new file mode 100644 index 0000000..8ce068b --- /dev/null +++ b/_posts/2021-09-23-quick-tour-of-garage.md @@ -0,0 +1,496 @@ +--- +layout: post +slug: a-quick-tour-of-garage +status: published +sitemap: true +title: A quick tour of Garage +description: Garage is a new lightweight and versatile object storage platform implementing the S3 API, let's quickly deploy it. +category: operation +tags: +--- + + +[Garage](https://garagehq.deuxfleurs.fr) is an object storage platform that can be used as a drop-in replacement for AWS S3. +It is designed to run on bare metal hardware and has no dependency on any cloud provider. +In this article, I quickly deploy a geo-distributed Garage cluster and use it as a backend for Nextcloud. + +I chose to use a cloud platform (Scaleway) to easily and quickly spawn geo-distributed machines. +To abstract our machines deployment, I wrote a small tool named [nuage](https://git.deuxfleurs.fr/quentin/nuage). + +You will need a working account on [Scaleway](https://console.scaleway.com). +Then, we will need to install some tools on our machine (be sure to have [go](https://golang.org) installed). + +Let's install Scaleway's CLI tool: + +```bash +sudo curl -o /usr/local/bin/scw -L "https://github.com/scaleway/scaleway-cli/releases/download/v2.3.1/scw-2.3.1-linux-x86_64" +sudo chmod +x /usr/local/bin/scw +scw init # enter your scaleway credentials +``` + +And my helper to easily deploy instances on Scaleway: + +```bash +go install git.deuxfleurs.fr/quentin/nuage@latest +export PATH="$PATH:$HOME/go/bin" +nuage # display how to use the tool +``` + +Now, we are ready to spawn some machines! + +## Spawn machines + +We start by creating our `nuage` inventory in a file named `garage_inventory.txt`: + +``` +fr-par-1 dev1-s debian_bullseye garage-fr-1 +fr-par-1 dev1-s debian_bullseye garage-fr-2 +pl-waw-1 dev1-s debian_bullseye garage-pl-1 +pl-waw-1 dev1-s debian_bullseye garage-pl-2 +nl-ams-1 dev1-s debian_bullseye garage-nl-1 +nl-ams-1 dev1-s debian_bullseye garage-nl-2 +``` + +Then let's pass it to `nuage`: + +```bash +nuage spawn < ./garage_inventory.txt +``` + +`nuage` will spawn 6 machines: + - 2 machines in Paris, France + - 2 machines in Warsaw, Poland + - 2 machines in Amsterdam, Netherlands + +All instances will run debian on Scaleway's cheap [dev1-s](https://www.scaleway.com/en/pricing/#development-instances) instances. +For the naming of our instances, we built it following this pattern: `garage--`. + +Now, we suppose that the instances are started and your able to login on them, for example: + +``` +ssh root@51.15.227.63 +``` + +## Some crypto + +Our garage instances will communicate together securely through TLS. +We need to generate some certificates locally that we will deploy on the remote instances later. +To ease the operation, we provide a small handler named `genkeys.sh` to generate all the needed keys: + +``` +wget https://git.deuxfleurs.fr/Deuxfleurs/garage/raw/tag/v0.3.0/genkeys.sh +chmod +x ./genkeys.sh +./genkeys.sh +``` + +Now you should have a folder named `pki` containing both the key and the certificate for your CA and your end-entity. +Ideally, each node would have its own end-entity certificate but to simplify the configuration, we will use only once in this tour. + +Let's create a script to deploy our pki: + +```bash +cat > deploy_pki.sh < /etc/garage/pki/garage-ca.crt < /etc/garage/pki/garage.crt < /etc/garage/pki/garage.key < /etc/garage/config.toml < /usr/local/bin/garagectl < /etc/systemd/system/garage.service < /etc/apache2/sites-available/nextcloud.conf < + Require all granted + AllowOverride All + Options FollowSymLinks MultiViews + + + Dav off + + +EOF +a2ensite nextcloud.conf +a2enmod rewrite +a2enmod headers +a2enmod env +a2enmod dir +a2enmod mime +systemctl restart apache2 +chown -R www-data:www-data /var/www/nextcloud/ +``` + +Then deploy it: + +``` +nuage run ./deploy_nextcloud.sh < ./nextcloud-inventory.txt +``` + +Then open in your browser Nextcloud, for me http://212.47.230.180/nextcloud. +Finish the installation by providing requested information. + +Now we will configure Nextcloud to use Garage as its primary object storage. +You can also [read its documentation](https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/primary_storage.html). + +First, we need to create the bucket and the key (and will also allow our own key): + +```bash +garagectl bucket create nextcloud +garagectl key new --name nextcloud +garagectl bucket allow nextcloud --read --write GK872ebec80feae4ad663e82ec +garagectl bucket allow nextcloud --read GKfd49e3906e5d2e3e23ee07f9 +``` + +We will SSH on the server and edit `config.php` + +```bash +ssh root@212.47.230.180 +vim /var/www/nextcloud/config/config.php +``` + +and add: + +```php + [ + 'class' => '\\OC\\Files\\ObjectStore\\S3', + 'arguments' => [ + 'bucket' => 'nextcloud', + 'autocreate' => false, + 'key' => 'GK872ebec80feae4ad663e82ec', + 'secret' => 'xxxxxxxxxxxxx', + 'hostname' => '51.158.182.206', + 'port' => 3900, + 'use_ssl' => false, + 'region' => 'garage', + // required for some non Amazon S3 implementations + 'use_path_style' => true + ], +], +``` + +*If you have some errors after reloading the page, run `tail -f /var/www/nextcloud/media/nextcloud.log`* + +Primary storage is only one way to integrate Garage in Nextcloud, it is also possible to integrate it through the "External storage" plugin. +This method is not covered here but you can refer to [Nextcloud's documentation](https://docs.nextcloud.com/server/latest/admin_manual/configuration_files/external_storage_configuration_gui.html). + +After uploading a file, you can see how nextcloud store them internally through awscli: + +``` +aws s3 ls nextcloud +``` + +Our current deployment has some drawbacks: we have a single point of failure with only one server and data are not sent encrypted. +One solution is to deploy garage on our server, locally, as a gateway. +First, we install it normally: + +```bash +nuage run ./deploy_pki.sh < ./nextcloud-inventory.txt +nuage run ./deploy_conf.sh < ./nextcloud-inventory.txt +nuage run ./deploy_bin.sh < ./nextcloud-inventory.txt +``` + +Then, we configure our new node as a gateway because we do not want to store data on it, we just want to use it to route data: + +```bash +garagectl status +garagectl node configure -z fr -g 2b145f7b4c15c2a4 +``` + +Then we edit Nextcloud's configuration `/var/www/nextcloud/config/config.php` to just change the hostname: + +```php + [ + 'class' => '\\OC\\Files\\ObjectStore\\S3', + 'arguments' => [ + /* other arguments */ + 'hostname' => '127.0.0.1', + /* other arguments */ + ], +] +``` + +Now we have a high availability backend as our local gateway will try to route our request to available servers. + +## Handle crashes + +Start by choosing a node you want to crash: + +```bash +nuage spawn < ./garage-inventory.txt +``` + +SSH on it, we will simulate its failure by just stopping it (there is no difference between graceful shutdown and crashes): + +```bash +ssh root@151.115.34.19 +systemctl stop garage +``` + +Connect on another node and note that the node is unavailable: + +```bash +ssh root@51.15.59.148 +garagectl status +``` + +For now, no re-balancing has been triggered. +Garage allows for transient failures. +If you want to re-balance, you have to explicitly remove the node from Garage. + +Now, let's assume this is only a transient failure, and let's restart it: + +```bash +ssh root@151.115.34.19 +systemctl start garage +journalctl -fu garage +``` + +Note how the repair is automatically triggered. +You can still manually trigger a repair if you want: + +```bash +garagectl repair --yes +``` + +Now let's assume that the machine burnt and all its disks are losts: + +```bash +systemctl stop garage +rm -r /var/lib/garage/ +systemctl start garage +garagectl status +``` + +Now our node is seen as a new one and its old ID is seen as failed. +We will replace the old node with this new one with a simple command: + +``` +garagectl node configure --replace 212027752f40c4d4 -c 1 -z pl 375690c499627ea8 +garagectl status +``` + +We do not cover this part, but you can also add or remove nodes at any time and trigger a re-balance. + +## Destroy our VM + +When you're done with this tour, just destroy the resources you created: + +```bash +nuage destroy < ./garage-inventory.txt +nuage destroy < ./nextcloud-inventory.txt +``` + +Thanks a lot, this is the end of my tour of Garage, see you next time :)