Deuxfleurs/garage
48
56
Fork 73
Code Issues 99 Pull requests 13 Releases 30 Wiki Activity

Merge pull request 'Add a mdbook documentation to present garage and help user on-boarding' (#45) from feature/mdbook into master
All checks were successful
continuous-integration/drone/push Build is passing
Details

Browse source 
Reviewed-on: #45
This commit is contained in:
Alex 2021-03-18 10:39:55 +01:00
commit 5b659b28ce
25 changed files with 784 additions and 257 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

117
README.md
View file

@ -18,119 +18,4 @@ Non-goals include:
Our main use case is to provide a distributed storage layer for small-scale self hosted services such as [Deuxfleurs](https://deuxfleurs.fr).
Check our [compatibility page](doc/Compatibility.md) to view details of the S3 API compatibility.
## Development
We propose the following quickstart to setup a full dev. environment as quickly as possible:
1. Setup a rust/cargo environment. eg. `dnf install rust cargo`
2. Install awscli v2 by following the guide [here](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html).
3. Run `cargo build` to build the project
4. Run `./script/dev-cluster.sh` to launch a test cluster (feel free to read the script)
5. Run `./script/dev-configure.sh` to configure your test cluster with default values (same datacenter, 100 tokens)
6. Run `./script/dev-bucket.sh` to create a bucket named `eprouvette` and an API key that will be stored in `/tmp/garage.s3`
7. Run `source ./script/dev-env-aws.sh` to configure your CLI environment
8. You can use `garage` to manage the cluster. Try `garage --help`.
9. You can use the `awsgrg` alias to add, remove, and delete files. Try `awsgrg help`, `awsgrg cp /proc/cpuinfo s3://eprouvette/cpuinfo.txt`, or `awsgrg ls s3://eprouvette`. `awsgrg` is a wrapper on the `aws s3` command pre-configured with the previously generated API key (the one in `/tmp/garage.s3`) and localhost as the endpoint.
Now you should be ready to start hacking on garage!
## S3 compatibility
Only a subset of S3 is supported: adding, listing, getting and deleting files in a bucket.
Bucket management, ACL and other advanced features are not (yet?) handled through the S3 API but through the `garage` CLI.
We primarily test `garage` against the `awscli` tool and `nextcloud`.
## Setting up Garage
Use the `genkeys.sh` script to generate TLS keys for encrypting communications between Garage nodes.
The script takes no arguments and will generate keys in `pki/`.
This script creates a certificate authority `garage-ca` which signs certificates for individual Garage nodes.
Garage nodes from a same cluster authenticate themselves by verifying that they have certificates signed by the same certificate authority.
Garage requires two locations to store its data: a metadata directory, and a data directory.
The metadata directory is used to store metadata such as object lists, and should ideally be located on an SSD drive.
The data directory is used to store the chunks of data of the objects stored in Garage.
In a typical deployment the data directory is stored on a standard HDD.
Garage does not handle TLS for its S3 API endpoint. This should be handled by adding a reverse proxy.
Create a configuration file with the following structure:
```
block_size = 1048576 # objects are split in blocks of maximum this number of bytes
metadata_dir = "/path/to/ssd/metadata/directory"
data_dir = "/path/to/hdd/data/directory"
rpc_bind_addr = "[::]:3901" # the port other Garage nodes will use to talk to this node
bootstrap_peers = [
# Ideally this list should contain the IP addresses of all other Garage nodes of the cluster.
# Use Ansible or any kind of configuration templating to generate this automatically.
"10.0.0.1:3901",
"10.0.0.2:3901",
"10.0.0.3:3901",
]
# optionnal: garage can find cluster nodes automatically using a Consul server
# garage only does lookup but does not register itself, registration should be handled externally by e.g. Nomad
consul_host = "localhost:8500" # optionnal: host name of a Consul server for automatic peer discovery
consul_service_name = "garage" # optionnal: service name to look up on Consul
max_concurrent_rpc_requests = 12
data_replication_factor = 3
meta_replication_factor = 3
meta_epidemic_fanout = 3
[rpc_tls]
# NOT RECOMMENDED: you can skip this section if you don't want to encrypt intra-cluster traffic
# Thanks to genkeys.sh, generating the keys and certificates is easy, so there is NO REASON NOT TO DO IT.
ca_cert = "/path/to/garage/pki/garage-ca.crt"
node_cert = "/path/to/garage/pki/garage.crt"
node_key = "/path/to/garage/pki/garage.key"
[s3_api]
api_bind_addr = "[::1]:3900" # the S3 API port, HTTP without TLS. Add a reverse proxy for the TLS part.
s3_region = "garage" # set this to anything. S3 API calls will fail if they are not made against the region set here.
[s3_web]
bind_addr = "[::1]:3902"
root_domain = ".garage.tld"
index = "index.html"
```
Build Garage using `cargo build --release`.
Then, run it using either `./target/release/garage server -c path/to/config_file.toml` or `cargo run --release -- server -c path/to/config_file.toml`.
Set the `RUST_LOG` environment to `garage=debug` to dump some debug information.
Set it to `garage=trace` to dump even more debug information.
Set it to `garage=warn` to show nothing except warnings and errors.
## Setting up cluster nodes
Once all your `garage` nodes are running, you will need to:
1. check that they are correctly talking to one another;
2. configure them with their physical location (in the case of a multi-dc deployment) and a number of "ring tokens" proportionnal to the storage space available on each node;
3. create some S3 API keys and buckets;
4. ???;
5. profit!
To run these administrative tasks, you will need to use the `garage` command line tool and it to connect to any of the cluster's nodes on the RPC port.
The `garage` CLI also needs TLS keys and certificates of its own to authenticate and be authenticated in the cluster.
A typicall invocation will be as follows:
```
./target/release/garage --ca-cert=pki/garage-ca.crt --client-cert=pki/garage-client.crt --client-key=pki/garage-client.key <...>
```
## Notes to self
### What to repair
- `tables`: to do a full sync of metadata, should not be necessary because it is done every hour by the system
- `versions` and `block_refs`: very time consuming, usefull if deletions have not been propagated, improves garbage collection
- `blocks`: very usefull to resync/rebalance blocks betweeen nodes
**[Go to the documentation](https://garagehq.deuxfleurs.fr)**

140
doc/Quickstart.md
View file

@ -1,140 +0,0 @@
# Quickstart on an existing deployment
First, chances are that your garage deployment is secured by TLS.
All your commands must be prefixed with their certificates.
I will define an alias once and for all to ease future commands.
Please adapt the path of the binary and certificates to your installation!
```
alias grg="/garage/garage --ca-cert /secrets/garage-ca.crt --client-cert /secrets/garage.crt --client-key /secrets/garage.key"
```
Now we can check that everything is going well by checking our cluster status:
```
grg status
```
Don't forget that `help` command and `--help` subcommands can help you anywhere, the CLI tool is self-documented! Two examples:
```
grg help
grg bucket allow --help
```
Fine, now let's create a bucket (we imagine that you want to deploy nextcloud):
```
grg bucket create nextcloud-bucket
```
Check that everything went well:
```
grg bucket list
grg bucket info nextcloud-bucket
```
Now we will generate an API key to access this bucket.
Note that API keys are independent of buckets: one key can access multiple buckets, multiple keys can access one bucket.
Now, let's start by creating a key only for our PHP application:
```
grg key new --name nextcloud-app-key
```
You will have the following output (this one is fake, `key_id` and `secret_key` were generated with the openssl CLI tool):
```
Key { key_id: "GK3515373e4c851ebaad366558", secret_key: "7d37d093435a41f2aab8f13c19ba067d9776c90215f56614adad6ece597dbb34", name: "nextcloud-app-key", name_timestamp: 1603280506694, deleted: false, authorized_buckets: [] }
```
Check that everything works as intended (be careful, info works only with your key identifier and not with its friendly name!):
```
grg key list
grg key info GK3515373e4c851ebaad366558
```
Now that we have a bucket and a key, we need to give permissions to the key on the bucket!
```
grg bucket allow --read --write nextcloud-bucket --key GK3515373e4c851ebaad366558
```
You can check at any times allowed keys on your bucket with:
```
grg bucket info nextcloud-bucket
```
Now, let's move to the S3 API!
We will use the `s3cmd` CLI tool.
You can install it via your favorite package manager.
Otherwise, check [their website](https://s3tools.org/s3cmd)
We will configure `s3cmd` with its interactive configuration tool, be careful not all endpoints are implemented!
Especially, the test run at the end does not work (yet).
```
$ s3cmd --configure
Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.
Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: GK3515373e4c851ebaad366558
Secret Key: 7d37d093435a41f2aab8f13c19ba067d9776c90215f56614adad6ece597dbb34
Default Region [US]: garage
Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: garage.deuxfleurs.fr
Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: garage.deuxfleurs.fr
Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:
When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:
On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:
New settings:
Access Key: GK3515373e4c851ebaad366558
Secret Key: 7d37d093435a41f2aab8f13c19ba067d9776c90215f56614adad6ece597dbb34
Default Region: garage
S3 Endpoint: garage.deuxfleurs.fr
DNS-style bucket+hostname:port template for accessing a bucket: garage.deuxfleurs.fr
Encryption password:
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0
Test access with supplied credentials? [Y/n] n
Save settings? [y/N] y
Configuration saved to '/home/quentin/.s3cfg'
```
Now, if everything works, the following commands should work:
```
echo hello world > hello.txt
s3cmd put hello.txt s3://nextcloud-bucket
s3cmd ls s3://nextcloud-bucket
s3cmd rm s3://nextcloud-bucket/hello.txt
```
That's all for now!

1
doc/book/.gitignore vendored Normal file
View file

@ -0,0 +1 @@
book

6
doc/book/book.toml Normal file
View file

@ -0,0 +1,6 @@
[book]
authors = ["Quentin Dufour"]
language = "en"
multilingual = false
src = "src"
title = "Garage Documentation"

31
doc/book/src/SUMMARY.md Normal file
View file

@ -0,0 +1,31 @@
# Summary
[The Garage Data Store](./intro.md)
- [Getting Started](./getting_started/index.md)
- [Get a binary](./getting_started/binary.md)
- [Configure the daemon](./getting_started/daemon.md)
- [Control the daemon](./getting_started/control.md)
- [Configure a cluster](./getting_started/cluster.md)
- [Create buckets and keys](./getting_started/bucket.md)
- [Handle files](./getting_started/files.md)
- [Cookbook](./cookbook/index.md)
- [Host a website](./cookbook/website.md)
- [Integrate as a media backend]()
- [Operate a cluster]()
- [Reference Manual](./reference_manual/index.md)
- [Garage CLI]()
- [S3 API](./reference_manual/s3_compatibility.md)
- [Design](./design/index.md)
- [Related Work](./design/related_work.md)
- [Internals](./design/internals.md)
- [Development](./development/index.md)
- [Setup your environment](./development/devenv.md)
- [Your first contribution]()
- [Working Documents](./working_documents/index.md)
- [Load Balancing Data](./working_documents/load_balancing.md)

5
doc/book/src/cookbook/index.md Normal file
View file

@ -0,0 +1,5 @@
# Cookbook
A cookbook, when you cook, is a collection of recipes.
Similarly, Garage's cookbook contains a collection of recipes that are known to works well!
This chapter could also be referred as "Tutorials" or "Best practises".

1
doc/book/src/cookbook/website.md Normal file
View file

@ -0,0 +1 @@
# Host a website

5
doc/book/src/design/index.md Normal file
View file

@ -0,0 +1,5 @@
# Design
The design section helps you to see Garage from a "big picture" perspective.
It will allow you to understand if Garage is a good fit for you,
how to better use it, how to contribute to it, what can Garage could and could not do, etc.

0
doc/Internals.md → doc/book/src/design/internals.md
View file

20
doc/Related Work.md → doc/book/src/design/related_work.md
View file

@ -1,3 +1,5 @@
# Related Work
## Context
Data storage is critical: it can lead to data loss if done badly and/or on hardware failure.
@ -8,7 +10,7 @@ But here we consider non specialized off the shelf machines that can be as low p
Distributed storage may help to solve both availability and scalability problems on these machines.
Many solutions were proposed, they can be categorized as block storage, file storage and object storage depending on the abstraction they provide.
## Related work
## Overview
Block storage is the most low level one, it's like exposing your raw hard drive over the network.
It requires very low latencies and stable network, that are often dedicated.
@ -36,3 +38,19 @@ However Pithos is not maintained anymore. More precisely the company that publis
Some tests conducted by the [ACIDES project](https://acides.org/) have shown that Openstack Swift consumes way more resources (CPU+RAM) that we can afford. Furthermore, people developing Swift have not designed their software for geo-distribution.
There were many attempts in research too. I am only thinking to [LBFS](https://pdos.csail.mit.edu/papers/lbfs:sosp01/lbfs.pdf) that was used as a basis for Seafile. But none of them have been effectively implemented yet.
## Existing software
**[Pithos](https://github.com/exoscale/pithos) :**
Pithos has been abandonned and should probably not used yet, in the following we explain why we did not pick their design.
Pithos was relying as a S3 proxy in front of Cassandra (and was working with Scylla DB too).
From its designers' mouth, storing data in Cassandra has shown its limitations justifying the project abandonment.
They built a closed-source version 2 that does not store blobs in the database (only metadata) but did not communicate further on it.
We considered there v2's design but concluded that it does not fit both our *Self-contained & lightweight* and *Simple* properties. It makes the development, the deployment and the operations more complicated while reducing the flexibility.
**[IPFS](https://ipfs.io/) :**
*Not written yet*
## Specific research papers
*Not yet written*

17
doc/book/src/development/devenv.md Normal file
View file

@ -0,0 +1,17 @@
# Setup your development environment
We propose the following quickstart to setup a full dev. environment as quickly as possible:
1. Setup a rust/cargo environment. eg. `dnf install rust cargo`
2. Install awscli v2 by following the guide [here](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html).
3. Run `cargo build` to build the project
4. Run `./script/dev-cluster.sh` to launch a test cluster (feel free to read the script)
5. Run `./script/dev-configure.sh` to configure your test cluster with default values (same datacenter, 100 tokens)
6. Run `./script/dev-bucket.sh` to create a bucket named `eprouvette` and an API key that will be stored in `/tmp/garage.s3`
7. Run `source ./script/dev-env-aws.sh` to configure your CLI environment
8. You can use `garage` to manage the cluster. Try `garage --help`.
9. You can use the `awsgrg` alias to add, remove, and delete files. Try `awsgrg help`, `awsgrg cp /proc/cpuinfo s3://eprouvette/cpuinfo.txt`, or `awsgrg ls s3://eprouvette`. `awsgrg` is a wrapper on the `aws s3` command pre-configured with the previously generated API key (the one in `/tmp/garage.s3`) and localhost as the endpoint.
Now you should be ready to start hacking on garage!

4
doc/book/src/development/index.md Normal file
View file

@ -0,0 +1,4 @@
# Development
Now that you are a Garage expert, you want to enhance it, you are in the right place!
We discuss here how to hack on Garage, how we manage its development, etc.

44
doc/book/src/getting_started/binary.md Normal file
View file

@ -0,0 +1,44 @@
# Get a binary
Currently, only two installations procedures are supported for Garage: from Docker (x86\_64 for Linux) and from source.
In the future, we plan to add a third one, by publishing a compiled binary (x86\_64 for Linux).
We did not test other architecture/operating system but, as long as your architecture/operating system is supported by Rust, you should be able to run Garage (feel free to report your tests!).
## From Docker
Our docker image is currently named `lxpz/garage_amd64` and is stored on the [Docker Hub](https://hub.docker.com/r/lxpz/garage_amd64/tags?page=1&ordering=last_updated).
We encourage you to use a fixed tag (eg. `v0.1.1d`) and not the `latest` tag.
For this example, we will use the latest published version at the time of the writing which is `v0.1.1d` but it's up to you
to check [the most recent versions on the Docker Hub](https://hub.docker.com/r/lxpz/garage_amd64/tags?page=1&ordering=last_updated).
For example:
```
sudo docker pull lxpz/garage_amd64:v0.1.1d
```
## From source
Garage is a standard Rust project.
First, you need `rust` and `cargo`.
On Debian:
```bash
sudo apt-get update
sudo apt-get install -y rustc cargo
```
Then, you can ask cargo to install the binary for you:
```bash
cargo install garage
```
That's all, `garage` should be in `$HOME/.cargo/bin`.
You can add this folder to your `$PATH` or copy the binary somewhere else on your system.
For the following, we will assume you copied it in `/usr/local/bin/garage`:
```bash
sudo cp $HOME/.cargo/bin/garage /usr/local/bin/garage
```

78
doc/book/src/getting_started/bucket.md Normal file
View file

@ -0,0 +1,78 @@
# Create buckets and keys
*We use a command named `garagectl` which is in fact an alias you must define as explained in the [Control the daemon](./daemon.md) section.*
In this section, we will suppose that we want to create a bucket named `nextcloud-bucket`
that will be accessed through a key named `nextcloud-app-key`.
Don't forget that `help` command and `--help` subcommands can help you anywhere, the CLI tool is self-documented! Two examples:
```
garagectl help
garagectl bucket allow --help
```
## Create a bucket
Fine, now let's create a bucket (we imagine that you want to deploy nextcloud):
```
garagectl bucket create nextcloud-bucket
```
Check that everything went well:
```
garagectl bucket list
garagectl bucket info nextcloud-bucket
```
## Create an API key
Now we will generate an API key to access this bucket.
Note that API keys are independent of buckets: one key can access multiple buckets, multiple keys can access one bucket.
Now, let's start by creating a key only for our PHP application:
```
garagectl key new --name nextcloud-app-key
```
You will have the following output (this one is fake, `key_id` and `secret_key` were generated with the openssl CLI tool):
```javascript
Key {
key_id: "GK3515373e4c851ebaad366558",
secret_key: "7d37d093435a41f2aab8f13c19ba067d9776c90215f56614adad6ece597dbb34",
name: "nextcloud-app-key",
name_timestamp: 1603280506694,
deleted: false,
authorized_buckets: []
}
```
Check that everything works as intended (be careful, info works only with your key identifier and not with its friendly name!):
```
garagectl key list
garagectl key info GK3515373e4c851ebaad366558
```
## Allow a key to access a bucket
Now that we have a bucket and a key, we need to give permissions to the key on the bucket!
```
garagectl bucket allow \
--read \
--write
nextcloud-bucket \
--key GK3515373e4c851ebaad366558
```
You can check at any times allowed keys on your bucket with:
```
garagectl bucket info nextcloud-bucket
```

72
doc/book/src/getting_started/cluster.md Normal file
View file

@ -0,0 +1,72 @@
# Configure a cluster
*We use a command named `garagectl` which is in fact an alias you must define as explained in the [Control the daemon](./daemon.md) section.*
In this section, we will inform garage of the disk space available on each node of the cluster
as well as the site (think datacenter) of each machine.
## Test cluster
As this part is not relevant for a test cluster, you can use this one-liner to create a basic topology:
```bash
garagectl status | grep UNCONFIGURED | grep -Po '^[0-9a-f]+' | while read id; do
garagectl node configure -d dc1 -n 10 $id
done
```
## Real-world cluster
For our example, we will suppose we have the following infrastructure (Tokens, Identifier and Datacenter are specific values to garage described in the following):
| Location | Name | Disk Space | `Tokens` | `Identifier` | `Datacenter` |
|----------|---------|------------|----------|--------------|--------------|
| Paris | Mercury | 1 To | `100` | `8781c5` | `par1` |
| Paris | Venus | 2 To | `200` | `2a638e` | `par1` |
| London | Earth | 2 To | `200` | `68143d` | `lon1` |
| Brussels | Mars | 1.5 To | `150` | `212f75` | `bru1` |
### Identifier
After its first launch, garage generates a random and unique identifier for each nodes, such as:
```
8781c50c410a41b363167e9d49cc468b6b9e4449b6577b64f15a249a149bdcbc
```
Often a shorter form can be used, containing only the beginning of the identifier, like `8781c5`,
which identifies the server "Mercury" located in "Paris" according to our previous table.
The most simple way to match an identifier to a node is to run:
```
garagectl status
```
It will display the IP address associated with each node; from the IP address you will be able to recognize the node.
### Tokens
Garage reasons on an arbitrary metric about disk storage that is named "tokens".
The number of tokens must be proportional to the disk space dedicated to the node.
Additionaly, ideally the number of tokens must be in the order of magnitude of 100
to provide a good trade-off between data load balancing and performances (*this sentence must be verified, it may be wrong*).
Here we chose 1 token = 10 Go but you are free to select the value that best fit your needs.
### Datacenter
Datacenter are simply a user-chosen identifier that identify a group of server that are located in the same place.
It is up to the system administrator deploying garage to identify what does "the same place" means.
Behind the scene, garage will try to store the same data on different sites to provide high availability despite a data center failure.
### Inject the topology
Given the information above, we will configure our cluster as follow:
```
garagectl node configure --datacenter par1 -n 100 -t mercury 8781c5
garagectl node configure --datacenter par1 -n 200 -t venus 2a638e
garagectl node configure --datacenter lon1 -n 200 -t earth 68143d
garagectl node configure --datacenter bru1 -n 150 -t mars 212f75
```

77
doc/book/src/getting_started/control.md Normal file
View file

@ -0,0 +1,77 @@
# Control the daemon
The `garage` binary has two purposes:
- it acts as a daemon when launched with `garage server ...`
- it acts as a control tool for the daemon when launched with any other command
In this section, we will see how to use the `garage` binary as a control tool for the daemon we just started.
You first need to get a shell having access to this binary, which depends of your configuration:
- with `docker-compose`, run `sudo docker-compose exec g1 bash` then `/garage/garage`
- with `docker`, run `sudo docker exec -ti garaged bash` then `/garage/garage`
- with `systemd`, simply run `/usr/local/bin/garage` if you followed previous instructions
*You can also install the binary on your machine to remotely control the cluster.*
## Talk to the daemon and create an alias
`garage` requires 4 options to talk with the daemon:
```
--ca-cert <ca-cert>
--client-cert <client-cert>
--client-key <client-key>
-h, --rpc-host <rpc-host>
```
The 3 first ones are certificates and keys needed by TLS, the last one is simply the address of garage's RPC endpoint.
Because we configure garage directly from the server, we do not need to set `--rpc-host`.
To avoid typing the 3 first options each time we want to run a command, we will create an alias.
### `docker-compose` alias
```bash
alias garagectl='/garage/garage \
--ca-cert /pki/garage-ca.crt \
--client-cert /pki/garage.crt \
--client-key /pki/garage.key'
```
### `docker` alias
```bash
alias garagectl='/garage/garage \
--ca-cert /etc/garage/pki/garage-ca.crt \
--client-cert /etc/garage/pki/garage.crt \
--client-key /etc/garage/pki/garage.key'
```
### raw binary alias
```bash
alias garagectl='/usr/local/bin/garage \
--ca-cert /etc/garage/pki/garage-ca.crt \
--client-cert /etc/garage/pki/garage.crt \
--client-key /etc/garage/pki/garage.key'
```
Of course, if your deployment does not match exactly one of this alias, feel free to adapt it to your needs!
## Test the alias
You can test your alias by running a simple command such as:
```
garagectl status
```
You should get something like that as result:
```
Healthy nodes:
2a638ed6c775b69a… 37f0ba978d27 [::ffff:172.20.0.101]:3901 UNCONFIGURED/REMOVED
68143d720f20c89d… 9795a2f7abb5 [::ffff:172.20.0.103]:3901 UNCONFIGURED/REMOVED
8781c50c410a41b3… 758338dde686 [::ffff:172.20.0.102]:3901 UNCONFIGURED/REMOVED
```
...which means that you are ready to configure your cluster!

222
doc/book/src/getting_started/daemon.md Normal file
View file

@ -0,0 +1,222 @@
# Configure the daemon
Garage is a software that can be run only in a cluster and requires at least 3 instances.
In our getting started guide, we document two deployment types:
- [Test deployment](#test-deployment) though `docker-compose`
- [Real-world deployment](#real-world-deployment) through `docker` or `systemd`
In any case, you first need to generate TLS certificates, as traffic is encrypted between Garage's nodes.
## Generating a TLS Certificate
To generate your TLS certificates, run on your machine:
```
wget https://git.deuxfleurs.fr/Deuxfleurs/garage/raw/branch/master/genkeys.sh
chmod +x genkeys.sh
./genkeys.sh
```
It will creates a folder named `pki` containing the keys that you will used for the cluster.
## Test deployment
Single machine deployment is only described through `docker-compose`.
Before starting, we recommend you create a folder for our deployment:
```bash
mkdir garage-single
cd garage-single
```
We start by creating a file named `docker-compose.yml` describing our network and our containers:
```yml
version: '3.4'
networks: { virtnet: { ipam: { config: [ subnet: 172.20.0.0/24 ]}}}
services:
g1:
image: lxpz/garage_amd64:v0.1.1d
networks: { virtnet: { ipv4_address: 172.20.0.101 }}
volumes:
- "./pki:/pki"
- "./config.toml:/garage/config.toml"
g2:
image: lxpz/garage_amd64:v0.1.1d
networks: { virtnet: { ipv4_address: 172.20.0.102 }}
volumes:
- "./pki:/pki"
- "./config.toml:/garage/config.toml"
g3:
image: lxpz/garage_amd64:v0.1.1d
networks: { virtnet: { ipv4_address: 172.20.0.103 }}
volumes:
- "./pki:/pki"
- "./config.toml:/garage/config.toml"
```
*We define a static network here which is not considered as a best practise on Docker.
The rational is that Garage only supports IP address and not domain names in its configuration, so we need to know the IP address in advance.*
and then create the `config.toml` file next to it as follow:
```toml
metadata_dir = "/garage/meta"
data_dir = "/garage/data"
rpc_bind_addr = "[::]:3901"
bootstrap_peers = [
"172.20.0.101:3901",
"172.20.0.102:3901",
"172.20.0.103:3901",
]
[rpc_tls]
ca_cert = "/pki/garage-ca.crt"
node_cert = "/pki/garage.crt"
node_key = "/pki/garage.key"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
[s3_web]
bind_addr = "[::]:3902"
root_domain = ".web.garage"
index = "index.html"
```
*Please note that we have not mounted `/garage/meta` or `/garage/data` on the host: data will be lost when the container will be destroyed.*
And that's all, you are ready to launch your cluster!
```
sudo docker-compose up
```
While your daemons are up, your cluster is still not configured yet.
However, you can check that your services are still listening as expected by querying them from your host:
```bash
curl http://172.20.0.{101,102,103}:3902
```
which should give you:
```
Not found
Not found
Not found
```
That's all, you are ready to [configure your cluster!](./cluster.md).
## Real-world deployment
Before deploying garage on your infrastructure, you must inventory your machines.
For our example, we will suppose the following infrastructure:
| Location | Name | IP Address | Disk Space |
|----------|---------|------------|------------|
| Paris | Mercury | fc00:1::1 | 1 To |
| Paris | Venus | fc00:1::2 | 2 To |
| London | Earth | fc00:1::2 | 2 To |
| Brussels | Mars | fc00:B::1 | 1.5 To |
On each machine, we will have a similar setup, especially you must consider the following folders/files:
- `/etc/garage/pki`: Garage certificates, must be generated on your computer and copied on the servers
- `/etc/garage/config.toml`: Garage daemon's configuration (defined below)
- `/etc/systemd/system/garage.service`: Service file to start garage at boot automatically (defined below, not required if you use docker)
- `/var/lib/garage/meta`: Contains Garage's metadata, put this folder on a SSD if possible
- `/var/lib/garage/data`: Contains Garage's data, this folder will grows and must be on a large storage, possibly big HDDs.
A valid `/etc/garage/config.toml` for our cluster would be:
```toml
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
rpc_bind_addr = "[::]:3901"
bootstrap_peers = [
"[fc00:1::1]:3901",
"[fc00:1::2]:3901",
"[fc00:B::1]:3901",
"[fc00:F::1]:3901",
]
[rpc_tls]
ca_cert = "/etc/garage/pki/garage-ca.crt"
node_cert = "/etc/garage/pki/garage.crt"
node_key = "/etc/garage/pki/garage.key"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
[s3_web]
bind_addr = "[::]:3902"
root_domain = ".web.garage"
index = "index.html"
```
Please make sure to change `bootstrap_peers` to **your** IP addresses!
### For docker users
On each machine, you can run the daemon with:
```bash
docker run \
-d \
--name garaged \
--restart always \
--network host \
-v /etc/garage/pki:/etc/garage/pki \
-v /etc/garage/config.toml:/garage/config.toml \
-v /var/lib/garage/meta:/var/lib/garage/meta \
-v /var/lib/garage/data:/var/lib/garage/data \
lxpz/garage_amd64:v0.1.1d
```
It should be restart automatically at each reboot.
Please note that we use host networking as otherwise Docker containers can no communicate with IPv6.
To upgrade, simply stop and remove this container and start again the command with a new version of garage.
### For systemd/raw binary users
Create a file named `/etc/systemd/system/garage.service`:
```toml
[Unit]
Description=Garage Data Store
After=network-online.target
Wants=network-online.target
[Service]
Environment='RUST_LOG=garage=info' 'RUST_BACKTRACE=1'
ExecStart=/usr/local/bin/garage server -c /etc/garage/config.toml
[Install]
WantedBy=multi-user.target
```
To start the service then automatically enable it at boot:
```bash
sudo systemctl start garage
sudo systemctl enable garage
```
To see if the service is running and to browse its logs:
```bash
sudo systemctl status garage
sudo journalctl -u garage
```
If you want to modify the service file, do not forget to run `systemctl daemon-reload`
to inform `systemd` of your modifications.

42
doc/book/src/getting_started/files.md Normal file
View file

@ -0,0 +1,42 @@
# Handle files
We recommend the use of MinIO Client to interact with Garage files (`mc`).
Instructions to install it and use it are provided on the [MinIO website](https://docs.min.io/docs/minio-client-quickstart-guide.html).
Before reading the following, you need a working `mc` command on your path.
## Configure `mc`
You need your access key and secret key created in the [previous section](bucket.md).
You also need to set the endpoint: it must match the IP address of one of the node of the cluster and the API port (3900 by default).
For this whole configuration, you must set an alias name: we chose `my-garage`, that you will used for all commands.
Adapt the following command accordingly and run it:
```bash
mc alias set \
my-garage \
http://172.20.0.101:3900 \
<access key> \
<secret key> \
--api S3v4
```
You must also add an environment variable to your configuration to inform MinIO of our region (`garage` by default).
The best way is to add the following snippet to your `$HOME/.bash_profile` or `$HOME/.bashrc` file:
```bash
export MC_REGION=garage
```
## Use `mc`
You can not list buckets from `mc` currently.
But the following commands and many more should work:
```bash
mc cp image.png my-garage/nextcloud-bucket
mc cp my-garage/nextcloud-bucket/image.png .
mc ls my-garage/nextcloud-bucket
mc mirror localdir/ my-garage/another-bucket
```

5
doc/book/src/getting_started/index.md Normal file
View file

@ -0,0 +1,5 @@
# Getting Started
Let's start your Garage journey!
In this chapter, we explain how to deploy a simple garage cluster and start interacting with it.
Our goal is to introduce you to Garage's workflows.

44
doc/book/src/img/logo.svg Normal file
View file

@ -0,0 +1,44 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="128" height="128" version="1.1" viewBox="0 0 33.867 33.867" xmlns="http://www.w3.org/2000/svg" xmlns:cc="http://creativecommons.org/ns#" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<metadata>
<rdf:RDF>
<cc:Work rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
<dc:title/>
</cc:Work>
</rdf:RDF>
</metadata>
<g stroke-width=".14689">
<path d="m20.613 10.981a2.2034 2.2034 0 0 1-0.73445-0.07638l-9.2042-2.4839a2.2342 2.2342 0 0 1-0.69332-0.32757z"/>
<g fill="#4e4e4e">
<path class="cls-1" d="m6.6028 26.612 1.3661-0.0088h0.01763q0.75796 0 0.75796 0.71389v2.3003a6.5748 6.5748 0 0 1-2.2886 0.37898q-1.2515 0-1.8861-0.8505t-0.63457-2.3179q0-1.4689 0.7888-2.2827a2.5823 2.5823 0 0 1 1.9301-0.81524 3.5371 3.5371 0 0 1 2.0667 0.64338 1.0385 1.0385 0 0 1-0.18068 0.46711 1.2603 1.2603 0 0 1-0.33932 0.35254 2.5926 2.5926 0 0 0-1.5027-0.51999 1.4175 1.4175 0 0 0-1.1854 0.54203q-0.42304 0.53909-0.42304 1.6966 0 2.1769 1.604 2.1769a4.4743 4.4743 0 0 0 0.97829-0.11457v-0.83728q0-0.3966 0.01763-0.58756h-0.64633a0.60519 0.60519 0 0 1-0.40101-0.11018 0.44067 0.44067 0 0 1-0.12779-0.35254 1.51 1.51 0 0 1 0.088134-0.47446z"/>
<path class="cls-1" d="m13.401 29.379a1.1413 1.1413 0 0 1-0.14689 0.31288 1.0664 1.0664 0 0 1-0.22474 0.25118 0.99592 0.99592 0 0 1-0.80937-0.51705 1.7847 1.7847 0 0 1-1.2603 0.56406q-0.67863 0-1.0282-0.3966a1.3573 1.3573 0 0 1-0.34372-0.9166q0-0.73445 0.48033-1.1149a1.9404 1.9404 0 0 1 1.2354-0.3687q0.40542 0 0.76677 0.03525v-0.2644q0-0.69626-0.66982-0.69626-0.47592 0-1.3485 0.31728a1.2368 1.2368 0 0 1-0.29378-0.78439 4.9164 4.9164 0 0 1 1.9096-0.3966 1.5526 1.5526 0 0 1 1.0752 0.37016q0.41423 0.37016 0.41423 1.1193v1.7979q-0.0029 0.48474 0.24384 0.68745zm-2.2122-0.22034a1.2471 1.2471 0 0 0 0.88134-0.42304v-0.77852a5.9182 5.9182 0 0 0-0.66982-0.03525 0.73445 0.73445 0 0 0-0.54643 0.18214 0.6331 0.6331 0 0 0-0.18508 0.46711 0.62282 0.62282 0 0 0 0.14689 0.44067 0.48768 0.48768 0 0 0 0.3731 0.14689z"/>
<path class="cls-1" d="m14.115 26.012a1.0547 1.0547 0 0 1 0.14689-0.32169 0.88134 0.88134 0 0 1 0.22474-0.25118 1.1017 1.1017 0 0 1 0.92982 0.78439q0.35254-0.78439 1.1369-0.78439a2.7028 2.7028 0 0 1 0.51118 0.06169 1.9786 1.9786 0 0 1-0.2644 1.0282 2.2357 2.2357 0 0 0-0.3966-0.05288q-0.53762 0-0.86372 0.57287v2.8174a3.0627 3.0627 0 0 1-0.53762 0.04407 3.3785 3.3785 0 0 1-0.55525-0.04407v-2.9525q-0.0059-0.6375-0.33197-0.90191z"/>
<path class="cls-1" d="m21.157 29.379a1.1413 1.1413 0 0 1-0.15423 0.31288 1.0664 1.0664 0 0 1-0.22474 0.25118 0.99592 0.99592 0 0 1-0.8079-0.51705 1.7847 1.7847 0 0 1-1.2603 0.56406q-0.67864 0-1.0282-0.3966a1.3573 1.3573 0 0 1-0.34372-0.9166q0-0.73445 0.48033-1.1149a1.9404 1.9404 0 0 1 1.2295-0.37457q0.40542 0 0.76677 0.03525v-0.2644q0-0.69626-0.66982-0.69626-0.47592 0-1.3485 0.31728a1.2368 1.2368 0 0 1-0.29378-0.7844 4.9164 4.9164 0 0 1 1.9096-0.3966 1.5526 1.5526 0 0 1 1.0752 0.37016q0.41423 0.37016 0.41423 1.1193v1.8038q0.0088 0.48474 0.25559 0.68745zm-2.2151-0.22034a1.2471 1.2471 0 0 0 0.88134-0.42304v-0.77852a5.9182 5.9182 0 0 0-0.66982-0.03525 0.73445 0.73445 0 0 0-0.54643 0.18508 0.6331 0.6331 0 0 0-0.18508 0.46711 0.62282 0.62282 0 0 0 0.14689 0.44067 0.48768 0.48768 0 0 0 0.3731 0.14395z"/>
<path class="cls-1" d="m22.241 29.344q-0.3966-0.60813-0.3966-1.679t0.50236-1.679a1.5188 1.5188 0 0 1 1.2074-0.60813 1.7039 1.7039 0 0 1 1.1898 0.44067 0.99739 0.99739 0 0 1 0.69626-0.37898 0.82552 0.82552 0 0 1 0.23356 0.24677 1.0282 1.0282 0 0 1 0.14689 0.30847q-0.24678 0.21152-0.24678 0.75796v2.4971q0 1.4013-0.4583 1.983-0.4583 0.58169-1.5071 0.58756a4.2598 4.2598 0 0 1-1.5776-0.29378 1.1854 1.1854 0 0 1 0.27322-0.80202 2.882 2.882 0 0 0 1.1854 0.27322q0.57728 0 0.79761-0.29378a1.322 1.322 0 0 0 0.22034-0.81084v-0.35254a1.6936 1.6936 0 0 1-1.1017 0.41423 1.3014 1.3014 0 0 1-1.1648-0.61106zm2.2651-0.71389v-2.0447a1.1355 1.1355 0 0 0-0.75796-0.36135 0.63604 0.63604 0 0 0-0.57728 0.37898 2.2988 2.2988 0 0 0-0.20712 1.0841q0 0.70508 0.18949 1.04a0.56406 0.56406 0 0 0 0.49796 0.33491 1.1193 1.1193 0 0 0 0.8549-0.43186z"/>
<path class="cls-1" d="m30.105 28.039h-2.4678a1.4924 1.4924 0 0 0 0.23356 0.80643q0.20712 0.28644 0.72711 0.28644a2.6778 2.6778 0 0 0 1.1546-0.30847 1.159 1.159 0 0 1 0.31728 0.66982 2.8467 2.8467 0 0 1-1.6966 0.50236q-0.99151 0-1.4234-0.64338-0.43186-0.64338-0.43186-1.6657 0-1.0282 0.47592-1.6657a1.5923 1.5923 0 0 1 1.3617-0.64338q0.88134 0 1.3617 0.53321a1.9434 1.9434 0 0 1 0.47593 1.344 3.4519 3.4519 0 0 1-0.08813 0.7844zm-1.701-1.8684q-0.7227 0-0.77558 1.0929h1.5335v-0.10576a1.25 1.25 0 0 0-0.18508-0.71389 0.64338 0.64338 0 0 0-0.567-0.27321z"/>
</g>
<path d="m17.034 3.0341a2.9114 2.9114 0 0 0-1.1462 0.24753l-11.697 5.1749a0.42304 0.42304 0 0 0-0.22169 0.56586 0.20418 0.20418 0 0 0 0.01757 0.04702l1.8769 3.7099h1.6288l-0.23151-1.2935c-0.0191-0.10429-0.18819-0.84337-0.3483-1.3751l5.4746 1.71c0.07196 0.34089 0.16746 0.65935 0.28112 0.9586h8.8765c0.0978-0.29932 0.17499-0.61834 0.22738-0.9586l5.4627-1.7053c-0.16011 0.53174-0.32713 1.2662-0.34623 1.3705l-0.23151 1.2935h1.6283l1.8593-3.6763 0.01757-0.03359 0.0181-0.04547a0.027909 0.027909 0 0 0 0-0.01188 0.39367 0.39367 0 0 0 0.01757-0.13643 0.41864 0.41864 0 0 0-0.26303-0.4191l-11.697-5.1749a2.9114 2.9114 0 0 0-1.2041-0.24753z" fill="#ffd952"/>
<path d="m17.034 5.4825a2.9114 2.9114 0 0 0-1.1462 0.24753l-11.697 5.1749a0.42304 0.42304 0 0 0-0.22169 0.56534 0.20418 0.20418 0 0 0 0.01757 0.04703l1.018 2.0118h2.1632c-0.068234-0.28802-0.15662-0.64282-0.25528-0.97049l3.1073 0.97048h14.121l3.0939-0.96583c-0.09841 0.32682-0.18541 0.67924-0.25321 0.96583h2.1627l1.0005-1.9782 0.01757-0.03359 0.0181-0.04547a0.027909 0.027909 0 0 0 0-0.01188 0.39367 0.39367 0 0 0 0.01757-0.13643 0.41864 0.41864 0 0 0-0.26303-0.41858l-11.697-5.1749a2.9114 2.9114 0 0 0-1.2041-0.24753z" fill="#49c8fa"/>
<path class="cls-2" d="m30.198 13.82a0.39367 0.39367 0 0 1-0.01762 0.13661 0.027909 0.027909 0 0 1 0 0.01175l-0.01762 0.04554-0.01762 0.03379-2.8306 5.5965c-0.39367 0.77705-1.1178 0.75355-0.99592-0.03232l0.56993-3.1817c0.0191-0.10429 0.18655-0.83874 0.34666-1.3705l-5.4629 1.7054c-0.85784 5.5716-8.1891 5.6641-9.3848 0l-5.4746-1.7098c0.16011 0.53174 0.32904 1.2706 0.34813 1.3749l0.56994 3.1816c0.12192 0.78586-0.60225 0.80937-0.99592 0.03232l-2.8482-5.6303a0.20418 0.20418 0 0 1-0.01763-0.04701 0.42304 0.42304 0 0 1 0.2218-0.56553l11.697-5.175a2.9114 2.9114 0 0 1 2.3502 0l11.697 5.175a0.41864 0.41864 0 0 1 0.26294 0.41864z" fill="#ffd952"/>
<path class="cls-3" d="m20.801 14.796 5.0574-2.0359a0.21446 0.21446 0 0 0 0-0.39807c-0.58756-0.24531-1.3132-0.52734-2.0242-0.82259-0.13073-0.05435-1.369 0.83434-1.4821 0.92541l-2.1799 1.7421c-0.52734 0.44214-0.07051 0.86959 0.62869 0.58903z" fill="#45c8ff"/>
<circle class="cls-3" cx="17.135" cy="16.785" r="2.6367" fill="#45c8ff"/>
<path d="m20.613 10.981a2.2034 2.2034 0 0 1-0.73445-0.07638l-9.2042-2.4839a2.2342 2.2342 0 0 1-0.69332-0.32757z"/>
<g fill="#4e4e4e">
<path class="cls-1" d="m6.6028 26.612 1.3661-0.0088h0.01763q0.75796 0 0.75796 0.71389v2.3003a6.5748 6.5748 0 0 1-2.2886 0.37898q-1.2515 0-1.8861-0.8505t-0.63457-2.3179q0-1.4689 0.7888-2.2827a2.5823 2.5823 0 0 1 1.9301-0.81524 3.5371 3.5371 0 0 1 2.0667 0.64338 1.0385 1.0385 0 0 1-0.18068 0.46711 1.2603 1.2603 0 0 1-0.33932 0.35254 2.5926 2.5926 0 0 0-1.5027-0.51999 1.4175 1.4175 0 0 0-1.1854 0.54203q-0.42304 0.53909-0.42304 1.6966 0 2.1769 1.604 2.1769a4.4743 4.4743 0 0 0 0.97829-0.11457v-0.83728q0-0.3966 0.01763-0.58756h-0.64633a0.60519 0.60519 0 0 1-0.40101-0.11018 0.44067 0.44067 0 0 1-0.12779-0.35254 1.51 1.51 0 0 1 0.088134-0.47446z"/>
<path class="cls-1" d="m13.401 29.379a1.1413 1.1413 0 0 1-0.14689 0.31288 1.0664 1.0664 0 0 1-0.22474 0.25118 0.99592 0.99592 0 0 1-0.80937-0.51705 1.7847 1.7847 0 0 1-1.2603 0.56406q-0.67863 0-1.0282-0.3966a1.3573 1.3573 0 0 1-0.34372-0.9166q0-0.73445 0.48033-1.1149a1.9404 1.9404 0 0 1 1.2354-0.3687q0.40542 0 0.76677 0.03525v-0.2644q0-0.69626-0.66982-0.69626-0.47592 0-1.3485 0.31728a1.2368 1.2368 0 0 1-0.29378-0.78439 4.9164 4.9164 0 0 1 1.9096-0.3966 1.5526 1.5526 0 0 1 1.0752 0.37016q0.41423 0.37016 0.41423 1.1193v1.7979q-0.0029 0.48474 0.24384 0.68745zm-2.2122-0.22034a1.2471 1.2471 0 0 0 0.88134-0.42304v-0.77852a5.9182 5.9182 0 0 0-0.66982-0.03525 0.73445 0.73445 0 0 0-0.54643 0.18214 0.6331 0.6331 0 0 0-0.18508 0.46711 0.62282 0.62282 0 0 0 0.14689 0.44067 0.48768 0.48768 0 0 0 0.3731 0.14689z"/>
<path class="cls-1" d="m14.115 26.012a1.0547 1.0547 0 0 1 0.14689-0.32169 0.88134 0.88134 0 0 1 0.22474-0.25118 1.1017 1.1017 0 0 1 0.92982 0.78439q0.35254-0.78439 1.1369-0.78439a2.7028 2.7028 0 0 1 0.51118 0.06169 1.9786 1.9786 0 0 1-0.2644 1.0282 2.2357 2.2357 0 0 0-0.3966-0.05288q-0.53762 0-0.86372 0.57287v2.8174a3.0627 3.0627 0 0 1-0.53762 0.04407 3.3785 3.3785 0 0 1-0.55525-0.04407v-2.9525q-0.0059-0.6375-0.33197-0.90191z"/>
<path class="cls-1" d="m21.157 29.379a1.1413 1.1413 0 0 1-0.15423 0.31288 1.0664 1.0664 0 0 1-0.22474 0.25118 0.99592 0.99592 0 0 1-0.8079-0.51705 1.7847 1.7847 0 0 1-1.2603 0.56406q-0.67864 0-1.0282-0.3966a1.3573 1.3573 0 0 1-0.34372-0.9166q0-0.73445 0.48033-1.1149a1.9404 1.9404 0 0 1 1.2295-0.37457q0.40542 0 0.76677 0.03525v-0.2644q0-0.69626-0.66982-0.69626-0.47592 0-1.3485 0.31728a1.2368 1.2368 0 0 1-0.29378-0.7844 4.9164 4.9164 0 0 1 1.9096-0.3966 1.5526 1.5526 0 0 1 1.0752 0.37016q0.41423 0.37016 0.41423 1.1193v1.8038q0.0088 0.48474 0.25559 0.68745zm-2.2151-0.22034a1.2471 1.2471 0 0 0 0.88134-0.42304v-0.77852a5.9182 5.9182 0 0 0-0.66982-0.03525 0.73445 0.73445 0 0 0-0.54643 0.18508 0.6331 0.6331 0 0 0-0.18508 0.46711 0.62282 0.62282 0 0 0 0.14689 0.44067 0.48768 0.48768 0 0 0 0.3731 0.14395z"/>
<path class="cls-1" d="m22.241 29.344q-0.3966-0.60813-0.3966-1.679t0.50236-1.679a1.5188 1.5188 0 0 1 1.2074-0.60813 1.7039 1.7039 0 0 1 1.1898 0.44067 0.99739 0.99739 0 0 1 0.69626-0.37898 0.82552 0.82552 0 0 1 0.23356 0.24677 1.0282 1.0282 0 0 1 0.14689 0.30847q-0.24678 0.21152-0.24678 0.75796v2.4971q0 1.4013-0.4583 1.983-0.4583 0.58169-1.5071 0.58756a4.2598 4.2598 0 0 1-1.5776-0.29378 1.1854 1.1854 0 0 1 0.27322-0.80202 2.882 2.882 0 0 0 1.1854 0.27322q0.57728 0 0.79761-0.29378a1.322 1.322 0 0 0 0.22034-0.81084v-0.35254a1.6936 1.6936 0 0 1-1.1017 0.41423 1.3014 1.3014 0 0 1-1.1648-0.61106zm2.2651-0.71389v-2.0447a1.1355 1.1355 0 0 0-0.75796-0.36135 0.63604 0.63604 0 0 0-0.57728 0.37898 2.2988 2.2988 0 0 0-0.20712 1.0841q0 0.70508 0.18949 1.04a0.56406 0.56406 0 0 0 0.49796 0.33491 1.1193 1.1193 0 0 0 0.8549-0.43186z"/>
<path class="cls-1" d="m30.105 28.039h-2.4678a1.4924 1.4924 0 0 0 0.23356 0.80643q0.20712 0.28644 0.72711 0.28644a2.6778 2.6778 0 0 0 1.1546-0.30847 1.159 1.159 0 0 1 0.31728 0.66982 2.8467 2.8467 0 0 1-1.6966 0.50236q-0.99151 0-1.4234-0.64338-0.43186-0.64338-0.43186-1.6657 0-1.0282 0.47592-1.6657a1.5923 1.5923 0 0 1 1.3617-0.64338q0.88134 0 1.3617 0.53321a1.9434 1.9434 0 0 1 0.47593 1.344 3.4519 3.4519 0 0 1-0.08813 0.7844zm-1.701-1.8684q-0.7227 0-0.77558 1.0929h1.5335v-0.10576a1.25 1.25 0 0 0-0.18508-0.71389 0.64338 0.64338 0 0 0-0.567-0.27321z"/>
</g>
<g>
<path d="m17.034 3.0341a2.9114 2.9114 0 0 0-1.1462 0.24753l-11.697 5.1749a0.42304 0.42304 0 0 0-0.22169 0.56586 0.20418 0.20418 0 0 0 0.01757 0.04702l1.8769 3.7099h1.6288l-0.23151-1.2935c-0.0191-0.10429-0.18819-0.84337-0.3483-1.3751l5.4746 1.71c0.07196 0.34089 0.16746 0.65935 0.28112 0.9586h8.8765c0.0978-0.29932 0.17499-0.61834 0.22738-0.9586l5.4627-1.7053c-0.16011 0.53174-0.32713 1.2662-0.34623 1.3705l-0.23151 1.2935h1.6283l1.8593-3.6763 0.01757-0.03359 0.0181-0.04547a0.027909 0.027909 0 0 0 0-0.01188 0.39367 0.39367 0 0 0 0.01757-0.13643 0.41864 0.41864 0 0 0-0.26303-0.4191l-11.697-5.1749a2.9114 2.9114 0 0 0-1.2041-0.24753z" fill="#ff9329"/>
<path d="m17.034 5.4825a2.9114 2.9114 0 0 0-1.1462 0.24753l-11.697 5.1749a0.42304 0.42304 0 0 0-0.22169 0.56534 0.20418 0.20418 0 0 0 0.01757 0.04703l1.018 2.0118h2.1632c-0.068234-0.28802-0.15662-0.64282-0.25528-0.97049l3.1073 0.97048h14.121l3.0939-0.96583c-0.09841 0.32682-0.18541 0.67924-0.25321 0.96583h2.1627l1.0005-1.9782 0.01757-0.03359 0.0181-0.04547a0.027909 0.027909 0 0 0 0-0.01188 0.39367 0.39367 0 0 0 0.01757-0.13643 0.41864 0.41864 0 0 0-0.26303-0.41858l-11.697-5.1749a2.9114 2.9114 0 0 0-1.2041-0.24753z" fill="#4e4e4e"/>
<path class="cls-2" d="m30.198 13.82a0.39367 0.39367 0 0 1-0.01762 0.13661 0.027909 0.027909 0 0 1 0 0.01175l-0.01762 0.04554-0.01762 0.03379-2.8306 5.5965c-0.39367 0.77705-1.1178 0.75355-0.99592-0.03232l0.56993-3.1817c0.0191-0.10429 0.18655-0.83874 0.34666-1.3705l-5.4629 1.7054c-0.85784 5.5716-8.1891 5.6641-9.3848 0l-5.4746-1.7098c0.16011 0.53174 0.32904 1.2706 0.34813 1.3749l0.56994 3.1816c0.12192 0.78586-0.60225 0.80937-0.99592 0.03232l-2.8482-5.6303a0.20418 0.20418 0 0 1-0.01763-0.04701 0.42304 0.42304 0 0 1 0.2218-0.56553l11.697-5.175a2.9114 2.9114 0 0 1 2.3502 0l11.697 5.175a0.41864 0.41864 0 0 1 0.26294 0.41864z" fill="#ff9329"/>
<path class="cls-3" d="m20.801 14.796 5.0574-2.0359a0.21446 0.21446 0 0 0 0-0.39807c-0.58756-0.24531-1.3132-0.52734-2.0242-0.82259-0.13073-0.05435-1.369 0.83434-1.4821 0.92541l-2.1799 1.7421c-0.52734 0.44214-0.07051 0.86959 0.62869 0.58903z" fill="#4e4e4e"/>
<circle class="cls-3" cx="17.135" cy="16.785" r="2.6367" fill="#4e4e4e"/>
</g>
</g>
</svg>
Side by side

After

Width:  |  Height:  |  Size: 13 KiB

95
doc/book/src/intro.md Normal file
View file

@ -0,0 +1,95 @@
![Garage's Logo](img/logo.svg)
# The Garage Geo-Distributed Data Store
Garage is a lightweight geo-distributed data store.
It comes from the observation that despite numerous object stores
many people have broken data management policies (backup/replication on a single site or none at all).
To promote better data management policies, with focused on the following desirable properties:
- **Self-contained & lightweight**: works everywhere and integrates well in existing environments to target hyperconverged infrastructures
- **Highly resilient**: highly resilient to network failures, network latency, disk failures, sysadmin failures
- **Simple**: simple to understand, simple to operate, simple to debug
- **Internet enabled**: made for multi-sites (eg. datacenter, offices, etc.) interconnected through a regular internet connection.
We also noted that the pursuit of some other goals are detrimental to our initial goals.
The following have been identified has non-goals, if it matters to you, you should not use Garage:
- **Extreme performances**: high performances constrain a lot the design and the infrastructure; we seek performances through minimalism only.
- **Feature extensiveness**: complete implementation of the S3 API or any other API to make garage a drop-in replacement is not targeted as it could lead to decisions impacting our desirable properties.
- **Storage optimizations**: erasure coding or any other coding technique both increase the difficulty of placing data and synchronizing; we limit ourselves to duplication.
- **POSIX/Filesystem compatibility**: we do not aim at being POSIX compatible or to emulate any kind of filesystem. Indeed, in a distributed environment, such syncronizations are translated in network messages that impose severe constraints on the deployment.
## Supported and planned protocols
Garage speaks (or will speak) the following protocols:
- [S3](https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html) - *SUPPORTED* - Enable applications to store large blobs such as pictures, video, images, documents, etc. S3 is versatile enough to also be used to publish a static website.
- [IMAP](https://github.com/go-pluto/pluto) - *PLANNED* - email storage is quite complex to get good oerformances.
To keep performances optimals, most imap servers only support on-disk storage.
We plan to add logic to Garage to make it a viable solution for email storage.
- *More to come*
## Use Cases
**[Deuxfleurs](https://deuxfleurs.fr) :** Garage is used by Deuxfleurs which is a non-profit hosting organization.
Especially, it is used to host their main website, this documentation and some of its members's blogs. Additionally,
Garage is used as a [backend for Nextcloud](https://docs.nextcloud.com/server/20/admin_manual/configuration_files/primary_storage.html). Deuxfleurs also plans to use Garage as their [Matrix's media backend](https://github.com/matrix-org/synapse-s3-storage-provider) and has the backend of [OCIS](https://github.com/owncloud/ocis).
*Are you using Garage? [Open a pull request](https://git.deuxfleurs.fr/Deuxfleurs/garage/) to add your organization here!*
## Comparison to existing software
**[Minio](https://min.io/) :** Minio shares our *self-contained & lightweight* goal but selected two of our non-goals: *storage optimizations* through erasure coding and *POSIX/Filesystem compatibility* through strong consistency.
However, by pursuing these two non-goals, minio do not reach our desirable properties.
First, it fails on the *simple* property: due to the erasure coding, minio has severe limitations on how drives can be added or deleted from a cluster.
Second, it fails on the *interned enabled* property: due to its strong consistency, minio is latency sensitive.
Furthermore, minio has no knowledge of "sites" and thus can not distribute data to minimize the failure of a given site.
**[Openstack Swift](https://docs.openstack.org/swift/latest/) :**
OpenStack Swift at least fails on the *self-contained & lightweight* goal.
Starting it requires around 8Gb of RAM, which is too much especially in an hyperconverged infrastructure.
It seems also to be far from *Simple*.
**[Ceph](https://ceph.io/ceph-storage/object-storage/) :**
This review holds for the whole Ceph stack, including the RADOS paper, Ceph Object Storage module, the RADOS Gateway, etc.
At is core, Ceph has been designed to provide *POSIX/Filesystem compatibility* which requires strong consistency, which in turn
makes Ceph latency sensitive and fails our *Internet enabled* goal.
Due to its industry oriented design, Ceph is also far from being *Simple* to operate and from being *self-contained & lightweight* which makes it hard to integrate it in an hyperconverged infrastructure.
In a certain way, Ceph and Minio are closer togethers than they are from Garage or OpenStack Swift.
*More comparisons are available in our [Related Work](design/related_work.md) chapter.*
## Other Resources
This website is not the only source of information about Garage!
We reference here other places on the Internet where you can learn more about Garage.
### Rust API (docs.rs)
If you encounter a specific bug in Garage or plan to patch it, you may jump directly to the source code documentation!
- [garage\_api](https://docs.rs/garage_api/latest/garage_api/) - contains the S3 standard API endpoint
- [garage\_model](https://docs.rs/garage_model/latest/garage_model/) - contains Garage's model built on the table abstraction
- [garage\_rpc](https://docs.rs/garage_rpc/latest/garage_rpc/) - contains Garage's federation protocol
- [garage\_table](https://docs.rs/garage_table/latest/garage_table/) - contains core Garage's CRDT datatypes
- [garage\_util](https://docs.rs/garage_util/latest/garage_util/) - contains garage entrypoints (daemon, cli)
- [garage\_web](https://docs.rs/garage_web/latest/garage_web/) - contains the S3 website endpoint
### Talks
We love to talk and hear about Garage, that's why we keep a log here:
- [(fr, 2020-12-02) Garage : jouer dans la cour des grands quand on est un hébergeur associatif](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/master/doc/20201202_talk/talk.pdf)
*Did you write or talk about Garage? [Open a pull request](https://git.deuxfleurs.fr/Deuxfleurs/garage/) to add a link here!*
## Community
If you want to discuss with us, you can join our Matrix channel at [#garage:deuxfleurs.fr](https://matrix.to/#/#garage:deuxfleurs.fr).
Our code and our issue tracker, which is the place where you should report bugs, are managed on [Deuxfleurs' Gitea](https://git.deuxfleurs.fr/Deuxfleurs/garage).
## License
Garage, all the source code, is released under the [AGPL v3 License](https://www.gnu.org/licenses/agpl-3.0.en.html).
Please note that if you patch Garage and then use it to provide any service over a network, you must share your code!

5
doc/book/src/reference_manual/index.md Normal file
View file

@ -0,0 +1,5 @@
# Reference Manual
A reference manual contains some extensive descriptions about the features and the behaviour of the software.
Reading of this chapter is recommended once you have a good knowledge/understanding of Garage.
It will be useful if you want to tune it or to use it in some exotic conditions.

0
doc/Compatibility.md → doc/book/src/reference_manual/s3_compatibility.md
View file

8
doc/book/src/working_documents/index.md Normal file
View file

@ -0,0 +1,8 @@
# Working Documents
Working documents are documents that reflect the fact that Garage is a software that evolves quickly.
They are a way to communicate our ideas, our changes, and so on before or while we are implementing them in Garage.
If you like to live on the edge, it could also serve as a documentation of our next features to be released.
Ideally, once the feature/patch has been merged, the working document should serve as a source to
update the rest of the documentation and then be removed.

2
doc/Load_Balancing.md → doc/book/src/working_documents/load_balancing.md
View file

@ -1,3 +1,5 @@
## Load Balancing Data (planned for version 0.2)
I have conducted a quick study of different methods to load-balance data over different Garage nodes using consistent hashing.
### Requirements