continuous-integration/drone/push Build is passing Details
Garage is a lightweight S3-compatible distributed object store, with the following goals:
- As self-contained as possible
- Easy to set up
- Highly resilient to network failures, network latency, disk failures, sysadmin failures
- Relatively simple
- Made for multi-datacenter deployments
- Extremely high performance
- Complete implementation of the S3 API
- Erasure coding (our replication model is simply to copy the data as is on several nodes, in different datacenters if possible)
Our main use case is to provide a distributed storage layer for small-scale self hosted services such as Deuxfleurs.
Check our compatibility page to view details of the S3 API compatibility.
We propose the following quickstart to setup a full dev. environment as quickly as possible:
- Setup a rust/cargo environment. eg.
dnf install rust cargo
- Install awscli v2 by following the guide here.
cargo buildto build the project
./script/dev-cluster.shto launch a test cluster (feel free to read the script)
./script/dev-configure.shto configure your test cluster with default values (same datacenter, 100 tokens)
./script/dev-bucket.shto create a bucket named
eprouvetteand an API key that will be stored in
source ./script/dev-env-aws.shto configure your CLI environment
- You can use
garageto manage the cluster. Try
- You can use the
awsgrgalias to add, remove, and delete files. Try
awsgrg cp /proc/cpuinfo s3://eprouvette/cpuinfo.txt, or
awsgrg ls s3://eprouvette.
awsgrgis a wrapper on the
aws s3command pre-configured with the previously generated API key (the one in
/tmp/garage.s3) and localhost as the endpoint.
Now you should be ready to start hacking on garage!
Only a subset of S3 is supported: adding, listing, getting and deleting files in a bucket.
Bucket management, ACL and other advanced features are not (yet?) handled through the S3 API but through the
We primarily test
garage against the
awscli tool and
Setting up Garage
genkeys.sh script to generate TLS keys for encrypting communications between Garage nodes.
The script takes no arguments and will generate keys in
This script creates a certificate authority
garage-ca which signs certificates for individual Garage nodes.
Garage nodes from a same cluster authenticate themselves by verifying that they have certificates signed by the same certificate authority.
Garage requires two locations to store its data: a metadata directory, and a data directory. The metadata directory is used to store metadata such as object lists, and should ideally be located on an SSD drive. The data directory is used to store the chunks of data of the objects stored in Garage. In a typical deployment the data directory is stored on a standard HDD.
Garage does not handle TLS for its S3 API endpoint. This should be handled by adding a reverse proxy.
Create a configuration file with the following structure:
block_size = 1048576 # objects are split in blocks of maximum this number of bytes metadata_dir = "/path/to/ssd/metadata/directory" data_dir = "/path/to/hdd/data/directory" rpc_bind_addr = "[::]:3901" # the port other Garage nodes will use to talk to this node bootstrap_peers = [ # Ideally this list should contain the IP addresses of all other Garage nodes of the cluster. # Use Ansible or any kind of configuration templating to generate this automatically. "10.0.0.1:3901", "10.0.0.2:3901", "10.0.0.3:3901", ] # optionnal: garage can find cluster nodes automatically using a Consul server # garage only does lookup but does not register itself, registration should be handled externally by e.g. Nomad consul_host = "localhost:8500" # optionnal: host name of a Consul server for automatic peer discovery consul_service_name = "garage" # optionnal: service name to look up on Consul max_concurrent_rpc_requests = 12 data_replication_factor = 3 meta_replication_factor = 3 meta_epidemic_fanout = 3 [rpc_tls] # NOT RECOMMENDED: you can skip this section if you don't want to encrypt intra-cluster traffic # Thanks to genkeys.sh, generating the keys and certificates is easy, so there is NO REASON NOT TO DO IT. ca_cert = "/path/to/garage/pki/garage-ca.crt" node_cert = "/path/to/garage/pki/garage.crt" node_key = "/path/to/garage/pki/garage.key" [s3_api] api_bind_addr = "[::1]:3900" # the S3 API port, HTTP without TLS. Add a reverse proxy for the TLS part. s3_region = "garage" # set this to anything. S3 API calls will fail if they are not made against the region set here. [s3_web] bind_addr = "[::1]:3902" root_domain = ".garage.tld" index = "index.html"
Build Garage using
cargo build --release.
Then, run it using either
./target/release/garage server -c path/to/config_file.toml or
cargo run --release -- server -c path/to/config_file.toml.
RUST_LOG environment to
garage=debug to dump some debug information.
Set it to
garage=trace to dump even more debug information.
Set it to
garage=warn to show nothing except warnings and errors.
Setting up cluster nodes
Once all your
garage nodes are running, you will need to:
- check that they are correctly talking to one another;
- configure them with their physical location (in the case of a multi-dc deployment) and a number of "ring tokens" proportionnal to the storage space available on each node;
- create some S3 API keys and buckets;
To run these administrative tasks, you will need to use the
garage command line tool and it to connect to any of the cluster's nodes on the RPC port.
garage CLI also needs TLS keys and certificates of its own to authenticate and be authenticated in the cluster.
A typicall invocation will be as follows:
./target/release/garage --ca-cert=pki/garage-ca.crt --client-cert=pki/garage-client.crt --client-key=pki/garage-client.key <...>
Notes to self
What to repair
tables: to do a full sync of metadata, should not be necessary because it is done every hour by the system
block_refs: very time consuming, usefull if deletions have not been propagated, improves garbage collection
blocks: very usefull to resync/rebalance blocks betweeen nodes