KokaKiwi/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues Pull requests Projects Releases Wiki Activity

Finish writing about Garage features, and fix from-source instructions

Browse source
This commit is contained in:
Alex 2022-09-15 13:23:57 +02:00
parent f6aebefcc9
commit 1d0a610690
Signed by untrusted user: lx
GPG key ID: 0E496D15096376BE
3 changed files with 110 additions and 49 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
doc/book/cookbook/exposing-websites.md
View file

@ -5,12 +5,14 @@ weight = 25
## Configuring a bucket for website access ## Configuring a bucket for website access
There are two methods to expose buckets as website: There are three methods to expose buckets as website:
1. using the PutBucketWebsite S3 API call, which is allowed for access keys that have the owner permission bit set 1. using the PutBucketWebsite S3 API call, which is allowed for access keys that have the owner permission bit set
2. from the Garage CLI, by an adminstrator of the cluster 2. from the Garage CLI, by an adminstrator of the cluster
3. using the Garage administration API
The `PutBucketWebsite` API endpoint [is documented](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketWebsite.html) in the official AWS docs. The `PutBucketWebsite` API endpoint [is documented](https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketWebsite.html) in the official AWS docs.
This endpoint can also be called [using `aws s3api`](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-website.html) on the command line. This endpoint can also be called [using `aws s3api`](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-website.html) on the command line.
The website configuration supported by Garage is only a subset of the possibilities on Amazon S3: redirections are not supported, only the index document and error document can be specified. The website configuration supported by Garage is only a subset of the possibilities on Amazon S3: redirections are not supported, only the index document and error document can be specified.

77
doc/book/cookbook/from-source.md
View file

@ -20,57 +20,76 @@ sudo apt-get update
sudo apt-get install build-essential sudo apt-get install build-essential
``` ```
## Using source from the Gitea repository (recommended) ## Building from source from the Gitea repository
The primary location for Garage's source code is the The primary location for Garage's source code is the
[Gitea repository](https://git.deuxfleurs.fr/Deuxfleurs/garage). [Gitea repository](https://git.deuxfleurs.fr/Deuxfleurs/garage),
which contains all of the released versions as well as the code
for the developpement of the next version.
Clone the repository and build Garage with the following commands: Clone the repository and enter it as follows:
```bash ```bash
git clone https://git.deuxfleurs.fr/Deuxfleurs/garage.git git clone https://git.deuxfleurs.fr/Deuxfleurs/garage.git
cd garage cd garage
cargo build
``` ```
Be careful, as this will make a debug build of Garage, which will be extremely slow! If you wish to build a specific version of Garage, check out the corresponding tag. For instance:
To make a release build, invoke `cargo build --release` (this takes much longer).
The binaries built this way are found in `target/{debug,release}/garage`.
## Using source from `crates.io`
Garage's source code is published on `crates.io`, Rust's official package repository.
This means you can simply ask `cargo` to download and build this source code for you:
```bash ```bash
cargo install garage git tag # List available tags
git checkout v0.8.0 # Change v0.8.0 with the version you wish to build
``` ```
That's all, `garage` should be in `$HOME/.cargo/bin`. Otherwise you will be building a developpement build from the `main` branch
that includes all of the changes to be released in the next version.
Be careful that such a build might be unstable or contain bugs,
and could be incompatible with nodes that run stable versions of Garage.
You can add this folder to your `$PATH` or copy the binary somewhere else on your system. Finally, build Garage with the following command:
For instance:
```bash ```bash
sudo cp $HOME/.cargo/bin/garage /usr/local/bin/garage cargo build --release
``` ```
The binary built this way can now be found in `target/release/garage`.
You may simply copy this binary to somewhere in your `$PATH` in order to
have the `garage` command available in your shell, for instance:
## Selecting features to activate in your build ```bash
sudo cp target/release/garage /usr/local/bin/garage
```
Garage supports a number of compilation options in the form of Cargo features, If you are planning to develop Garage,
you might be interested in producing debug builds, which compile faster but run slower:
this can be done by removing the `--release` flag, and the resulting build can then
be found in `target/debug/garage`.
## List of available Cargo feature flags
Garage supports a number of compilation options in the form of Cargo feature flags,
which can be used to provide builds adapted to your system and your use case. which can be used to provide builds adapted to your system and your use case.
The following features are available: To produce a build with a given set of features, invoke the `cargo build` command
as follows:
| Feature | Enabled | Description | ```bash
| ------- | ------- | ----------- | # This will build the default feature set plus feature1, feature2 and feature3
| `bundled-libs` | BY DEFAULT | Use bundled version of sqlite3, zstd, lmdb and libsodium | cargo build --release --features feature1,feature2,feature3
| `system-libs` | optional | Use system version of sqlite3, zstd, lmdb and libsodium if available (exclusive with `bundled-libs`, build using `cargo build --no-default-features --features system-libs`) | # This will build ONLY feature1, feature2 and feature3
| `k2v` | optional | Enable the experimental K2V API (if used, all nodes on your Garage cluster must have it enabled as well) | cargo build --release --no-default-features \
| `kubernetes-discovery` | optional | Enable automatic registration and discovery of cluster nodes through the Kubernetes API | --features feature1,feature2,feature3
| `metrics` | BY DEFAULT | Enable collection of metrics in Prometheus format on the admin API | ```
The following feature flags are available in v0.8.0:
| Feature flag | Enabled | Description |
| ------------ | ------- | ----------- |
| `bundled-libs` | *by default* | Use bundled version of sqlite3, zstd, lmdb and libsodium |
| `system-libs` | optional | Use system version of sqlite3, zstd, lmdb and libsodium<br>if available (exclusive with `bundled-libs`, build using<br>`cargo build --no-default-features --features system-libs`) |
| `k2v` | optional | Enable the experimental K2V API (if used, all nodes on your<br>Garage cluster must have it enabled as well) |
| `kubernetes-discovery` | optional | Enable automatic registration and discovery<br>of cluster nodes through the Kubernetes API |
| `metrics` | *by default* | Enable collection of metrics in Prometheus format on the admin API |
| `telemetry-otlp` | optional | Enable collection of execution traces using OpenTelemetry | | `telemetry-otlp` | optional | Enable collection of execution traces using OpenTelemetry |
| `sled` | BY DEFAULT | Enable using Sled to store Garage's metadata | | `sled` | *by default* | Enable using Sled to store Garage's metadata |
| `lmdb` | optional | Enable using LMDB to store Garage's metadata | | `lmdb` | optional | Enable using LMDB to store Garage's metadata |
| `sqlite` | optional | Enable using Sqlite3 to store Garage's metadata | | `sqlite` | optional | Enable using Sqlite3 to store Garage's metadata |

74
doc/book/reference-manual/features.md
View file

@ -26,6 +26,11 @@ a storage plan that best exploits the available storage capacity while satisfyin
To learn more about geo-distributed Garage clusters, To learn more about geo-distributed Garage clusters,
read our documentation on [setting up a real-world deployment](@/documentation/cookbook/real-world.md). read our documentation on [setting up a real-world deployment](@/documentation/cookbook/real-world.md).
### Standalone/self-contained
Garage is extremely simple to deploy, and does not depend on any external service to run.
This makes setting up and administering storage clusters, we hope, as easy as it could be.
### Flexible topology ### Flexible topology
A Garage cluster can very easily evolve over time, as storage nodes are added or removed. A Garage cluster can very easily evolve over time, as storage nodes are added or removed.
@ -46,7 +51,7 @@ This is particularly usefull when nodes are far from one another and talk to one
### Several replication modes ### Several replication modes
Garage supports a variety of replication modes, with 1 copy, 2 copies or 3 copies of your data, Garage supports a variety of replication modes, with 1 copy, 2 copies or 3 copies of your data,
and with various levels of consistency. and with various levels of consistency, in order to adapt to a variety of usage scenarios.
Read our reference page on [supported replication modes](@/documentation/reference-manual/configuration.md#replication-mode) Read our reference page on [supported replication modes](@/documentation/reference-manual/configuration.md#replication-mode)
to select the replication mode best suited to your use case (hint: in most cases, `replication_mode = "3"` is what you want). to select the replication mode best suited to your use case (hint: in most cases, `replication_mode = "3"` is what you want).
@ -54,32 +59,67 @@ to select the replication mode best suited to your use case (hint: in most cases
A storage bucket can easily be configured to be served directly by Garage as a static web site. A storage bucket can easily be configured to be served directly by Garage as a static web site.
Domain names for multiple websites directly map to bucket names, making it easy to build Domain names for multiple websites directly map to bucket names, making it easy to build
a platform for your user's to autonomously build and host their websites over Garage. a platform for your users to autonomously build and host their websites over Garage.
Surprisingly, none of the other alternative S3 implementations we surveyed (such as Minio Surprisingly, none of the other alternative S3 implementations we surveyed (such as Minio
or CEPH) support publishing static websites from S3 buckets, a feature that is however or CEPH) support publishing static websites from S3 buckets, a feature that is however
directly inherited from S3 on AWS. directly inherited from S3 on AWS.
Read more on our [dedicated documentation page](@/documentation/cookbook/exposing-websites.md).
### Bucket names as aliases ### Bucket names as aliases
- the same bucket may have multiple names (useful when exposing websites for example) In Garage, a bucket may have several names, known as aliases.
Aliases can easily be added and removed on demand:
this allows to easily rename buckets if needed
without having to copy all of their content, something that cannot be done on AWS.
For buckets served as static websites, having multiple aliases for a bucket can allow
exposing the same content under different domain names.
- bucket renaming is possible Garage also supports bucket aliases which are local to a single user:
this allows different users to have different buckets with the same name, thus avoiding naming collisions.
This can be helpfull for instance if you want to write an application that creates per-user buckets with always the same name.
- Scoped buckets: 2 users can have a different bucket with the same name -> avoid collision. Helpful if you want to write an application that creates per-user bucket always with the same name. This feature is totally invisible to S3 clients and does not break compatibility with AWS.
### Standalone/self contained
### Integration with Kubernetes and Nomad
Many node discovery methods: Kubernetes integration, Nomad integration through Consul
### Support for changing IP addresses
(as long as all nodes don't change their IP at the same time)
### Cluster administration API ### Cluster administration API
Garage provides a fully-fledged REST API to administer your cluster programatically.
Functionnality included in the admin API include: setting up and monitoring
cluster nodes, managing access credentials, and managing storage buckets and bucket aliases.
A full reference of the administration API is available [here](@/documentation/reference-manual/admin-api.md).
### Metrics and traces ### Metrics and traces
### (experimental) K2V API Garage makes some internal metrics available in the Prometheus data format,
which allows you to build interactive dashboards to visualize the load and internal state of your storage cluster.
For developpers and performance-savvy administrators,
Garage also supports exporting traces of what it does internally in OpenTelemetry format.
This allows to monitor the time spent at various steps of the processing of requests,
in order to detect potential performance bottlenecks.
### Kubernetes and Nomad integrations
Garage can automatically discover other nodes in the cluster thanks to integration
with orchestrators such as Kubernetes and Nomad (when used with Consul).
This eases the configuration of your cluster as it removes one step where nodes need
to be manually connected to one another.
### Support for changing IP addresses
As long as all of your nodes don't thange their IP address at the same time,
Garage should be able to tolerate nodes with changing/dynamic IP addresses,
as nodes will regularly exchange the IP addresses of their peers and try to
reconnect using newer addresses when existing connections are broken.
### K2V API (experimental)
As part of an ongoing research project, Garage can expose an experimental key/value storage API called K2V.
K2V is made for the storage and retrieval of many small key/value pairs that need to be processed in bulk.
This completes the S3 API with an alternative that can be used to easily store and access metadata
related to objects stored in an S3 bucket.
In the context of our research project, [Aérogramme](https://aerogramme.deuxfleurs.fr),
K2V is used to provide metadata and log storage for operations on encrypted e-mail storage.
Learn more on the specification of K2V [here](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)
and on how to enable it in Garage [here](@/documentation/reference-manual/k2v.md).