S3-compatible object store for small self-hosted geo-distributed deployments
https://garagehq.deuxfleurs.fr/
Alex
309d7aef3f
continuous-integration/drone/push Build is passing
Details
Performance improvements included in this PR: - [x] Use `Bytes` at a few places where appropriate, instead of `Vec<u8>`, to reduce the number of copies - [x] StreamChunker now accumulates incoming slices in a `Vec<Bytes>` instead of a `VecDeque<u8>`. Replaces calls to `.extend()` and `.drain()` that were quite costly by a simple `concat()` on a vec of slices which is much more optimized - [x] Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using `spawn_blocking` - [x] Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel - [x] Better LMDB defaults: we now put flags `NoSync` and `NoMetaSync` to avoid `fsync` at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads Other changes included in this PR: - [x] Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12 - [x] switch to `tracing_subscriber` for logs, which allows to have timing of each event Reviewed-on: #342 |
||
|---|---|---|
| doc | ||
| nix | ||
| script | ||
| src | ||
| .dockerignore | ||
| .drone.yml | ||
| .gitattributes | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.nix | ||
| Cargo.toml | ||
| Dockerfile | ||
| LICENSE | ||
| Makefile | ||
| README.md | ||
| default.nix | ||
| k2v_test.py | ||
| rustfmt.toml | ||
| shell.nix | ||
README.md
Garage 
[ Website and documentation | Binary releases | Git repository | Matrix channel ]
Garage is a lightweight S3-compatible distributed object store, with the following goals:
- As self-contained as possible
- Easy to set up
- Highly resilient to network failures, network latency, disk failures, sysadmin failures
- Relatively simple
- Made for multi-datacenter deployments
Non-goals include:
- Extremely high performance
- Complete implementation of the S3 API
- Erasure coding (our replication model is simply to copy the data as is on several nodes, in different datacenters if possible)
Our main use case is to provide a distributed storage layer for small-scale self hosted services such as Deuxfleurs.