forked from Deuxfleurs/garage
S3-compatible object store for small self-hosted geo-distributed deployments
Alex
309d7aef3f
Performance improvements included in this PR: - [x] Use `Bytes` at a few places where appropriate, instead of `Vec<u8>`, to reduce the number of copies - [x] StreamChunker now accumulates incoming slices in a `Vec<Bytes>` instead of a `VecDeque<u8>`. Replaces calls to `.extend()` and `.drain()` that were quite costly by a simple `concat()` on a vec of slices which is much more optimized - [x] Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using `spawn_blocking` - [x] Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel - [x] Better LMDB defaults: we now put flags `NoSync` and `NoMetaSync` to avoid `fsync` at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads Other changes included in this PR: - [x] Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12 - [x] switch to `tracing_subscriber` for logs, which allows to have timing of each event Reviewed-on: Deuxfleurs/garage#342 |
||
---|---|---|
doc | ||
nix | ||
script | ||
src | ||
.dockerignore | ||
.drone.yml | ||
.gitattributes | ||
.gitignore | ||
Cargo.lock | ||
Cargo.nix | ||
Cargo.toml | ||
default.nix | ||
Dockerfile | ||
k2v_test.py | ||
LICENSE | ||
Makefile | ||
README.md | ||
rustfmt.toml | ||
shell.nix |
Garage
[ Website and documentation | Binary releases | Git repository | Matrix channel ]
Garage is a lightweight S3-compatible distributed object store, with the following goals:
- As self-contained as possible
- Easy to set up
- Highly resilient to network failures, network latency, disk failures, sysadmin failures
- Relatively simple
- Made for multi-datacenter deployments
Non-goals include:
- Extremely high performance
- Complete implementation of the S3 API
- Erasure coding (our replication model is simply to copy the data as is on several nodes, in different datacenters if possible)
Our main use case is to provide a distributed storage layer for small-scale self hosted services such as Deuxfleurs.