This PR includes work from @jirutka :
- [x] Allow linking against system-provided libraries (libsodium, libsqlite, libzstd) #370
- [x] Make OTLP exporter optional and allow building without Prometheus exporter (/metrics) #372
And also:
- [x] Update `.nix` files
- [x] Remove heed default-features
- [x] Bump versions of all Garage crates to 0.8.0
- [x] Make db engines (lmdb, sled, sqlite) optionnal
- [x] Add documentation for available features
- [x] Directly include code of previous versions used for migration in order to reduce dependencies
- [x] Read variable `GIT_VERSION` from garage main instead of in crate garage_util to make builds faster
- [x] Report features used in the build somewhere? (in `garage --version` or something)
- [x] Check we `warn!` correctly if we try to use deactivated feature
- [x] Allow not to launch S3 endpoint if not in config
Reviewed-on: #373
Garage currently uses the legacy resolver "1". The new one is used
by default if the root package specifies 'edition = 2021', which
Garage does not (yet).
The problem with the legacy resolver is, among others, that features
enabled by dev-dependencies are propagated to normal dependencies.
This affects e.g. hyper - one of the dev-dependencies enables "http2"
feature that adds many extra dependencies. If we build garage without
opentelemetry-otlp (this is enabled in the following commit), there's
no normal dependency enabling "http2" feature.
See https://doc.rust-lang.org/cargo/reference/resolver.html#feature-resolver-version-2
Included in this PR:
- [x] Small refactor, resync code is moved to a separate `block/resync.rs` file
- [x] Block resync tranquility is no longer in config file, it is set dynamically using `garage worker set resync-tranquility` (this parameter is persisted over Garage restarts)
- [x] Up to 4 block resync workers can be activated to run simultaneously to speed up big resyncs, this parameter is set dynamically using `garage worker set resync-n-workers`
Reviewed-on: #369
Unfortunately, rusqlite uses the opposite logic for enabling/disabling
bundled libraries to others (libsodium-sys, zstd-sys). Cargo features
are very limited and doesn't allow to enable feature A in a dependency
iff feature B is disabled.
Note, lmdb-rkv-sys doesn't need any special treatment because it
automatically links against system liblmdb if found via pkgconf.
Linux distros should build garage with
`--no-default-features --features system-libs` to disable bundled-libs
and enable system-libs.
If this feature is enabled, libsodium-sys and zstd-sys will link
dynamically against system-provided libraries instead of building
and linking statically the bundled (possibly outdated and vulnerable)
copies of them. This feature is intended mainly for linux package
maintainers.
By default, structopt reports the value provided by
the env var CARGO_PKG_VERSION, feeded by Cargo when reading
Cargo.toml. However for Garage we use a versioning based on git,
so we often report a version that is behind the real version.
In this commit, we create garage_util::version::garage() that
reports the right version and configure all structopt subcommands
to call this function instead of using the env var.
- [x] New background worker trait
- [x] Adapt all current workers to use new API
- [x] Command to list currently running workers, and whether they are active, idle, or dead
- [x] Error reporting
- Optimizations
- [x] Merkle updater: several items per iteration
- [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on
- scrub:
- [x] have only one worker with a channel to start/pause/cancel
- [x] automatic scrub
- [x] ability to view and change tranquility from CLI
- [x] persistence of a few info
- [ ] Testing
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #332
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
