Commit graph

1160 commits

Author SHA1 Message Date
0009fd136c Merge pull request 'Make block resync speed dynamically configurable' (#369) from resync-ajustable-speed into main
All checks were successful
continuous-integration/drone/push Build is passing
Included in this PR:

- [x] Small refactor, resync code is moved to a separate `block/resync.rs` file
- [x] Block resync tranquility is no longer in config file, it is set dynamically using `garage worker set resync-tranquility` (this parameter is persisted over Garage restarts)
- [x] Up to 4 block resync workers can be activated to run simultaneously to speed up big resyncs, this parameter is set dynamically using `garage worker set resync-n-workers`

Reviewed-on: #369
2022-09-05 12:35:08 +02:00
7511ba5530 Allow linking against system-provided libsqlite
Some checks are pending
continuous-integration/drone/push Build is pending
continuous-integration/drone/pr Build is pending
Unfortunately, rusqlite uses the opposite logic for enabling/disabling
bundled libraries to others (libsodium-sys, zstd-sys). Cargo features
are very limited and doesn't allow to enable feature A in a dependency
iff feature B is disabled.

Note, lmdb-rkv-sys doesn't need any special treatment because it
automatically links against system liblmdb if found via pkgconf.

Linux distros should build garage with
`--no-default-features --features system-libs` to disable bundled-libs
and enable system-libs.
2022-09-03 19:15:57 +02:00
a6e40b75ea Add feature "system-libs" to enable linking against system libraries
If this feature is enabled, libsodium-sys and zstd-sys will link
dynamically against system-provided libraries instead of building
and linking statically the bundled (possibly outdated and vulnerable)
copies of them. This feature is intended mainly for linux package
maintainers.
2022-09-03 18:44:34 +02:00
e1751c8a9c
fix clippy
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-02 17:24:26 +02:00
5d4b937a00
Ability to have up to 4 concurrently working resync workers
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2022-09-02 17:18:13 +02:00
5e8baa433d
Make BlockManagerLocked fully private again
All checks were successful
continuous-integration/drone/push Build is passing
2022-09-02 16:52:22 +02:00
47be652a1f
block manager: refactor: split resync into separate file
All checks were successful
continuous-integration/drone/push Build is passing
2022-09-02 16:47:15 +02:00
943d76c583
Ability to dynamically set resync tranquility
All checks were successful
continuous-integration/drone/push Build is passing
2022-09-02 15:34:21 +02:00
6226f5ceca
Update to netapp 0.4.5 - fixed ping
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-02 14:33:12 +02:00
13b5f28c7e
Make use of BytesBuf from new Netapp
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-02 13:46:42 +02:00
1ef87ac4cb
cargo fmt
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-02 13:38:29 +02:00
99b532b85b
Apply PRIO_SECONDARY to block data transfers
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2022-09-01 16:35:43 +02:00
e648bf7b69
update cargo.nix
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-01 16:31:04 +02:00
df094bd807
Less strict timeouts 2022-09-01 16:30:44 +02:00
f3bf34b6a1
update netapp: straming + fix-ping
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-01 14:23:54 +02:00
bc977f9a7a
Update to Netapp with OrderTag support and exploit OrderTags
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-09-01 12:58:20 +02:00
4b726b0941
netapp recv with unbounded channel removes deadlock
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-01 09:47:28 +02:00
70231d68b2
Fix bytes_read counter
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-08-31 19:44:27 +02:00
e598231ca4
update netapp git commit
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-08-31 19:27:25 +02:00
c9bc9d89de
Merge branch 'lx-perf-improvements' into netapp-stream-body
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2022-08-31 17:42:31 +02:00
eb97e13a6a
update cargo.nix
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-08-31 17:42:00 +02:00
efbca67ce4
Add env filter to tracing subscriber
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2022-08-31 14:39:12 +02:00
44cd98d2e4
Tracing-subscriber: write to stderr
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-08-31 14:28:17 +02:00
dd5304f6fc
Replace logging crate pretty_env_logger by tracing_subscriber::fmt
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-08-31 14:24:41 +02:00
322dafc761
Try to fix clippy
Some checks reported errors
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build was killed
continuous-integration/drone Build was killed
2022-08-29 17:32:45 +02:00
5d065b8a0f
cargo2nix fix to fetchCrateGit
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-08-29 17:24:53 +02:00
52749e28f7
Merge branch 'lx-perf-improvements' into netapp-stream-body
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
continuous-integration/drone Build is failing
2022-08-29 16:48:43 +02:00
4da67b0035
Update drone signature
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
continuous-integration/drone Build is passing
2022-08-29 16:48:31 +02:00
1921f4f7e6
Merge branch 'lx-perf-improvements' into netapp-stream-body
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-08-29 16:45:05 +02:00
ebc20a8798
Merge branch 'main' into lx-perf-improvements
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-08-29 16:44:13 +02:00
532eca7ff9
Add some documentation for Caddy
Some checks are pending
continuous-integration/drone/pr Build is passing
continuous-integration/drone Build is passing
continuous-integration/drone/push Build is pending
2022-08-12 10:33:41 +02:00
2c7bae935a
Configure structopt to report the right version
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone Build is passing
continuous-integration/drone/push Build is passing
By default, structopt reports the value provided by
the env var CARGO_PKG_VERSION, feeded by Cargo when reading
Cargo.toml. However for Garage we use a versioning based on git,
so we often report a version that is behind the real version.
In this commit, we create garage_util::version::garage() that
reports the right version and configure all structopt subcommands
to call this function instead of using the env var.
2022-08-11 10:21:45 +02:00
8cd02639dc
drone: set TARGET env as needed by "to_s3" func
All checks were successful
continuous-integration/drone Build is passing
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-08-03 11:19:26 +02:00
e935861854
Factor out node request order selection logic & use in manager
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
continuous-integration/drone Build is failing
2022-07-29 12:25:03 +02:00
f0ee3056d3
Update cargo.nix 2022-07-29 12:25:03 +02:00
126b037307
update netapp 2022-07-29 12:25:03 +02:00
33750c04ed
Update cargo.nix 2022-07-29 12:25:03 +02:00
68087ee13d
Fix clippy 2022-07-29 12:25:03 +02:00
605a630333
Use streaming in block manager 2022-07-29 12:25:02 +02:00
a35d4da721
update netapp to 0.5 2022-07-29 12:25:02 +02:00
8e7e680afe
First adaptation to WIP netapp with streaming body 2022-07-29 12:25:02 +02:00
16f6a1a65d
fix clippy
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-07-29 12:24:49 +02:00
ad35b18bb1
Faster chunker 2022-07-29 12:24:49 +02:00
49154a78d8
Update cargo.nix 2022-07-29 12:24:48 +02:00
ff4771c36a
cargo fmt 2022-07-29 12:24:48 +02:00
381eb9a5a1
Fix tests 2022-07-29 12:24:48 +02:00
2cad656a03
More make clippy happy 2022-07-29 12:24:48 +02:00
0176da3ad2
Make clippy happy 2022-07-29 12:24:48 +02:00
40150527b8
Update cargo.nix 2022-07-29 12:24:48 +02:00
2f111e6b3d
Performance improvements:
- reduce contention on mutation_lock by having 256 of them
- better lmdb defaults
2022-07-29 12:24:48 +02:00