performance improvements #342

Merged
lx merged 21 commits from lx-perf-improvements into main 3 months ago
lx commented 5 months ago
Owner

Performance improvements included in this PR:

  • Use Bytes at a few places where appropriate, instead of Vec<u8>, to reduce the number of copies
    • StreamChunker now accumulates incoming slices in a Vec<Bytes> instead of a VecDeque<u8>. Replaces calls to .extend() and .drain() that were quite costly by a simple concat() on a vec of slices which is much more optimized
  • Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using spawn_blocking
  • Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel
  • Better LMDB defaults: we now put flags NoSync and NoMetaSync to avoid fsync at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads

Other changes included in this PR:

  • Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12
  • switch to tracing_subscriber for logs, which allows to have timing of each event
Performance improvements included in this PR: - [x] Use `Bytes` at a few places where appropriate, instead of `Vec<u8>`, to reduce the number of copies - [x] StreamChunker now accumulates incoming slices in a `Vec<Bytes>` instead of a `VecDeque<u8>`. Replaces calls to `.extend()` and `.drain()` that were quite costly by a simple `concat()` on a vec of slices which is much more optimized - [x] Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using `spawn_blocking` - [x] Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel - [x] Better LMDB defaults: we now put flags `NoSync` and `NoMetaSync` to avoid `fsync` at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads Other changes included in this PR: - [x] Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12 - [x] switch to `tracing_subscriber` for logs, which allows to have timing of each event
lx added 2 commits 5 months ago
8b63147775
Performance improvements:
lx added 1 commit 5 months ago
dbb7e7192c
Update cargo.nix
lx added 1 commit 5 months ago
b003bc0baf
Make clippy happy
lx added 1 commit 5 months ago
0957427d87
More make clippy happy
lx added 1 commit 5 months ago
22187e03ab
Fix tests
lx added 1 commit 5 months ago
11e00a7e34
cargo fmt
lx added 1 commit 5 months ago
44de628599
Update cargo.nix
lx added 1 commit 4 months ago
120aa063f0
Faster chunker
lx added 1 commit 4 months ago
2010b7a44e
fix clippy
lx added 17 commits 4 months ago
lx added 10 commits 3 months ago
lx force-pushed lx-perf-improvements from 1b3df6017d to ebc20a8798 3 months ago
lx added 1 commit 3 months ago
4da67b0035
Update drone signature
lx added 1 commit 3 months ago
dd5304f6fc
Replace logging crate pretty_env_logger by tracing_subscriber::fmt
lx added 1 commit 3 months ago
44cd98d2e4
Tracing-subscriber: write to stderr
lx added 1 commit 3 months ago
efbca67ce4
Add env filter to tracing subscriber
lx added 1 commit 3 months ago
eb97e13a6a
update cargo.nix
lx added 1 commit 3 months ago
6226f5ceca
Update to netapp 0.4.5 - fixed ping
lx added 1 commit 3 months ago
07e6bcde85
Merge branch 'main' into lx-perf-improvements
lx added 1 commit 3 months ago
d23b3a14fc
Merge branch 'main' into lx-perf-improvements
lx added 1 commit 3 months ago
d9d199a6c9
Merge branch 'main' into lx-perf-improvements
lx changed title from WIP: performance improvements to performance improvements 3 months ago
lx added 1 commit 3 months ago
f91fab8582
Simplify+improve async hasher by using bounded channel
lx merged commit 309d7aef3f into main 3 months ago
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
The pull request has been merged as 309d7aef3f.
Sign in to join this conversation.
Loading…
There is no content yet.