performance improvements #342

Merged
lx merged 21 commits from lx-perf-improvements into main 2022-09-12 14:38:44 +00:00
Owner

Performance improvements included in this PR:

  • Use Bytes at a few places where appropriate, instead of Vec<u8>, to reduce the number of copies
    • StreamChunker now accumulates incoming slices in a Vec<Bytes> instead of a VecDeque<u8>. Replaces calls to .extend() and .drain() that were quite costly by a simple concat() on a vec of slices which is much more optimized
  • Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using spawn_blocking
  • Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel
  • Better LMDB defaults: we now put flags NoSync and NoMetaSync to avoid fsync at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads

Other changes included in this PR:

  • Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12
  • switch to tracing_subscriber for logs, which allows to have timing of each event
Performance improvements included in this PR: - [x] Use `Bytes` at a few places where appropriate, instead of `Vec<u8>`, to reduce the number of copies - [x] StreamChunker now accumulates incoming slices in a `Vec<Bytes>` instead of a `VecDeque<u8>`. Replaces calls to `.extend()` and `.drain()` that were quite costly by a simple `concat()` on a vec of slices which is much more optimized - [x] Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using `spawn_blocking` - [x] Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel - [x] Better LMDB defaults: we now put flags `NoSync` and `NoMetaSync` to avoid `fsync` at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads Other changes included in this PR: - [x] Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12 - [x] switch to `tracing_subscriber` for logs, which allows to have timing of each event
lx added 17 commits 2022-07-29 10:24:54 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
continuous-integration/drone Build is passing Details
fcb04843f7
Run clippy in nix, leveraging nix caching ability
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
continuous-integration/drone Build is passing Details
a184f0d0b5
Migrate to nix-daemon builders
2f111e6b3d
Performance improvements:
- reduce contention on mutation_lock by having 256 of them
- better lmdb defaults
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
16f6a1a65d
fix clippy
lx force-pushed lx-perf-improvements from 1b3df6017d to ebc20a8798 2022-08-29 14:44:21 +00:00 Compare
lx added 1 commit 2022-08-29 14:48:38 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone Build is passing Details
4da67b0035
Update drone signature
lx added 1 commit 2022-08-31 12:24:54 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
dd5304f6fc
Replace logging crate pretty_env_logger by tracing_subscriber::fmt
lx added 1 commit 2022-08-31 12:28:28 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
44cd98d2e4
Tracing-subscriber: write to stderr
lx added 1 commit 2022-08-31 12:43:37 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
efbca67ce4
Add env filter to tracing subscriber
lx added 1 commit 2022-08-31 15:42:11 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
eb97e13a6a
update cargo.nix
lx added 1 commit 2022-09-02 12:33:23 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
6226f5ceca
Update to netapp 0.4.5 - fixed ping
lx added 1 commit 2022-09-05 10:40:26 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
07e6bcde85
Merge branch 'main' into lx-perf-improvements
lx added 1 commit 2022-09-06 19:53:47 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
d23b3a14fc
Merge branch 'main' into lx-perf-improvements
lx added 1 commit 2022-09-08 13:49:26 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
d9d199a6c9
Merge branch 'main' into lx-perf-improvements
lx changed title from WIP: performance improvements to performance improvements 2022-09-12 14:14:57 +00:00
lx added 1 commit 2022-09-12 14:24:05 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
f91fab8582
Simplify+improve async hasher by using bounded channel
lx merged commit 309d7aef3f into main 2022-09-12 14:38:44 +00:00
Sign in to join this conversation.
No description provided.