performance improvements #342

Merged

lx merged 21 commits from lx-perf-improvements into main 2022-09-12 14:38:44 +00:00

Conversation 0 Commits 21 Files changed 16 +654 -318

lx commented 2022-07-18 16:44:50 +00:00

Owner

Copy link

Performance improvements included in this PR:

• Use Bytes at a few places where appropriate, instead of Vec<u8>, to reduce the number of copies
  • StreamChunker now accumulates incoming slices in a Vec<Bytes> instead of a VecDeque<u8>. Replaces calls to .extend() and .drain() that were quite costly by a simple concat() on a vec of slices which is much more optimized
• Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using spawn_blocking
• Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel
• Better LMDB defaults: we now put flags NoSync and NoMetaSync to avoid fsync at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads

Other changes included in this PR:

• Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12
• switch to tracing_subscriber for logs, which allows to have timing of each event

Performance improvements included in this PR: - [x] Use `Bytes` at a few places where appropriate, instead of `Vec<u8>`, to reduce the number of copies - [x] StreamChunker now accumulates incoming slices in a `Vec<Bytes>` instead of a `VecDeque<u8>`. Replaces calls to `.extend()` and `.drain()` that were quite costly by a simple `concat()` on a vec of slices which is much more optimized - [x] Hashing (b2, sha256, md5) is now done on a Tokio thread dedicated to cpu-intensive tasks, using `spawn_blocking` - [x] Block manager now uses 256 independant locks instead of one big lock for writing, reduces contention when writing several/many objects in parallel - [x] Better LMDB defaults: we now put flags `NoSync` and `NoMetaSync` to avoid `fsync` at each transaction (extremely slow). Also increased number of LMDB readers to accomodate more intensive workloads Other changes included in this PR: - [x] Update to hashing and MAC crates: md5 and sha2 from 0.9 to 0.10, hmac from 0.10 to 0.12 - [x] switch to `tracing_subscriber` for logs, which allows to have timing of each event

lx added 17 commits 2022-07-29 10:24:54 +00:00

Fail if compiled binary is dynamic 76cb34a0ae

Put log-lines in nix.conf 9c9e483375

Fix: compile aarch64+armv6 as static binaries a49d0ea19f

Bump Nix image to 22.05 96561c48a1

Refactor default.nix to follow Nix Flakes patterns 5fb8584247

Run clippy in nix, leveraging nix caching ability fcb04843f7

Migrate to nix-daemon builders a184f0d0b5

Compute hashes on dedicated threads 1b2e1296eb

Performance improvements: ... 2f111e6b3d

- reduce contention on mutation_lock by having 256 of them
- better lmdb defaults

Update cargo.nix 40150527b8

Make clippy happy 0176da3ad2

More make clippy happy 2cad656a03

Fix tests 381eb9a5a1

cargo fmt ff4771c36a

Update cargo.nix 49154a78d8

Faster chunker ad35b18bb1

fix clippy 16f6a1a65d

lx force-pushed lx-perf-improvements from 1b3df6017d to ebc20a8798 2022-08-29 14:44:21 +00:00 Compare

lx added 1 commit 2022-08-29 14:48:38 +00:00

Update drone signature 4da67b0035

lx added 1 commit 2022-08-31 12:24:54 +00:00

Replace logging crate pretty_env_logger by tracing_subscriber::fmt dd5304f6fc

lx added 1 commit 2022-08-31 12:28:28 +00:00

Tracing-subscriber: write to stderr 44cd98d2e4

lx added 1 commit 2022-08-31 12:43:37 +00:00

Add env filter to tracing subscriber efbca67ce4

lx added 1 commit 2022-08-31 15:42:11 +00:00

update cargo.nix eb97e13a6a

lx added 1 commit 2022-09-02 12:33:23 +00:00

Update to netapp 0.4.5 - fixed ping 6226f5ceca

lx added 1 commit 2022-09-05 10:40:26 +00:00

Merge branch 'main' into lx-perf-improvements 07e6bcde85

lx added 1 commit 2022-09-06 19:53:47 +00:00

Merge branch 'main' into lx-perf-improvements d23b3a14fc

lx added 1 commit 2022-09-08 13:49:26 +00:00

Merge branch 'main' into lx-perf-improvements d9d199a6c9

lx changed title from WIP: performance improvements to performance improvements 2022-09-12 14:14:57 +00:00

lx added 1 commit 2022-09-12 14:24:05 +00:00

Simplify+improve async hasher by using bounded channel f91fab8582

lx merged commit 309d7aef3f into main 2022-09-12 14:38:44 +00:00

lx referenced this pull request from a commit 2022-09-12 14:38:44 +00:00

Merge pull request 'performance improvements' (#342) from lx-perf-improvements into main

lx referenced this pull request 2022-09-14 11:20:46 +00:00

Poor I/O performance when writing to HDD #288

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

action

check-aws

The behaviour should be checked against AWS S3

  

action

discussion-needed

We need to discuss if the feature is desirable and/or better specify the envisioned behavior or technical implementation

  

action

for-external-contributors

Core team does not plan working on this in the near future however an external contribution will be reviewed & merged

  

action

for-newcomers

An issue that should be newcomer friendly.

  

action

more-info-needed

More information is needed to investigate this issue

  

action

need-funding

This issue represents lots of work, we need to find money to pay our core team work

  

action

triage-required

It's not very clear what we want to do with this issue yet, more thoughts is required

  

kind

correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

kind

ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

kind

improvement

Feature requests that do not fall in other categories

  

kind

performance

Deceptive performance observed OR idea to improve performances

  

kind

testing

Functional tests, integration tests, Jepsen, etc.

  

kind

usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

  

kind

wrong-behavior

Unintended behavior

  

prio

critical

Could impact integrity or availability of production clusters

  

prio

low

For issues that we don't want to spend time on until made absolutely necessary

  

scope

admin-api

Issues concerning the administration API

  

scope

background-healing

Background healing covers repair tasks, anti-entropy, merkle trees, etc. ; everything related to replicating data appropriately.

  

scope

build

Issues with Nix, Drone, Rust Compiler, etc.

  

scope

documentation

  

scope

k8s

Kubernetes related, including Helm Charts

  

scope

layout

Linked to the cluster layout and storage management

  

scope

metadata

Metadata Engine (LMDB, sqlite, etc.)

  

scope

ops

Operations related to operations/maintenance issue, tooling related (like CLI missing info ; impossible, too dangerous, too complicated migration, etc.)

  

scope

rpc

Linked to RPC

  

scope

s3-api

Related to the S3 API (especially compatibility issues).

  

scope

security

Cryptography-related, auth-related, authorization-related issues

  

scope

telemetry

OpenTelemetry (traces, metrics, etc.)

No labels

action

check-aws

action

discussion-needed

action

for-external-contributors

action

for-newcomers

action

more-info-needed

action

need-funding

action

triage-required

kind

correctness

kind

ideas

kind

improvement

kind

performance

kind

testing

kind

usability

kind

wrong-behavior

prio

critical

prio

low

scope

admin-api

scope

background-healing

scope

build

scope

documentation

scope

k8s

scope

layout

scope

metadata

scope

ops

scope

rpc

scope

s3-api

scope

security

scope

telemetry

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#342

No description provided.