Background task manager #332

Merged

lx merged 35 commits from background-task-manager into main 2022-07-08 11:30:32 +00:00

Conversation 0 Commits 35 Files changed 27 +2050 -738

lx commented 2022-06-21 12:08:05 +00:00

Owner

Copy link

• New background worker trait
• Adapt all current workers to use new API
• Command to list currently running workers, and whether they are active, idle, or dead
• Error reporting
• Optimizations
  • Merkle updater: several items per iteration
  • Use tokio::task::spawn_blocking where appropriate so that CPU-intensive tasks don't block other things going on
• scrub:
  • have only one worker with a channel to start/pause/cancel
  • automatic scrub
  • ability to view and change tranquility from CLI
  • persistence of a few info
• Testing

- [x] New background worker trait - [x] Adapt all current workers to use new API - [x] Command to list currently running workers, and whether they are active, idle, or dead - [x] Error reporting - Optimizations - [x] Merkle updater: several items per iteration - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on - scrub: - [x] have only one worker with a channel to start/pause/cancel - [x] automatic scrub - [x] ability to view and change tranquility from CLI - [x] persistence of a few info - [ ] Testing

lx added 3 commits 2022-06-21 12:08:06 +00:00

First try on background worker manager e12bc3b595

New worker semantics applied to garage_table 3119ea59b0

Block manager with new worker 269f996fd0

lx added 1 commit 2022-06-21 12:24:11 +00:00

New worker for index counter propagator b8338dea56

lx added 1 commit 2022-06-21 13:28:12 +00:00

Online repair new workers, except blocks and scrub a855c54bdb

lx added 1 commit 2022-06-21 14:00:20 +00:00

Block repair with new worker semantics ba1ace6cf6

lx added 1 commit 2022-06-21 14:27:21 +00:00

rename things 14337d2a56

lx added 1 commit 2022-06-21 15:18:38 +00:00

Rename things, garage worker list cmd 10c886111e

lx added 1 commit 2022-06-21 15:22:26 +00:00

fix clippy 708dab6e0f

lx added 1 commit 2022-06-21 16:04:22 +00:00

fix clippy 8fb8569912

lx added 1 commit 2022-06-21 16:07:07 +00:00

Update Cargo.nix f82cf164f5

lx added 2 commits 2022-06-24 08:31:39 +00:00

(to test) error reporting and throttling at higher layer 59b43914d4

Error reporting 95ffba343f

lx added 2 commits 2022-06-24 09:05:22 +00:00

Slightly improve blocking code, add info to resync worker 0837b3dacd

Smaller batches for index counter propagation e7810e9cb3

lx added 1 commit 2022-06-27 09:58:36 +00:00

(makefile with run release) 928394cc32

lx added 1 commit 2022-06-27 10:01:36 +00:00

Fix clippy lint fdfe7dd60d

lx added 1 commit 2022-06-27 10:06:29 +00:00

Update cargo.nix fc50724256

lx added 1 commit 2022-06-27 14:53:12 +00:00

Report progress of scrub and block repair 0e5175abee

lx added 1 commit 2022-06-27 15:58:16 +00:00

Only one scrub worker (wip) 247dbcd598

lx added 1 commit 2022-06-28 14:09:45 +00:00

update cargo.nix f1c972289d

lx added 1 commit 2022-06-28 14:59:53 +00:00

Persist scrub worker thing in Persister b053fc0518

lx added 1 commit 2022-06-28 15:11:16 +00:00

Count corruptions in scrub worker b0a181e17e

lx added 1 commit 2022-06-28 15:31:36 +00:00

cargo fmt fc2bc8b0ca

lx added 1 commit 2022-06-28 15:43:07 +00:00

Small things 368414a261

lx added 1 commit 2022-06-28 15:44:04 +00:00

remove useless info!() c4be56704d

lx added 1 commit 2022-07-04 10:46:51 +00:00

Merge branch 'main' into background-task-manager 17d0254c7c

lx added 1 commit 2022-07-04 10:53:52 +00:00

add delays in k2v test_items_and_indices 5ee9cb7768

lx changed title from WIP: Background task manager to Background task manager 2022-07-04 11:03:04 +00:00

lx added 1 commit 2022-07-08 07:55:41 +00:00

cleanup 0200eae679

lx added 1 commit 2022-07-08 08:10:42 +00:00

Small refactoring 4cc9a648ab

lx added 1 commit 2022-07-08 08:16:56 +00:00

whoops and small refactoring 4312623930

lx added 1 commit 2022-07-08 08:32:51 +00:00

fix comments 0f660b086c

lx added 1 commit 2022-07-08 08:40:01 +00:00

Rename WorkerStatus to WorkerState ... d1cf1a0fa6

because it's a state in a state machine

lx added 4 commits 2022-07-08 08:43:29 +00:00

Fix poll item when item didn't change b6d59ec19a

Publish k2v-client crate to crates.io (#337) ... fe3fa83de7

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #337
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>

add delays in k2v test_items_and_indices aab34bfe54

Merge branch 'main' into background-task-manager 18d014dcc5

lx added 1 commit 2022-07-08 08:45:27 +00:00

cargo fmt 08cd5f2f1d

lx merged commit 4f38cadf6e into main 2022-07-08 11:30:32 +00:00

lx referenced this pull request from a commit 2022-07-08 11:30:32 +00:00

Background task manager (#332)

lx referenced this pull request 2022-09-14 11:12:42 +00:00

Improved monitoring of repair operations #207

lx referenced this pull request 2022-09-14 11:15:27 +00:00

Automatic scrub regularly #255

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

action

check-aws

The behaviour should be checked against AWS S3

  

action

discussion-needed

We need to discuss if the feature is desirable and/or better specify the envisioned behavior or technical implementation

  

action

for-external-contributors

Core team does not plan working on this in the near future however an external contribution will be reviewed & merged

  

action

for-newcomers

An issue that should be newcomer friendly.

  

action

more-info-needed

More information is needed to investigate this issue

  

action

need-funding

This issue represents lots of work, we need to find money to pay our core team work

  

action

triage-required

It's not very clear what we want to do with this issue yet, more thoughts is required

  

kind

correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

kind

ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

kind

improvement

Feature requests that do not fall in other categories

  

kind

performance

Deceptive performance observed OR idea to improve performances

  

kind

testing

Functional tests, integration tests, Jepsen, etc.

  

kind

usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

  

kind

wrong-behavior

Unintended behavior

  

prio

critical

Could impact integrity or availability of production clusters

  

prio

low

For issues that we don't want to spend time on until made absolutely necessary

  

scope

admin-api

Issues concerning the administration API

  

scope

background-healing

Background healing covers repair tasks, anti-entropy, merkle trees, etc. ; everything related to replicating data appropriately.

  

scope

build

Issues with Nix, Drone, Rust Compiler, etc.

  

scope

documentation

  

scope

k8s

Kubernetes related, including Helm Charts

  

scope

layout

Linked to the cluster layout and storage management

  

scope

metadata

Metadata Engine (LMDB, sqlite, etc.)

  

scope

ops

Operations related to operations/maintenance issue, tooling related (like CLI missing info ; impossible, too dangerous, too complicated migration, etc.)

  

scope

rpc

Linked to RPC

  

scope

s3-api

Related to the S3 API (especially compatibility issues).

  

scope

security

Cryptography-related, auth-related, authorization-related issues

  

scope

telemetry

OpenTelemetry (traces, metrics, etc.)

No labels

action

check-aws

action

discussion-needed

action

for-external-contributors

action

for-newcomers

action

more-info-needed

action

need-funding

action

triage-required

kind

correctness

kind

ideas

kind

improvement

kind

performance

kind

testing

kind

usability

kind

wrong-behavior

prio

critical

prio

low

scope

admin-api

scope

background-healing

scope

build

scope

documentation

scope

k8s

scope

layout

scope

metadata

scope

ops

scope

rpc

scope

s3-api

scope

security

scope

telemetry

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#332

No description provided.