Some improvements to Garage internals #451

Merged

lx merged 14 commits from internals-rework into main 2023-01-03 11:37:32 +00:00

Conversation 0 Commits 14 Files Changed 27 +442 -559

lx commented 2022-12-14 11:01:37 +00:00

Owner

Copy Link

• Proper queueing for delayed inserts, now persisted to disk
• Launch background workers in separate step, so that it is not done for offline repair
• Get rid of background job worker and one-shot background spawn functions
• Simplified worker exit logic, make it more aggressive (expect all queues to be persisted)
• Online repair workers: retry on error
• Small refactorings

- [x] Proper queueing for delayed inserts, now persisted to disk - [x] Launch background workers in separate step, so that it is not done for offline repair - [x] Get rid of background job worker and one-shot background spawn functions - [x] Simplified worker exit logic, make it more aggressive (expect all queues to be persisted) - [x] Online repair workers: retry on error - [x] Small refactorings

lx added 2 commits 2022-12-14 11:01:37 +00:00

f8e528c15d

Small refactor of tables internals

83c8467e23

Proper queueing for delayed inserts, now backed to disk

lx added 2 commits 2022-12-14 11:51:49 +00:00

2183518edc

Spawn all background workers in a separate step

d56c472712

Refactor background runner and get rid of job worker

lx added 1 commit 2022-12-14 11:57:46 +00:00

a19bfef508

Improve error message on rpc connection failure

lx added 1 commit 2022-12-14 11:58:34 +00:00

0d6b05bb6c

Update cargo.nix

lx added 1 commit 2022-12-14 12:54:34 +00:00

d4af27f920

Add missing notify

lx added 1 commit 2022-12-14 14:25:46 +00:00

dfc131850a

Simplified and more aggressive worker exit logic

lx added 1 commit 2022-12-14 15:08:27 +00:00

510b620108

Get rid of background::spawn

lx added 1 commit 2022-12-14 15:11:29 +00:00

e6f14ab5cf

better error message handling

lx added 1 commit 2022-12-14 15:17:04 +00:00

13c8662126

factorize

lx added 1 commit 2022-12-14 15:31:51 +00:00

1fcd0b371b

online repair workers: retry on error

lx closed this pull request 2023-01-02 12:42:25 +00:00

lx reopened this pull request 2023-01-02 12:44:03 +00:00

lx changed target branch from cli-improvements to main 2023-01-02 12:44:11 +00:00

lx added 1 commit 2023-01-02 12:50:51 +00:00

6b857a9b8c

cargo fmt

lx added 1 commit 2023-01-02 14:07:50 +00:00

939a6d67e8

Merge branch 'main' into internals-rework

lx changed title from WIP: Some improvements to Garage internals to Some improvements to Garage internals 2023-01-03 11:37:24 +00:00

lx merged commit 582b076179 into main 2023-01-03 11:37:32 +00:00

lx referenced this issue from a commit 2023-01-03 11:37:32 +00:00

Merge pull request 'Some improvements to Garage internals' (#451) from internals-rework into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

AdminAPI

Issues concerning the administration API

  

Bug

  

Check AWS

The behaviour should be checked against AWS S3

  

CI

Issues with Nix, Drone, Rust Compiler, etc.

  

Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

Critical

  

Documentation

  

Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

Improvement

  

Low priority

For issues that we don't want to spend time on until made absolutely necessary

  

Newcomer

A bug that should be newcomer friendly.

  

Performance

  

S3 Compatibility

  

Testing

  

Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No Label

AdminAPI

Bug

Check AWS

CI

Correctness

Critical

Documentation

Ideas

Improvement

Low priority

Newcomer

Performance

S3 Compatibility

Testing

Usability

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#451

No description provided.