NLnet task 3 #667

Merged

lx merged 60 commits from nlnet-task3 into next-0.10 2024-01-11 10:58:08 +00:00

Conversation 1 Commits 60 Files changed 41 +3284 -1668

lx commented 2023-11-08 14:42:02 +00:00

Owner

Copy link

This PR tracks the progress of NLnet task 3, fixing #495 - read-after-write consistency when layout changes.

Subtask 1 (Jepsen testing): See PR #544 for details.

• Jepsen testing was achieved
  • Without layout changes, Jepsen found no consistency anomalies
  • When introducing layout changes, anomalies began to occur

Subtask 2 (write and prove solution):

• Think of a solution.
• Write down some notes on the solution and sketch a proof.

Subtask 3 (implementation): Mostly in this PR:

• Change layout data structure to keep track of layout history and update trackers (that track the sync status of all nodes in the cluster)
• Naively adapt existing code to new layout data structures
• Add layout manager that handles update trackers
• Adapt the resync procedure to take into account the layout history, and to correctly update the update trackers
• Adapt the table read/write functions to access correct nodes given current layout history and values of update trakcers
  • Table::insert
  • Table::insert_many
  • BlockManager::rpc_put_block
• Layout ACK locking while writes are in progress
• Make sure data blocks also can still be found, since block moving is not taken into account in sync_map
• Caching of current ack and sync values in layout history / update tracker
• Adapt System::health()
• Update the CLI to make info from the update trackers visible
• Update the admin API as well
• Add a command to consider a node or set of nodes "dead" and forcibly advance the update trackers for that node

--

FIXES:

• gateways must also ack!
• improve recovery procedure when nodes are unavailable, by allowing individual node sync operations to be marked as completed in sync_map even if not all sync targets were completed successfully, as long as a quorum among all write sets was achieved
• similarly, ack_sync can progress even if not all nodes have finished a sync, as long as a quorum among all write sets of all partitions was achieved

--

• Hand testing
• Proof-reading

LEFT FOR FUTURE PRs:

• Add garage layout skip-dead-nodes equivalent to admin API?
• ack_map could be forcibly increased after a certain delay (15 minutes?), to avoid user having to call garage layout skip-dead-nodes when some nodes are unavailable
• Can GC be improved to benefit from the update trackers? See #151. Also just fix GC that is probably broken (but always has been?)
• Improve/fix/refactor K2V RPCs (for insertion and for polling)

Subtask 4 (re-testing): WIP, some is also in PR #544

• Jepsen

  • Final results: BEFORE PATCH, AFTER PATCH

This PR tracks the progress of NLnet task 3, fixing #495 - read-after-write consistency when layout changes. **Subtask 1 (Jepsen testing):** See PR #544 for details. - [x] Jepsen testing was achieved - [x] Without layout changes, Jepsen found no consistency anomalies - [x] When introducing layout changes, anomalies began to occur **Subtask 2 (write and prove solution):** - [x] Think of a solution. - [x] Write down some notes on the solution and sketch a proof. **Subtask 3 (implementation):** Mostly in this PR: - [x] Change layout data structure to keep track of layout history and update trackers (that track the sync status of all nodes in the cluster) - [x] Naively adapt existing code to new layout data structures - [x] Add layout manager that handles update trackers - [x] Adapt the resync procedure to take into account the layout history, and to correctly update the update trackers - [x] Adapt the table read/write functions to access correct nodes given current layout history and values of update trakcers - [x] Table::insert - [x] Table::insert_many - [x] BlockManager::rpc_put_block - [x] Layout ACK locking while writes are in progress - [x] Make sure data blocks also can still be found, since block moving is not taken into account in sync_map - [x] Caching of current ack and sync values in layout history / update tracker - [x] Adapt `System::health()` - [x] Update the CLI to make info from the update trackers visible - [x] Update the admin API as well - [x] Add a command to consider a node or set of nodes "dead" and forcibly advance the update trackers for that node -- FIXES: - [x] gateways must also ack! - [x] improve recovery procedure when nodes are unavailable, by allowing individual node sync operations to be marked as completed in sync_map even if not all sync targets were completed successfully, as long as a quorum among all write sets was achieved - [x] similarly, ack_sync can progress even if not all nodes have finished a sync, as long as a quorum among all write sets of all partitions was achieved -- - [x] Hand testing - [x] Proof-reading LEFT FOR FUTURE PRs: - Add `garage layout skip-dead-nodes` equivalent to admin API? - ack_map could be forcibly increased after a certain delay (15 minutes?), to avoid user having to call `garage layout skip-dead-nodes` when some nodes are unavailable - Can GC be improved to benefit from the update trackers? See #151. Also just fix GC that is probably broken (but always has been?) - Improve/fix/refactor K2V RPCs (for insertion and for polling) **Subtask 4 (re-testing):** WIP, some is also in PR #544 - [x] Jepsen - Final results: [BEFORE PATCH](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/fa9247f11b89c960dffe82d6bf990ed4335788e3/script/jepsen.garage/results/Results-2023-12-13-tsfix2.png), [AFTER PATCH](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/fa9247f11b89c960dffe82d6bf990ed4335788e3/script/jepsen.garage/results/Results-2023-12-13-task3c.png)

lx added 2 commits 2023-11-08 14:42:02 +00:00

garage_rpc: reorder functions in layout.rs 0962313ebd

remove Ring and use ClusterLayout everywhere 12d1dbfc6b

lx added 1 commit 2023-11-08 15:41:18 +00:00

avoid using layout_watch in System directly 4a9c94514f

lx added 2 commits 2023-11-08 18:28:56 +00:00

WIP: garage_rpc: store layout version history fe9af1dcaa

cluster layout: adapt all uses of ClusterLayout to LayoutHistory 8dccee3ccf

lx added 1 commit 2023-11-08 18:31:17 +00:00

bump garage protocol version tag to 0x000A (0.10) 1da0a5676e

lx added 1 commit 2023-11-09 10:20:06 +00:00

layout: use separate CRDT for staged layout changes 523d2ecb95

lx added 1 commit 2023-11-09 11:55:58 +00:00

wip: split out layout management from System into separate LayoutManager 8a2b1dd422

lx added 2 commits 2023-11-09 13:12:22 +00:00

layout: more refactoring 19ef1ec8e7

layout: refactor to use a RwLock on LayoutHistory bfb1845fdc

lx added 1 commit 2023-11-09 13:53:54 +00:00

layout: separate code path for synchronizing update trackers only 94caf9c0c1

lx added 1 commit 2023-11-09 14:32:17 +00:00

layout: begin managing the update tracker values 03ebf18830

lx added 1 commit 2023-11-09 14:42:29 +00:00

layout admin: add missing calls to update_hash bad7cc812e

lx added 1 commit 2023-11-09 14:53:00 +00:00

layout: fix tracker bugs 9d95f6f704

lx changed title from WIP: NLnet task 3 to WIP: NLnet task 3 subtask 3 (implementation) 2023-11-09 14:54:11 +00:00

lx added 1 commit 2023-11-09 15:32:50 +00:00

layout: add helpers to LayoutHistory and prepare integration with Table df36cf3099

lx added 1 commit 2023-11-11 11:08:56 +00:00

table sync: adapt to new layout history ce89d1ddab

lx force-pushed nlnet-task3 from 0f319a1c04 to df24bb806d 2023-11-11 11:44:34 +00:00 Compare

lx added 1 commit 2023-11-11 12:11:16 +00:00

layout: fix test 9a491fa137

lx added 1 commit 2023-11-14 11:48:56 +00:00

layout: some refactoring of nongateway nodes 8e292e06b3

lx force-pushed nlnet-task3 from 74109c699e to 1aab1f4e68 2023-11-14 12:12:41 +00:00 Compare

lx added 1 commit 2023-11-14 12:29:40 +00:00

layout: fixes in schema 83a11374ca

lx added 2 commits 2023-11-14 13:28:36 +00:00

system: add todo wrt new layout 866196750f

layout: prepare for write sets 3b361d2959

lx added 1 commit 2023-11-14 14:41:13 +00:00

table: take into account multiple write sets in inserts 90e1619b1e

lx added 1 commit 2023-11-14 14:45:09 +00:00

layout: fix test 7ef2c23120

lx added 1 commit 2023-11-15 11:16:24 +00:00

layout history merge: rm invalid versions when valid versions are added b3e729f4b8

lx added 2 commits 2023-11-15 12:07:56 +00:00

integration test: print stdout and stderr on subcommand crash 46007bf01d

rpc: fix write set quorums acd49de9f9

lx added 2 commits 2023-11-15 13:21:24 +00:00

layout: wip cache global mins 65066c7064

layout: add helper for cached/external values to centralize recomputation 393c4d4515

lx added 1 commit 2023-11-15 14:40:58 +00:00

layou: implement ack locking 33c8a489b0

lx added 1 commit 2023-11-15 14:57:10 +00:00

layout: fix test d4df03424f

lx added 1 commit 2023-11-16 12:27:04 +00:00

layout: split helper in separate file; more precise difference tracking ad5c6f779f

lx added 1 commit 2023-11-16 12:52:09 +00:00

layout: refactor digests and add "!=" assertions before epidemic bcast 707442f5de

lx added 2 commits 2023-11-16 15:42:00 +00:00

rpc_helper: don't use tokio::spawn for individual requests 22f38808e7

table: implement write sets for insert_many 3ecd14b9f6

lx changed title from WIP: NLnet task 3 subtask 3 (implementation) to WIP: NLnet task 3 2023-11-16 15:43:39 +00:00

lx added 1 commit 2023-11-27 10:53:32 +00:00

block manager: read_block using old layout versions if necessary d6d239fc79

lx added 1 commit 2023-11-27 11:10:46 +00:00

rpc: update system::health to take into account write sets for all partitions 78362140f5

lx added 1 commit 2023-11-27 12:19:18 +00:00

cli: show when nodes are draining metadata 539a920313

lx force-pushed nlnet-task3 from a081e74bd7 to c539077d30 2023-11-27 15:22:34 +00:00 Compare

lx added 1 commit 2023-11-28 10:13:01 +00:00

rpc helper: write comments + small refactoring of tracing 539af6eac4

lx added 1 commit 2023-11-28 13:25:21 +00:00

admin: more info in admin GetClusterStatus c04dd8788a

lx added 1 commit 2023-12-07 09:30:52 +00:00

layout updates: fix the set of nodes among which minima are calculated c8356a91d9

lx force-pushed nlnet-task3 from 9358658cc5 to 95eb13eb08 2023-12-07 09:57:29 +00:00 Compare

lx added 1 commit 2023-12-07 10:27:47 +00:00

table sync: use write quorums to report global success or failure of sync d90de365b3

lx added 1 commit 2023-12-07 10:56:22 +00:00

layout cli: safer skip-dead-nodes command aa59059a91

lx force-pushed nlnet-task3 from 2b93daa78d to 9cecea64d4 2023-12-07 13:51:31 +00:00 Compare

lx added 1 commit 2023-12-07 14:16:15 +00:00

fix build with discovery features 431b28e0cf

lx added 1 commit 2023-12-08 09:36:53 +00:00

rpc: fix system::health 91b874c4ef

lx added 1 commit 2023-12-08 10:24:50 +00:00

cli: improvements to the layout commands when multiple layouts are live 7f2541101f

lx added 1 commit 2023-12-08 10:51:13 +00:00

layout version: refactor get_node_zone 063294dd56

lx added 1 commit 2023-12-08 11:03:07 +00:00

layout: move block_read_nodes_of to rpc_helper to avoid double-locking ... 5dd200c015

(in theory, this could have caused a deadlock)

lx added 1 commit 2023-12-08 11:18:26 +00:00

rpc helper: small refactorings 64a6e557a4

lx added 1 commit 2023-12-08 13:16:06 +00:00

layout: refactoring, merge two files 4dbf254512

lx added 1 commit 2023-12-08 13:54:32 +00:00

table: fix insert_many to not send duplicates f8df90b79b

lx added 1 commit 2023-12-11 13:58:07 +00:00

table: remove redundant tracing in insert_many e4f493b481

lx added 1 commit 2023-12-11 14:32:03 +00:00

fix some clippy lints 85b5a6bcd1

lx added 1 commit 2023-12-11 14:46:50 +00:00

layout: refactor/fix bad while loop adccce1145

lx added 1 commit 2023-12-11 15:09:47 +00:00

layout: refactoring and fix in layout helper 0041b013a4

lx changed target branch from main to next-0.10 2024-01-11 10:57:14 +00:00

lx changed title from WIP: NLnet task 3 to NLnet task 3 2024-01-11 10:57:29 +00:00

lx merged commit 8a6ec1d611 into next-0.10 2024-01-11 10:58:08 +00:00

lx deleted branch nlnet-task3 2024-01-11 10:58:09 +00:00

lx referenced this pull request from a commit 2024-01-11 10:58:10 +00:00

Merge pull request 'NLnet task 3' (#667) from nlnet-task3 into next-0.10

lx referenced this pull request 2024-01-11 11:03:30 +00:00

Garage v1.0 #683

veretcle commented 2024-10-18 13:59:48 +00:00

First-time contributor

Copy link

Add garage layout skip-dead-nodes equivalent to admin API?

Is there any plan to effectively add this API endpoint?

> Add garage layout skip-dead-nodes equivalent to admin API? Is there any plan to effectively add this API endpoint?

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

action

check-aws

The behaviour should be checked against AWS S3

  

action

discussion-needed

We need to discuss if the feature is desirable and/or better specify the envisioned behavior or technical implementation

  

action

for-external-contributors

Core team does not plan working on this in the near future however an external contribution will be reviewed & merged

  

action

for-newcomers

An issue that should be newcomer friendly.

  

action

more-info-needed

More information is needed to investigate this issue

  

action

need-funding

This issue represents lots of work, we need to find money to pay our core team work

  

action

triage-required

It's not very clear what we want to do with this issue yet, more thoughts is required

  

kind

correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

kind

ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

kind

improvement

Feature requests that do not fall in other categories

  

kind

performance

Deceptive performance observed OR idea to improve performances

  

kind

testing

Functional tests, integration tests, Jepsen, etc.

  

kind

usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

  

kind

wrong-behavior

Unintended behavior

  

prio

critical

Could impact integrity or availability of production clusters

  

prio

low

For issues that we don't want to spend time on until made absolutely necessary

  

scope

admin-api

Issues concerning the administration API

  

scope

background-healing

Background healing covers repair tasks, anti-entropy, merkle trees, etc. ; everything related to replicating data appropriately.

  

scope

build

Issues with Nix, Drone, Rust Compiler, etc.

  

scope

documentation

  

scope

k8s

Kubernetes related, including Helm Charts

  

scope

layout

Linked to the cluster layout and storage management

  

scope

metadata

Metadata Engine (LMDB, sqlite, etc.)

  

scope

ops

Operations related to operations/maintenance issue, tooling related (like CLI missing info ; impossible, too dangerous, too complicated migration, etc.)

  

scope

rpc

Linked to RPC

  

scope

s3-api

Related to the S3 API (especially compatibility issues).

  

scope

security

Cryptography-related, auth-related, authorization-related issues

  

scope

telemetry

OpenTelemetry (traces, metrics, etc.)

No labels

action

check-aws

action

discussion-needed

action

for-external-contributors

action

for-newcomers

action

more-info-needed

action

need-funding

action

triage-required

kind

correctness

kind

ideas

kind

improvement

kind

performance

kind

testing

kind

usability

kind

wrong-behavior

prio

critical

prio

low

scope

admin-api

scope

background-healing

scope

build

scope

documentation

scope

k8s

scope

layout

scope

metadata

scope

ops

scope

rpc

scope

s3-api

scope

security

scope

telemetry

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#667

No description provided.