Jepsen testing (NLnet task 3 subtask 1) #544

Merged

lx merged 41 commits from jepsen into main 2024-01-11 10:52:13 +00:00

Conversation 0 Commits 41 Files changed 22 +1045

lx commented 2023-04-19 10:52:09 +00:00

Owner

Copy link

• Connect Garage and Jepsen : cluster configuration, layout creation, S3 get/put/delete requests

• Verify GetObject/PutObject register properties

  • test reg : GetObject/PutObject does NOT act as a linearizable register, for at least the following two reasons:

    • timestamps were not strictly increasing (fixed in #543, not sure we want to merge?)
    • reads are not monotonic (will not fix)

  • Create a consistency model and a checker that correctly models the valid states for GetObject/PutObject, check if #543 changes something or not

• Verify PutObject/ListObjects set properties

  • test set1 : Jepsen's set test : items are written, time passes, a read occurs, the read contains all written items, check passes

  • test set2 : Jepsen's set-full test, check fails with linearizability contrainst, passes without

  • test set2 : custom checker for read-after-write in sets, check passes

• Introduce layout changes and check that things break (#495)

• Fix #495, validate fix

  • Final results: BEFORE PATCH, AFTER PATCH

  • Notes on Jepsen testing

Left for future work:

• Testing of K2V operations

- [x] Connect Garage and Jepsen : cluster configuration, layout creation, S3 get/put/delete requests - Verify GetObject/PutObject register properties - [x] test `reg` : GetObject/PutObject does NOT act as a linearizable register, for at least the following two reasons: - timestamps were not strictly increasing (fixed in #543, not sure we want to merge?) - reads are not monotonic (will not fix) - [x] Create a consistency model and a checker that correctly models the valid states for GetObject/PutObject, check if #543 changes something or not - Verify PutObject/ListObjects set properties - [x] test `set1` : Jepsen's `set` test : items are written, time passes, a read occurs, the read contains all written items, check passes - [x] test `set2` : Jepsen's `set-full` test, check fails with linearizability contrainst, passes without - [x] test `set2` : custom checker for read-after-write in sets, check passes - [x] Introduce layout changes and check that things break (#495) - [x] Fix #495, validate fix - Final results: [BEFORE PATCH](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/fa9247f11b89c960dffe82d6bf990ed4335788e3/script/jepsen.garage/results/Results-2023-12-13-tsfix2.png), [AFTER PATCH](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/fa9247f11b89c960dffe82d6bf990ed4335788e3/script/jepsen.garage/results/Results-2023-12-13-task3c.png) - [Notes on Jepsen testing](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/fa9247f11b89c960dffe82d6bf990ed4335788e3/script/jepsen.garage/README.md) Left for future work: - Testing of K2V operations

❤️ 1

lx force-pushed jepsen from b633033164 to ce1cb543e8 2023-04-19 10:57:28 +00:00 Compare

lx added the

kind

correctness

label 2023-06-05 10:16:38 +00:00

lx added this to the v1.0 milestone 2023-06-14 12:23:44 +00:00

lx force-pushed jepsen from 6dcb9d31ff to 9003883e64 2023-10-18 10:00:54 +00:00 Compare

lx force-pushed jepsen from 9003883e64 to 9ec4cca334 2023-10-18 10:03:20 +00:00 Compare

lx added 1 commit 2023-10-18 12:01:29 +00:00

jepsen: update to garage 0.9.0 ef5ca86dfc

lx added 2 commits 2023-10-18 13:34:23 +00:00

jepsen: update jepsen and fix garage key info 012ade5d4b

refactor jepsen setup logic 84d43501ce

lx added 1 commit 2023-10-18 14:30:53 +00:00

refactor jepsen code ddd3de7fce

lx added 1 commit 2023-10-18 15:51:54 +00:00

make jepsen test more robust: handle errors and timeouts, fixed access key b3bf16ee27

lx added 2 commits 2023-10-19 14:45:38 +00:00

jepsen: refactoring 74e50edddd

jepsen: investigating listobjects error da8b170748

lx added 1 commit 2023-10-19 21:41:10 +00:00

jepsen: fix the list-objects call (?) ef662822c9

lx added 1 commit 2023-10-20 10:13:27 +00:00

jepsen: wip checker for register-like behavior 4ba18ce9cc

lx added 1 commit 2023-10-20 10:57:07 +00:00

jepsen: errors in reg2 workload under investigation 4b93ce179a

lx added 1 commit 2023-10-20 11:37:07 +00:00

jepsen: reg2 failure seems to happen only with deleteobject d148b83d4f

lx added 1 commit 2023-10-20 13:00:26 +00:00

jepsen: register crdt read-after-write is fixed with deleteobject patch f5b0972781

lx added 1 commit 2023-10-20 13:48:51 +00:00

jepsen: add cluster reconfiguration nemesis 654775308e

lx added 1 commit 2023-10-20 13:53:57 +00:00

jepsen: code path for nemesis final generator 9030c1eef8

lx added 1 commit 2023-10-20 13:55:17 +00:00

jepsen: update readme fb6c9a1243

lx added 1 commit 2023-10-24 09:40:01 +00:00

jepsen: more testing d2c365767b

lx added 1 commit 2023-10-24 13:44:18 +00:00

jepsen: set1 and set2 don't fail anymore ?? d13bde5e26

lx added 1 commit 2023-10-24 14:40:03 +00:00

jepsen: fix nemesis to actually generate many operations d7ab2c639e

lx added 1 commit 2023-10-24 15:45:34 +00:00

jepsen: got a failure with set1 4fa2646a75

lx added 3 commits 2023-10-25 10:13:42 +00:00

jepsen: reconfigure nemesis + add db nemesis db921cc05f

jepsen: fix set2 test omg finally this is so stupid cfbfa09d24

jepsen: failures with set2 test in --scenario r fd85010a40

lx added 1 commit 2023-10-25 12:05:06 +00:00

jepsen: use 7 nodes 9df7fa0bcd

lx added 1 commit 2023-10-25 12:43:34 +00:00

jepsen: testing 5b1f50be65

lx added 1 commit 2023-11-16 11:57:45 +00:00

jepsen: add patch and use more complete names 18e5811159

lx changed title from WIP: Jepsen testing to WIP: Jepsen testing (NLnet task 3 subtask 1) 2023-11-16 15:43:28 +00:00

lx referenced this pull request 2023-11-16 15:44:48 +00:00

NLnet task 3 #667

lx added 1 commit 2023-11-16 17:09:28 +00:00

jepsen: nlnet task3a seems to fix things 92dd2bbe15

lx added 1 commit 2023-12-14 15:24:07 +00:00

jepsen: updated results, confirming that task3 works fa9247f11b

lx changed title from WIP: Jepsen testing (NLnet task 3 subtask 1) to Jepsen testing (NLnet task 3 subtask 1) 2024-01-11 10:51:56 +00:00

lx merged commit 723e56b37f into main 2024-01-11 10:52:13 +00:00

lx referenced this pull request from a commit 2024-01-11 10:52:15 +00:00

Merge pull request 'Jepsen testing (NLnet task 3 subtask 1)' (#544) from jepsen into main

lx referenced this pull request 2024-01-11 11:03:30 +00:00

Garage v1.0 #683

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

action

check-aws

The behaviour should be checked against AWS S3

  

action

discussion-needed

We need to discuss if the feature is desirable and/or better specify the envisioned behavior or technical implementation

  

action

for-external-contributors

Core team does not plan working on this in the near future however an external contribution will be reviewed & merged

  

action

for-newcomers

An issue that should be newcomer friendly.

  

action

more-info-needed

More information is needed to investigate this issue

  

action

need-funding

This issue represents lots of work, we need to find money to pay our core team work

  

action

triage-required

It's not very clear what we want to do with this issue yet, more thoughts is required

  

kind

correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

kind

ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

kind

improvement

Feature requests that do not fall in other categories

  

kind

performance

Deceptive performance observed OR idea to improve performances

  

kind

testing

Functional tests, integration tests, Jepsen, etc.

  

kind

usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

  

kind

wrong-behavior

Unintended behavior

  

prio

critical

Could impact integrity or availability of production clusters

  

prio

low

For issues that we don't want to spend time on until made absolutely necessary

  

scope

admin-api

Issues concerning the administration API

  

scope

background-healing

Background healing covers repair tasks, anti-entropy, merkle trees, etc. ; everything related to replicating data appropriately.

  

scope

build

Issues with Nix, Drone, Rust Compiler, etc.

  

scope

documentation

  

scope

k8s

Kubernetes related, including Helm Charts

  

scope

layout

Linked to the cluster layout and storage management

  

scope

metadata

Metadata Engine (LMDB, sqlite, etc.)

  

scope

ops

Operations related to operations/maintenance issue, tooling related (like CLI missing info ; impossible, too dangerous, too complicated migration, etc.)

  

scope

rpc

Linked to RPC

  

scope

s3-api

Related to the S3 API (especially compatibility issues).

  

scope

security

Cryptography-related, auth-related, authorization-related issues

  

scope

telemetry

OpenTelemetry (traces, metrics, etc.)

No labels

action

check-aws

action

discussion-needed

action

for-external-contributors

action

for-newcomers

action

more-info-needed

action

need-funding

action

triage-required

kind

correctness

kind

ideas

kind

improvement

kind

performance

kind

testing

kind

usability

kind

wrong-behavior

prio

critical

prio

low

scope

admin-api

scope

background-healing

scope

build

scope

documentation

scope

k8s

scope

layout

scope

metadata

scope

ops

scope

rpc

scope

s3-api

scope

security

scope

telemetry

Milestone

Clear milestone

No items

No milestone

v1.0

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#544

No description provided.