Jepsen testing (NLnet task 3 subtask 1) #544

Merged

lx merged 41 commits from jepsen into main 2024-01-11 10:52:13 +00:00

Conversation 0 Commits 41 Files Changed 22 +1045

lx commented 2023-04-19 10:52:09 +00:00

Owner

Copy Link

• Connect Garage and Jepsen : cluster configuration, layout creation, S3 get/put/delete requests

• Verify GetObject/PutObject register properties

  • test reg : GetObject/PutObject does NOT act as a linearizable register, for at least the following two reasons:

    • timestamps were not strictly increasing (fixed in #543, not sure we want to merge?)
    • reads are not monotonic (will not fix)

  • Create a consistency model and a checker that correctly models the valid states for GetObject/PutObject, check if #543 changes something or not

• Verify PutObject/ListObjects set properties

  • test set1 : Jepsen's set test : items are written, time passes, a read occurs, the read contains all written items, check passes

  • test set2 : Jepsen's set-full test, check fails with linearizability contrainst, passes without

  • test set2 : custom checker for read-after-write in sets, check passes

• Introduce layout changes and check that things break (#495)

• Fix #495, validate fix

  • Final results: BEFORE PATCH, AFTER PATCH

  • Notes on Jepsen testing

Left for future work:

• Testing of K2V operations

- [x] Connect Garage and Jepsen : cluster configuration, layout creation, S3 get/put/delete requests - Verify GetObject/PutObject register properties - [x] test `reg` : GetObject/PutObject does NOT act as a linearizable register, for at least the following two reasons: - timestamps were not strictly increasing (fixed in #543, not sure we want to merge?) - reads are not monotonic (will not fix) - [x] Create a consistency model and a checker that correctly models the valid states for GetObject/PutObject, check if #543 changes something or not - Verify PutObject/ListObjects set properties - [x] test `set1` : Jepsen's `set` test : items are written, time passes, a read occurs, the read contains all written items, check passes - [x] test `set2` : Jepsen's `set-full` test, check fails with linearizability contrainst, passes without - [x] test `set2` : custom checker for read-after-write in sets, check passes - [x] Introduce layout changes and check that things break (#495) - [x] Fix #495, validate fix - Final results: [BEFORE PATCH](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/fa9247f11b89c960dffe82d6bf990ed4335788e3/script/jepsen.garage/results/Results-2023-12-13-tsfix2.png), [AFTER PATCH](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/fa9247f11b89c960dffe82d6bf990ed4335788e3/script/jepsen.garage/results/Results-2023-12-13-task3c.png) - [Notes on Jepsen testing](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/fa9247f11b89c960dffe82d6bf990ed4335788e3/script/jepsen.garage/README.md) Left for future work: - Testing of K2V operations

❤️ 1

lx added 7 commits 2023-04-19 10:52:10 +00:00

1f3d92fb84 jepsen: wip

71910ef109 jepsen ssh into containers seem to work ?

f4a6e78459 jepsen connects to vagrant vms

6dd7195d4e jepsen: s3 gets and puts

ab736be9f0 better match exceptions

7d0523883a even without nemesis, s3 get/put/delete is not linearizable (is this normal?)

cc5a677bac the fix for increasing timestamps does not make things linearizable

lx force-pushed jepsen from b633033164 to ce1cb543e8 2023-04-19 10:57:28 +00:00 Compare

lx added 1 commit 2023-04-19 12:14:32 +00:00

e95e2e0dfe refactor jepsen test

lx added 1 commit 2023-04-19 13:27:37 +00:00

2b70fe47e2 Add garage set test (this one works :p)

lx added 2 commits 2023-04-19 14:16:46 +00:00

4e0584a438 two set workloads with different checkers

70f042131f set tests with independant tests together

lx added 1 commit 2023-04-19 20:02:15 +00:00

a549122acd Check read-after-write property for sets

lx added 1 commit 2023-04-19 20:17:27 +00:00

6dcb9d31ff reformatting

lx added the

Correctness

label 2023-06-05 10:16:38 +00:00

lx added this to the v1.0 milestone 2023-06-14 12:23:44 +00:00

lx force-pushed jepsen from 6dcb9d31ff to 9003883e64 2023-10-18 10:00:54 +00:00 Compare

lx force-pushed jepsen from 9003883e64 to 9ec4cca334 2023-10-18 10:03:20 +00:00 Compare

lx added 1 commit 2023-10-18 12:01:29 +00:00

ef5ca86dfc jepsen: update to garage 0.9.0

lx added 2 commits 2023-10-18 13:34:23 +00:00

012ade5d4b jepsen: update jepsen and fix garage key info

84d43501ce refactor jepsen setup logic

lx added 1 commit 2023-10-18 14:30:53 +00:00

ddd3de7fce refactor jepsen code

lx added 1 commit 2023-10-18 15:51:54 +00:00

b3bf16ee27 make jepsen test more robust: handle errors and timeouts, fixed access key

lx added 2 commits 2023-10-19 14:45:38 +00:00

74e50edddd jepsen: refactoring

da8b170748 jepsen: investigating listobjects error

lx added 1 commit 2023-10-19 21:41:10 +00:00

ef662822c9 jepsen: fix the list-objects call (?)

lx added 1 commit 2023-10-20 10:13:27 +00:00

4ba18ce9cc jepsen: wip checker for register-like behavior

lx added 1 commit 2023-10-20 10:57:07 +00:00

4b93ce179a jepsen: errors in reg2 workload under investigation

lx added 1 commit 2023-10-20 11:37:07 +00:00

d148b83d4f jepsen: reg2 failure seems to happen only with deleteobject

lx added 1 commit 2023-10-20 13:00:26 +00:00

f5b0972781 jepsen: register crdt read-after-write is fixed with deleteobject patch

lx added 1 commit 2023-10-20 13:48:51 +00:00

654775308e jepsen: add cluster reconfiguration nemesis

lx added 1 commit 2023-10-20 13:53:57 +00:00

9030c1eef8 jepsen: code path for nemesis final generator

lx added 1 commit 2023-10-20 13:55:17 +00:00

fb6c9a1243 jepsen: update readme

lx added 1 commit 2023-10-24 09:40:01 +00:00

d2c365767b jepsen: more testing

lx added 1 commit 2023-10-24 13:44:18 +00:00

d13bde5e26 jepsen: set1 and set2 don't fail anymore ??

lx added 1 commit 2023-10-24 14:40:03 +00:00

d7ab2c639e jepsen: fix nemesis to actually generate many operations

lx added 1 commit 2023-10-24 15:45:34 +00:00

4fa2646a75 jepsen: got a failure with set1

lx added 3 commits 2023-10-25 10:13:42 +00:00

db921cc05f jepsen: reconfigure nemesis + add db nemesis

cfbfa09d24 jepsen: fix set2 test omg finally this is so stupid

fd85010a40 jepsen: failures with set2 test in --scenario r

lx added 1 commit 2023-10-25 12:05:06 +00:00

9df7fa0bcd jepsen: use 7 nodes

lx added 1 commit 2023-10-25 12:43:34 +00:00

5b1f50be65 jepsen: testing

lx added 1 commit 2023-11-16 11:57:45 +00:00

18e5811159

jepsen: add patch and use more complete names

lx changed title from WIP: Jepsen testing to WIP: Jepsen testing (NLnet task 3 subtask 1) 2023-11-16 15:43:28 +00:00

lx referenced this pull request 2023-11-16 15:44:48 +00:00

NLnet task 3 #667

lx added 1 commit 2023-11-16 17:09:28 +00:00

92dd2bbe15 jepsen: nlnet task3a seems to fix things

lx added 1 commit 2023-12-14 15:24:07 +00:00

fa9247f11b jepsen: updated results, confirming that task3 works

lx changed title from WIP: Jepsen testing (NLnet task 3 subtask 1) to Jepsen testing (NLnet task 3 subtask 1) 2024-01-11 10:51:56 +00:00

lx merged commit 723e56b37f into main 2024-01-11 10:52:13 +00:00

lx referenced this issue from a commit 2024-01-11 10:52:15 +00:00

Merge pull request 'Jepsen testing (NLnet task 3 subtask 1)' (#544) from jepsen into main

lx referenced this pull request 2024-01-11 11:03:30 +00:00

Garage v1.0 #683

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

AdminAPI

Issues concerning the administration API

  

Bug

  

Check AWS

The behaviour should be checked against AWS S3

  

CI

Issues with Nix, Drone, Rust Compiler, etc.

  

Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

Critical

  

Documentation

  

Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

Improvement

  

Low priority

For issues that we don't want to spend time on until made absolutely necessary

  

Newcomer

A bug that should be newcomer friendly.

  

Performance

  

S3 Compatibility

  

Testing

  

Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No Label

AdminAPI

Bug

Check AWS

CI

Correctness

Critical

Documentation

Ideas

Improvement

Low priority

Newcomer

Performance

S3 Compatibility

Testing

Usability

Milestone

Clear milestone

No items

No Milestone

v1.0

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#544

No description provided.