5.7 KiB
jepsen.garage
Jepsen checking of Garage consistency properties.
Usage
Requirements:
- vagrant
- VirtualBox, configured so that nodes can take an IP in a private network
192.168.56.0/24 - a user that can create VirtualBox VMs
- leiningen
- gnuplot
Set up VMs:
vagrant up
Run tests (this one should fail):
lein run test --nodes-file nodes.vagrant --time-limit 64 --concurrency 50 --rate 50 --workload reg
These ones are working:
lein run test --nodes-file nodes.vagrant --time-limit 64 --rate 50 --concurrency 50 --workload set1
lein run test --nodes-file nodes.vagrant --time-limit 64 --rate 50 --concurrency 50 --workload set2
Results
Register linear, without timestamp patch
Command: lein run test --nodes-file nodes.vagrant --time-limit 60 --rate 20 --concurrency 20 --workload reg1 --ops-per-key 100
Results: fails with a simple clock-scramble nemesis.
Explanation: without the timestamp patch, nodes will create objects using their local clock only as a timestamp, so the ordering will be all over the place if clocks are scrambled.
Register linear, with timestamp patch
Command: lein run test --nodes-file nodes.vagrant --time-limit 60 --rate 20 --concurrency 20 --workload reg1 --ops-per-key 100 --patch tsfix1
Results:
- No failure with clock-scramble nemesis
- Fails with clock-scramble nemesis + partition nemesis
Explanation: S3 objects are not meant to behave like linearizable registers. TODO explain using a counter-example
Read-after-write CRDT register model, without timestamp patch
Command: lein run test --nodes-file nodes.vagrant --time-limit 60 --rate 100 --concurrency 100 --workload reg2 --ops-per-key 100
Results: fails with a simple clock-scramble nemesis.
Explanation: old values are not overwritten correctly when their timestamps are in the future.
Read-after-write CRDT register model, with timestamp patch (v2 with DeleteObject fix as well)
Command: lein run test --nodes-file nodes.vagrant --time-limit 60 --rate 100 --concurrency 100 --workload reg2 --ops-per-key 100 --patch tsfix2
Results:
- No failures with clock-scramble nemesis + partition nemesis
- Fails with layout reconfiguration nemesis (TODO: test more and investigate)
Set, basic test (write some items, then read)
Command: lein run test --nodes-file nodes.vagrant --time-limit 60 --rate 100 --concurrency 100 --workload set1 --ops-per-key 100
Results:
- For now, no failures with clock-scramble nemesis + partition nemesis
- TODO: layout reconfiguration nemesis (does not fail yet! but it should)
Set, continuous test (interspersed reads and writes)
TODO
Investigating (and fixing) errors
Segfaults
They are due to the download being interrupted in the middle (^C during first launch on clean VMs), the garage binary is truncated.
Add :force? to the cached-wget! call in daemon.clj to re-download the binary.
In jepsen.garage: prefix wierdness
In store/garage set1/20231019T163358.615+0200:
INFO [2023-10-19 16:35:20,977] clojure-agent-send-off-pool-207 - jepsen.garage.set list results for prefix set20/ : (set13/0 set13/1 set13/10 set13/11 set13/12 set13/13 set13/14 set13/15 set13/16 set13/17 set13/18 set13/19 set13/2 set13/20 set13/21 set13/22 set13/23 set13/24 set13/25 set13/26 set13/27 set13/28 set13/29 set13/3 set13/30 set13/31 set13/32 set13/33 set13/34 set13/35 set13/36 set13/37 set13/38 set13/39 set13/4 set13/40 set13/41 set13/42 set13/43 set13/44 set13/45 set13/46 set13/47 set13/48 set13/49 set13/5 set13/50 set13/51 set13/52 set13/53 set13/54 set13/55 set13/56 set13/57 set13/58 set13/59 set13/6 set13/60 set13/61 set13/62 set13/63 set13/64 set13/65 set13/66 set13/67 set13/68 set13/69 set13/7 set13/70 set13/71 set13/72 set13/73 set13/74 set13/75 set13/76 set13/77 set13/78 set13/79 set13/8 set13/80 set13/81 set13/82 set13/83 set13/84 set13/85 set13/86 set13/87 set13/88 set13/89 set13/9 set13/90 set13/91 set13/92 set13/93 set13/94 set13/95 set13/96 set13/97 set13/98 set13/99) (node: http://192.168.56.25:3900 )
After inspecting, the actual S3 call made was with prefix "set13/", so at least this is not an error in Garage itself but in the jepsen code.
Finally found out that this was due to closures not correctly capturing their context in the list function in s3api.clj (wtf clojure?) Not sure exactly where it came from but it seems to have been fixed by making list-inner a separate function and not a sub-function, and passing all values that were previously in the context (creds and prefix) as additional arguments.
reg2 test inconsistency, even with timestamp fix
The reg2 test is our custom checker for CRDT read-after-write on individual object keys, acting as registers which can be updated. The test fails without the timestamp fix, which is expected as the clock scrambler will prevent nodes from having a correct ordering of objects.
With the timestamp fix, the happenned-before relationship should at least be respected, meaning that when a PutObject call starts after another PutObject call has ended, the second call should overwrite the value of the first call, and that value should not be readable by future GetObject calls. However, we observed inconsistencies even with the timestamp fix.
The inconsistencies seemed to always happenned after writing a nil value, which translates to a DeleteObject call instead of a PutObject. By removing the possibility of writing nil values, therefore only doing PutObject calls, the issue disappears. There is therefore an issue to fix in DeleteObject.
The issue in DeleteObject seems to have been fixed by commit c82d91c6bccf307186332b6c5c6fc0b128b1b2b1
License
Copyright © 2023 Alex Auvolat
This program and the accompanying materials are made available under the terms of the GNU Affero General Public License v3.0.