Bucket and access key tables are not read-after-write consistent #147

New Issue

Open

opened 2021-11-04 16:22:47 +00:00 by lx · 0 comments

lx commented

2021-11-04 16:22:47 +00:00

Owner

The read quorums are currently set to 1 for these tables.

For reads, having a quorum of 1 is usefull to read faster: we can just read from the local node, no cost involved. This is not something we are willing to compromise on as it would make Garage drastically slower

To have read-after-write consistency and respect this property, we would need to have a quorum of n (number of nodes) for all write operations. But then we tolerate zero failures...

The fact that the quorum is currently set to 1 is the cause of a bug in the smoke test: if we allow (or disallow) website access for a bucket, it is propagated asynchronously to other nodes, therefore the check that an http request makes a 200 or a 404 doesn't always return the result that it should taking into account the last udpate. The issue can be avoided by running all steps of the test on the same node, however from a distributed point of view the issue persists and would certainly show up in Jepsen testing or so.

The read quorums are currently set to 1 for these tables. For reads, having a quorum of 1 is usefull to read faster: we can just read from the local node, no cost involved. This is not something we are willing to compromise on as it would make Garage drastically slower To have read-after-write consistency and respect this property, we would need to have a quorum of n (number of nodes) for all write operations. But then we tolerate zero failures... The fact that the quorum is currently set to 1 is the cause of a bug in the smoke test: if we allow (or disallow) website access for a bucket, it is propagated asynchronously to other nodes, therefore the check that an http request makes a 200 or a 404 doesn't always return the result that it should taking into account the last udpate. The issue can be avoided by running all steps of the test on the same node, however from a distributed point of view the issue persists and would certainly show up in Jepsen testing or so.

lx added the

Bug

label 2021-11-04 16:22:47 +00:00

lx added the

Correctness

label 2021-11-08 15:35:52 +00:00

lx changed title from ~~Bucket and key tables are not sequentially consistent~~ to Bucket and key tables are not read-after-write consistent

2023-12-01 11:44:41 +00:00

lx changed title from ~~Bucket and key tables are not read-after-write consistent~~ to Bucket and access key tables are not read-after-write consistent

2023-12-01 11:47:32 +00:00