K2V #293

Merged
lx merged 68 commits from k2v into main 2022-05-10 11:16:58 +00:00
Owner

Specification:

Making a Gitea PR as this is easier to edit and to comment than on cryptpad

View spec at this URL

  • Specify the structure of K2V triples
  • Specify the DVVS format used for causality detection
  • Specify the K2V index (just a counter of number of values per partition key)
  • Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
  • Specify index endpoint: ReadIndex
  • Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
  • Move to JSON objects instead of tuples
  • Specify endpoints for polling for updates on single values (PollItem)

Implementation:

  • Table for K2V items, causal contexts
  • Indexing mechanism and table for K2V index
  • Make API handlers a bit more generic
  • K2V API endpoint
  • K2V API router
  • ReadItem
  • InsertItem
  • DeleteItem
  • PollItem
  • ReadIndex
  • InsertBatch
  • ReadBatch
  • DeleteBatch

Testing:

  • Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
  • Actual tests:
    • Adapt testing framework
    • Simple test with InsertItem + ReadItem
    • Test with several Insert/Read/DeleteItem + ReadIndex
    • Test all combinations of return formats for ReadItem
    • Test with ReadBatch, InsertBatch, DeleteBatch
    • Test with PollItem
    • Test error codes
  • Fix most broken stuff
    • test PollItem broken randomly
    • when invalid causality tokens are given, errors should be 4xx not 5xx

Improvements:

  • Descending range queries
    • Specify
    • Implement
    • Add test
  • Batch updates to index counter
  • Put K2V behind experimental feature flag
**Specification:** Making a Gitea PR as this is easier to edit and to comment than on cryptpad View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [ ] Put K2V behind `experimental` feature flag
lx added 1 commit 2022-04-11 11:06:14 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
0dfaa45643
First draft of the K2V specification
lx added 1 commit 2022-04-11 12:37:11 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
ae3d6c9e84
Specify stuff about causality tokens (aka contexts)
lx added 1 commit 2022-04-11 15:02:14 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
7e8e650f38
Document indexing
lx added 1 commit 2022-04-11 15:09:39 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
834e564efa
Move to json objects
lx added 1 commit 2022-04-11 16:08:05 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
a455f48ef4
switch json fields to camelCase
lx added 1 commit 2022-04-11 16:38:58 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
30f0cab22c
Spec for PollItem
lx added 2 commits 2022-04-13 13:34:56 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
2e45b541a2
First incarnation of what one of the K2V tables could be
lx added 1 commit 2022-04-13 14:46:06 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
6c990b2bf2
rename api to s3::api in many places to make place for k2v::api
lx added 2 commits 2022-04-13 15:38:42 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
e6f40c4cfa
remove unused imports
lx added 1 commit 2022-04-13 15:42:03 +00:00
continuous-integration/drone/pr Build was killed Details
continuous-integration/drone/push Build is failing Details
03bfdfb4ef
Fix clippy
lx changed title from WIP: K2V specification to WIP: K2V 2022-04-13 15:42:28 +00:00
lx added 1 commit 2022-04-13 15:47:07 +00:00
continuous-integration/drone/pr Build was killed Details
continuous-integration/drone/push Build is failing Details
f9de5478d6
update cargo.nix
lx added 1 commit 2022-04-14 09:37:14 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
46700e6673
Fix test
lx added 1 commit 2022-04-14 12:10:15 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
b8562d6e3c
Split off router macros to b reused for K2V router
lx added 2 commits 2022-04-14 12:44:27 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
69f14245bb
Remove useless imports and dead code
lx added 1 commit 2022-04-14 13:03:01 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
025db41bba
cargo fmt
lx added 2 commits 2022-04-14 14:22:27 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
12dd378fca
fix clippy
lx added 1 commit 2022-04-14 14:42:30 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
da14343ea7
prepare k2v rpc
lx added 1 commit 2022-04-15 10:14:43 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
a9a1d5532d
RPC code to insert single values in K2V item table
lx added 1 commit 2022-04-15 13:49:48 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
7cdec31e10
Implement DeleteItem and InsertItem
lx added 1 commit 2022-04-15 14:44:17 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
fe4a8e85be
Fix display bug in garage status (nothing to do with k2v)
lx added 1 commit 2022-04-15 15:59:50 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
3a66cf4f7d
Add generic index counter
lx added 1 commit 2022-04-21 09:56:34 +00:00
continuous-integration/drone/push Build encountered an error Details
continuous-integration/drone/pr Build encountered an error Details
e9b796356a
Integrate index counter with k2v item table
lx added 1 commit 2022-04-21 10:01:56 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
cc8b13f808
fix clippy
lx added 1 commit 2022-04-21 10:16:19 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
d3a9075cd3
Add prefix parameter to APIs that use listings
lx added 1 commit 2022-04-21 11:27:26 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
87a8315546
First implementation of ReadIndex that might work if I'm lucky
lx added 1 commit 2022-04-21 11:33:45 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
bf94344ae0
signatures for service k2v different than for s3
lx added 1 commit 2022-04-21 11:43:02 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
3b650cd2a7
update Cargo.nix
lx added 1 commit 2022-04-21 14:36:18 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
574d88a02f
Make "s3" service a parameter in more places
lx added 1 commit 2022-04-21 15:03:57 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
cec08a23af
Fix signatures and add basic code that makes a request
(and it crashes yeah yeah yeah)
lx added 1 commit 2022-04-21 15:06:22 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
f35b7c6ba1
Fix cargo fmt
lx added 1 commit 2022-04-22 13:07:48 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
91faae679f
Fix bugs, test does interesting things
lx added 1 commit 2022-04-22 13:29:25 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
362e7570a3
Filter out correctly tombstones in index
lx force-pushed k2v from 26f44e5d85 to 3d4d59e714 2022-04-22 14:14:59 +00:00 Compare
lx added 1 commit 2022-04-22 14:32:11 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
99e7c3396c
Also count conflicts
lx added 1 commit 2022-04-22 16:00:24 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
140994c830
Implement ReadBatch
lx added 1 commit 2022-04-22 16:05:53 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
ae0e6c6d27
more strictness; cargo fmt+clippy
lx added 1 commit 2022-04-26 11:37:48 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
7a876cf94d
Implement DeleteBatch
lx added 1 commit 2022-04-26 12:30:46 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
aedb1c1751
First K2V automated test
lx added 1 commit 2022-04-26 12:35:16 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
e9e76f6fc5
Remove a bit of noise
lx added 1 commit 2022-04-26 13:30:28 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
0902d655ce
Test with many InsertItem, DeleteItem, ReadItem and ReadIndex
lx added 1 commit 2022-04-26 15:03:41 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
f6d5d8c532
Implement PollItem
lx added 1 commit 2022-04-26 15:14:26 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
bc8047a8da
Add test for PollItem
lx added 1 commit 2022-04-27 08:16:34 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is failing Details
8fa25e882b
udate cargo.nix
lx added 2 commits 2022-04-27 09:02:43 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
9ed52f36cc
Add tests for all possible item return formats
lx added 1 commit 2022-04-27 09:03:18 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
c2e91cc4de
cargo fmt
lx added 1 commit 2022-04-27 09:05:45 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
7a9f40d12f
IMPORTANT TWEAK: deduplicate returned values
lx added 1 commit 2022-04-27 09:13:29 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is passing Details
ab57510ffd
Add spec details on pitfalls of the API
lx added 1 commit 2022-04-27 12:31:35 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
d7e2eb166d
Add test for batch operations
lx added 1 commit 2022-04-28 08:32:33 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is passing Details
3ac6970a24
Add EnumerationOrder parameter to table range queries
lx added 1 commit 2022-04-28 08:59:46 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
3c8e4df397
Specify and implement reverse for ReadIndex and ReadBatch
lx added 1 commit 2022-04-28 09:02:51 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is passing Details
226439f2da
Add informations about how `reverse` works
lx added 1 commit 2022-04-28 09:28:33 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
7362618b97
Test reverse and actually implement it correctly
lx added 1 commit 2022-04-28 09:36:11 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
fdbed3f1e8
Encode causality tokens using URL-safe b64
this should fix the pollitem test where the ct was passed in the url
lx added 1 commit 2022-04-28 09:44:33 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
754919170f
Better handling of causal context parse errors
lx added 1 commit 2022-04-28 09:57:33 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
17883bbe4c
Test a few API error codes
lx added 1 commit 2022-04-28 11:48:10 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
41b58d7e25
Batch propagation of index counter updates
lx added 1 commit 2022-04-28 11:49:40 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
1747132875
fix clippy
lx requested review from quentin 2022-04-28 11:51:41 +00:00
lx added 2 commits 2022-04-28 12:47:18 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
fe37f45007
Actually do some batching in index counter
trinity-1686a reviewed 2022-05-02 13:31:49 +00:00
trinity-1686a left a comment
Owner

quick review of k2v spec. Overall looks good, see comments for details

quick review of k2v spec. Overall looks good, see comments for details
@ -0,0 +13,4 @@
- K2V triplets exist separately from S3 objects. K2V triples don't exist for
the S3 API, and S3 objects don't exist for the K2V API.
- Values stored for triples have associated causality information, that enables

are triples and triplets the same thing? If they are different we should clarify how, and if they are the same, we should use only one word to name them.

are triples and triplets the same thing? If they are different we should clarify how, and if they are the same, we should use only one word to name them.
lx marked this conversation as resolved
@ -0,0 +172,4 @@
- **Internal server errors on updates do not mean that the update isn't stored.**
K2V will return an internal server error when it cannot reach a quorum of nodes on
which to save an updated value. However the value may still be stored on just one
node, which whill then propagate it to other nodes asynchronously via anti-entropy.

which will

which **will**
lx marked this conversation as resolved
@ -0,0 +653,4 @@
prefix: null,
start: null,
end: null,
singleItem: false,

maybe singleItem should be true by default for delete operation, that would be safer against faulty API usage (it would probably require being the same for SEARCH, for the sake of coherence)

maybe `singleItem` should be true by default for delete operation, that would be safer against faulty API usage (it would probably require being the same for SEARCH, for the sake of coherence)
trinity-1686a reviewed 2022-05-02 13:48:51 +00:00
@ -0,0 +30,4 @@
- a partition key (`pk`), an utf8 string that defines in what partition the triple is
stored; triples in different partitions cannot be listed together, they must
be the object of different ReadItem or ReadBatch queries

ReadItem reads a single triplet, so I don't think it's affected? Also, are other *Batch affected? I assume no, but this should probably be explicited

ReadItem reads a single triplet, so I don't think it's affected? Also, are other \*Batch affected? I assume no, but this should probably be explicited
lx marked this conversation as resolved
lx added 1 commit 2022-05-03 19:56:30 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
3ce6ffa270
Corrections and clarifications in K2V spec
lx added 1 commit 2022-05-09 09:11:03 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
08fbb922ca
Put K2V behind a feature flag
lx force-pushed k2v from e4443d720d to eaca736a4f 2022-05-09 09:24:29 +00:00 Compare
lx added 1 commit 2022-05-10 11:14:49 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build was killed Details
aa5b54fe19
Merge branch 'main' into k2v
lx changed title from WIP: K2V to K2V 2022-05-10 11:16:04 +00:00
lx removed review request for quentin 2022-05-10 11:16:12 +00:00
lx merged commit 5768bf3622 into main 2022-05-10 11:16:58 +00:00
lx deleted branch k2v 2022-05-10 11:17:00 +00:00
trinity-1686a added this to the (deleted) milestone 2022-05-23 13:18:59 +00:00
Sign in to join this conversation.
No description provided.