K2V #293

Merged
lx merged 68 commits from k2v into main 2022-05-10 11:16:58 +00:00
Owner

Specification:

Making a Gitea PR as this is easier to edit and to comment than on cryptpad

View spec at this URL

  • Specify the structure of K2V triples
  • Specify the DVVS format used for causality detection
  • Specify the K2V index (just a counter of number of values per partition key)
  • Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
  • Specify index endpoint: ReadIndex
  • Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
  • Move to JSON objects instead of tuples
  • Specify endpoints for polling for updates on single values (PollItem)

Implementation:

  • Table for K2V items, causal contexts
  • Indexing mechanism and table for K2V index
  • Make API handlers a bit more generic
  • K2V API endpoint
  • K2V API router
  • ReadItem
  • InsertItem
  • DeleteItem
  • PollItem
  • ReadIndex
  • InsertBatch
  • ReadBatch
  • DeleteBatch

Testing:

  • Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
  • Actual tests:
    • Adapt testing framework
    • Simple test with InsertItem + ReadItem
    • Test with several Insert/Read/DeleteItem + ReadIndex
    • Test all combinations of return formats for ReadItem
    • Test with ReadBatch, InsertBatch, DeleteBatch
    • Test with PollItem
    • Test error codes
  • Fix most broken stuff
    • test PollItem broken randomly
    • when invalid causality tokens are given, errors should be 4xx not 5xx

Improvements:

  • Descending range queries
    • Specify
    • Implement
    • Add test
  • Batch updates to index counter
  • Put K2V behind experimental feature flag
**Specification:** Making a Gitea PR as this is easier to edit and to comment than on cryptpad View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [ ] Put K2V behind `experimental` feature flag
lx added 1 commit 2022-04-11 11:06:14 +00:00
First draft of the K2V specification
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
0dfaa45643
lx added 1 commit 2022-04-11 12:37:11 +00:00
Specify stuff about causality tokens (aka contexts)
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
ae3d6c9e84
lx added 1 commit 2022-04-11 15:02:14 +00:00
Document indexing
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
7e8e650f38
lx added 1 commit 2022-04-11 15:09:39 +00:00
Move to json objects
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
834e564efa
lx added 1 commit 2022-04-11 16:08:05 +00:00
switch json fields to camelCase
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
a455f48ef4
lx added 1 commit 2022-04-11 16:38:58 +00:00
Spec for PollItem
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
30f0cab22c
lx added 2 commits 2022-04-13 13:34:56 +00:00
First incarnation of what one of the K2V tables could be
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2e45b541a2
lx added 1 commit 2022-04-13 14:46:06 +00:00
rename api to s3::api in many places to make place for k2v::api
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
6c990b2bf2
lx added 2 commits 2022-04-13 15:38:42 +00:00
remove unused imports
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
e6f40c4cfa
lx added 1 commit 2022-04-13 15:42:03 +00:00
Fix clippy
Some checks reported errors
continuous-integration/drone/pr Build was killed
continuous-integration/drone/push Build is failing
03bfdfb4ef
lx changed title from WIP: K2V specification to WIP: K2V 2022-04-13 15:42:28 +00:00
lx added 1 commit 2022-04-13 15:47:07 +00:00
update cargo.nix
Some checks reported errors
continuous-integration/drone/pr Build was killed
continuous-integration/drone/push Build is failing
f9de5478d6
lx added 1 commit 2022-04-14 09:37:14 +00:00
Fix test
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
46700e6673
lx added 1 commit 2022-04-14 12:10:15 +00:00
Split off router macros to b reused for K2V router
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
b8562d6e3c
lx added 2 commits 2022-04-14 12:44:27 +00:00
Remove useless imports and dead code
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
69f14245bb
lx added 1 commit 2022-04-14 13:03:01 +00:00
cargo fmt
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
025db41bba
lx added 2 commits 2022-04-14 14:22:27 +00:00
fix clippy
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
12dd378fca
lx added 1 commit 2022-04-14 14:42:30 +00:00
prepare k2v rpc
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
da14343ea7
lx added 1 commit 2022-04-15 10:14:43 +00:00
RPC code to insert single values in K2V item table
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
a9a1d5532d
lx added 1 commit 2022-04-15 13:49:48 +00:00
Implement DeleteItem and InsertItem
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
7cdec31e10
lx added 1 commit 2022-04-15 14:44:17 +00:00
Fix display bug in garage status (nothing to do with k2v)
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
fe4a8e85be
lx added 1 commit 2022-04-15 15:59:50 +00:00
Add generic index counter
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
3a66cf4f7d
lx added 1 commit 2022-04-21 09:56:34 +00:00
Integrate index counter with k2v item table
Some checks reported errors
continuous-integration/drone/push Build encountered an error
continuous-integration/drone/pr Build encountered an error
e9b796356a
lx added 1 commit 2022-04-21 10:01:56 +00:00
fix clippy
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
cc8b13f808
lx added 1 commit 2022-04-21 10:16:19 +00:00
Add prefix parameter to APIs that use listings
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
d3a9075cd3
lx added 1 commit 2022-04-21 11:27:26 +00:00
First implementation of ReadIndex that might work if I'm lucky
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
87a8315546
lx added 1 commit 2022-04-21 11:33:45 +00:00
signatures for service k2v different than for s3
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
bf94344ae0
lx added 1 commit 2022-04-21 11:43:02 +00:00
update Cargo.nix
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
3b650cd2a7
lx added 1 commit 2022-04-21 14:36:18 +00:00
Make "s3" service a parameter in more places
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
574d88a02f
lx added 1 commit 2022-04-21 15:03:57 +00:00
Fix signatures and add basic code that makes a request
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
cec08a23af
(and it crashes yeah yeah yeah)
lx added 1 commit 2022-04-21 15:06:22 +00:00
Fix cargo fmt
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
f35b7c6ba1
lx added 1 commit 2022-04-22 13:07:48 +00:00
Fix bugs, test does interesting things
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
91faae679f
lx added 1 commit 2022-04-22 13:29:25 +00:00
Filter out correctly tombstones in index
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
362e7570a3
lx force-pushed k2v from 26f44e5d85 to 3d4d59e714 2022-04-22 14:14:59 +00:00 Compare
lx added 1 commit 2022-04-22 14:32:11 +00:00
Also count conflicts
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
99e7c3396c
lx added 1 commit 2022-04-22 16:00:24 +00:00
Implement ReadBatch
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
140994c830
lx added 1 commit 2022-04-22 16:05:53 +00:00
more strictness; cargo fmt+clippy
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
ae0e6c6d27
lx added 1 commit 2022-04-26 11:37:48 +00:00
Implement DeleteBatch
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
7a876cf94d
lx added 1 commit 2022-04-26 12:30:46 +00:00
First K2V automated test
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
aedb1c1751
lx added 1 commit 2022-04-26 12:35:16 +00:00
Remove a bit of noise
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
e9e76f6fc5
lx added 1 commit 2022-04-26 13:30:28 +00:00
Test with many InsertItem, DeleteItem, ReadItem and ReadIndex
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
0902d655ce
lx added 1 commit 2022-04-26 15:03:41 +00:00
Implement PollItem
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
f6d5d8c532
lx added 1 commit 2022-04-26 15:14:26 +00:00
Add test for PollItem
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
bc8047a8da
lx added 1 commit 2022-04-27 08:16:34 +00:00
udate cargo.nix
Some checks failed
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is failing
8fa25e882b
lx added 2 commits 2022-04-27 09:02:43 +00:00
Add tests for all possible item return formats
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
9ed52f36cc
lx added 1 commit 2022-04-27 09:03:18 +00:00
cargo fmt
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
c2e91cc4de
lx added 1 commit 2022-04-27 09:05:45 +00:00
IMPORTANT TWEAK: deduplicate returned values
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
7a9f40d12f
lx added 1 commit 2022-04-27 09:13:29 +00:00
Add spec details on pitfalls of the API
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is passing
ab57510ffd
lx added 1 commit 2022-04-27 12:31:35 +00:00
Add test for batch operations
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
d7e2eb166d
lx added 1 commit 2022-04-28 08:32:33 +00:00
Add EnumerationOrder parameter to table range queries
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is passing
3ac6970a24
lx added 1 commit 2022-04-28 08:59:46 +00:00
Specify and implement reverse for ReadIndex and ReadBatch
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
3c8e4df397
lx added 1 commit 2022-04-28 09:02:51 +00:00
Add informations about how reverse works
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is passing
226439f2da
lx added 1 commit 2022-04-28 09:28:33 +00:00
Test reverse and actually implement it correctly
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
7362618b97
lx added 1 commit 2022-04-28 09:36:11 +00:00
Encode causality tokens using URL-safe b64
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
fdbed3f1e8
this should fix the pollitem test where the ct was passed in the url
lx added 1 commit 2022-04-28 09:44:33 +00:00
Better handling of causal context parse errors
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
754919170f
lx added 1 commit 2022-04-28 09:57:33 +00:00
Test a few API error codes
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
17883bbe4c
lx added 1 commit 2022-04-28 11:48:10 +00:00
Batch propagation of index counter updates
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
41b58d7e25
lx added 1 commit 2022-04-28 11:49:40 +00:00
fix clippy
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
1747132875
lx requested review from quentin 2022-04-28 11:51:41 +00:00
lx added 2 commits 2022-04-28 12:47:18 +00:00
Actually do some batching in index counter
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
fe37f45007
trinity-1686a reviewed 2022-05-02 13:31:49 +00:00
trinity-1686a left a comment
Owner

quick review of k2v spec. Overall looks good, see comments for details

quick review of k2v spec. Overall looks good, see comments for details
@ -0,0 +13,4 @@
- K2V triplets exist separately from S3 objects. K2V triples don't exist for
the S3 API, and S3 objects don't exist for the K2V API.
- Values stored for triples have associated causality information, that enables

are triples and triplets the same thing? If they are different we should clarify how, and if they are the same, we should use only one word to name them.

are triples and triplets the same thing? If they are different we should clarify how, and if they are the same, we should use only one word to name them.
lx marked this conversation as resolved
@ -0,0 +172,4 @@
- **Internal server errors on updates do not mean that the update isn't stored.**
K2V will return an internal server error when it cannot reach a quorum of nodes on
which to save an updated value. However the value may still be stored on just one
node, which whill then propagate it to other nodes asynchronously via anti-entropy.

which will

which **will**
lx marked this conversation as resolved
@ -0,0 +653,4 @@
prefix: null,
start: null,
end: null,
singleItem: false,

maybe singleItem should be true by default for delete operation, that would be safer against faulty API usage (it would probably require being the same for SEARCH, for the sake of coherence)

maybe `singleItem` should be true by default for delete operation, that would be safer against faulty API usage (it would probably require being the same for SEARCH, for the sake of coherence)
trinity-1686a reviewed 2022-05-02 13:48:51 +00:00
@ -0,0 +30,4 @@
- a partition key (`pk`), an utf8 string that defines in what partition the triple is
stored; triples in different partitions cannot be listed together, they must
be the object of different ReadItem or ReadBatch queries

ReadItem reads a single triplet, so I don't think it's affected? Also, are other *Batch affected? I assume no, but this should probably be explicited

ReadItem reads a single triplet, so I don't think it's affected? Also, are other \*Batch affected? I assume no, but this should probably be explicited
lx marked this conversation as resolved
lx added 1 commit 2022-05-03 19:56:30 +00:00
Corrections and clarifications in K2V spec
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
3ce6ffa270
lx added 1 commit 2022-05-09 09:11:03 +00:00
Put K2V behind a feature flag
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
08fbb922ca
lx force-pushed k2v from e4443d720d to eaca736a4f 2022-05-09 09:24:29 +00:00 Compare
lx added 1 commit 2022-05-10 11:14:49 +00:00
Merge branch 'main' into k2v
Some checks reported errors
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build was killed
aa5b54fe19
lx changed title from WIP: K2V to K2V 2022-05-10 11:16:04 +00:00
lx removed review request for quentin 2022-05-10 11:16:12 +00:00
lx merged commit 5768bf3622 into main 2022-05-10 11:16:58 +00:00
lx referenced this pull request from a commit 2022-05-10 11:16:58 +00:00
lx deleted branch k2v 2022-05-10 11:17:00 +00:00
trinity-1686a added this to the (deleted) milestone 2022-05-23 13:18:59 +00:00
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#293
No description provided.