K2V #293

Merged
lx merged 68 commits from k2v into main 3 weeks ago
lx commented 2 months ago
Owner

Specification:

Making a Gitea PR as this is easier to edit and to comment than on cryptpad

View spec at this URL

  • Specify the structure of K2V triples
  • Specify the DVVS format used for causality detection
  • Specify the K2V index (just a counter of number of values per partition key)
  • Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
  • Specify index endpoint: ReadIndex
  • Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
  • Move to JSON objects instead of tuples
  • Specify endpoints for polling for updates on single values (PollItem)

Implementation:

  • Table for K2V items, causal contexts
  • Indexing mechanism and table for K2V index
  • Make API handlers a bit more generic
  • K2V API endpoint
  • K2V API router
  • ReadItem
  • InsertItem
  • DeleteItem
  • PollItem
  • ReadIndex
  • InsertBatch
  • ReadBatch
  • DeleteBatch

Testing:

  • Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
  • Actual tests:
    • Adapt testing framework
    • Simple test with InsertItem + ReadItem
    • Test with several Insert/Read/DeleteItem + ReadIndex
    • Test all combinations of return formats for ReadItem
    • Test with ReadBatch, InsertBatch, DeleteBatch
    • Test with PollItem
    • Test error codes
  • Fix most broken stuff
    • test PollItem broken randomly
    • when invalid causality tokens are given, errors should be 4xx not 5xx

Improvements:

  • Descending range queries
    • Specify
    • Implement
    • Add test
  • Batch updates to index counter
  • Put K2V behind experimental feature flag
**Specification:** Making a Gitea PR as this is easier to edit and to comment than on cryptpad View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [ ] Put K2V behind `experimental` feature flag
lx added 1 commit 2 months ago
0dfaa45643
First draft of the K2V specification
lx added 1 commit 2 months ago
ae3d6c9e84
Specify stuff about causality tokens (aka contexts)
lx added 1 commit 2 months ago
7e8e650f38
Document indexing
lx added 1 commit 2 months ago
834e564efa
Move to json objects
lx added 1 commit 2 months ago
a455f48ef4
switch json fields to camelCase
lx added 1 commit 2 months ago
30f0cab22c
Spec for PollItem
lx added 2 commits 2 months ago
2e45b541a2
First incarnation of what one of the K2V tables could be
lx added 1 commit 2 months ago
6c990b2bf2
rename api to s3::api in many places to make place for k2v::api
lx added 2 commits 2 months ago
e6f40c4cfa
remove unused imports
lx added 1 commit 2 months ago
03bfdfb4ef
Fix clippy
lx changed title from WIP: K2V specification to WIP: K2V 2 months ago
lx added 1 commit 2 months ago
f9de5478d6
update cargo.nix
lx added 1 commit 1 month ago
46700e6673
Fix test
lx added 1 commit 1 month ago
b8562d6e3c
Split off router macros to b reused for K2V router
lx added 2 commits 1 month ago
69f14245bb
Remove useless imports and dead code
lx added 1 commit 1 month ago
025db41bba
cargo fmt
lx added 2 commits 1 month ago
12dd378fca
fix clippy
lx added 1 commit 1 month ago
da14343ea7
prepare k2v rpc
lx added 1 commit 1 month ago
a9a1d5532d
RPC code to insert single values in K2V item table
lx added 1 commit 1 month ago
7cdec31e10
Implement DeleteItem and InsertItem
lx added 1 commit 1 month ago
fe4a8e85be
Fix display bug in garage status (nothing to do with k2v)
lx added 1 commit 1 month ago
3a66cf4f7d
Add generic index counter
lx added 1 commit 1 month ago
e9b796356a
Integrate index counter with k2v item table
lx added 1 commit 1 month ago
cc8b13f808
fix clippy
lx added 1 commit 1 month ago
d3a9075cd3
Add prefix parameter to APIs that use listings
lx added 1 commit 1 month ago
87a8315546
First implementation of ReadIndex that might work if I'm lucky
lx added 1 commit 1 month ago
bf94344ae0
signatures for service k2v different than for s3
lx added 1 commit 1 month ago
3b650cd2a7
update Cargo.nix
lx added 1 commit 1 month ago
574d88a02f
Make "s3" service a parameter in more places
lx added 1 commit 1 month ago
cec08a23af
Fix signatures and add basic code that makes a request
lx added 1 commit 1 month ago
f35b7c6ba1
Fix cargo fmt
lx added 1 commit 1 month ago
91faae679f
Fix bugs, test does interesting things
lx added 1 commit 1 month ago
362e7570a3
Filter out correctly tombstones in index
lx added 1 commit 1 month ago
26f44e5d85
Implement InsertBatch
lx force-pushed k2v from 26f44e5d85 to 3d4d59e714 1 month ago
lx added 1 commit 1 month ago
99e7c3396c
Also count conflicts
lx added 1 commit 1 month ago
140994c830
Implement ReadBatch
lx added 1 commit 1 month ago
ae0e6c6d27
more strictness; cargo fmt+clippy
lx added 1 commit 1 month ago
7a876cf94d
Implement DeleteBatch
lx added 1 commit 1 month ago
aedb1c1751
First K2V automated test
lx added 1 commit 1 month ago
e9e76f6fc5
Remove a bit of noise
lx added 1 commit 1 month ago
0902d655ce
Test with many InsertItem, DeleteItem, ReadItem and ReadIndex
lx added 1 commit 1 month ago
f6d5d8c532
Implement PollItem
lx added 1 commit 1 month ago
bc8047a8da
Add test for PollItem
lx added 1 commit 1 month ago
8fa25e882b
udate cargo.nix
lx added 2 commits 1 month ago
9ed52f36cc
Add tests for all possible item return formats
lx added 1 commit 1 month ago
c2e91cc4de
cargo fmt
lx added 1 commit 1 month ago
7a9f40d12f
IMPORTANT TWEAK: deduplicate returned values
lx added 1 commit 1 month ago
ab57510ffd
Add spec details on pitfalls of the API
lx added 1 commit 1 month ago
d7e2eb166d
Add test for batch operations
lx added 1 commit 1 month ago
3ac6970a24
Add EnumerationOrder parameter to table range queries
lx added 1 commit 1 month ago
3c8e4df397
Specify and implement reverse for ReadIndex and ReadBatch
lx added 1 commit 1 month ago
226439f2da
Add informations about how `reverse` works
lx added 1 commit 1 month ago
7362618b97
Test reverse and actually implement it correctly
lx added 1 commit 1 month ago
fdbed3f1e8
Encode causality tokens using URL-safe b64
lx added 1 commit 1 month ago
754919170f
Better handling of causal context parse errors
lx added 1 commit 1 month ago
17883bbe4c
Test a few API error codes
lx added 1 commit 1 month ago
41b58d7e25
Batch propagation of index counter updates
lx added 1 commit 1 month ago
1747132875
fix clippy
lx requested review from quentin 1 month ago
lx added 2 commits 1 month ago
fe37f45007
Actually do some batching in index counter
trinity-1686a reviewed 4 weeks ago
trinity-1686a left a comment

quick review of k2v spec. Overall looks good, see comments for details

- K2V triplets exist separately from S3 objects. K2V triples don't exist for
the S3 API, and S3 objects don't exist for the K2V API.
- Values stored for triples have associated causality information, that enables
Poster
Collaborator

are triples and triplets the same thing? If they are different we should clarify how, and if they are the same, we should use only one word to name them.

are triples and triplets the same thing? If they are different we should clarify how, and if they are the same, we should use only one word to name them.
lx marked this conversation as resolved
- **Internal server errors on updates do not mean that the update isn't stored.**
K2V will return an internal server error when it cannot reach a quorum of nodes on
which to save an updated value. However the value may still be stored on just one
node, which whill then propagate it to other nodes asynchronously via anti-entropy.
Poster
Collaborator

which will

which **will**
lx marked this conversation as resolved
prefix: null,
start: null,
end: null,
singleItem: false,
Poster
Collaborator

maybe singleItem should be true by default for delete operation, that would be safer against faulty API usage (it would probably require being the same for SEARCH, for the sake of coherence)

maybe `singleItem` should be true by default for delete operation, that would be safer against faulty API usage (it would probably require being the same for SEARCH, for the sake of coherence)
trinity-1686a reviewed 4 weeks ago
- a partition key (`pk`), an utf8 string that defines in what partition the triple is
stored; triples in different partitions cannot be listed together, they must
be the object of different ReadItem or ReadBatch queries
Poster
Collaborator

ReadItem reads a single triplet, so I don't think it's affected? Also, are other *Batch affected? I assume no, but this should probably be explicited

ReadItem reads a single triplet, so I don't think it's affected? Also, are other \*Batch affected? I assume no, but this should probably be explicited
lx marked this conversation as resolved
lx added 1 commit 4 weeks ago
3ce6ffa270
Corrections and clarifications in K2V spec
lx added 1 commit 3 weeks ago
08fbb922ca
Put K2V behind a feature flag
lx added 66 commits 3 weeks ago
328b52bd93
Encode causality tokens using URL-safe b64
e4443d720d
Put K2V behind a feature flag
lx force-pushed k2v from e4443d720d to eaca736a4f 3 weeks ago
lx added 1 commit 3 weeks ago
aa5b54fe19
Merge branch 'main' into k2v
lx changed title from WIP: K2V to K2V 3 weeks ago
lx removed review request for quentin 3 weeks ago
lx merged commit 5768bf3622 into main 3 weeks ago
lx referenced this issue from a commit 3 weeks ago
lx deleted branch k2v 3 weeks ago
trinity-1686a added this to the K2V - partition Key+sorting Key+Value milestone 5 days ago
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build was killed
The pull request has been merged as 5768bf3622.
Sign in to join this conversation.
Loading…
There is no content yet.