Deuxfleurs/garage
34
65
Fork 83
Code Issues95 Pull requests16 Releases32 Wiki Activity

K2V #293

Merged
lx merged 68 commits from k2v into main 2022-05-10 11:16:58 +00:00
Conversation 0 Commits 68 Files changed 83 +6491 -1226
lx commented 2022-04-11 11:06:07 +00:00
Owner

Specification:

Making a Gitea PR as this is easier to edit and to comment than on cryptpad

View spec at this URL

  • Specify the structure of K2V triples
  • Specify the DVVS format used for causality detection
  • Specify the K2V index (just a counter of number of values per partition key)
  • Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
  • Specify index endpoint: ReadIndex
  • Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
  • Move to JSON objects instead of tuples
  • Specify endpoints for polling for updates on single values (PollItem)

Implementation:

  • Table for K2V items, causal contexts
  • Indexing mechanism and table for K2V index
  • Make API handlers a bit more generic
  • K2V API endpoint
  • K2V API router
  • ReadItem
  • InsertItem
  • DeleteItem
  • PollItem
  • ReadIndex
  • InsertBatch
  • ReadBatch
  • DeleteBatch

Testing:

  • Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
  • Actual tests:
    • Adapt testing framework
    • Simple test with InsertItem + ReadItem
    • Test with several Insert/Read/DeleteItem + ReadIndex
    • Test all combinations of return formats for ReadItem
    • Test with ReadBatch, InsertBatch, DeleteBatch
    • Test with PollItem
    • Test error codes
  • Fix most broken stuff
    • test PollItem broken randomly
    • when invalid causality tokens are given, errors should be 4xx not 5xx

Improvements:

  • Descending range queries
    • Specify
    • Implement
    • Add test
  • Batch updates to index counter
  • Put K2V behind experimental feature flag
**Specification:** Making a Gitea PR as this is easier to edit and to comment than on cryptpad View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [ ] Put K2V behind `experimental` feature flag
lx added 1 commit 2022-04-11 11:06:14 +00:00
lx added 1 commit 2022-04-11 12:37:11 +00:00
lx added 1 commit 2022-04-11 15:02:14 +00:00
lx added 1 commit 2022-04-11 15:09:39 +00:00
lx added 1 commit 2022-04-11 16:08:05 +00:00
lx added 1 commit 2022-04-11 16:38:58 +00:00
lx added 2 commits 2022-04-13 13:34:56 +00:00
lx added 1 commit 2022-04-13 14:46:06 +00:00
lx added 2 commits 2022-04-13 15:38:42 +00:00
lx added 1 commit 2022-04-13 15:42:03 +00:00
lx changed title from WIP: K2V specification to WIP: K2V 2022-04-13 15:42:28 +00:00
lx added 1 commit 2022-04-13 15:47:07 +00:00
lx added 1 commit 2022-04-14 09:37:14 +00:00
lx added 1 commit 2022-04-14 12:10:15 +00:00
lx added 2 commits 2022-04-14 12:44:27 +00:00
lx added 1 commit 2022-04-14 13:03:01 +00:00
lx added 2 commits 2022-04-14 14:22:27 +00:00
lx added 1 commit 2022-04-14 14:42:30 +00:00
lx added 1 commit 2022-04-15 10:14:43 +00:00
lx added 1 commit 2022-04-15 13:49:48 +00:00
lx added 1 commit 2022-04-15 14:44:17 +00:00
lx added 1 commit 2022-04-15 15:59:50 +00:00
lx added 1 commit 2022-04-21 09:56:34 +00:00
lx added 1 commit 2022-04-21 10:01:56 +00:00
lx added 1 commit 2022-04-21 10:16:19 +00:00
lx added 1 commit 2022-04-21 11:27:26 +00:00
lx added 1 commit 2022-04-21 11:33:45 +00:00
lx added 1 commit 2022-04-21 11:43:02 +00:00
lx added 1 commit 2022-04-21 14:36:18 +00:00
lx added 1 commit 2022-04-21 15:03:57 +00:00 
(and it crashes yeah yeah yeah)
lx added 1 commit 2022-04-21 15:06:22 +00:00
lx added 1 commit 2022-04-22 13:07:48 +00:00
lx added 1 commit 2022-04-22 13:29:25 +00:00
lx force-pushed k2v from 26f44e5d85 to 3d4d59e714 2022-04-22 14:14:59 +00:00 Compare
lx added 1 commit 2022-04-22 14:32:11 +00:00
lx added 1 commit 2022-04-22 16:00:24 +00:00
lx added 1 commit 2022-04-22 16:05:53 +00:00
lx added 1 commit 2022-04-26 11:37:48 +00:00
lx added 1 commit 2022-04-26 12:30:46 +00:00
lx added 1 commit 2022-04-26 12:35:16 +00:00
lx added 1 commit 2022-04-26 13:30:28 +00:00
lx added 1 commit 2022-04-26 15:03:41 +00:00
lx added 1 commit 2022-04-26 15:14:26 +00:00
lx added 1 commit 2022-04-27 08:16:34 +00:00
lx added 2 commits 2022-04-27 09:02:43 +00:00
lx added 1 commit 2022-04-27 09:03:18 +00:00
lx added 1 commit 2022-04-27 09:05:45 +00:00
lx added 1 commit 2022-04-27 09:13:29 +00:00
lx added 1 commit 2022-04-27 12:31:35 +00:00
lx added 1 commit 2022-04-28 08:32:33 +00:00
lx added 1 commit 2022-04-28 08:59:46 +00:00
lx added 1 commit 2022-04-28 09:02:51 +00:00
lx added 1 commit 2022-04-28 09:28:33 +00:00
lx added 1 commit 2022-04-28 09:36:11 +00:00 
this should fix the pollitem test where the ct was passed in the url
lx added 1 commit 2022-04-28 09:44:33 +00:00
lx added 1 commit 2022-04-28 09:57:33 +00:00
lx added 1 commit 2022-04-28 11:48:10 +00:00
lx added 1 commit 2022-04-28 11:49:40 +00:00
requested review from quentin 2022-04-28 11:51:41 +00:00
lx added 2 commits 2022-04-28 12:47:18 +00:00
trinity-1686a reviewed 2022-05-02 13:31:49 +00:00
trinity-1686a left a comment
Owner

quick review of k2v spec. Overall looks good, see comments for details

quick review of k2v spec. Overall looks good, see comments for details
doc/drafts/k2v-spec.md Outdated
@ -0,0 +13,4 @@
- K2V triplets exist separately from S3 objects. K2V triples don't exist for
the S3 API, and S3 objects don't exist for the K2V API.
- Values stored for triples have associated causality information, that enables
trinity-1686a commented 2022-05-02 13:12:05 +00:00
Owner

are triples and triplets the same thing? If they are different we should clarify how, and if they are the same, we should use only one word to name them.

are triples and triplets the same thing? If they are different we should clarify how, and if they are the same, we should use only one word to name them.
lx marked this conversation as resolved
doc/drafts/k2v-spec.md Outdated
@ -0,0 +172,4 @@
- **Internal server errors on updates do not mean that the update isn't stored.**
K2V will return an internal server error when it cannot reach a quorum of nodes on
which to save an updated value. However the value may still be stored on just one
node, which whill then propagate it to other nodes asynchronously via anti-entropy.
trinity-1686a commented 2022-05-02 13:31:13 +00:00
Owner

which will

which **will**
lx marked this conversation as resolved
doc/drafts/k2v-spec.md Outdated
@ -0,0 +653,4 @@
prefix: null,
start: null,
end: null,
singleItem: false,
trinity-1686a commented 2022-05-02 13:25:24 +00:00
Owner

maybe singleItem should be true by default for delete operation, that would be safer against faulty API usage (it would probably require being the same for SEARCH, for the sake of coherence)

maybe `singleItem` should be true by default for delete operation, that would be safer against faulty API usage (it would probably require being the same for SEARCH, for the sake of coherence)
trinity-1686a reviewed 2022-05-02 13:48:51 +00:00
doc/drafts/k2v-spec.md Outdated
@ -0,0 +30,4 @@
- a partition key (`pk`), an utf8 string that defines in what partition the triple is
stored; triples in different partitions cannot be listed together, they must
be the object of different ReadItem or ReadBatch queries
trinity-1686a commented 2022-05-02 13:48:51 +00:00
Owner

ReadItem reads a single triplet, so I don't think it's affected? Also, are other *Batch affected? I assume no, but this should probably be explicited

ReadItem reads a single triplet, so I don't think it's affected? Also, are other \*Batch affected? I assume no, but this should probably be explicited
lx marked this conversation as resolved
lx added 1 commit 2022-05-03 19:56:30 +00:00
lx added 1 commit 2022-05-09 09:11:03 +00:00
lx force-pushed k2v from e4443d720d to eaca736a4f 2022-05-09 09:24:29 +00:00 Compare
lx added 1 commit 2022-05-10 11:14:49 +00:00
lx changed title from WIP: K2V to K2V 2022-05-10 11:16:04 +00:00
removed review request for quentin 2022-05-10 11:16:12 +00:00
lx merged commit 5768bf3622 into main 2022-05-10 11:16:58 +00:00
lx referenced this pull request from a commit 2022-05-10 11:16:58 +00:00
First implementation of K2V (#293)
lx deleted branch k2v 2022-05-10 11:17:00 +00:00
trinity-1686a added this to the (deleted) milestone 2022-05-23 13:18:59 +00:00
Sign in to join this conversation.
No reviewers
No labels
action
check-aws
action
discussion-needed
action
for-external-contributors
action
for-newcomers
action
more-info-needed
action
need-funding
action
triage-required
kind
correctness
kind/experimental
kind
ideas
kind
improvement
kind
performance
kind
testing
kind
usability
kind
wrong-behavior
prio
critical
prio
low
scope
admin-api
scope
admin-sdk
scope
background-healing
scope
build
scope
documentation
scope
k8s
scope
layout
scope
metadata
scope
ops
scope
rpc
scope
s3-api
scope
security
scope
telemetry
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#293
No description provided.
Delete branch "k2v"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?