Commit graph

16 commits

Author SHA1 Message Date
Alex 5768bf3622 First implementation of K2V (#293)
All checks were successful
continuous-integration/drone/push Build is passing
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-10 13:16:57 +02:00
Alex 5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
Alex 2090a6187f
Add tranquilizer mechanism to improve on token bucket mechanism
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-11-04 13:26:59 +01:00
Alex 6b47c294f5
Refactoring on repair commands
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-10-27 11:14:55 +02:00
Trinity Pointard 28c015d9ff
add cli parameter to verify local bloc integrity
All checks were successful
continuous-integration/drone/push Build is passing
reuse code for listing local blocks
add disk i/o speed limit on integrity check
2021-10-27 10:31:03 +02:00
Alex 6a8439fd13 Some improvements in background worker but we terminate late
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-15 23:14:12 +01:00
Alex 4d4117f2b4 Refactor block resync loop; make workers infaillible
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-15 20:09:44 +01:00
Alex 097c339d98 Fix race condition
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-15 15:26:29 +01:00
Alex 046b649bcc (not well tested) use merkle tree for sync
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-11 18:28:27 +01:00
Alex 94f3d28774 WIP big refactoring
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-11 16:54:15 +01:00
Alex f319a7d374 Refactor model stuff, including cleaner CRDTs 2021-03-10 16:21:56 +01:00
Alex bec26a1312 Rename garage_core to garage_model 2020-07-07 13:59:22 +02:00
Alex fa13cf6996 Repair: do not mark deleted when upstream object is not found
With the previous behaviour, repairing could see some data as absent
and decide that the object or version was deleted,
thus going on to delete the version and blocks.
In the case where read_quorum + write_quorum <= replication_factor
however, entries may not yet be returned by the get, thus data would
have been deleted that should hot have been. The new behavior is more
cautious and just skips the entry when the warning is emitted.
2020-05-04 13:30:42 +00:00
Alex 0a283e4e70 Fix deletion propagation 2020-04-26 18:59:17 +00:00
Alex 0e49e0c8b5 Add key table to repair procedure 2020-04-26 16:22:22 +00:00
Alex d8f5e643bc Split code for modular compilation 2020-04-24 10:10:01 +00:00
Renamed from src/store/repair.rs (Browse further)