Commit graph

28 commits

Author SHA1 Message Date
5768bf3622 First implementation of K2V (#293)
All checks were successful
continuous-integration/drone/push Build is passing
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-10 13:16:57 +02:00
5d404dcd54
Add missing opentelemetry features 2022-04-08 14:21:04 +02:00
94f1e48fff Update to netapp 0.4.2 (a tiny fix)
All checks were successful
continuous-integration/drone/push Build is passing
2022-04-07 11:50:03 +02:00
9d0ed78887 Add feature flag for Kubernetes discovery 2022-03-24 16:57:43 +01:00
db46cdef79
Update netapp to v0.4.1
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-03-15 17:09:57 +01:00
203e8d2c34
Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
818daa5c78
Refactor how durations are measured 2022-03-14 10:53:35 +01:00
bb04d94fa9
Update to Netapp 0.4 which supports distributed tracing 2022-03-14 10:52:30 +01:00
8c2fb0c066
Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
9d44127245
add support for kubernetes service discovery
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
This commit adds support to discover garage instances running in
kubernetes.

Once enabled by setting `kubernetes_namespace` and
`kubernetes_service_name` garage will create a Custom Resources
`garagenodes.deuxfleurs.fr` with nodes public key as the resource name.
and IP and Port information as spec in the namespace configured by
`kubernetes_namespace`.

For discovering nodes the resources are filtered with the optionally set
`kubernetes_service_name` which sets a label
`garage.deuxfleurs.fr/service` on the resources.

This allows to separate multiple garage deployments in a single
namespace.

the `kubernetes_skip_crd` variable allows to disable the creation of the
CRD by garage itself. The user must deploy this manually.
2022-03-12 13:05:52 +01:00
5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
c94406f428
Improve how node roles are assigned in Garage
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
2021-11-16 16:05:53 +01:00
43e13a501d
Use published netapp crate instead of git repo
All checks were successful
continuous-integration/drone/push Build is passing
2021-10-26 10:36:57 +02:00
4067797d01
First port of Garage to Netapp 2021-10-22 15:55:18 +02:00
b9127dd6f8
Prepare for v0.3.0 and add migration path from v0.2.1.x
Some checks reported errors
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build was killed
2021-05-28 15:29:58 +02:00
f859d15062 update to v0.2.1
Some checks are pending
continuous-integration/drone/push Build is pending
2021-03-19 13:39:18 +01:00
fd8f4caa81 Support old CPUs
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-19 12:19:40 +01:00
4c26a0b9c1 Update Cargo.toml files with AGPL license info
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/tag Build is passing
2021-03-18 21:59:17 +01:00
dead945c8f Prepare for release 0.2
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-18 19:33:15 +01:00
f4346cc5f4 Update dependencies
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-16 15:58:40 +01:00
0cd5b2ae19 WIP migrate to tokio 1
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-15 22:36:41 +01:00
3bf2df622a Time and metadata improvements
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-15 16:21:41 +01:00
20e6e9fa20 Update sled & try to debug deadlock (but its in sled...)
All checks were successful
continuous-integration/drone/push Build is passing
2021-02-23 21:27:28 +01:00
e8e4418ca7 Add blake2 and xxhash hash functions
Some checks reported errors
continuous-integration/drone/push Build was killed
2021-02-23 17:52:28 +01:00
1d1d497e2b Bump everything to 0.1.1 2021-01-15 17:54:48 +01:00
cc65cdc0fe Add license, description and repository to .toml files 2020-07-07 14:14:58 +02:00
d867bbcfb5 Implement DeleteObjects 2020-05-01 15:52:35 +00:00
d8f5e643bc Split code for modular compilation 2020-04-24 10:10:01 +00:00