Commit graph

762 commits

Author SHA1 Message Date
Alex 2d5541b0e7
Integrate LMDB with rest
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-03 16:40:38 +02:00
Alex f7a1c70089
Implement iterator for LMDB 2022-06-03 16:18:56 +02:00
Alex bd2997a453
LMDB: use Heed wrapper (much more complete)
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-03 15:53:20 +02:00
Alex 4e72c713f1
Start LMDB adapter, with fixed semantics
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-03 15:31:07 +02:00
Alex 16e0a655d0
Update rmp-serde
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-03 14:16:30 +02:00
Alex d3d7df098b
small fixes 2022-06-03 14:00:13 +02:00
Alex 3d18c9e183
whoops
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-03 13:35:02 +02:00
Alex 9bb58638f0
Fix block repair to not deadlock with sqlite 2022-06-03 12:56:35 +02:00
Alex 295bc2741f
fix clipy lint
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-03 12:52:41 +02:00
Alex 4bf706b170
Slightly prettier code 2022-06-03 12:35:12 +02:00
Alex bd9ff432d7
Garage works on sqlite, but it's a hack
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-03 12:12:25 +02:00
Alex df0877bbba
Conversion utility
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-03 11:44:41 +02:00
Alex cc0d984118
Fix most clippy lints
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-03 11:14:24 +02:00
Alex f25309e58f
Change value type to be a dyn thing
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-03 10:44:54 +02:00
Alex 7b79906350
Drop later 2022-06-03 04:44:40 +02:00
Alex 6ebb54cddb
Just refactor a bit
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-02 23:46:28 +02:00
Alex 0e4f336887
Horrible implementation of range for sqlite
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-02 23:43:32 +02:00
Alex c439cb11a9
Sqlite iter with unsafe code
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-02 23:14:10 +02:00
Alex 364061453c
Add back Send 2022-06-02 22:05:57 +02:00
Alex 43704afb29
Begin sqlite adapter
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-02 19:58:47 +02:00
Alex f29b91232f
Use Cell instead of ArcSwap
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-02 18:11:32 +02:00
Alex fbd5b64ff3
Complete sled abstraction
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-02 17:38:30 +02:00
Alex 6805e184e9
Do not put sled in garage's cargo.toml
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-06-02 17:01:04 +02:00
Alex 9f0f5b2e37
Adapt Garage to use new DB abstraction 2022-06-02 16:58:00 +02:00
Alex 04901093e7
Implement iter() and range() on db
Some checks failed
continuous-integration/drone/push Build is failing
2022-06-02 15:25:24 +02:00
Alex fd8d5c37f7
First iteration of a generic DB layer
Some checks failed
continuous-integration/drone/push Build is failing
2022-06-02 14:59:26 +02:00
Alex b54a938724 Fix garage_version() now that GIT_VERSION is read in crate garage_rpc
Some checks reported errors
continuous-integration/drone/tag Build is passing
continuous-integration/drone Build is passing
continuous-integration/drone/push Build was killed
2022-06-02 12:00:10 +02:00
Alex ff06d3f082
Fix Content-Type headers for {admin,k2v} errors and admin responses
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
Fix #315
2022-05-25 17:09:33 +02:00
Alex 93eab8eaa3 Fixes to S3 compatibility page (#314)
All checks were successful
continuous-integration/drone/push Build is passing
Mention PostObject is implemented, fix english mistakes

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #314
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-25 16:54:44 +02:00
Quentin 43ddc933f9
Update Ceph S3 endpoints compatibility
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-05-25 15:20:08 +02:00
Alex 9f303f6308
Shorter page title
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-05-24 15:47:42 +02:00
Alex 3be43f3372
Add lost content for Restic with Garage
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
Suggested-by: Quentin <quentin@deuxfleurs.fr>
2022-05-24 15:32:42 +02:00
Alex 2da448b43f
Add documentation for new Admin API and a few infos on K2V
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-05-24 15:28:37 +02:00
Alex b2a2d3859f K2V client improvements (#307)
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone Build is passing
- [x] Better distinguish error types
- [x] Parse error messages received from server
- [x] Remove `src/` folder layer, we don't have that for other crates

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #307
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-24 12:48:05 +02:00
Alex 382e74c798 First version of admin API (#298)
All checks were successful
continuous-integration/drone/push Build is passing
**Spec:**

- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases

View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md>

**Code:**

- [x] Refactor code for admin api to use common api code that was created for K2V

**General endpoints:**

- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout

**Key-related endpoints:**

- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey

**Bucket-related endpoints:**

- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite

**Operations on key/bucket permissions:**

- [x] BucketAllowKey
- [x] BucketDenyKey

**Operations on bucket aliases:**

- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket

**And also:**

- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design

Fix #231
Fix #295

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-24 12:16:39 +02:00
trinity-1686a 64c193e3db Add a K2V client library and CLI (#303)
All checks were successful
continuous-integration/drone/push Build is passing
lib.rs could use getting split in modules, but I'm not sure how exactly

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Reviewed-on: #303
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-05-18 22:24:09 +02:00
Alex c692f55d5c
K2V: Fix end parameter and add tests (fix #305)
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-05-17 11:50:23 +02:00
Alex 7b474855e3
Make background runner terminate correctly 2022-05-17 11:38:31 +02:00
Alex 176715c5b2
Fix ReadIndex spec and add JSON5 remark to doc
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-05-16 11:54:37 +02:00
Alex 5768bf3622 First implementation of K2V (#293)
All checks were successful
continuous-integration/drone/push Build is passing
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-10 13:16:57 +02:00
Alex def78c5e6f
Update netapp to 0.4.4, fix #300
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-05-09 12:08:47 +02:00
Alex 277a20ec44 Fix layout show to not show changes when there are no changes (#297)
All checks were successful
continuous-integration/drone/push Build is passing
fixes #295, partially

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #297
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-09 11:14:55 +02:00
Alex c9ef3e461b
fix clippy
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-04-19 12:50:40 +02:00
Alex c93008d333
Prettier code for defragmentation 2022-04-19 12:50:40 +02:00
Alex e5341ca47b
Defragmentation in UploadPartCopy: first pass (not pretty but it compiles) 2022-04-19 12:50:36 +02:00
withinboredom a4f9f19ac3 remove size limitation in UploadPartCopy (#280)
All checks were successful
continuous-integration/drone/push Build is passing
This removes the >1mb s3_copy restriction.

This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248).

Co-authored-by: Rob Landers <landers.robert@gmail.com>
Reviewed-on: #280
Co-authored-by: withinboredom <landers.robert@gmail.com>
Co-committed-by: withinboredom <landers.robert@gmail.com>
2022-04-19 12:49:43 +02:00
Baptiste Jonglez 47e57518ec Add documentation on running Kopia with Garage
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-04-10 13:04:07 +02:00
Alex dffcd9f4b1
update Cargo.nix
All checks were successful
continuous-integration/drone/tag Build is passing
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
2022-04-08 14:35:09 +02:00
Alex 5d404dcd54
Add missing opentelemetry features 2022-04-08 14:21:04 +02:00
Quentin 62f0715abe Add/Fix OpenTelemetry
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-04-07 16:12:35 +02:00