jpds/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues 30 Pull requests Projects Releases Packages Wiki Activity
745 commits 43 branches 48 tags 59 MiB
Commit graph

513 commits

Author SHA1 Message Date
Alex Auvolat
96b11524d5
Error refactoring 2022-05-13 15:04:53 +02:00
Alex Auvolat
c0fb9fd0fe
Common error type and admin error type that uses it 2022-05-13 14:30:30 +02:00
Alex Auvolat
983037d965
Possibility of different error types for different APIs 2022-05-13 13:51:34 +02:00
Alex Auvolat
e4e1f8f0d6
Fix clippy 2022-05-12 17:11:45 +02:00
Alex Auvolat
e7ddba53e3
Slightly more detailed error reporting from helper 2022-05-12 17:10:25 +02:00
Alex Auvolat
ed76893581
Simplify 2022-05-12 11:21:23 +02:00
Alex Auvolat
fc2f73ddb5
BucketAllowKey and BucketDenyKey 2022-05-12 11:19:41 +02:00
Alex Auvolat
fe399a3265
DeleteBucket 2022-05-12 11:02:36 +02:00
Alex Auvolat
de1a5b87b6
CreateBucket 2022-05-12 10:45:09 +02:00
Alex Auvolat
2b93a01d2b
ListBucket and GetBucketInfo 2022-05-12 10:20:34 +02:00
Alex Auvolat
393b76ecba
Implement CreateKey, DeleteKey and rudimentary UpdateKey 2022-05-11 11:40:26 +02:00
Alex Auvolat
5c00c9fb46
First key endpoints: ListKeys and GetKeyInfo 2022-05-11 11:10:28 +02:00
Alex Auvolat
f97a7845e9
Add API access key admin endpoints 2022-05-11 10:27:40 +02:00
Alex Auvolat
bb6ec9ebd9
Update Cargo.nix and improve log message 2022-05-10 13:36:35 +02:00
Alex Auvolat
dd54d0b2b1
Refactor code for apply/revert, implement Update/Apply/RevertLayout 2022-05-10 13:25:10 +02:00
Alex Auvolat
01c4876fb4
Specify remaining cluster-related endpoints 2022-05-10 13:25:10 +02:00
Alex Auvolat
ec03e3d16c
Fmt & cleanup 2022-05-10 13:25:10 +02:00
Alex Auvolat
7a19daafbd
Implement /status Admin endpoint 2022-05-10 13:25:10 +02:00
Alex Auvolat
99fcfa3844
Make background runner terminate correctly 2022-05-10 13:25:10 +02:00
Alex Auvolat
633958c7b1
Refactor admin API to be in api/admin and use common code 2022-05-10 13:25:06 +02:00
Alex
5768bf3622 First implementation of K2V (#293) 
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-10 13:16:57 +02:00
Alex Auvolat
def78c5e6f
Update netapp to 0.4.4, fix #300 2022-05-09 12:08:47 +02:00
Alex
277a20ec44 Fix layout show to not show changes when there are no changes (#297) 
fixes #295, partially

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#297
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-09 11:14:55 +02:00
Alex Auvolat
c9ef3e461b
fix clippy 2022-04-19 12:50:40 +02:00
Alex Auvolat
c93008d333
Prettier code for defragmentation 2022-04-19 12:50:40 +02:00
Alex Auvolat
e5341ca47b
Defragmentation in UploadPartCopy: first pass (not pretty but it compiles) 2022-04-19 12:50:36 +02:00
withinboredom
a4f9f19ac3 remove size limitation in UploadPartCopy (#280) 
This removes the >1mb s3_copy restriction.

This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248).

Co-authored-by: Rob Landers <landers.robert@gmail.com>
Reviewed-on: Deuxfleurs/garage#280
Co-authored-by: withinboredom <landers.robert@gmail.com>
Co-committed-by: withinboredom <landers.robert@gmail.com>
 2022-04-19 12:49:43 +02:00
Alex Auvolat
5d404dcd54
Add missing opentelemetry features 2022-04-08 14:21:04 +02:00
Alex Auvolat
94f1e48fff Update to netapp 0.4.2 (a tiny fix) 2022-04-07 11:50:03 +02:00
Alex Auvolat
cb5836d53c Bring maximum exponential backoff time down from 16h to 1h 2022-04-07 11:49:29 +02:00
Alex Auvolat
0091002ef2
New replication modes and their documentation 2022-03-28 16:26:04 +02:00
Alex Auvolat
8f9cf3a5d1
fix a clippy lint 2022-03-28 15:48:55 +02:00
Alex Auvolat
913f7754bb
Add blocks in errored state to garage stats 2022-03-28 15:47:23 +02:00
Alex Auvolat
42dde54126
Log admin GET requests at debug level instead of info 
to reduce noise in logs
 2022-03-28 15:46:52 +02:00
Alex Auvolat
9d0ed78887 Add feature flag for Kubernetes discovery 2022-03-24 16:57:43 +01:00
Alex Auvolat
509d256c58
Make layout optimization work in relative terms 2022-03-24 15:27:14 +01:00
Alex Auvolat
2814d41842
Allow garage layout assign to assign to several nodes at once 2022-03-24 15:27:13 +01:00
Alex Auvolat
7e0e2ffda2
Slight change and add comment to layout assignation algo 2022-03-24 15:27:13 +01:00
Alex Auvolat
413ab0eaed
Small change to partition assignation algorithm 
This change helps ensure that nodes for each partition are spread
over all datacenters, a property that wasn't ensured previously
when going from a 2 DC deployment to a 3 DC deployment
 2022-03-24 15:27:10 +01:00
Alex Auvolat
3dc9214172
Add lots of comments on how the resync queue works 
(I don't really want to change/refactor that code though)
 2022-03-23 10:25:39 +01:00
Alex Auvolat
077dd1cde9
Clippy 2022-03-23 10:25:39 +01:00
Alex Auvolat
e480aaf338
Make background tranquility a configurable parameter 2022-03-23 10:25:19 +01:00
Alex Auvolat
8fd6745745
Move block RC code to separate rc.rs 2022-03-23 10:25:19 +01:00
Alex Auvolat
c3982a90b6
Move DataBlock out of manager.rs 2022-03-23 10:25:19 +01:00
Alex Auvolat
c1d9854d2c
Move block manager to separate module 2022-03-23 10:25:15 +01:00
trinity-1686a
8565f7dc31 cleanup 2022-03-23 10:22:37 +01:00
trinity-1686a
8db6b84559 add test for create bucket and put website with streaming signature 2022-03-23 10:22:37 +01:00
trinity-1686a
1eb7fdb08f add test framework for arbitraty S3 requests 
and implement some basic test with it
 2022-03-23 10:22:36 +01:00
KokaKiwi
e934934f14 garage_api: Update streaming payload stream unit tests 2022-03-23 10:22:36 +01:00
KokaKiwi
98545a16dd garage_api: Handle streaming payload early in request handling 2022-03-23 10:22:36 +01:00