jpds/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues 3 Pull requests Packages Projects Releases Wiki Activity
723 commits 43 branches 48 tags 63 MiB
Commit graph

27 commits

Author SHA1 Message Date
Alex 5768bf3622 First implementation of K2V (#293) 
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-10 13:16:57 +02:00
Alex c9ef3e461b
fix clippy 2022-04-19 12:50:40 +02:00
Alex c93008d333
Prettier code for defragmentation 2022-04-19 12:50:40 +02:00
Alex e5341ca47b
Defragmentation in UploadPartCopy: first pass (not pretty but it compiles) 2022-04-19 12:50:36 +02:00
withinboredom a4f9f19ac3 remove size limitation in UploadPartCopy (#280) 
This removes the >1mb s3_copy restriction.

This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248).

Co-authored-by: Rob Landers <landers.robert@gmail.com>
Reviewed-on: Deuxfleurs/garage#280
Co-authored-by: withinboredom <landers.robert@gmail.com>
Co-committed-by: withinboredom <landers.robert@gmail.com>
 2022-04-19 12:49:43 +02:00
trinity-1686a f6f8b7f1ad Support for PostObject (#222) 
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html)

- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#222
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-02-21 23:02:30 +01:00
trinity-1686a e55fa38c99 Add date verification to presigned urls (#196) 
fix #96
fix #162 by returning Forbidden instead Bad Request

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-01-18 12:22:31 +01:00
Alex f7349f4005
Add quotes in returned etags 2022-01-13 14:03:33 +01:00
Alex 1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures 2022-01-13 14:03:30 +01:00
Alex 6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
Alex 3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187) 2022-01-13 13:56:55 +01:00
Alex 168a90dfb5
Fix some error codes 2022-01-05 17:07:36 +01:00
Alex 5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
Alex 6ccffc3162
Improved XML serialization 
- Use quick_xml and serde for all XML response returned by the S3 API.
- Include tests for all structs used to generate XML
- Remove old manual XML escaping function which was unsafe
 2021-05-06 22:37:15 +02:00
Trinity Pointard 84856e84e5
fix clippy warnings on api 2021-05-03 22:11:41 +02:00
Alex 3bf2df622a Time and metadata improvements 2021-03-15 16:21:41 +01:00
Alex 097c339d98 Fix race condition 2021-03-15 15:26:29 +01:00
Alex f319a7d374 Refactor model stuff, including cleaner CRDTs 2021-03-10 16:21:56 +01:00
Alex 40763fd749 Cargo fmt 2021-02-23 18:46:25 +01:00
Alex 1de96248e0 add application/xml header and missing xml escapes 2021-02-19 23:40:18 +01:00
Quentin 5a5592c176 Replace with option syntaxic sugar 2020-11-11 16:12:42 +01:00
Alex a50fa70d45 Refactor error management in API part 2020-11-08 15:05:28 +01:00
Alex 44dba0e53c cargo fmt 2020-07-08 17:34:37 +02:00
Alex 64a6eda0d2 Migrate S3 api to use new model 2020-07-08 17:33:24 +02:00
Alex f22ecb60a8 Update to Hyper 0.13.6 that accepts non-Sync streams in wrap_stream. 
Simplifies code and makes it possible to publish on crates.io
 2020-07-07 17:15:53 +02:00
Alex bec26a1312 Rename garage_core to garage_model 2020-07-07 13:59:22 +02:00
Alex 0957d0fdfa Work on API 2020-04-28 10:18:14 +00:00