Commit graph

43 commits

Author SHA1 Message Date
5d4b6f2173
Faster GetObject workflow for getting entire objects
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-19 12:19:59 +02:00
d9d199a6c9
Merge branch 'main' into lx-perf-improvements
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-08 15:49:17 +02:00
431dee050f
Remove opentelemetry-otlp dep in api/
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-09-06 17:25:44 +02:00
48ffaaadfc
Bump versions to 0.8.0 (compatibility is broken already)
Some checks failed
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2022-09-06 16:47:56 +02:00
ea36b9ff90 Allow building without Prometheus exporter (/metrics endpoint)
Some checks failed
continuous-integration/drone/pr Build is failing
prometheus and opentelemetry-prometheus add 7 extra dependencies in
total and increases the size of the garage binary by ~7 % (with
fat LTO).
2022-09-06 01:15:09 +02:00
e7af006c1c Make OTLP exporter optional via feature "telemetry-otlp"
opentelemetry-otlp add 48 (!) extra dependencies and increases the
size of the garage binary by ~11 % (with fat LTO).
2022-09-06 01:14:47 +02:00
1b2e1296eb
Compute hashes on dedicated threads 2022-07-29 12:24:44 +02:00
382e74c798 First version of admin API (#298)
All checks were successful
continuous-integration/drone/push Build is passing
**Spec:**

- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases

View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md>

**Code:**

- [x] Refactor code for admin api to use common api code that was created for K2V

**General endpoints:**

- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout

**Key-related endpoints:**

- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey

**Bucket-related endpoints:**

- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite

**Operations on key/bucket permissions:**

- [x] BucketAllowKey
- [x] BucketDenyKey

**Operations on bucket aliases:**

- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket

**And also:**

- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design

Fix #231
Fix #295

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-24 12:16:39 +02:00
5768bf3622 First implementation of K2V (#293)
All checks were successful
continuous-integration/drone/push Build is passing
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-10 13:16:57 +02:00
c1d9854d2c
Move block manager to separate module 2022-03-23 10:25:15 +01:00
203e8d2c34
Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
f0d0cd9a20
Remove strum crate dependency; add protobuf nix dependency 2022-03-14 10:53:00 +01:00
8c2fb0c066
Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
f6f8b7f1ad Support for PostObject (#222)
All checks were successful
continuous-integration/drone/push Build is passing
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html)

- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #222
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-02-21 23:02:30 +01:00
b45dcc1925 Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156)
All checks were successful
continuous-integration/drone/push Build is passing
Closes #64.

Reviewed-on: #156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 10:55:31 +01:00
0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
c94406f428
Improve how node roles are assigned in Garage
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
2021-11-16 16:05:53 +01:00
9c58ec28d3 add support for vhost-style s3 bucket 2021-11-16 15:41:41 +01:00
1b450c4b49
Improvements to CLI and various fixes for netapp version
Discovery via consul, persist peer list to file
2021-10-22 16:55:24 +02:00
4067797d01
First port of Garage to Netapp 2021-10-22 15:55:18 +02:00
b9127dd6f8
Prepare for v0.3.0 and add migration path from v0.2.1.x
Some checks reported errors
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build was killed
2021-05-28 15:29:58 +02:00
631c36b3ff S3 API: support ListBuckets
All checks were successful
continuous-integration/drone/push Build is passing
2021-05-03 21:55:30 +02:00
f859d15062 update to v0.2.1
Some checks are pending
continuous-integration/drone/push Build is pending
2021-03-19 13:39:18 +01:00
4c26a0b9c1 Update Cargo.toml files with AGPL license info
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/tag Build is passing
2021-03-18 21:59:17 +01:00
dead945c8f Prepare for release 0.2
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-18 19:33:15 +01:00
f4346cc5f4 Update dependencies
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-16 15:58:40 +01:00
0cd5b2ae19 WIP migrate to tokio 1
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-15 22:36:41 +01:00
e818f51073 Forgot a bump 2021-01-15 18:36:51 +01:00
4a5bbbb810 Propose ETag fix 2020-12-05 19:23:46 +01:00
435d5f9205 Fix base64/hex checksum comparison 2020-11-22 11:04:33 +01:00
a50fa70d45 Refactor error management in API part 2020-11-08 15:05:28 +01:00
6c7f9704ea Implement correct ETag for objects created with PutObject 2020-07-13 16:51:30 +02:00
86bf4dedac Add support for model migrations 2020-07-08 16:10:53 +02:00
f22ecb60a8 Update to Hyper 0.13.6 that accepts non-Sync streams in wrap_stream.
Simplifies code and makes it possible to publish on crates.io
2020-07-07 17:15:53 +02:00
3b0b11085e Add versions to dependencies 2020-07-07 14:18:47 +02:00
cc65cdc0fe Add license, description and repository to .toml files 2020-07-07 14:14:58 +02:00
bec26a1312 Rename garage_core to garage_model 2020-07-07 13:59:22 +02:00
b46a7788d1 Implement HTTP ranges in get 2020-05-04 13:09:23 +00:00
d867bbcfb5 Implement DeleteObjects 2020-05-01 15:52:35 +00:00
3324971701 Slightly improved S3 compatibility
- ListBucket does not require any of the parameters (delimiter,
    prefix, max-keys, etc)
- URLs are properly percent_decoded
- PutObject and DeleteObject calls now answer correctly
    (empty body, version id in the x-amz-version-id: header)
2020-05-01 14:30:50 +00:00
f2e05986c4 Starting to be S3 compatible 2020-04-24 17:46:52 +00:00
d8f5e643bc Split code for modular compilation 2020-04-24 10:10:01 +00:00