Commit graph

349 commits

Author SHA1 Message Date
ceac3713d6 modifications in several files to :
- have consistent error return types
- store the zone redundancy in a Lww
- print the error and message in the CLI (TODO: for the server Api, should msg be returned in the body response?)
2022-10-05 15:29:48 +02:00
1778e4b318
Fix span name for api server requests 2022-09-26 16:21:30 +02:00
782630fc27
Initialize metrics exporter earlier (fix #389) 2022-09-20 17:50:22 +02:00
5d4b6f2173
Faster GetObject workflow for getting entire objects 2022-09-19 12:19:59 +02:00
76f42a1a2b
Properly return HTTP 204 when deleting non-existent object (fix #227) 2022-09-14 17:07:55 +02:00
38be811b1c
Fix clippy lint that says we should implement Eq 2022-09-13 16:08:00 +02:00
44733474bb
Remove/change println! in server code (fix #358) 2022-09-13 16:01:55 +02:00
ff30891999
Use streaming block API for get with Range requests 2022-09-13 15:13:07 +02:00
7f54706b95
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-09-08 15:50:56 +02:00
d9d199a6c9
Merge branch 'main' into lx-perf-improvements 2022-09-08 15:49:17 +02:00
ceb1f0229a
Move version back into util 2022-09-07 18:36:46 +02:00
2559f63e9b
Make all HTTP services optionnal 2022-09-07 17:54:16 +02:00
28d86e7602
Report build features in garage --help 2022-09-07 17:05:21 +02:00
db61f41030
Move GIT_VERSION injection later in build chain to reduce build times 2022-09-07 11:59:56 +02:00
907054775d
Faster copy, better get error message 2022-09-06 22:25:23 +02:00
431dee050f
Remove opentelemetry-otlp dep in api/ 2022-09-06 17:25:44 +02:00
48ffaaadfc
Bump versions to 0.8.0 (compatibility is broken already) 2022-09-06 16:47:56 +02:00
ea36b9ff90 Allow building without Prometheus exporter (/metrics endpoint)
prometheus and opentelemetry-prometheus add 7 extra dependencies in
total and increases the size of the garage binary by ~7 % (with
fat LTO).
2022-09-06 01:15:09 +02:00
e7af006c1c Make OTLP exporter optional via feature "telemetry-otlp"
opentelemetry-otlp add 48 (!) extra dependencies and increases the
size of the garage binary by ~11 % (with fat LTO).
2022-09-06 01:14:47 +02:00
13b5f28c7e
Make use of BytesBuf from new Netapp 2022-09-02 13:46:42 +02:00
bc977f9a7a
Update to Netapp with OrderTag support and exploit OrderTags 2022-09-01 12:58:20 +02:00
68087ee13d
Fix clippy 2022-07-29 12:25:03 +02:00
605a630333
Use streaming in block manager 2022-07-29 12:25:02 +02:00
16f6a1a65d
fix clippy 2022-07-29 12:24:49 +02:00
ad35b18bb1
Faster chunker 2022-07-29 12:24:49 +02:00
2cad656a03
More make clippy happy 2022-07-29 12:24:48 +02:00
2f111e6b3d
Performance improvements:
- reduce contention on mutation_lock by having 256 of them
- better lmdb defaults
2022-07-29 12:24:48 +02:00
1b2e1296eb
Compute hashes on dedicated threads 2022-07-29 12:24:44 +02:00
77e3fd6db2 improve internal item counter mechanisms and implement bucket quotas (#326)
- [x] Refactoring of internal counting API
- [x] Repair procedure for counters (it's an offline procedure!!!)
- [x] New counter for objects in buckets
- [x] Add quotas to buckets struct
- [x] Add CLI to manage bucket quotas
- [x] Add admin API to manage bucket quotas
- [x] Apply quotas by adding checks on put operations
- [x] Proof-read

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#326
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-06-15 20:20:28 +02:00
d544a0e0e0
Send CORS headers for all requests 2022-06-13 10:19:52 +02:00
b44d3fc796 Abstract database behind generic interface and implement alternative drivers (#322)
- [x] Design interface
- [x] Implement Sled backend
  - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
  - [x] `.updated()` trigger on Garage tables

Fix #284

**Bugs**

- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work

**Known issues for various back-ends**

- Sled:
  - Eats all my RAM and also all my disk space
  - `.len()` has to traverse the whole table
  - Is actually quite slow on some operations
  - And is actually pretty bad code...
- Sqlite:
  - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  - (adapter uses a bunch of unsafe code)
- Heed (LMDB):
  - Not suited for 32-bit machines as it has to map the whole DB in memory.
  - (adpater uses a tiny bit of unsafe code)

**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.

**Converting databases** is actually quite easy. For example from Sled to LMDB:

```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```

Then, just add this to your `config.toml`:

```toml
db_engine = "lmdb"
```

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-06-08 10:01:44 +02:00
ff06d3f082
Fix Content-Type headers for {admin,k2v} errors and admin responses
Fix #315
2022-05-25 17:09:33 +02:00
382e74c798 First version of admin API (#298)
**Spec:**

- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases

View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md>

**Code:**

- [x] Refactor code for admin api to use common api code that was created for K2V

**General endpoints:**

- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout

**Key-related endpoints:**

- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey

**Bucket-related endpoints:**

- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite

**Operations on key/bucket permissions:**

- [x] BucketAllowKey
- [x] BucketDenyKey

**Operations on bucket aliases:**

- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket

**And also:**

- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design

Fix #231
Fix #295

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-24 12:16:39 +02:00
c692f55d5c
K2V: Fix end parameter and add tests (fix #305) 2022-05-17 11:50:23 +02:00
5768bf3622 First implementation of K2V (#293)
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-10 13:16:57 +02:00
c9ef3e461b
fix clippy 2022-04-19 12:50:40 +02:00
c93008d333
Prettier code for defragmentation 2022-04-19 12:50:40 +02:00
e5341ca47b
Defragmentation in UploadPartCopy: first pass (not pretty but it compiles) 2022-04-19 12:50:36 +02:00
a4f9f19ac3 remove size limitation in UploadPartCopy (#280)
This removes the >1mb s3_copy restriction.

This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248).

Co-authored-by: Rob Landers <landers.robert@gmail.com>
Reviewed-on: Deuxfleurs/garage#280
Co-authored-by: withinboredom <landers.robert@gmail.com>
Co-committed-by: withinboredom <landers.robert@gmail.com>
2022-04-19 12:49:43 +02:00
8f9cf3a5d1
fix a clippy lint 2022-03-28 15:48:55 +02:00
c1d9854d2c
Move block manager to separate module 2022-03-23 10:25:15 +01:00
1eb7fdb08f add test framework for arbitraty S3 requests
and implement some basic test with it
2022-03-23 10:22:36 +01:00
e934934f14 garage_api: Update streaming payload stream unit tests 2022-03-23 10:22:36 +01:00
98545a16dd garage_api: Handle streaming payload early in request handling 2022-03-23 10:22:36 +01:00
ba6b56ae68
Fix some new clippy lints 2022-03-14 12:27:49 +01:00
203e8d2c34
Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
d9a35359bf
Add metrics to web endpoint 2022-03-14 10:53:50 +01:00
2a5609b292
Add metrics to API endpoint 2022-03-14 10:53:36 +01:00
f0d0cd9a20
Remove strum crate dependency; add protobuf nix dependency 2022-03-14 10:53:00 +01:00
bb04d94fa9
Update to Netapp 0.4 which supports distributed tracing 2022-03-14 10:52:30 +01:00
8c2fb0c066
Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
8a5bbc3b0b
More permissive OPTIONS on S3 API 2022-03-01 11:15:16 +01:00
97f245f218
Add tracing output to signature calculation 2022-02-28 12:22:39 +01:00
8129a98291
Process CORS earlier in pipeline 2022-02-28 12:22:39 +01:00
f6f8b7f1ad Support for PostObject (#222)
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html)

- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#222
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-02-21 23:02:30 +01:00
c99f55c420
Add restriction on part ordering in CompleteMultipartUpload 2022-01-25 12:45:00 +01:00
acdf893362
Fix partnumber 2022-01-25 12:25:23 +01:00
338b1b83ee
Implement part_number for GetObject 2022-01-24 21:04:42 +01:00
6dab836f3a
Multipart improvements
- support part_number for HeadObject
- add checks in complete_multipart_upload
2022-01-24 21:04:40 +01:00
513a6b15f9
Handle OPTIONS on website endpoint 2022-01-24 12:32:28 +01:00
ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
2022-01-24 11:58:00 +01:00
820924534a use clamp instead of min(max()) 2022-01-24 11:56:59 +01:00
440374524b Implement ListParts 2022-01-21 10:42:30 +01:00
fe003d6fbc Add ListPartsResult structure 2022-01-20 16:38:55 +01:00
e55fa38c99 Add date verification to presigned urls (#196)
fix #96
fix #162 by returning Forbidden instead Bad Request

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-01-18 12:22:31 +01:00
178e35f868 refactor s3_router and api_server to make unused Endpoint parameters more obvious 2022-01-17 15:50:24 +01:00
fdcddbe168 Fix Multipart Upload with WinSCP (#164) (#193)
Closes #164.

Reviewed-on: Deuxfleurs/garage#193
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 11:18:40 +01:00
b45dcc1925 Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156)
Closes #64.

Reviewed-on: Deuxfleurs/garage#156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 10:55:31 +01:00
d4dd2e2640
Make use of website config, return error document on error 2022-01-13 14:25:19 +01:00
3ea8ca1b9e
Implement GetBucketWebsite 2022-01-13 14:23:52 +01:00
f7349f4005
Add quotes in returned etags 2022-01-13 14:03:33 +01:00
1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures 2022-01-13 14:03:30 +01:00
6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187) 2022-01-13 13:56:55 +01:00
b4592a00fe Implement ListMultipartUploads (#171)
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
2022-01-12 19:04:55 +01:00
3586c7257c
Clarify error message 2022-01-10 14:10:04 +01:00
7ee11f0eb6
Fix unit tests 2022-01-05 17:34:48 +01:00
168a90dfb5
Fix some error codes 2022-01-05 17:07:36 +01:00
135858d067
Implement DeleteBucket 2022-01-05 16:28:19 +01:00
8395030e48
Implement CreateBucket 2022-01-05 15:56:48 +01:00
677ab60cc1
Small changes in key model and refactoring 2022-01-04 18:59:17 +01:00
df35feba18
New buckets for 0.6.0: make bucket id a SK and not a HK, CLI updates 2022-01-04 12:53:14 +01:00
1bcd6fabbd
New buckets for 0.6.0: small changes
- Fix bucket delete

- fix merge of bucket creation date

- Replace deletable with option in aliases
    Rationale: if two aliases point to conflicting bucket, resolving
    by making an arbitrary choice risks making data accessible when it
    shouldn't be. We'd rather resolve to deleting the alias until
    someone puts it back.
2022-01-04 12:52:47 +01:00
2140cd7205
Remove website redirects 2022-01-04 12:52:46 +01:00
beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
d8ab5bdc3e
New buckets for 0.6.0: fix model and migration 2022-01-04 12:47:28 +01:00
b76d0580a0
Fix forgotten flag 2022-01-04 12:47:28 +01:00
b1cfd16913
New buckets for 0.6.0: small fixes, including:
- ensure bucket names are correct aws s3 names
- when making aliases, ensure timestamps of links in both ways are the
  same
- fix small remarks by trinity
- don't have a separate website_access field
2022-01-04 12:46:41 +01:00
0bbb6673e7
Model changes 2022-01-04 12:45:52 +01:00
5b1117e582
New model for buckets 2022-01-04 12:45:46 +01:00
ca7b438f3f less strict ListBuckets (#178)
fix #175

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#178
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 15:05:36 +01:00
60d4459926 BucketWebsite (#174)
fix #77

this does not store anything but a on/off switch for website, and does not implement GetBucketWebsite as it would require storing more. GetBucketWebsite should be pretty easy to implement once data is stored though.

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#174
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-15 10:41:39 +01:00
c4ac8835d3 add proper request router for s3 api (#163)
fix #161

Current request router was organically grown, and is getting messier and messier with each addition.
This router cover exaustively existing API endpoints (with exceptions listed in [#161(comment)](Deuxfleurs/garage#161 (comment)) either because new and old api endpoint can't feasabily be differentied, or it's more lambda than s3).

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#163
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-12-06 15:17:47 +01:00
7f26ed55cd Improved handling of HTTP ranges
- correct HTTP code when range syntax is invalid (fix #140)
- when multiple ranges are given, simply ignore and send whole file

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#157
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2021-11-29 11:52:42 +01:00
8811bb08e6
In ListBuckets, hide entry if no perms 2021-11-22 12:10:28 +01:00
c94406f428
Improve how node roles are assigned in Garage
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
2021-11-16 16:05:53 +01:00
53888995bd update doc and comments 2021-11-16 15:41:41 +01:00
396fe4c702 clippy 2021-11-16 15:41:41 +01:00
02158ee666 fix issue where list on vhost-bucket would list bucket instead of bucket content 2021-11-16 15:41:41 +01:00
9c58ec28d3 add support for vhost-style s3 bucket 2021-11-16 15:41:41 +01:00
ada7899b24
Fix clippy lints (fix #121) 2021-10-26 10:20:05 +02:00
de4276202a
Improve CLI, adapt tests, update documentation 2021-10-25 14:21:48 +02:00
1b450c4b49
Improvements to CLI and various fixes for netapp version
Discovery via consul, persist peer list to file
2021-10-22 16:55:24 +02:00
4067797d01
First port of Garage to Netapp 2021-10-22 15:55:18 +02:00
f6060b92aa
Fix HTTP return code for DeleteObject (fix #98) 2021-10-11 14:24:49 +02:00
f3a097abdf
WIP: try to fix #93, and improve S3 ListObjects (v1 and v2) API calls 2021-10-11 11:15:47 +02:00
b9127dd6f8
Prepare for v0.3.0 and add migration path from v0.2.1.x 2021-05-28 15:29:58 +02:00
5fdabf3e75
Add basic support for the "Versioning" command 2021-05-14 22:33:26 +02:00
6ccffc3162
Improved XML serialization
- Use quick_xml and serde for all XML response returned by the S3 API.
- Include tests for all structs used to generate XML
- Remove old manual XML escaping function which was unsafe
2021-05-06 22:37:15 +02:00
e4b9e4e24d
rename types to CamelCase 2021-05-03 22:15:09 +02:00
84856e84e5
fix clippy warnings on api 2021-05-03 22:11:41 +02:00
631c36b3ff S3 API: support ListBuckets 2021-05-03 21:55:30 +02:00
ee2a3d363b
Remove STREAMING-AWS4-HMAC-SHA256-PAYLOAD (see #64) 2021-05-03 17:30:40 +02:00
dcfc32cf85
Many S3 compatibility improvements:
- return XML errors
- implement AuthorizationHeaderMalformed error to redirect clients to
  correct location (used by minio client)
- implement GetBucketLocation
- fix DeleteObjects XML parsing and response
2021-04-28 01:05:40 +02:00
642186c530
Fix #59 (& issue) 2021-04-27 23:10:43 +02:00
74373aebcf
make most requested changes 2021-04-27 16:47:08 +02:00
1e3df189d0
document api crate 2021-04-27 16:37:10 +02:00
f859d15062 update to v0.2.1 2021-03-19 13:39:18 +01:00
4c26a0b9c1 Update Cargo.toml files with AGPL license info 2021-03-18 21:59:17 +01:00
6edbc65847 Add trinity's comment in the code 2021-03-18 19:46:43 +01:00
bfa0ff8f82 Merge pull request 'add support for caching headers' (#49) from trinity-1686a/garage:cache-headers into master
Reviewed-on: Deuxfleurs/garage#49
2021-03-18 19:45:02 +01:00
dead945c8f Prepare for release 0.2 2021-03-18 19:33:15 +01:00
b4c903371c add support for caching headers 2021-03-18 15:46:33 +01:00
f4346cc5f4 Update dependencies 2021-03-16 15:58:40 +01:00
0cd5b2ae19 WIP migrate to tokio 1 2021-03-15 22:36:41 +01:00
667e4e72a8 Small fixes 2021-03-15 19:51:16 +01:00
3bf2df622a Time and metadata improvements 2021-03-15 16:21:41 +01:00
097c339d98 Fix race condition 2021-03-15 15:26:29 +01:00
bdcbdd1cd8 Fix list API bug 2021-03-15 14:46:37 +01:00
831eb35763 cargo fmt 2021-03-12 21:52:19 +01:00
c475471e7a Implement table gc, currently for block_ref and version only 2021-03-12 19:57:37 +01:00
7fdaf7aef0 Fix merkle updater not being notified; improved logging 2021-03-12 14:37:46 +01:00
3214dd52dd Very minor changes 2021-03-10 21:50:09 +01:00
af7600f989 Correctly implement CompleteMultipartUpload with etag check of parts 2021-03-10 17:01:05 +01:00
0fd7df8fa0 Switch to blake2 sum for identifying blocks by their data 2021-03-10 16:33:31 +01:00
f319a7d374 Refactor model stuff, including cleaner CRDTs 2021-03-10 16:21:56 +01:00
40763fd749 Cargo fmt 2021-02-23 18:46:25 +01:00
b1b640ae8b rename hash() to sha256sum(), we might want to change it at some places 2021-02-21 15:24:30 +01:00
3bcbbe1e31 More precise logging (warn only when returning a 500) 2021-02-20 00:30:39 +01:00
10b983b8e7 Add verification of part numbers in CompleteMultipartUpload (WIP #30) 2021-02-20 00:13:07 +01:00
1de96248e0 add application/xml header and missing xml escapes 2021-02-19 23:40:18 +01:00
5d1fa591d9 Add compatibility list 2021-02-19 19:10:23 +01:00
e64ecbdccd S3 compatibility: return 404 instead of 400 on some multipart commands 2021-02-19 18:51:05 +01:00
55a2a636ca Implement ListObjectsV2 2021-02-19 16:44:06 +01:00
02d512f3fd Fix #28, extra headers being ignored (because of profound stupidity) 2021-02-19 12:38:22 +01:00
76390085ef Small improvements in the S3 put workflow 2021-02-19 12:11:02 +01:00
36814be447 Fix S3 ListObjects result and replace println!s by debug!s 2021-01-16 16:05:54 +01:00
e818f51073 Forgot a bump 2021-01-15 18:36:51 +01:00
022b386a50 Improved compatibility on list API call 2020-12-06 15:39:03 +01:00
4a5bbbb810 Propose ETag fix 2020-12-05 19:23:46 +01:00
76b489f3d3 Reformulate patch 2020-12-05 19:20:07 +01:00
bd7e3d1bd1 Fix Content-Length 2020-12-05 18:57:22 +01:00
9f46fb699a Content-range fix 2020-12-05 16:37:59 +01:00
d54f15b2c6 Small optimisation 2020-11-29 17:07:14 +01:00
fed97f37e1 ETag patch 2020-11-29 16:38:01 +01:00
601ae25ad2 Small refactorings 2020-11-29 16:21:28 +01:00
17dc610f8a Also check hash for < 3KB files 2020-11-22 11:14:46 +01:00
435d5f9205 Fix base64/hex checksum comparison 2020-11-22 11:04:33 +01:00
f8a04852a2 Convert bucket table to better CRDT representation 2020-11-20 23:53:54 +01:00
e02e9e035e Begin improve model to use better CRDTs 2020-11-20 23:53:54 +01:00
e9fd265ce6 Slight refactoring to make things clearer with DeletedFilter 2020-11-20 20:11:04 +01:00
7d7b9e95a9 Simplify and_then(Some) as map() and remove move 2020-11-11 16:36:48 +01:00
5a5592c176 Replace with option syntaxic sugar 2020-11-11 16:12:42 +01:00
a50fa70d45 Refactor error management in API part 2020-11-08 15:05:28 +01:00
6a8b972f3a Modif parse_bucket_key to fail when bucket is "" 2020-11-08 13:39:44 +01:00
e1415f5976 Add a second test 2020-11-07 15:34:53 +01:00
9a50ce12a8 Fix formatting 2020-11-07 13:59:30 +01:00
8f4ada1965 Add a test for parse_bucket_key 2020-11-07 13:53:32 +01:00
5faf069e33 trace test 2020-07-15 15:41:49 +02:00
27a0d0d859 pretty 2020-07-15 15:37:35 +02:00
1c70552f95 Validate content MD5 and SHA256 sums for PutObject and UploadPart 2020-07-15 15:31:13 +02:00
6c7f9704ea Implement correct ETag for objects created with PutObject 2020-07-13 16:51:30 +02:00
9305e5e87f More headers taken into account 2020-07-09 17:04:43 +02:00
44dba0e53c cargo fmt 2020-07-08 17:34:37 +02:00
64a6eda0d2 Migrate S3 api to use new model 2020-07-08 17:33:24 +02:00
86bf4dedac Add support for model migrations 2020-07-08 16:10:53 +02:00
86fb7bbba5 Apply cargo fmt; add trace output when request signature is bad 2020-07-08 13:33:02 +02:00
f22ecb60a8 Update to Hyper 0.13.6 that accepts non-Sync streams in wrap_stream.
Simplifies code and makes it possible to publish on crates.io
2020-07-07 17:15:53 +02:00
3b0b11085e Add versions to dependencies 2020-07-07 14:18:47 +02:00
cc65cdc0fe Add license, description and repository to .toml files 2020-07-07 14:14:58 +02:00
bec26a1312 Rename garage_core to garage_model 2020-07-07 13:59:22 +02:00
b46a7788d1 Implement HTTP ranges in get 2020-05-04 13:09:23 +00:00
d867bbcfb5 Implement DeleteObjects 2020-05-01 15:52:35 +00:00
3324971701 Slightly improved S3 compatibility
- ListBucket does not require any of the parameters (delimiter,
    prefix, max-keys, etc)
- URLs are properly percent_decoded
- PutObject and DeleteObject calls now answer correctly
    (empty body, version id in the x-amz-version-id: header)
2020-05-01 14:30:50 +00:00
3686f100b7 Compatibility fixes 2020-04-28 10:35:04 +00:00
0957d0fdfa Work on API 2020-04-28 10:18:14 +00:00
0877a5500c Abort multipart upload 2020-04-26 20:46:33 +00:00
81ecc4999e Implement multipart uploads 2020-04-26 20:39:32 +00:00
1999c0ae51 Update delete code 2020-04-26 19:11:19 +00:00
9cb870f950 Prepare for multipart uploads 2020-04-26 18:55:13 +00:00
ea7e4748ed S3 compatibility: fix bucket listing and HEAD and PUT on bucket 2020-04-26 16:22:33 +00:00
e3203f998b Remove leading / in keys; better delimiter handling 2020-04-24 22:28:15 +02:00
be4831d768 Less verbosity 2020-04-24 19:27:27 +00:00
a52db67954 xml escape 2020-04-24 18:56:00 +00:00
91b2d1fcc1 Some basic S3 functionnality 2020-04-24 18:47:11 +00:00
f2e05986c4 Starting to be S3 compatible 2020-04-24 17:46:52 +00:00
d8f5e643bc Split code for modular compilation 2020-04-24 10:10:01 +00:00
44a1089d95 Make table objects slightly more fool-proof; add key table 2020-04-23 18:16:52 +00:00
c9c6b0dbd4 Reorganize code 2020-04-23 17:05:46 +00:00