Alex
8041d9a827
s3: add xml structures to serialize/deserialize lifecycle configs
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-08-29 17:44:17 +02:00
Alex
2e90e1c124
Merge branch 'main' into next
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone/tag Build was killed
continuous-integration/drone Build is passing
2023-08-29 11:32:42 +02:00
Alex
cece1be1bb
bump version to 0.8.3
continuous-integration/drone/pr Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
2023-08-28 13:17:26 +02:00
Alex
9b4ce4a8ad
admin api: refactor caddy check api code
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2023-08-28 12:17:10 +02:00
Jonathan Davies
f607ac6792
garage/api: cargo: Updated idna dependency to 0.4.
2023-08-28 09:30:27 +00:00
Quentin
24e533f262
support {s3,web}.root_domains in /check endpoint
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-08-08 11:05:42 +02:00
Jonathan Davies
59bfc68f2e
post_object.rs: Fixed typos / grammar.
continuous-integration/drone/pr Build is passing
2023-08-01 15:31:39 +01:00
Jonathan Davies
b94ba47f29
api/Cargo.toml: Updated roxmltree to 0.18.
2023-06-24 14:15:26 +01:00
Alex
8ef42c9609
admin docs: reformatting, key admin: add check
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-06-14 17:19:25 +02:00
Alex
a83a092c03
admin: uniformize layout api and improve code
2023-06-14 17:12:37 +02:00
Alex
7895f99d3a
admin and cli: hide secret keys unless asked
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-06-14 16:56:15 +02:00
Alex
4a82f6380e
admin api: move all endpoints to v1/ by default (v0/ still supported)
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-06-14 14:15:51 +02:00
Alex
28cc9f178a
admin api: make name optionnal for CreateKey
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-06-14 13:56:37 +02:00
Alex
35c108b85d
admin api: switch GetClusterHealth to camelcase ( fix #381 again)
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-06-14 13:53:19 +02:00
Alex
52376d47ca
admin api: change cluster status/layout to use lists and not maps ( fix #377 )
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-06-14 13:45:27 +02:00
Alex
5670367126
multipartupload in test: add forgotten timestamp
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-06-13 23:10:46 +02:00
Alex
90b2d43eb4
Merge branch 'main' into next
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2023-06-13 17:14:11 +02:00
Alex
bf19a44fd9
admin API: add missing camelCase conversions ( fix #381 )
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2023-06-13 16:15:50 +02:00
Alex
942c1f1bfe
multipart uploads: save timestamp
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2023-06-13 10:48:22 +02:00
Alex
a6cc563bdd
UploadPart: automatic cleanup of version (and reference blocked) when interrupted
2023-06-09 16:23:37 +02:00
Alex
c14d3735e5
Add test for multipart uploads and fix part renumbering
2023-06-09 16:23:37 +02:00
Alex
412ab77b08
comments and clippy lint fixes
2023-06-09 16:23:37 +02:00
Alex
511e07ecd4
fix mpu counter (add missing workers) and report info at appropriate places
2023-06-09 16:23:37 +02:00
Alex
8644376ac2
fix test; simplify code
2023-06-09 16:23:37 +02:00
Alex
7ad7dae5d4
fix s3 list test
2023-06-09 16:23:37 +02:00
Alex
82e75c0e29
Adapt S3 API code to use new multipart upload models
...
- Create and PutPart
- completemultipartupload
- upload part copy
- list_parts
2023-06-09 16:23:37 +02:00
Jonathan Davies
8a945ee996
payload.rs: Surround / in inverted commas.
continuous-integration/drone/pr Build is passing
2023-06-06 16:26:06 +01:00
Jonathan Davies
180992d0f1
payload.rs: Fixed typo in error message.
2023-06-06 16:25:29 +01:00
Alex
746b0090e4
k2v signature verification: double urlencoding (see comment in source code)
2023-05-18 11:18:06 +02:00
Jonathan Davies
c783194e8b
*: apply clippy recommendations.
continuous-integration/drone/pr Build is passing
2023-05-09 20:49:34 +01:00
Alex
351d734e6c
Merge branch 'main' into next
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-05-09 12:40:08 +02:00
Jakub Jirutka
d2deee0b8b
Declare garage crates using workspace.dependencies
...
This will allow to really disable "sled" feature without declaring
`default-features = false` in every Cargo.toml where garage_db and
garage_model is used.
See https://doc.rust-lang.org/cargo/reference/workspaces.html#the-dependencies-table
2023-05-09 08:46:15 +00:00
Alex
fa78d806e3
Merge branch 'main' into next
2023-04-25 12:34:26 +02:00
Jonathan Davies
c168383113
api/Cargo.toml: Bumped quick-xml to version 0.26.
continuous-integration/drone/pr Build is failing
2023-04-23 20:14:28 +01:00
Alex
0a1ddcf630
Prepare for v0.8.2
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2023-03-13 18:46:31 +01:00
Jonathan Davies
9ea154ae9c
admin/cluster.rs: Added rust_version.
continuous-integration/drone/pr Build is failing
2023-03-10 14:46:54 +00:00
Jonathan Davies
e4e5196066
api/generic_server.rs: Use new handle_forwarded_for_headers() function.
2023-03-06 11:43:35 +00:00
Jonathan Davies
94d70bec69
generic_server.rs: Added support for logging X-Forwarded-For header.
...
continuous-integration/drone/pr Build is passing
Fixes : #460
2023-02-04 15:19:21 +00:00
Jonathan Davies
1b6ec74748
error.rs: Corrected error messages to say unexpected scope.
2023-02-02 16:20:31 +00:00
Jonathan Davies
9c354f0a8f
Improved bucket authorization response strings.
continuous-integration/drone/pr Build is passing
2023-01-29 20:34:41 +00:00
Jonathan Davies
004bb5b4f1
api_server.rs: Adapted to use query string per Caddy upstream change.
2023-01-29 20:34:37 +00:00
Alex
8e93d69974
More clippy fixes
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2023-01-26 17:26:32 +01:00
Alex
590a0a8450
Merge branch 'main' into k2v-watch-range-2
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is failing
2023-01-26 16:46:40 +01:00
Jonathan Davies
20c1cdf662
Cargo.toml: Loosen tracing dependency to just 0.1.
2023-01-26 11:13:11 +00:00
Jonathan Davies
63e22e71f2
api/Cargo.toml: Updated idna from 0.2 to 0.3.
2023-01-26 11:13:11 +00:00
Jonathan Davies
36944f1839
Cargo.toml: Updated base64 from 0.13 to 0.21.
2023-01-26 11:13:07 +00:00
Jonathan Davies
93c3f8fc8c
api/Cargo.toml: Updated url from 2.1 to 2.3.
2023-01-23 19:16:58 +00:00
Jonathan Davies
1c435fce09
api/Cargo.toml: Updated httpdate from 0.3 to 1.0.
2023-01-23 19:16:54 +00:00
Jonathan Davies
dead123892
api/Cargo.toml: Updated pin-project to 1.0.12.
2023-01-23 18:39:35 +00:00
Jonathan Davies
100b01e859
Implemented website hosting authorization endpoint.
...
Fixes : #468
2023-01-13 09:38:58 +00:00
Alex
cbfae673e8
PollRange & PollItem: min timeout = 1 sec
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-11 15:03:08 +01:00
Alex
bba13f40fc
Correctly return bad requests when seeh marker is invalid
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build was killed
2023-01-11 12:27:19 +01:00
Alex
b83517d521
Implement PollRange API endpoint
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-10 15:22:25 +01:00
Alex
a48e2e0cb2
K2V: Subscription to ranges of items
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-10 10:30:59 +01:00
Alex
570e5e5bbb
Merge branch 'main' into next
continuous-integration/drone/push Build is passing
2023-01-04 11:34:43 +01:00
Alex
0650a43cf1
PutObject: better cleanup on Drop (incl. when request is interrupted in the middle)
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-03 17:05:17 +01:00
Alex
426d8784da
cleanup
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-01-03 15:08:37 +01:00
Alex
6775569525
Bump everything to v0.8.1
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-02 14:15:33 +01:00
Alex
1af4a5ed56
Merge pull request 'Fix router keyword handling ( fix #442 )' ( #446 ) from router-keywords-fix into main
...
continuous-integration/drone/push Build is passing
Reviewed-on: #446
2022-12-15 08:40:26 +00:00
Alex
a0abf41762
Fix router keyword handling ( fix #442 )
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-12-12 12:05:37 +01:00
Alex
2ac75018a1
Properly enforce allow_create_bucket
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-12-12 12:03:54 +01:00
Alex
2c2e65ad8b
Merge commit 'ec12d6c' into next
continuous-integration/drone/push Build is passing
2022-12-11 18:41:15 +01:00
Alex
533afcf4e1
simplify
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-12-11 18:17:08 +01:00
Alex
5ea5fd2130
Always return 200 OK on /v0/health, reinstate admin api doc as draft and complete it
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-12-11 18:11:28 +01:00
Alex
d7868c48a4
Separate /health (simple text answer) and /v0/health (full json answer, authenticated)
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-12-05 15:38:32 +01:00
Alex
280d1be7b1
Refactor health check and add ability to return it in json
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-12-05 15:28:57 +01:00
Alex
2065f011ca
Implement /health admin API endpoint to check node health
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-12-05 14:59:15 +01:00
Quentin
dc50fa3b34
Fix typo in admin API on BucketInfo
continuous-integration/drone/push Build was killed
continuous-integration/drone/pr Build was killed
2022-11-11 16:56:56 +01:00
Alex
217abdca18
Fix HTTP return code
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-11-08 15:38:53 +01:00
Alex
d75b37b018
Return more info when layout's .check() fails, fix compilation, fix test
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2022-11-08 14:58:39 +01:00
Alex
ea5afc2511
Style improvements
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2022-11-07 20:11:30 +01:00
Alex
28d7a49f63
Merge branch 'main' into optimal-layout
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2022-11-07 12:20:59 +01:00
Alex
5670599372
Merge pull request 'Use status code 204 No Content for empty responses' ( #403 ) from tobikris/garage:http-no-content into main
...
continuous-integration/drone/push Build is passing
Reviewed-on: #403
2022-10-18 14:20:44 +00:00
Tobias Krischer
7865003323
Use status code 204 No Content for empty responses
continuous-integration/drone/pr Build is passing
2022-10-17 10:55:26 +02:00
Alex
8d04ae7014
cargo2nix unstable (patched), rust 1.63.0, nixpkgs 22.05 (32-bit builds are broken)
continuous-integration/drone/pr Build encountered an error
continuous-integration/drone/push Build is failing
2022-10-14 14:30:48 +02:00
Mendes
4abab246f1
cargo fmt
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-10-10 17:21:13 +02:00
Mendes
ceac3713d6
modifications in several files to :
...
- have consistent error return types
- store the zone redundancy in a Lww
- print the error and message in the CLI (TODO: for the server Api, should msg be returned in the body response?)
2022-10-05 15:29:48 +02:00
Alex
1778e4b318
Fix span name for api server requests
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-26 16:21:30 +02:00
Alex
782630fc27
Initialize metrics exporter earlier ( fix #389 )
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-20 17:50:22 +02:00
Alex
5d4b6f2173
Faster GetObject workflow for getting entire objects
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-19 12:19:59 +02:00
Alex
76f42a1a2b
Properly return HTTP 204 when deleting non-existent object ( fix #227 )
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-14 17:07:55 +02:00
Alex
38be811b1c
Fix clippy lint that says we should implement Eq
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-13 16:08:00 +02:00
Alex
44733474bb
Remove/change println! in server code ( fix #358 )
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-13 16:01:55 +02:00
Alex
ff30891999
Use streaming block API for get with Range requests
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-13 15:13:07 +02:00
Alex
7f54706b95
Merge branch 'lx-perf-improvements' into netapp-stream-body
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-08 15:50:56 +02:00
Alex
d9d199a6c9
Merge branch 'main' into lx-perf-improvements
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-09-08 15:49:17 +02:00
Alex
ceb1f0229a
Move version back into util
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-07 18:36:46 +02:00
Alex
2559f63e9b
Make all HTTP services optionnal
continuous-integration/drone/push Build is pending
continuous-integration/drone/pr Build is pending
2022-09-07 17:54:16 +02:00
Alex
28d86e7602
Report build features in garage --help
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is passing
2022-09-07 17:05:21 +02:00
Alex
db61f41030
Move GIT_VERSION injection later in build chain to reduce build times
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is passing
2022-09-07 11:59:56 +02:00
Alex
907054775d
Faster copy, better get error message
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-06 22:25:23 +02:00
Alex
431dee050f
Remove opentelemetry-otlp dep in api/
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-09-06 17:25:44 +02:00
Alex
48ffaaadfc
Bump versions to 0.8.0 (compatibility is broken already)
continuous-integration/drone/pr Build is failing
continuous-integration/drone/push Build is failing
2022-09-06 16:47:56 +02:00
Jakub Jirutka
ea36b9ff90
Allow building without Prometheus exporter (/metrics endpoint)
...
continuous-integration/drone/pr Build is failing
prometheus and opentelemetry-prometheus add 7 extra dependencies in
total and increases the size of the garage binary by ~7 % (with
fat LTO).
2022-09-06 01:15:09 +02:00
Jakub Jirutka
e7af006c1c
Make OTLP exporter optional via feature "telemetry-otlp"
...
opentelemetry-otlp add 48 (!) extra dependencies and increases the
size of the garage binary by ~11 % (with fat LTO).
2022-09-06 01:14:47 +02:00
Alex
13b5f28c7e
Make use of BytesBuf from new Netapp
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-09-02 13:46:42 +02:00
Alex
bc977f9a7a
Update to Netapp with OrderTag support and exploit OrderTags
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2022-09-01 12:58:20 +02:00
Alex
68087ee13d
Fix clippy
2022-07-29 12:25:03 +02:00
Alex
605a630333
Use streaming in block manager
2022-07-29 12:25:02 +02:00
Alex
16f6a1a65d
fix clippy
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-07-29 12:24:49 +02:00
Alex
ad35b18bb1
Faster chunker
2022-07-29 12:24:49 +02:00
Alex
2cad656a03
More make clippy happy
2022-07-29 12:24:48 +02:00
Alex
2f111e6b3d
Performance improvements:
...
- reduce contention on mutation_lock by having 256 of them
- better lmdb defaults
2022-07-29 12:24:48 +02:00
Alex
1b2e1296eb
Compute hashes on dedicated threads
2022-07-29 12:24:44 +02:00
Alex
77e3fd6db2
improve internal item counter mechanisms and implement bucket quotas ( #326 )
...
continuous-integration/drone/push Build is passing
- [x] Refactoring of internal counting API
- [x] Repair procedure for counters (it's an offline procedure!!!)
- [x] New counter for objects in buckets
- [x] Add quotas to buckets struct
- [x] Add CLI to manage bucket quotas
- [x] Add admin API to manage bucket quotas
- [x] Apply quotas by adding checks on put operations
- [x] Proof-read
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #326
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-06-15 20:20:28 +02:00
Quentin
d544a0e0e0
Send CORS headers for all requests
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-06-13 10:19:52 +02:00
Alex
b44d3fc796
Abstract database behind generic interface and implement alternative drivers ( #322 )
...
continuous-integration/drone/push Build is passing
- [x] Design interface
- [x] Implement Sled backend
- [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
- [x] `.updated()` trigger on Garage tables
Fix #284
**Bugs**
- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work
**Known issues for various back-ends**
- Sled:
- Eats all my RAM and also all my disk space
- `.len()` has to traverse the whole table
- Is actually quite slow on some operations
- And is actually pretty bad code...
- Sqlite:
- Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
- (adapter uses a bunch of unsafe code)
- Heed (LMDB):
- Not suited for 32-bit machines as it has to map the whole DB in memory.
- (adpater uses a tiny bit of unsafe code)
**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.
**Converting databases** is actually quite easy. For example from Sled to LMDB:
```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```
Then, just add this to your `config.toml`:
```toml
db_engine = "lmdb"
```
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-06-08 10:01:44 +02:00
Alex
ff06d3f082
Fix Content-Type headers for {admin,k2v} errors and admin responses
...
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
Fix #315
2022-05-25 17:09:33 +02:00
Alex
382e74c798
First version of admin API ( #298 )
...
continuous-integration/drone/push Build is passing
**Spec:**
- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases
View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md >
**Code:**
- [x] Refactor code for admin api to use common api code that was created for K2V
**General endpoints:**
- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout
**Key-related endpoints:**
- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey
**Bucket-related endpoints:**
- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite
**Operations on key/bucket permissions:**
- [x] BucketAllowKey
- [x] BucketDenyKey
**Operations on bucket aliases:**
- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket
**And also:**
- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design
Fix #231
Fix #295
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-24 12:16:39 +02:00
Alex
c692f55d5c
K2V: Fix end
parameter and add tests ( fix #305 )
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-05-17 11:50:23 +02:00
Alex
5768bf3622
First implementation of K2V ( #293 )
...
continuous-integration/drone/push Build is passing
**Specification:**
View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md )
- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)
**Implementation:**
- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch
**Testing:**
- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
- [x] Adapt testing framework
- [x] Simple test with InsertItem + ReadItem
- [x] Test with several Insert/Read/DeleteItem + ReadIndex
- [x] Test all combinations of return formats for ReadItem
- [x] Test with ReadBatch, InsertBatch, DeleteBatch
- [x] Test with PollItem
- [x] Test error codes
- [ ] Fix most broken stuff
- [x] test PollItem broken randomly
- [x] when invalid causality tokens are given, errors should be 4xx not 5xx
**Improvements:**
- [x] Descending range queries
- [x] Specify
- [x] Implement
- [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-10 13:16:57 +02:00
Alex
c9ef3e461b
fix clippy
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-04-19 12:50:40 +02:00
Alex
c93008d333
Prettier code for defragmentation
2022-04-19 12:50:40 +02:00
Alex
e5341ca47b
Defragmentation in UploadPartCopy: first pass (not pretty but it compiles)
2022-04-19 12:50:36 +02:00
withinboredom
a4f9f19ac3
remove size limitation in UploadPartCopy ( #280 )
...
continuous-integration/drone/push Build is passing
This removes the >1mb s3_copy restriction.
This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248 ).
Co-authored-by: Rob Landers <landers.robert@gmail.com>
Reviewed-on: #280
Co-authored-by: withinboredom <landers.robert@gmail.com>
Co-committed-by: withinboredom <landers.robert@gmail.com>
2022-04-19 12:49:43 +02:00
Alex
8f9cf3a5d1
fix a clippy lint
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-03-28 15:48:55 +02:00
Alex
c1d9854d2c
Move block manager to separate module
2022-03-23 10:25:15 +01:00
trinity-1686a
1eb7fdb08f
add test framework for arbitraty S3 requests
...
and implement some basic test with it
2022-03-23 10:22:36 +01:00
Jill
e934934f14
garage_api: Update streaming payload stream unit tests
2022-03-23 10:22:36 +01:00
Jill
98545a16dd
garage_api: Handle streaming payload early in request handling
2022-03-23 10:22:36 +01:00
Alex
ba6b56ae68
Fix some new clippy lints
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-03-14 12:27:49 +01:00
Alex
203e8d2c34
Bump version to 0.7 because of incompatible Netapp
2022-03-14 10:54:24 +01:00
Alex
d9a35359bf
Add metrics to web endpoint
2022-03-14 10:53:50 +01:00
Alex
2a5609b292
Add metrics to API endpoint
2022-03-14 10:53:36 +01:00
Alex
f0d0cd9a20
Remove strum crate dependency; add protobuf nix dependency
2022-03-14 10:53:00 +01:00
Alex
bb04d94fa9
Update to Netapp 0.4 which supports distributed tracing
2022-03-14 10:52:30 +01:00
Alex
8c2fb0c066
Add tracing integration with opentelemetry
2022-03-14 10:52:13 +01:00
Alex
8a5bbc3b0b
More permissive OPTIONS on S3 API
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone Build is passing
2022-03-01 11:15:16 +01:00
Alex
97f245f218
Add tracing output to signature calculation
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-02-28 12:22:39 +01:00
Alex
8129a98291
Process CORS earlier in pipeline
2022-02-28 12:22:39 +01:00
trinity-1686a
f6f8b7f1ad
Support for PostObject ( #222 )
...
continuous-integration/drone/push Build is passing
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html )
- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data
Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #222
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-02-21 23:02:30 +01:00
Alex
c99f55c420
Add restriction on part ordering in CompleteMultipartUpload
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone Build is passing
2022-01-25 12:45:00 +01:00
Alex
acdf893362
Fix partnumber
2022-01-25 12:25:23 +01:00
Alex
338b1b83ee
Implement part_number for GetObject
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-24 21:04:42 +01:00
Alex
6dab836f3a
Multipart improvements
...
- support part_number for HeadObject
- add checks in complete_multipart_upload
2022-01-24 21:04:40 +01:00
Alex
513a6b15f9
Handle OPTIONS on website endpoint
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-24 12:32:28 +01:00
Alex
ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general
...
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
2022-01-24 11:58:00 +01:00
Trinity Pointard
820924534a
use clamp instead of min(max())
continuous-integration/drone/push Build is passing
2022-01-24 11:56:59 +01:00
Quentin
440374524b
Implement ListParts
2022-01-21 10:42:30 +01:00
Quentin
fe003d6fbc
Add ListPartsResult structure
2022-01-20 16:38:55 +01:00
trinity-1686a
e55fa38c99
Add date verification to presigned urls ( #196 )
...
continuous-integration/drone/push Build is passing
fix #96
fix #162 by returning Forbidden instead Bad Request
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: #196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-01-18 12:22:31 +01:00
trinity-1686a
178e35f868
refactor s3_router and api_server to make unused Endpoint parameters more obvious
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-17 15:50:24 +01:00
Jill
fdcddbe168
Fix Multipart Upload with WinSCP ( #164 ) ( #193 )
...
continuous-integration/drone/push Build is passing
Closes #164 .
Reviewed-on: #193
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 11:18:40 +01:00
Jill
b45dcc1925
Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD ( #64 ) ( #156 )
...
continuous-integration/drone/push Build is passing
Closes #64 .
Reviewed-on: #156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
2022-01-17 10:55:31 +01:00
Alex
d4dd2e2640
Make use of website config, return error document on error
2022-01-13 14:25:19 +01:00
Alex
3ea8ca1b9e
Implement GetBucketWebsite
2022-01-13 14:23:52 +01:00
Alex
f7349f4005
Add quotes in returned etags
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-01-13 14:03:33 +01:00
Alex
1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures
2022-01-13 14:03:30 +01:00
Alex
6617a72220
Implement UploadPartCopy
2022-01-13 13:58:47 +01:00
Alex
3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy ( fix #187 )
2022-01-13 13:56:55 +01:00