networkException/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Pull requests Activity
1231 commits 26 branches 60 tags 36 MiB
Commit graph

748 commits

Author SHA1 Message Date
Alex Auvolat
2da8786f54
move things around 2022-10-18 19:13:52 +02:00
Alex Auvolat
5d8d393054
Load TLS certificates only once 2022-10-18 19:11:16 +02:00
Alex Auvolat
002b9fc50c
Add TLS support for Consul discovery + refactoring 2022-10-18 18:38:20 +02:00
Alex
5670599372 Merge pull request 'Use status code 204 No Content for empty responses' (#403) from tobikris/garage:http-no-content into main 
Reviewed-on: Deuxfleurs/garage#403
 2022-10-18 14:20:44 +00:00
Tobias Krischer
7865003323
Use status code 204 No Content for empty responses 2022-10-17 10:55:26 +02:00
Alex Auvolat
c050a59fd0
Fix conditional testing in garage_db 2022-10-14 18:27:18 +02:00
Alex Auvolat
fcaee3bea0
definitively expunge openssl from dependencies everywhere 2022-10-14 18:10:36 +02:00
Alex Auvolat
8d04ae7014
cargo2nix unstable (patched), rust 1.63.0, nixpkgs 22.05 (32-bit builds are broken) 2022-10-14 14:30:48 +02:00
Alex Auvolat
ad917ffd3f
Fix instant substractions that might have panicked 2022-09-29 15:53:54 +02:00
Alex Auvolat
1f97ce37e6
Shutdown properly on SIGTERM/SIGHUP and on Windows signals 2022-09-28 10:41:59 +02:00
Alex Auvolat
1778e4b318
Fix span name for api server requests 2022-09-26 16:21:30 +02:00
Alex Auvolat
782630fc27
Initialize metrics exporter earlier (fix #389) 2022-09-20 17:50:22 +02:00
Alex Auvolat
ded444f6c9
Ability to have custom timeouts in request strategy (not used) 2022-09-20 16:01:41 +02:00
Alex Auvolat
357b72f4ff
Merge branch 'main' into configurable-timeouts 2022-09-20 15:19:58 +02:00
Alex Auvolat
1f7b050b7d
Change a warn! into a debug! 2022-09-20 11:49:48 +02:00
Alex Auvolat
56592e1853
RPC performance changes 
- configurable ping timeout
- single, much higher, configurable RPC timeout
- no more concurrency semaphore
 2022-09-19 20:31:00 +02:00
Alex Auvolat
5d4b6f2173
Faster GetObject workflow for getting entire objects 2022-09-19 12:19:59 +02:00
Alex Auvolat
76f42a1a2b
Properly return HTTP 204 when deleting non-existent object (fix #227) 2022-09-14 17:07:55 +02:00
Alex Auvolat
e46dc2a8ef
Allow for hostnames in bootstrap_peers and rpc_public_addr (fix #353) 2022-09-14 16:09:38 +02:00
Alex Auvolat
ab722cb40f
Add checks on replication_factor of layouts we use (fix #363, fix #364) 2022-09-13 16:22:23 +02:00
Alex Auvolat
38be811b1c
Fix clippy lint that says we should implement Eq 2022-09-13 16:08:00 +02:00
Alex Auvolat
44733474bb
Remove/change println! in server code (fix #358) 2022-09-13 16:01:55 +02:00
Alex Auvolat
07febd3ecd
Ensure data dir is created immediately when Garage starts (fix #349) 2022-09-13 15:57:27 +02:00
Alex
11bdc971e2 Merge pull request 'use netapp streaming body' (#343) from netapp-stream-body into main 
Reviewed-on: Deuxfleurs/garage#343
 2022-09-13 15:26:08 +02:00
Alex Auvolat
ff30891999
Use streaming block API for get with Range requests 2022-09-13 15:13:07 +02:00
Alex Auvolat
28a4af73ca
Use netapp 0.5 published from crates.io 2022-09-13 13:11:44 +02:00
Alex Auvolat
b823151a0b
improvements in block manager 2022-09-12 16:57:38 +02:00
Alex Auvolat
f91fab8582
Simplify+improve async hasher by using bounded channel 2022-09-12 16:23:43 +02:00
Alex Auvolat
7f54706b95
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-09-08 15:50:56 +02:00
Alex Auvolat
d9d199a6c9
Merge branch 'main' into lx-perf-improvements 2022-09-08 15:49:17 +02:00
Alex Auvolat
ceb1f0229a
Move version back into util 2022-09-07 18:36:46 +02:00
Alex Auvolat
f310fce34b
Inject GIT_VERSION even later 2022-09-07 18:30:15 +02:00
Alex Auvolat
06df301de5
Fix merge 2022-09-07 18:16:01 +02:00
Alex Auvolat
8adc654713
Merge branch 'main' into improve-deps 2022-09-07 18:13:27 +02:00
Alex Auvolat
107853334b
Fix build error 2022-09-07 18:10:19 +02:00
Alex Auvolat
1449204439
Add warnings when features are not included in build 2022-09-07 18:02:13 +02:00
Alex Auvolat
2e00809af5
Error messages when system-libs XOR bundled-libs != 1 2022-09-07 17:57:12 +02:00
Alex Auvolat
2559f63e9b
Make all HTTP services optionnal 2022-09-07 17:54:16 +02:00
Alex Auvolat
28d86e7602
Report build features in garage --help 2022-09-07 17:05:21 +02:00
Alex Auvolat
db61f41030
Move GIT_VERSION injection later in build chain to reduce build times 2022-09-07 11:59:56 +02:00
Alex Auvolat
907054775d
Faster copy, better get error message 2022-09-06 22:25:23 +02:00
Alex Auvolat
6b958979bd
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-09-06 22:13:01 +02:00
Alex Auvolat
c2cc08852b
Reenable node ordering 2022-09-06 19:31:42 +02:00
Alex Auvolat
6f02c36a89
cargo fmt 2022-09-06 17:59:41 +02:00
Alex Auvolat
0f5689c169
Include code from v0.5.1 directly to remove dependencies 2022-09-06 17:52:50 +02:00
Alex Auvolat
1e92e9f782
Disable k2v tests when feature is disabled 2022-09-06 17:29:46 +02:00
Alex Auvolat
431dee050f
Remove opentelemetry-otlp dep in api/ 2022-09-06 17:25:44 +02:00
Alex Auvolat
b886c75450
Make all DB engines optional build features 2022-09-06 17:09:43 +02:00
Alex Auvolat
48ffaaadfc
Bump versions to 0.8.0 (compatibility is broken already) 2022-09-06 16:47:56 +02:00
Alex Auvolat
454d8474ef
Fix clippy 2022-09-06 15:43:50 +02:00
Jakub Jirutka
ea36b9ff90 Allow building without Prometheus exporter (/metrics endpoint) 
prometheus and opentelemetry-prometheus add 7 extra dependencies in
total and increases the size of the garage binary by ~7 % (with
fat LTO).
 2022-09-06 01:15:09 +02:00
Jakub Jirutka
e7af006c1c Make OTLP exporter optional via feature "telemetry-otlp" 
opentelemetry-otlp add 48 (!) extra dependencies and increases the
size of the garage binary by ~11 % (with fat LTO).
 2022-09-06 01:14:47 +02:00
Alex Auvolat
729a910e14
Remove Heed default features 2022-09-05 16:40:13 +02:00
Alex Auvolat
07e6bcde85
Merge branch 'main' into lx-perf-improvements 2022-09-05 12:40:17 +02:00
Jakub Jirutka
7511ba5530 Allow linking against system-provided libsqlite 
Unfortunately, rusqlite uses the opposite logic for enabling/disabling
bundled libraries to others (libsodium-sys, zstd-sys). Cargo features
are very limited and doesn't allow to enable feature A in a dependency
iff feature B is disabled.

Note, lmdb-rkv-sys doesn't need any special treatment because it
automatically links against system liblmdb if found via pkgconf.

Linux distros should build garage with
`--no-default-features --features system-libs` to disable bundled-libs
and enable system-libs.
 2022-09-03 19:15:57 +02:00
Jakub Jirutka
a6e40b75ea Add feature "system-libs" to enable linking against system libraries 
If this feature is enabled, libsodium-sys and zstd-sys will link
dynamically against system-provided libraries instead of building
and linking statically the bundled (possibly outdated and vulnerable)
copies of them. This feature is intended mainly for linux package
maintainers.
 2022-09-03 18:44:34 +02:00
Alex Auvolat
e1751c8a9c
fix clippy 2022-09-02 17:24:26 +02:00
Alex Auvolat
5d4b937a00
Ability to have up to 4 concurrently working resync workers 2022-09-02 17:18:13 +02:00
Alex Auvolat
5e8baa433d
Make BlockManagerLocked fully private again 2022-09-02 16:52:22 +02:00
Alex Auvolat
47be652a1f
block manager: refactor: split resync into separate file 2022-09-02 16:47:15 +02:00
Alex Auvolat
943d76c583
Ability to dynamically set resync tranquility 2022-09-02 15:34:21 +02:00
Alex Auvolat
6226f5ceca
Update to netapp 0.4.5 - fixed ping 2022-09-02 14:33:12 +02:00
Alex Auvolat
13b5f28c7e
Make use of BytesBuf from new Netapp 2022-09-02 13:46:42 +02:00
Alex Auvolat
1ef87ac4cb
cargo fmt 2022-09-02 13:38:29 +02:00
Alex Auvolat
99b532b85b
Apply PRIO_SECONDARY to block data transfers 2022-09-01 16:35:43 +02:00
Alex Auvolat
df094bd807
Less strict timeouts 2022-09-01 16:30:44 +02:00
Alex Auvolat
bc977f9a7a
Update to Netapp with OrderTag support and exploit OrderTags 2022-09-01 12:58:20 +02:00
Alex Auvolat
70231d68b2
Fix bytes_read counter 2022-08-31 19:44:27 +02:00
Alex Auvolat
c9bc9d89de
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-08-31 17:42:31 +02:00
Alex Auvolat
efbca67ce4
Add env filter to tracing subscriber 2022-08-31 14:39:12 +02:00
Alex Auvolat
44cd98d2e4
Tracing-subscriber: write to stderr 2022-08-31 14:28:17 +02:00
Alex Auvolat
dd5304f6fc
Replace logging crate pretty_env_logger by tracing_subscriber::fmt 2022-08-31 14:24:41 +02:00
Alex Auvolat
322dafc761
Try to fix clippy 2022-08-29 17:32:45 +02:00
Alex Auvolat
1921f4f7e6
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-08-29 16:45:05 +02:00
Alex Auvolat
ebc20a8798
Merge branch 'main' into lx-perf-improvements 2022-08-29 16:44:13 +02:00
Quentin Dufour
2c7bae935a
Configure structopt to report the right version 
By default, structopt reports the value provided by
the env var CARGO_PKG_VERSION, feeded by Cargo when reading
Cargo.toml. However for Garage we use a versioning based on git,
so we often report a version that is behind the real version.
In this commit, we create garage_util::version::garage() that
reports the right version and configure all structopt subcommands
to call this function instead of using the env var.
 2022-08-11 10:21:45 +02:00
Alex Auvolat
e935861854
Factor out node request order selection logic & use in manager 2022-07-29 12:25:03 +02:00
Alex Auvolat
68087ee13d
Fix clippy 2022-07-29 12:25:03 +02:00
Alex Auvolat
605a630333
Use streaming in block manager 2022-07-29 12:25:02 +02:00
Alex Auvolat
a35d4da721
update netapp to 0.5 2022-07-29 12:25:02 +02:00
Alex Auvolat
8e7e680afe
First adaptation to WIP netapp with streaming body 2022-07-29 12:25:02 +02:00
Alex Auvolat
16f6a1a65d
fix clippy 2022-07-29 12:24:49 +02:00
Alex Auvolat
ad35b18bb1
Faster chunker 2022-07-29 12:24:49 +02:00
Alex Auvolat
ff4771c36a
cargo fmt 2022-07-29 12:24:48 +02:00
Alex Auvolat
381eb9a5a1
Fix tests 2022-07-29 12:24:48 +02:00
Alex Auvolat
2cad656a03
More make clippy happy 2022-07-29 12:24:48 +02:00
Alex Auvolat
0176da3ad2
Make clippy happy 2022-07-29 12:24:48 +02:00
Alex Auvolat
2f111e6b3d
Performance improvements: 
- reduce contention on mutation_lock by having 256 of them
- better lmdb defaults
 2022-07-29 12:24:48 +02:00
Alex Auvolat
1b2e1296eb
Compute hashes on dedicated threads 2022-07-29 12:24:44 +02:00
Alex Auvolat
ac03fa7937
Uniformize tracing::* imports (hopefully fixes 32-bit build) 2022-07-15 18:31:19 +02:00
Alex
4f38cadf6e Background task manager (#332) 
- [x] New background worker trait
- [x] Adapt all current workers to use new API
- [x] Command to list currently running workers, and whether they are active, idle, or dead
- [x] Error reporting
- Optimizations
  - [x] Merkle updater: several items per iteration
  - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on
- scrub:
  - [x] have only one worker with a channel to start/pause/cancel
  - [x] automatic scrub
  - [x] ability to view and change tranquility from CLI
  - [x] persistence of a few info
- [ ] Testing

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#332
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-08 13:30:26 +02:00
Alex Auvolat
aab34bfe54
add delays in k2v test_items_and_indices 2022-07-08 10:41:57 +02:00
Alex
fe3fa83de7 Publish k2v-client crate to crates.io (#337) 
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#337
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-04 18:27:25 +02:00
Alex Auvolat
b6d59ec19a
Fix poll item when item didn't change 2022-07-04 14:00:02 +02:00
Alex
0850bac874 Add poll command to k2v-cli (#335) 
Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#335
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-04 12:45:32 +02:00
Alex
77e3fd6db2 improve internal item counter mechanisms and implement bucket quotas (#326) 
- [x] Refactoring of internal counting API
- [x] Repair procedure for counters (it's an offline procedure!!!)
- [x] New counter for objects in buckets
- [x] Add quotas to buckets struct
- [x] Add CLI to manage bucket quotas
- [x] Add admin API to manage bucket quotas
- [x] Apply quotas by adding checks on put operations
- [x] Proof-read

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#326
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-15 20:20:28 +02:00
Quentin Dufour
d544a0e0e0
Send CORS headers for all requests 2022-06-13 10:19:52 +02:00
Alex Auvolat
138e13071b
Fix garage_db build on 32-bit systems 2022-06-09 14:55:20 +02:00
Alex
b44d3fc796 Abstract database behind generic interface and implement alternative drivers (#322) 
- [x] Design interface
- [x] Implement Sled backend
  - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
  - [x] `.updated()` trigger on Garage tables

Fix #284

**Bugs**

- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work

**Known issues for various back-ends**

- Sled:
  - Eats all my RAM and also all my disk space
  - `.len()` has to traverse the whole table
  - Is actually quite slow on some operations
  - And is actually pretty bad code...
- Sqlite:
  - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  - (adapter uses a bunch of unsafe code)
- Heed (LMDB):
  - Not suited for 32-bit machines as it has to map the whole DB in memory.
  - (adpater uses a tiny bit of unsafe code)

**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.

**Converting databases** is actually quite easy. For example from Sled to LMDB:

```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```

Then, just add this to your `config.toml`:

```toml
db_engine = "lmdb"
```

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-08 10:01:44 +02:00
Alex Auvolat
ff06d3f082
Fix Content-Type headers for {admin,k2v} errors and admin responses 
Fix #315
 2022-05-25 17:09:33 +02:00
Alex
b2a2d3859f K2V client improvements (#307) 
- [x] Better distinguish error types
- [x] Parse error messages received from server
- [x] Remove `src/` folder layer, we don't have that for other crates

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#307
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-24 12:48:05 +02:00
Alex
382e74c798 First version of admin API (#298) 
**Spec:**

- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases

View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md>

**Code:**

- [x] Refactor code for admin api to use common api code that was created for K2V

**General endpoints:**

- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout

**Key-related endpoints:**

- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey

**Bucket-related endpoints:**

- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite

**Operations on key/bucket permissions:**

- [x] BucketAllowKey
- [x] BucketDenyKey

**Operations on bucket aliases:**

- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket

**And also:**

- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design

Fix #231
Fix #295

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-24 12:16:39 +02:00
trinity-1686a
64c193e3db Add a K2V client library and CLI (#303) 
lib.rs could use getting split in modules, but I'm not sure how exactly

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#303
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-05-18 22:24:09 +02:00
Alex Auvolat
c692f55d5c
K2V: Fix end parameter and add tests (fix #305) 2022-05-17 11:50:23 +02:00
Alex Auvolat
7b474855e3
Make background runner terminate correctly 2022-05-17 11:38:31 +02:00
Alex
5768bf3622 First implementation of K2V (#293) 
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-10 13:16:57 +02:00
Alex Auvolat
def78c5e6f
Update netapp to 0.4.4, fix #300 2022-05-09 12:08:47 +02:00
Alex
277a20ec44 Fix layout show to not show changes when there are no changes (#297) 
fixes #295, partially

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#297
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-09 11:14:55 +02:00
Alex Auvolat
c9ef3e461b
fix clippy 2022-04-19 12:50:40 +02:00
Alex Auvolat
c93008d333
Prettier code for defragmentation 2022-04-19 12:50:40 +02:00
Alex Auvolat
e5341ca47b
Defragmentation in UploadPartCopy: first pass (not pretty but it compiles) 2022-04-19 12:50:36 +02:00
withinboredom
a4f9f19ac3 remove size limitation in UploadPartCopy (#280) 
This removes the >1mb s3_copy restriction.

This restriction doesn't seem to be documented anywhere (I could be wrong). It also causes some software to fail (such as #248).

Co-authored-by: Rob Landers <landers.robert@gmail.com>
Reviewed-on: Deuxfleurs/garage#280
Co-authored-by: withinboredom <landers.robert@gmail.com>
Co-committed-by: withinboredom <landers.robert@gmail.com>
 2022-04-19 12:49:43 +02:00
Alex Auvolat
5d404dcd54
Add missing opentelemetry features 2022-04-08 14:21:04 +02:00
Alex Auvolat
94f1e48fff Update to netapp 0.4.2 (a tiny fix) 2022-04-07 11:50:03 +02:00
Alex Auvolat
cb5836d53c Bring maximum exponential backoff time down from 16h to 1h 2022-04-07 11:49:29 +02:00
Alex Auvolat
0091002ef2
New replication modes and their documentation 2022-03-28 16:26:04 +02:00
Alex Auvolat
8f9cf3a5d1
fix a clippy lint 2022-03-28 15:48:55 +02:00
Alex Auvolat
913f7754bb
Add blocks in errored state to garage stats 2022-03-28 15:47:23 +02:00
Alex Auvolat
42dde54126
Log admin GET requests at debug level instead of info 
to reduce noise in logs
 2022-03-28 15:46:52 +02:00
Alex Auvolat
9d0ed78887 Add feature flag for Kubernetes discovery 2022-03-24 16:57:43 +01:00
Alex Auvolat
509d256c58
Make layout optimization work in relative terms 2022-03-24 15:27:14 +01:00
Alex Auvolat
2814d41842
Allow garage layout assign to assign to several nodes at once 2022-03-24 15:27:13 +01:00
Alex Auvolat
7e0e2ffda2
Slight change and add comment to layout assignation algo 2022-03-24 15:27:13 +01:00
Alex Auvolat
413ab0eaed
Small change to partition assignation algorithm 
This change helps ensure that nodes for each partition are spread
over all datacenters, a property that wasn't ensured previously
when going from a 2 DC deployment to a 3 DC deployment
 2022-03-24 15:27:10 +01:00
Alex Auvolat
3dc9214172
Add lots of comments on how the resync queue works 
(I don't really want to change/refactor that code though)
 2022-03-23 10:25:39 +01:00
Alex Auvolat
077dd1cde9
Clippy 2022-03-23 10:25:39 +01:00
Alex Auvolat
e480aaf338
Make background tranquility a configurable parameter 2022-03-23 10:25:19 +01:00
Alex Auvolat
8fd6745745
Move block RC code to separate rc.rs 2022-03-23 10:25:19 +01:00
Alex Auvolat
c3982a90b6
Move DataBlock out of manager.rs 2022-03-23 10:25:19 +01:00
Alex Auvolat
c1d9854d2c
Move block manager to separate module 2022-03-23 10:25:15 +01:00
trinity-1686a
8565f7dc31 cleanup 2022-03-23 10:22:37 +01:00
trinity-1686a
8db6b84559 add test for create bucket and put website with streaming signature 2022-03-23 10:22:37 +01:00
trinity-1686a
1eb7fdb08f add test framework for arbitraty S3 requests 
and implement some basic test with it
 2022-03-23 10:22:36 +01:00
KokaKiwi
e934934f14 garage_api: Update streaming payload stream unit tests 2022-03-23 10:22:36 +01:00
KokaKiwi
98545a16dd garage_api: Handle streaming payload early in request handling 2022-03-23 10:22:36 +01:00
Alex Auvolat
db46cdef79
Update netapp to v0.4.1 2022-03-15 17:09:57 +01:00
Alex Auvolat
ba6b56ae68
Fix some new clippy lints 2022-03-14 12:27:49 +01:00
Alex Auvolat
0af314b295
Add comment for fsync 2022-03-14 11:54:00 +01:00
Alex Auvolat
d78bf379fb
Fix resync queue to not drop items 2022-03-14 11:51:37 +01:00
Alex Auvolat
f7e6f4616f
Spawn a single resync worker 2022-03-14 11:51:37 +01:00
Alex Auvolat
dc5ec4ecf9
Add appropriate fsync() calls in write_block 
to ensure that data is persisted properly
 2022-03-14 11:51:32 +01:00
Alex Auvolat
fe62d01b7e
Implement exponential backoff for resync retries 2022-03-14 11:41:20 +01:00
Alex Auvolat
9b2b531f4d
Make admin server optional 2022-03-14 10:54:25 +01:00
Alex Auvolat
2377a92f6b
Add wrapper over sled tree to count items (used for big queues) 2022-03-14 10:54:25 +01:00
Alex Auvolat
203e8d2c34
Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
Alex Auvolat
f869ca625d
Add spans to table calls, change span names in RPC 2022-03-14 10:54:12 +01:00
Alex Auvolat
0cc31ee169
add missing netapp telemetry feature 2022-03-14 10:54:11 +01:00
Alex Auvolat
dc8d0496cc
Refactoring: rename config files, make modifications less invasive 2022-03-14 10:53:51 +01:00
Alex Auvolat
d9a35359bf
Add metrics to web endpoint 2022-03-14 10:53:50 +01:00
Alex Auvolat
2a5609b292
Add metrics to API endpoint 2022-03-14 10:53:36 +01:00
Alex Auvolat
818daa5c78
Refactor how durations are measured 2022-03-14 10:53:35 +01:00
Alex Auvolat
f0d0cd9a20
Remove strum crate dependency; add protobuf nix dependency 2022-03-14 10:53:00 +01:00
Alex Auvolat
55d4471599
Remove ... at end of hex IDs 2022-03-14 10:52:31 +01:00
Alex Auvolat
bb04d94fa9
Update to Netapp 0.4 which supports distributed tracing 2022-03-14 10:52:30 +01:00
Alex Auvolat
8c2fb0c066
Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
Alex Auvolat
2cab84b1fe
Add many metrics in table/ and rpc/ 2022-03-14 10:51:50 +01:00
Maximilien R
1e2cf26373
Implement basic metrics in table 2022-03-14 10:51:17 +01:00
mricher
e349af13a7
Update dependencies and add admin module with metrics 
- Global dependencies updated in Cargo.lock
- New module created in src/admin to host:
  - the (future) admin REST API
  - the metric collection
- add configuration block

No metrics implemented yet
 2022-03-14 10:51:12 +01:00
Max Audron
9d44127245
add support for kubernetes service discovery 
This commit adds support to discover garage instances running in
kubernetes.

Once enabled by setting `kubernetes_namespace` and
`kubernetes_service_name` garage will create a Custom Resources
`garagenodes.deuxfleurs.fr` with nodes public key as the resource name.
and IP and Port information as spec in the namespace configured by
`kubernetes_namespace`.

For discovering nodes the resources are filtered with the optionally set
`kubernetes_service_name` which sets a label
`garage.deuxfleurs.fr/service` on the resources.

This allows to separate multiple garage deployments in a single
namespace.

the `kubernetes_skip_crd` variable allows to disable the creation of the
CRD by garage itself. The user must deploy this manually.
 2022-03-12 13:05:52 +01:00
Quentin Dufour
c00b2c9948 Functional tests for admin commands 2022-03-07 17:32:07 +01:00
Quentin Dufour
8df1e186de Functional tests for website endpoints 2022-03-07 17:32:07 +01:00
Quentin Dufour
2ef60b8417 Functional test for multipart endpoints 2022-03-07 17:32:07 +01:00
Quentin Dufour
1e639ec67c Functional test for ListMultipartUploads 2022-03-07 17:32:07 +01:00
Quentin Dufour
cfea1e0315 Functional tests for bucket endpoints 2022-03-07 17:32:02 +01:00
Quentin Dufour
05eb79929e Functional tests for object operations 2022-03-07 17:05:10 +01:00
Quentin Dufour
0f4e0e8bb9 Move ListObjects tests to Rust 2022-03-07 17:05:10 +01:00
Alex Auvolat
8a5bbc3b0b
More permissive OPTIONS on S3 API 2022-03-01 11:15:16 +01:00
Alex Auvolat
97f245f218
Add tracing output to signature calculation 2022-02-28 12:22:39 +01:00
Alex Auvolat
8129a98291
Process CORS earlier in pipeline 2022-02-28 12:22:39 +01:00
trinity-1686a
f6f8b7f1ad Support for PostObject (#222) 
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html)

- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#222
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-02-21 23:02:30 +01:00
Jill
84613e66a2
garage(tests): Remove RNG stuff 2022-02-11 10:50:55 +01:00
Jill
c8b30ebc79
garage(tests): Remove superfluous test 2022-02-10 17:55:51 +01:00
Jill
d7decda3f4
garage(tests): Add random suffix to created buckets. 2022-02-10 17:55:50 +01:00
Jill
cd13ea461b
garage(tests): Add some unsafe-usage doc and tweaks 2022-02-10 17:55:50 +01:00
Quentin Dufour
5d19f3d2d7
Add integration tests to Drone 2022-02-10 17:55:50 +01:00
Jill
3baa841d6f
tests: Fix garage integration test 2022-02-10 17:55:49 +01:00
Jill
dd407e7041
tests: Add garage integration tests (base) 2022-02-10 17:55:49 +01:00
Alex Auvolat
af261e1789 Fix a bug when a migration is followed by a rebalance 
Nodes would stabilize on different encoding formats for the values,
some having the pre-migration format and some having the post-migration
format. This would be reflected in the Merkle trees never converging
and thus having an infinite resync loop.
 2022-02-10 17:38:27 +01:00
Alex Auvolat
ae2f32baf1
Hide deleted key in bucket info (fix #211) 2022-02-02 17:12:48 +01:00
Alex Auvolat
c99f55c420
Add restriction on part ordering in CompleteMultipartUpload 2022-01-25 12:45:00 +01:00
Alex Auvolat
acdf893362
Fix partnumber 2022-01-25 12:25:23 +01:00
Alex Auvolat
338b1b83ee
Implement part_number for GetObject 2022-01-24 21:04:42 +01:00
Alex Auvolat
6dab836f3a
Multipart improvements 
- support part_number for HeadObject
- add checks in complete_multipart_upload
 2022-01-24 21:04:40 +01:00
Alex Auvolat
513a6b15f9
Handle OPTIONS on website endpoint 2022-01-24 12:32:28 +01:00
Alex Auvolat
ea7fb901eb
Implement {Put,Get,Delete}BucketCors and CORS in general 
- OPTIONS request against API endpoint
- Returning corresponding CORS headers on API calls
- Returning corresponding CORS headers on website GET's
 2022-01-24 11:58:00 +01:00
Trinity Pointard
820924534a use clamp instead of min(max()) 2022-01-24 11:56:59 +01:00
Quentin Dufour
440374524b Implement ListParts 2022-01-21 10:42:30 +01:00
Quentin Dufour
fe003d6fbc Add ListPartsResult structure 2022-01-20 16:38:55 +01:00
trinity-1686a
e55fa38c99 Add date verification to presigned urls (#196) 
fix #96
fix #162 by returning Forbidden instead Bad Request

Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#196
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-01-18 12:22:31 +01:00
trinity-1686a
178e35f868 refactor s3_router and api_server to make unused Endpoint parameters more obvious 2022-01-17 15:50:24 +01:00
Jill
fdcddbe168 Fix Multipart Upload with WinSCP (#164) (#193) 
Closes #164.

Reviewed-on: Deuxfleurs/garage#193
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
 2022-01-17 11:18:40 +01:00
Jill
b45dcc1925 Support STREAMING-AWS4-HMAC-SHA256-PAYLOAD (#64) (#156) 
Closes #64.

Reviewed-on: Deuxfleurs/garage#156
Co-authored-by: Jill <kokakiwi@deuxfleurs.fr>
Co-committed-by: Jill <kokakiwi@deuxfleurs.fr>
 2022-01-17 10:55:31 +01:00
Alex Auvolat
d4dd2e2640
Make use of website config, return error document on error 2022-01-13 14:25:19 +01:00
Alex Auvolat
9eb211948e
Allow setting index document and error document on the CLI 2022-01-13 14:25:19 +01:00
Alex Auvolat
3ea8ca1b9e
Implement GetBucketWebsite 2022-01-13 14:23:52 +01:00
Alex Auvolat
f7349f4005
Add quotes in returned etags 2022-01-13 14:03:33 +01:00
Alex Auvolat
1ee8f596ee
Testing for UploadPartCopies and bugfixes in AWS signatures 2022-01-13 14:03:30 +01:00
Alex Auvolat
6617a72220
Implement UploadPartCopy 2022-01-13 13:58:47 +01:00
Alex Auvolat
3770a34e3d
Implement x-amz-copy-if-xxx copy preconditions and return more headers on copy (fix #187) 2022-01-13 13:56:55 +01:00
Quentin
b4592a00fe Implement ListMultipartUploads (#171) 
Implement ListMultipartUploads, also refactor ListObjects and ListObjectsV2.

It took me some times as I wanted to propose the following things:
  - Using an iterator instead of the loop+goto pattern. I find it easier to read and it should enable some optimizations. For example, when consuming keys of a common prefix, we do many [redundant checks](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/api/s3_list.rs#L125-L156) while the only thing to do is to [check if the following key is still part of the common prefix](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/feature/s3-multipart-compat/src/api/s3_list.rs#L476).
  - Try to name things (see ExtractionResult and RangeBegin enums) and to separate concerns (see ListQuery and Accumulator)
  - An IO closure to make unit tests possibles.
  - Unit tests, to track regressions and document how to interact with the code
  - Integration tests with `s3api`. In the future, I would like to move them in Rust with the aws rust SDK.

Merging of the logic of ListMultipartUploads and ListObjects was not a goal but a consequence of the previous modifications.

Some points that we might want to discuss:
  - ListObjectsV1, when using pagination and delimiters, has a weird behavior (it lists multiple times the same prefix) with `aws s3api` due to the fact that it can not use our optimization to skip the whole prefix. It is independant from my refactor and can be tested with the commented `s3api` tests in `test-smoke.sh`. It probably has the same weird behavior on the official AWS S3 implementation.
  - Considering ListMultipartUploads, I had to "abuse" upload id marker to support prefix skipping. I send an `upload-id-marker` with the hardcoded value `include` to emulate your "including" token.
  - Some ways to test ListMultipartUploads with existing software (my tests are limited to s3api for now).

Co-authored-by: Quentin Dufour <quentin@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#171
Co-authored-by: Quentin <quentin@dufour.io>
Co-committed-by: Quentin <quentin@dufour.io>
 2022-01-12 19:04:55 +01:00