kaiyou/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Pull requests Releases Activity
1822 commits 17 branches 48 tags 92 MiB
Commit graph

163 commits

Author SHA1 Message Date
Alex Auvolat
19639705e6 Mark sled as deprecated, make lmdb default, and improve sqlite and lmdb defaults 2023-05-17 14:30:53 +02:00
Alex Auvolat
217d429937 fix clippy lint in format-table crate 2023-05-17 13:06:37 +02:00
Alex Auvolat
a1cec2cd60 Split format_table into separate crate and reduce k2v-client dependencies 2023-05-17 13:01:37 +02:00
Alex Auvolat
e4f955d672 fix base64 uses 2023-05-16 19:46:56 +02:00
Alex Auvolat
2e6bb3f766 update Cargo.nix 2023-05-11 11:34:18 +02:00
Alex Auvolat
351d734e6c Merge branch 'main' into next 2023-05-09 12:40:08 +02:00
Alex Auvolat
9e0a9c1c15 move git-version dependency to main crate to reduce rebuilds 2023-05-09 11:35:32 +02:00
Jakub Jirutka
833cf082da Remove unnecessary/unused "timeago" features 
To decrease dependency bloat and binary size.
 2023-05-07 01:03:54 +02:00
Alex Auvolat
fa78d806e3 Merge branch 'main' into next 2023-04-25 12:34:26 +02:00
Alex Auvolat
6894878146 update cargo.nix 2023-04-24 10:26:14 +02:00
Jonathan Davies
c168383113 api/Cargo.toml: Bumped quick-xml to version 0.26. 2023-04-23 20:14:28 +01:00
Alex Auvolat
0a1ddcf630 Prepare for v0.8.2 2023-03-13 18:46:31 +01:00
Alex Auvolat
f37ec584b6 Merge branch 'main' into rustversion-label 2023-03-13 16:14:13 +01:00
Alex Auvolat
3b498c7c47
update cargo.nix 2023-03-13 13:59:02 +01:00
Alex Auvolat
40fa1242f0 update cargo.nix 2023-03-10 18:15:06 +01:00
Alex Auvolat
e2173d00a9 Update cargo.nix 2023-01-30 11:47:34 +01:00
Alex Auvolat
590a0a8450 Merge branch 'main' into k2v-watch-range-2 2023-01-26 16:46:40 +01:00
Alex Auvolat
94d559ae00 Merge branch 'main' into report-disk-usage 2023-01-26 16:20:41 +01:00
Alex Auvolat
654999e254 Update Cargo.nix 2023-01-26 15:50:54 +01:00
Alex Auvolat
0da054194b Update Cargo.nix 2023-01-26 14:47:15 +00:00
Alex Auvolat
efb6b6e868 Disk space report 
Report available disk space on nodes and calculate cluster-wide available space in `garage stats` (fix #479)
 2023-01-26 15:04:32 +01:00
Alex Auvolat
dac254a6e7
Merge branch 'main' into k2v-watch-range-2 2023-01-11 17:09:37 +01:00
Alex Auvolat
32aab06929
k2v-client libary poll_range and CLI poll-range 2023-01-11 11:14:29 +01:00
Felix Scheinost
d6ea0cbefa Add tests for rpc_secret_file 2023-01-07 14:19:36 +01:00
Alex Auvolat
570e5e5bbb
Merge branch 'main' into next 2023-01-04 11:34:43 +01:00
Alex Auvolat
8d5505514f
Make it explicit when using nonversioned encoding 2023-01-03 15:27:36 +01:00
Alex Auvolat
a81200d345
Update cargo.nix 2023-01-03 14:45:47 +01:00
Alex Auvolat
939a6d67e8
Merge branch 'main' into internals-rework 2023-01-02 15:07:44 +01:00
Alex Auvolat
6775569525
Bump everything to v0.8.1 2023-01-02 14:15:33 +01:00
Alex Auvolat
0d6b05bb6c
Update cargo.nix 2022-12-14 12:58:24 +01:00
Alex Auvolat
2c2e65ad8b
Merge commit 'ec12d6c' into next 2022-12-11 18:41:15 +01:00
Alex Auvolat
26121bb619
Fix logs appearing twice 2022-11-29 22:23:27 +01:00
Alex Auvolat
0cfb56d33e
update cargo.nix 2022-11-21 14:47:18 +01:00
Alex Auvolat
d75b37b018
Return more info when layout's .check() fails, fix compilation, fix test 2022-11-08 14:58:39 +01:00
Alex Auvolat
28d7a49f63
Merge branch 'main' into optimal-layout 2022-11-07 12:20:59 +01:00
Alex Auvolat
e03d9062f7
Show a nice message and a backtrace when Garage panics 2022-11-04 16:39:02 +01:00
Alex Auvolat
5b18fd8201
Add garage bucket cleanup-incomplete-uploads command 2022-11-04 11:55:59 +01:00
Alex Auvolat
57b5c2c754
Change reqwest rustls features 2022-10-18 22:11:27 +02:00
Alex Auvolat
002b9fc50c
Add TLS support for Consul discovery + refactoring 2022-10-18 18:38:20 +02:00
Alex Auvolat
00cf076412
Fix cargo2nix feature discovery 2022-10-18 12:15:45 +02:00
Alex Auvolat
fcaee3bea0
definitively expunge openssl from dependencies everywhere 2022-10-14 18:10:36 +02:00
Alex Auvolat
8d04ae7014
cargo2nix unstable (patched), rust 1.63.0, nixpkgs 22.05 (32-bit builds are broken) 2022-10-14 14:30:48 +02:00
Alex Auvolat
357b72f4ff
Merge branch 'main' into configurable-timeouts 2022-09-20 15:19:58 +02:00
Alex Auvolat
1f7b050b7d
Change a warn! into a debug! 2022-09-20 11:49:48 +02:00
Alex Auvolat
56592e1853
RPC performance changes 
- configurable ping timeout
- single, much higher, configurable RPC timeout
- no more concurrency semaphore
 2022-09-19 20:31:00 +02:00
Alex Auvolat
5d4b6f2173
Faster GetObject workflow for getting entire objects 2022-09-19 12:19:59 +02:00
Alex Auvolat
e46dc2a8ef
Allow for hostnames in bootstrap_peers and rpc_public_addr (fix #353) 2022-09-14 16:09:38 +02:00
Alex Auvolat
28a4af73ca
Use netapp 0.5 published from crates.io 2022-09-13 13:11:44 +02:00
Alex Auvolat
7f54706b95
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-09-08 15:50:56 +02:00
Alex Auvolat
d9d199a6c9
Merge branch 'main' into lx-perf-improvements 2022-09-08 15:49:17 +02:00
Alex Auvolat
ceb1f0229a
Move version back into util 2022-09-07 18:36:46 +02:00
Alex Auvolat
28d86e7602
Report build features in garage --help 2022-09-07 17:05:21 +02:00
Alex Auvolat
db61f41030
Move GIT_VERSION injection later in build chain to reduce build times 2022-09-07 11:59:56 +02:00
Alex Auvolat
6b958979bd
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-09-06 22:13:01 +02:00
Alex Auvolat
4024822585
Update netapp to lastest git version with LAS scheduling 2022-09-06 19:45:00 +02:00
Alex Auvolat
0f5689c169
Include code from v0.5.1 directly to remove dependencies 2022-09-06 17:52:50 +02:00
Alex Auvolat
431dee050f
Remove opentelemetry-otlp dep in api/ 2022-09-06 17:25:44 +02:00
Alex Auvolat
2c2b93acdf
Update Nix files with optional db engines 2022-09-06 17:20:10 +02:00
Alex Auvolat
48ffaaadfc
Bump versions to 0.8.0 (compatibility is broken already) 2022-09-06 16:47:56 +02:00
Alex Auvolat
8d77a76df1
Update .nix files 2022-09-06 15:49:41 +02:00
Alex Auvolat
6226f5ceca
Update to netapp 0.4.5 - fixed ping 2022-09-02 14:33:12 +02:00
Alex Auvolat
e648bf7b69
update cargo.nix 2022-09-01 16:31:04 +02:00
Alex Auvolat
f3bf34b6a1
update netapp: straming + fix-ping 2022-09-01 14:23:54 +02:00
Alex Auvolat
4b726b0941
netapp recv with unbounded channel removes deadlock 2022-09-01 09:47:28 +02:00
Alex Auvolat
e598231ca4
update netapp git commit 2022-08-31 19:27:25 +02:00
Alex Auvolat
c9bc9d89de
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-08-31 17:42:31 +02:00
Alex Auvolat
eb97e13a6a
update cargo.nix 2022-08-31 17:42:00 +02:00
Alex Auvolat
dd5304f6fc
Replace logging crate pretty_env_logger by tracing_subscriber::fmt 2022-08-31 14:24:41 +02:00
Alex Auvolat
1921f4f7e6
Merge branch 'lx-perf-improvements' into netapp-stream-body 2022-08-29 16:45:05 +02:00
Alex Auvolat
ebc20a8798
Merge branch 'main' into lx-perf-improvements 2022-08-29 16:44:13 +02:00
Quentin Dufour
2c7bae935a
Configure structopt to report the right version 
By default, structopt reports the value provided by
the env var CARGO_PKG_VERSION, feeded by Cargo when reading
Cargo.toml. However for Garage we use a versioning based on git,
so we often report a version that is behind the real version.
In this commit, we create garage_util::version::garage() that
reports the right version and configure all structopt subcommands
to call this function instead of using the env var.
 2022-08-11 10:21:45 +02:00
Alex Auvolat
f0ee3056d3
Update cargo.nix 2022-07-29 12:25:03 +02:00
Alex Auvolat
33750c04ed
Update cargo.nix 2022-07-29 12:25:03 +02:00
Alex Auvolat
49154a78d8
Update cargo.nix 2022-07-29 12:24:48 +02:00
Alex Auvolat
40150527b8
Update cargo.nix 2022-07-29 12:24:48 +02:00
Alex
4f38cadf6e Background task manager (#332) 
- [x] New background worker trait
- [x] Adapt all current workers to use new API
- [x] Command to list currently running workers, and whether they are active, idle, or dead
- [x] Error reporting
- Optimizations
  - [x] Merkle updater: several items per iteration
  - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on
- scrub:
  - [x] have only one worker with a channel to start/pause/cancel
  - [x] automatic scrub
  - [x] ability to view and change tranquility from CLI
  - [x] persistence of a few info
- [ ] Testing

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#332
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-08 13:30:26 +02:00
Alex
77e3fd6db2 improve internal item counter mechanisms and implement bucket quotas (#326) 
- [x] Refactoring of internal counting API
- [x] Repair procedure for counters (it's an offline procedure!!!)
- [x] New counter for objects in buckets
- [x] Add quotas to buckets struct
- [x] Add CLI to manage bucket quotas
- [x] Add admin API to manage bucket quotas
- [x] Apply quotas by adding checks on put operations
- [x] Proof-read

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#326
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-15 20:20:28 +02:00
Alex
b44d3fc796 Abstract database behind generic interface and implement alternative drivers (#322) 
- [x] Design interface
- [x] Implement Sled backend
  - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
  - [x] `.updated()` trigger on Garage tables

Fix #284

**Bugs**

- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work

**Known issues for various back-ends**

- Sled:
  - Eats all my RAM and also all my disk space
  - `.len()` has to traverse the whole table
  - Is actually quite slow on some operations
  - And is actually pretty bad code...
- Sqlite:
  - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  - (adapter uses a bunch of unsafe code)
- Heed (LMDB):
  - Not suited for 32-bit machines as it has to map the whole DB in memory.
  - (adpater uses a tiny bit of unsafe code)

**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.

**Converting databases** is actually quite easy. For example from Sled to LMDB:

```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```

Then, just add this to your `config.toml`:

```toml
db_engine = "lmdb"
```

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-08 10:01:44 +02:00
Alex
b2a2d3859f K2V client improvements (#307) 
- [x] Better distinguish error types
- [x] Parse error messages received from server
- [x] Remove `src/` folder layer, we don't have that for other crates

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#307
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-24 12:48:05 +02:00
Alex
382e74c798 First version of admin API (#298) 
**Spec:**

- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases

View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md>

**Code:**

- [x] Refactor code for admin api to use common api code that was created for K2V

**General endpoints:**

- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout

**Key-related endpoints:**

- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey

**Bucket-related endpoints:**

- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite

**Operations on key/bucket permissions:**

- [x] BucketAllowKey
- [x] BucketDenyKey

**Operations on bucket aliases:**

- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket

**And also:**

- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design

Fix #231
Fix #295

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-24 12:16:39 +02:00
trinity-1686a
64c193e3db Add a K2V client library and CLI (#303) 
lib.rs could use getting split in modules, but I'm not sure how exactly

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#303
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-05-18 22:24:09 +02:00
Alex
5768bf3622 First implementation of K2V (#293) 
**Specification:**

View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md)

- [x] Specify the structure of K2V triples
- [x] Specify the DVVS format used for causality detection
- [x] Specify the K2V index (just a counter of number of values per partition key)
- [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem
- [x] Specify index endpoint: ReadIndex
- [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch
- [x] Move to JSON objects instead of tuples
- [x] Specify endpoints for polling for updates on single values (PollItem)

**Implementation:**

- [x] Table for K2V items, causal contexts
- [x] Indexing mechanism and table for K2V index
- [x] Make API handlers a bit more generic
- [x] K2V API endpoint
- [x] K2V API router
- [x] ReadItem
- [x] InsertItem
- [x] DeleteItem
- [x] PollItem
- [x] ReadIndex
- [x] InsertBatch
- [x] ReadBatch
- [x] DeleteBatch

**Testing:**

- [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values)
- [x] Actual tests:
  - [x] Adapt testing framework
  - [x] Simple test with InsertItem + ReadItem
  - [x] Test with several Insert/Read/DeleteItem + ReadIndex
  - [x] Test all combinations of return formats for ReadItem
  - [x] Test with ReadBatch, InsertBatch, DeleteBatch
  - [x] Test with PollItem
  - [x] Test error codes
- [ ] Fix most broken stuff
  - [x] test PollItem broken randomly
  - [x] when invalid causality tokens are given, errors should be 4xx not 5xx

**Improvements:**

- [x] Descending range queries
  - [x] Specify
  - [x] Implement
  - [x] Add test
- [x] Batch updates to index counter
- [x] Put K2V behind `k2v` feature flag

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#293
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-10 13:16:57 +02:00
Alex Auvolat
def78c5e6f
Update netapp to 0.4.4, fix #300 2022-05-09 12:08:47 +02:00
Alex Auvolat
dffcd9f4b1
update Cargo.nix 2022-04-08 14:35:09 +02:00
Alex Auvolat
94f1e48fff Update to netapp 0.4.2 (a tiny fix) 2022-04-07 11:50:03 +02:00
Alex Auvolat
9d0ed78887 Add feature flag for Kubernetes discovery 2022-03-24 16:57:43 +01:00
Alex Auvolat
2d13f0aa13
run cargo2nix 2022-03-23 10:25:37 +01:00
trinity-1686a
1eb7fdb08f add test framework for arbitraty S3 requests 
and implement some basic test with it
 2022-03-23 10:22:36 +01:00
Alex Auvolat
db46cdef79
Update netapp to v0.4.1 2022-03-15 17:09:57 +01:00
Alex Auvolat
203e8d2c34
Bump version to 0.7 because of incompatible Netapp 2022-03-14 10:54:24 +01:00
Alex Auvolat
0cc31ee169
add missing netapp telemetry feature 2022-03-14 10:54:11 +01:00
Alex Auvolat
d9a35359bf
Add metrics to web endpoint 2022-03-14 10:53:50 +01:00
Alex Auvolat
f0d0cd9a20
Remove strum crate dependency; add protobuf nix dependency 2022-03-14 10:53:00 +01:00
Alex Auvolat
bb04d94fa9
Update to Netapp 0.4 which supports distributed tracing 2022-03-14 10:52:30 +01:00
Alex Auvolat
8c2fb0c066
Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
Alex Auvolat
2cab84b1fe
Add many metrics in table/ and rpc/ 2022-03-14 10:51:50 +01:00
Max Audron
9d44127245
add support for kubernetes service discovery 
This commit adds support to discover garage instances running in
kubernetes.

Once enabled by setting `kubernetes_namespace` and
`kubernetes_service_name` garage will create a Custom Resources
`garagenodes.deuxfleurs.fr` with nodes public key as the resource name.
and IP and Port information as spec in the namespace configured by
`kubernetes_namespace`.

For discovering nodes the resources are filtered with the optionally set
`kubernetes_service_name` which sets a label
`garage.deuxfleurs.fr/service` on the resources.

This allows to separate multiple garage deployments in a single
namespace.

the `kubernetes_skip_crd` variable allows to disable the creation of the
CRD by garage itself. The user must deploy this manually.
 2022-03-12 13:05:52 +01:00
Quentin Dufour
8df1e186de Functional tests for website endpoints 2022-03-07 17:32:07 +01:00
trinity-1686a
f6f8b7f1ad Support for PostObject (#222) 
Add support for [PostObject](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html)

- [x] routing PostObject properly
- [x] parsing multipart body
- [x] validating signature
- [x] validating policy
- [x] validating content length
- [x] actually saving data

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com>
Reviewed-on: Deuxfleurs/garage#222
Reviewed-by: Alex <alex@adnab.me>
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-02-21 23:02:30 +01:00
Jill
84613e66a2
garage(tests): Remove RNG stuff 2022-02-11 10:50:55 +01:00