Commit graph

57 commits

Author SHA1 Message Date
44454aac01
[rm-sled] Remove the Sled database engine 2024-03-08 14:11:02 +01:00
57acc60082
[sse-c] Implement SSE-C encryption 2024-03-07 15:43:47 +01:00
bbde9bc912
Merge branch 'main' into next-0.10 2024-03-04 15:56:10 +01:00
8670140358
[rel-0.9.3] Bump version to 0.9.3 2024-03-04 14:00:55 +01:00
6a7623e90d
[rel-0.9.2] Bump version to v0.9.2 2024-03-01 16:54:39 +01:00
eaac4924ef
[fix-auth-ct-eq] use argon2 hashing and verification for admin/metrics token checking 2024-02-29 13:07:15 +01:00
eb4a6ce106
Merge branch 'main' into next-0.10 2024-02-15 14:06:34 +01:00
5ea24254a9
[import-netapp] import Netapp code into Garage codebase 2024-02-15 12:15:07 +01:00
1b0f167d2f
[fix-cargo-toml] fix cargo warnings in Cargo.toml files 2024-02-15 10:54:58 +01:00
cf2af186fc
Merge branch 'main' into next-0.10 2024-02-13 11:36:28 +01:00
c2e1e172d4
[dep-upgrade-202402] update toml, kube and k8s-openapi 2024-02-08 23:29:56 +01:00
2b92e8d7c6
[lto-nix] enable LTO for release builds using Nix 2024-02-08 10:22:23 +01:00
Jonathan Davies
620664ee9c Cargo.toml: Enable full LTO in release builds and thin in dev builds. 2024-02-07 16:11:27 +00:00
e524e7a30d
[dep-upgrade-202402] rename BytesBody into ErrorBody for clarity 2024-02-07 14:45:52 +01:00
81ccd4586e
[dep-upgrade-202402] upgrade to http/hyper 1.x for tests 2024-02-05 19:57:35 +01:00
0bb5b77530
[dep-upgrade-202402] wip: port to http/hyper crates v1 2024-02-05 18:49:54 +01:00
6e69a1fffc
[dep-upgrade-202402] prepare migration to http/hyper 1.0 2024-02-05 14:44:12 +01:00
6e4229e29c
[dep-upgrade-202402] update aws-sdk dependencies 2024-02-05 14:02:45 +01:00
c0a7552015
[dep-upgrade-202402] upgrade easy dependencies 2024-02-05 13:58:23 +01:00
fe1af5d98b
[dep-upgrade-202402] refactor dependencies: move all as workspace deps 2024-02-05 13:02:02 +01:00
ee57dd922b
Bump version to 0.9.1 2024-01-16 16:28:17 +01:00
db48dd3d6c
bump crate versions to 0.10.0 2024-01-11 12:05:51 +01:00
952c9570c4 bump version to v0.9.0 2023-10-10 14:08:11 +02:00
51eac97260 update version to 0.8.4 2023-09-05 23:28:12 +02:00
cece1be1bb bump version to 0.8.3 2023-08-28 13:17:26 +02:00
c3d3b837eb bump k2v-client to v0.0.4 2023-05-22 10:47:15 +02:00
e2ce5970c6 Add basic k2v_client integration tests 2023-05-22 10:45:06 +02:00
217d429937 fix clippy lint in format-table crate 2023-05-17 13:06:37 +02:00
a1cec2cd60 Split format_table into separate crate and reduce k2v-client dependencies 2023-05-17 13:01:37 +02:00
75759a163c Allow to really disable sled feature 2023-05-09 08:46:15 +00:00
d2deee0b8b Declare garage crates using workspace.dependencies
This will allow to really disable "sled" feature without declaring
`default-features = false` in every Cargo.toml where garage_db and
garage_model is used.

See https://doc.rust-lang.org/cargo/reference/workspaces.html#the-dependencies-table
2023-05-09 08:46:15 +00:00
db72812f01 Use the new cargo feature resolver "2"
Garage currently uses the legacy resolver "1". The new one is used
by default if the root package specifies 'edition = 2021', which
Garage does not (yet).

The problem with the legacy resolver is, among others, that features
enabled by dev-dependencies are propagated to normal dependencies.
This affects e.g. hyper - one of the dev-dependencies enables "http2"
feature that adds many extra dependencies. If we build garage without
opentelemetry-otlp (this is enabled in the following commit), there's
no normal dependency enabling "http2" feature.

See https://doc.rust-lang.org/cargo/reference/resolver.html#feature-resolver-version-2
2022-09-06 01:14:19 +02:00
b44d3fc796 Abstract database behind generic interface and implement alternative drivers ()
- [x] Design interface
- [x] Implement Sled backend
  - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
  - [x] `.updated()` trigger on Garage tables

Fix 

**Bugs**

- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work

**Known issues for various back-ends**

- Sled:
  - Eats all my RAM and also all my disk space
  - `.len()` has to traverse the whole table
  - Is actually quite slow on some operations
  - And is actually pretty bad code...
- Sqlite:
  - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  - (adapter uses a bunch of unsafe code)
- Heed (LMDB):
  - Not suited for 32-bit machines as it has to map the whole DB in memory.
  - (adpater uses a tiny bit of unsafe code)

**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.

**Converting databases** is actually quite easy. For example from Sled to LMDB:

```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```

Then, just add this to your `config.toml`:

```toml
db_engine = "lmdb"
```

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: 
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-06-08 10:01:44 +02:00
382e74c798 First version of admin API ()
**Spec:**

- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases

View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md>

**Code:**

- [x] Refactor code for admin api to use common api code that was created for K2V

**General endpoints:**

- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout

**Key-related endpoints:**

- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey

**Bucket-related endpoints:**

- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite

**Operations on key/bucket permissions:**

- [x] BucketAllowKey
- [x] BucketDenyKey

**Operations on bucket aliases:**

- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket

**And also:**

- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design

Fix 
Fix 

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: 
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-24 12:16:39 +02:00
64c193e3db Add a K2V client library and CLI ()
lib.rs could use getting split in modules, but I'm not sure how exactly

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Reviewed-on: 
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-05-18 22:24:09 +02:00
43945234ae
Add missing src/block to toplevel cargo.toml 2022-03-23 10:26:10 +01:00
mricher
e349af13a7
Update dependencies and add admin module with metrics
- Global dependencies updated in Cargo.lock
- New module created in src/admin to host:
  - the (future) admin REST API
  - the metric collection
- add configuration block

No metrics implemented yet
2022-03-14 10:51:12 +01:00
cea871d944 Skeleton to the new web API 2020-11-02 15:48:39 +01:00
bec26a1312 Rename garage_core to garage_model 2020-07-07 13:59:22 +02:00
d8f5e643bc Split code for modular compilation 2020-04-24 10:10:01 +00:00
c0335ac690 Remove a few features in dependencies 2020-04-21 20:37:02 +00:00
cc4f2f1cfb Pretty logging 2020-04-21 12:54:55 +00:00
e325c7f47a Add hostname to node info 2020-04-19 19:08:48 +02:00
69f1d8fef2 WIP
TODOs:
- ensure sync goes both way
- finish sending blocks to other nodes when they need them before deleting
2020-04-17 17:09:57 +02:00
4fe8329847 Enable debug info in release profile 2020-04-16 20:46:43 +02:00
c788fc9f9e Cleanup 2020-04-12 19:18:31 +02:00
d2814b5c33 TLS works \o/
So, the issues were:
- webpki does not support IP addresses as DNS names in URLs,
  so I hacked the HttpsConnector to always provide a fixed string
  as the DNS name for server certificate validation
- the certificate requied a SAN section which was complicated to build
  but eventually the solution is there in genkeys.sh
2020-04-12 19:00:30 +02:00
d1e8f78b2c Trying to do TLS 2020-04-12 15:51:19 +02:00
dcf58499a4 table::insert_many, version_table::updated 2020-04-11 19:43:29 +02:00
ff4fb97568 (Try to) disable LTO ? 2020-04-10 22:55:01 +02:00