KokaKiwi/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues Pull requests Projects Releases Wiki Activity
1974 commits 13 branches 1 tag 58 MiB
Commit graph

64 commits

Author SHA1 Message Date
Florian Klink
a0f6bc5b7f add rpc_public_addr_subnet config option 
In case `rpc_public_addr` is not set, but autodiscovery is used, this
allows filtering the list of automatically discovered IPs to a specific
subnet.

For example, if nodes should pick *their* IP inside a specific subnet,
but you don't want to explicitly write the IP down (as it's dynamic, or
you want to share configs across nodes), you can use this option.
 2024-06-05 08:41:36 +02:00
Alex Auvolat
afad62939e
[next-0.10] bump version number to 1.0 2024-03-28 15:19:44 +01:00
Alex Auvolat
f7cd4eb600
Merge branch 'main' into next-0.10 2024-03-26 16:34:40 +01:00
Alex Auvolat
74949c69cb
[s3-checksum] implement x-amz-checksum-* headers 2024-03-26 15:01:34 +01:00
Alex Auvolat
fe8a7819fa
[syslog] Add support to logging to syslog 
Original patch by Jakub Jirutka for Alpine Linux port.
 2024-03-20 14:22:18 +01:00
Alex Auvolat
0038ca8a78
Merge branch 'main' into next-0.10 2024-03-18 20:19:30 +01:00
Alex Auvolat
e8f9718ccd
[sqlite-r2d2] implement connection pooling in sqlite backend 2024-03-18 18:05:25 +01:00
Alex Auvolat
44454aac01
[rm-sled] Remove the Sled database engine 2024-03-08 14:11:02 +01:00
Alex Auvolat
57acc60082
[sse-c] Implement SSE-C encryption 2024-03-07 15:43:47 +01:00
Alex Auvolat
bbde9bc912
Merge branch 'main' into next-0.10 2024-03-04 15:56:10 +01:00
Alex Auvolat
8670140358
[rel-0.9.3] Bump version to 0.9.3 2024-03-04 14:00:55 +01:00
Alex Auvolat
6a7623e90d
[rel-0.9.2] Bump version to v0.9.2 2024-03-01 16:54:39 +01:00
Alex Auvolat
eaac4924ef
[fix-auth-ct-eq] use argon2 hashing and verification for admin/metrics token checking 2024-02-29 13:07:15 +01:00
Alex Auvolat
eb4a6ce106
Merge branch 'main' into next-0.10 2024-02-15 14:06:34 +01:00
Alex Auvolat
5ea24254a9
[import-netapp] import Netapp code into Garage codebase 2024-02-15 12:15:07 +01:00
Alex Auvolat
1b0f167d2f
[fix-cargo-toml] fix cargo warnings in Cargo.toml files 2024-02-15 10:54:58 +01:00
Alex Auvolat
cf2af186fc
Merge branch 'main' into next-0.10 2024-02-13 11:36:28 +01:00
Alex Auvolat
c2e1e172d4
[dep-upgrade-202402] update toml, kube and k8s-openapi 2024-02-08 23:29:56 +01:00
Alex Auvolat
2b92e8d7c6
[lto-nix] enable LTO for release builds using Nix 2024-02-08 10:22:23 +01:00
Jonathan Davies
620664ee9c Cargo.toml: Enable full LTO in release builds and thin in dev builds. 2024-02-07 16:11:27 +00:00
Alex Auvolat
e524e7a30d
[dep-upgrade-202402] rename BytesBody into ErrorBody for clarity 2024-02-07 14:45:52 +01:00
Alex Auvolat
81ccd4586e
[dep-upgrade-202402] upgrade to http/hyper 1.x for tests 2024-02-05 19:57:35 +01:00
Alex Auvolat
0bb5b77530
[dep-upgrade-202402] wip: port to http/hyper crates v1 2024-02-05 18:49:54 +01:00
Alex Auvolat
6e69a1fffc
[dep-upgrade-202402] prepare migration to http/hyper 1.0 2024-02-05 14:44:12 +01:00
Alex Auvolat
6e4229e29c
[dep-upgrade-202402] update aws-sdk dependencies 2024-02-05 14:02:45 +01:00
Alex Auvolat
c0a7552015
[dep-upgrade-202402] upgrade easy dependencies 2024-02-05 13:58:23 +01:00
Alex Auvolat
fe1af5d98b
[dep-upgrade-202402] refactor dependencies: move all as workspace deps 2024-02-05 13:02:02 +01:00
Alex Auvolat
ee57dd922b
Bump version to 0.9.1 2024-01-16 16:28:17 +01:00
Alex Auvolat
db48dd3d6c
bump crate versions to 0.10.0 2024-01-11 12:05:51 +01:00
Alex Auvolat
952c9570c4 bump version to v0.9.0 2023-10-10 14:08:11 +02:00
Alex Auvolat
51eac97260 update version to 0.8.4 2023-09-05 23:28:12 +02:00
Alex Auvolat
cece1be1bb bump version to 0.8.3 2023-08-28 13:17:26 +02:00
Alex Auvolat
c3d3b837eb bump k2v-client to v0.0.4 2023-05-22 10:47:15 +02:00
Alex Auvolat
e2ce5970c6 Add basic k2v_client integration tests 2023-05-22 10:45:06 +02:00
Alex Auvolat
217d429937 fix clippy lint in format-table crate 2023-05-17 13:06:37 +02:00
Alex Auvolat
a1cec2cd60 Split format_table into separate crate and reduce k2v-client dependencies 2023-05-17 13:01:37 +02:00
Jakub Jirutka
75759a163c Allow to really disable sled feature 2023-05-09 08:46:15 +00:00
Jakub Jirutka
d2deee0b8b Declare garage crates using workspace.dependencies 
This will allow to really disable "sled" feature without declaring
`default-features = false` in every Cargo.toml where garage_db and
garage_model is used.

See https://doc.rust-lang.org/cargo/reference/workspaces.html#the-dependencies-table
 2023-05-09 08:46:15 +00:00
Jakub Jirutka
db72812f01 Use the new cargo feature resolver "2" 
Garage currently uses the legacy resolver "1". The new one is used
by default if the root package specifies 'edition = 2021', which
Garage does not (yet).

The problem with the legacy resolver is, among others, that features
enabled by dev-dependencies are propagated to normal dependencies.
This affects e.g. hyper - one of the dev-dependencies enables "http2"
feature that adds many extra dependencies. If we build garage without
opentelemetry-otlp (this is enabled in the following commit), there's
no normal dependency enabling "http2" feature.

See https://doc.rust-lang.org/cargo/reference/resolver.html#feature-resolver-version-2
 2022-09-06 01:14:19 +02:00
Alex
b44d3fc796 Abstract database behind generic interface and implement alternative drivers (#322) 
- [x] Design interface
- [x] Implement Sled backend
  - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
  - [x] `.updated()` trigger on Garage tables

Fix #284

**Bugs**

- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work

**Known issues for various back-ends**

- Sled:
  - Eats all my RAM and also all my disk space
  - `.len()` has to traverse the whole table
  - Is actually quite slow on some operations
  - And is actually pretty bad code...
- Sqlite:
  - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  - (adapter uses a bunch of unsafe code)
- Heed (LMDB):
  - Not suited for 32-bit machines as it has to map the whole DB in memory.
  - (adpater uses a tiny bit of unsafe code)

**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.

**Converting databases** is actually quite easy. For example from Sled to LMDB:

```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```

Then, just add this to your `config.toml`:

```toml
db_engine = "lmdb"
```

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-08 10:01:44 +02:00
Alex
382e74c798 First version of admin API (#298) 
**Spec:**

- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases

View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md>

**Code:**

- [x] Refactor code for admin api to use common api code that was created for K2V

**General endpoints:**

- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout

**Key-related endpoints:**

- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey

**Bucket-related endpoints:**

- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite

**Operations on key/bucket permissions:**

- [x] BucketAllowKey
- [x] BucketDenyKey

**Operations on bucket aliases:**

- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket

**And also:**

- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design

Fix #231
Fix #295

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-05-24 12:16:39 +02:00
trinity-1686a
64c193e3db Add a K2V client library and CLI (#303) 
lib.rs could use getting split in modules, but I'm not sure how exactly

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Reviewed-on: Deuxfleurs/garage#303
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
 2022-05-18 22:24:09 +02:00
Alex Auvolat
43945234ae
Add missing src/block to toplevel cargo.toml 2022-03-23 10:26:10 +01:00
mricher
e349af13a7
Update dependencies and add admin module with metrics 
- Global dependencies updated in Cargo.lock
- New module created in src/admin to host:
  - the (future) admin REST API
  - the metric collection
- add configuration block

No metrics implemented yet
 2022-03-14 10:51:12 +01:00
Quentin
cea871d944 Skeleton to the new web API 2020-11-02 15:48:39 +01:00
Alex Auvolat
bec26a1312 Rename garage_core to garage_model 2020-07-07 13:59:22 +02:00
Alex Auvolat
d8f5e643bc Split code for modular compilation 2020-04-24 10:10:01 +00:00
Alex Auvolat
c0335ac690 Remove a few features in dependencies 2020-04-21 20:37:02 +00:00
Alex Auvolat
cc4f2f1cfb Pretty logging 2020-04-21 12:54:55 +00:00
Alex Auvolat
e325c7f47a Add hostname to node info 2020-04-19 19:08:48 +02:00