Commit graph

65 commits

Author SHA1 Message Date
f3589fee77
update Cargo dependencies
Some checks failed
ci/woodpecker/push/debug Pipeline failed
ci/woodpecker/pr/debug Pipeline failed
Some aws-* crates have been pinned because newer versions don't compile
with rustc 1.73. These crates are only used for integration testing, no
crates which are part of the final Garage version are concerned.
2024-08-25 11:27:46 +02:00
a0f6bc5b7f add rpc_public_addr_subnet config option
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
In case `rpc_public_addr` is not set, but autodiscovery is used, this
allows filtering the list of automatically discovered IPs to a specific
subnet.

For example, if nodes should pick *their* IP inside a specific subnet,
but you don't want to explicitly write the IP down (as it's dynamic, or
you want to share configs across nodes), you can use this option.
2024-06-05 08:41:36 +02:00
afad62939e
[next-0.10] bump version number to 1.0
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/deployment/debug Pipeline was successful
ci/woodpecker/deployment/release/4 Pipeline was successful
ci/woodpecker/deployment/release/3 Pipeline was successful
ci/woodpecker/deployment/release/1 Pipeline was successful
ci/woodpecker/deployment/release/2 Pipeline was successful
ci/woodpecker/deployment/publish Pipeline was successful
2024-03-28 15:19:44 +01:00
f7cd4eb600
Merge branch 'main' into next-0.10
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/cron/debug Pipeline was successful
ci/woodpecker/cron/release/4 Pipeline was successful
ci/woodpecker/cron/release/1 Pipeline was successful
ci/woodpecker/cron/release/3 Pipeline was successful
ci/woodpecker/cron/release/2 Pipeline was successful
ci/woodpecker/cron/publish Pipeline was successful
2024-03-26 16:34:40 +01:00
74949c69cb
[s3-checksum] implement x-amz-checksum-* headers
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
2024-03-26 15:01:34 +01:00
fe8a7819fa
[syslog] Add support to logging to syslog
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
Original patch by Jakub Jirutka for Alpine Linux port.
2024-03-20 14:22:18 +01:00
0038ca8a78
Merge branch 'main' into next-0.10
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/cron/debug Pipeline was successful
ci/woodpecker/cron/release/4 Pipeline was successful
ci/woodpecker/cron/release/3 Pipeline was successful
ci/woodpecker/cron/release/2 Pipeline was successful
ci/woodpecker/cron/release/1 Pipeline was successful
ci/woodpecker/cron/publish Pipeline was successful
2024-03-18 20:19:30 +01:00
e8f9718ccd
[sqlite-r2d2] implement connection pooling in sqlite backend 2024-03-18 18:05:25 +01:00
44454aac01
[rm-sled] Remove the Sled database engine
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
2024-03-08 14:11:02 +01:00
57acc60082
[sse-c] Implement SSE-C encryption 2024-03-07 15:43:47 +01:00
bbde9bc912
Merge branch 'main' into next-0.10
Some checks failed
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline failed
ci/woodpecker/cron/release/3 Pipeline was successful
ci/woodpecker/cron/release/2 Pipeline was successful
ci/woodpecker/cron/debug Pipeline was successful
ci/woodpecker/cron/release/1 Pipeline was successful
ci/woodpecker/cron/release/4 Pipeline was successful
ci/woodpecker/cron/publish Pipeline was successful
2024-03-04 15:56:10 +01:00
8670140358
[rel-0.9.3] Bump version to 0.9.3
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
2024-03-04 14:00:55 +01:00
6a7623e90d
[rel-0.9.2] Bump version to v0.9.2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
2024-03-01 16:54:39 +01:00
eaac4924ef
[fix-auth-ct-eq] use argon2 hashing and verification for admin/metrics token checking 2024-02-29 13:07:15 +01:00
eb4a6ce106
Merge branch 'main' into next-0.10
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/deployment/debug Pipeline was successful
ci/woodpecker/deployment/release/2 Pipeline was successful
ci/woodpecker/deployment/release/1 Pipeline was successful
ci/woodpecker/deployment/release/3 Pipeline was successful
ci/woodpecker/deployment/release/4 Pipeline was successful
ci/woodpecker/deployment/publish Pipeline was successful
2024-02-15 14:06:34 +01:00
5ea24254a9
[import-netapp] import Netapp code into Garage codebase 2024-02-15 12:15:07 +01:00
1b0f167d2f
[fix-cargo-toml] fix cargo warnings in Cargo.toml files
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/deployment/release/2 Pipeline was successful
ci/woodpecker/deployment/debug Pipeline was successful
ci/woodpecker/deployment/release/3 Pipeline was successful
ci/woodpecker/deployment/release/4 Pipeline was successful
ci/woodpecker/deployment/release/1 Pipeline was successful
ci/woodpecker/deployment/publish Pipeline was successful
2024-02-15 10:54:58 +01:00
cf2af186fc
Merge branch 'main' into next-0.10
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
2024-02-13 11:36:28 +01:00
c2e1e172d4
[dep-upgrade-202402] update toml, kube and k8s-openapi 2024-02-08 23:29:56 +01:00
2b92e8d7c6
[lto-nix] enable LTO for release builds using Nix
Some checks reported errors
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build was killed
2024-02-08 10:22:23 +01:00
Jonathan Davies
620664ee9c Cargo.toml: Enable full LTO in release builds and thin in dev builds.
All checks were successful
continuous-integration/drone/pr Build is passing
2024-02-07 16:11:27 +00:00
e524e7a30d
[dep-upgrade-202402] rename BytesBody into ErrorBody for clarity 2024-02-07 14:45:52 +01:00
81ccd4586e
[dep-upgrade-202402] upgrade to http/hyper 1.x for tests
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2024-02-05 19:57:35 +01:00
0bb5b77530
[dep-upgrade-202402] wip: port to http/hyper crates v1
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2024-02-05 18:49:54 +01:00
6e69a1fffc
[dep-upgrade-202402] prepare migration to http/hyper 1.0
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2024-02-05 14:44:12 +01:00
6e4229e29c
[dep-upgrade-202402] update aws-sdk dependencies
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2024-02-05 14:02:45 +01:00
c0a7552015
[dep-upgrade-202402] upgrade easy dependencies
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2024-02-05 13:58:23 +01:00
fe1af5d98b
[dep-upgrade-202402] refactor dependencies: move all as workspace deps
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2024-02-05 13:02:02 +01:00
ee57dd922b
Bump version to 0.9.1
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2024-01-16 16:28:17 +01:00
db48dd3d6c
bump crate versions to 0.10.0
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2024-01-11 12:05:51 +01:00
952c9570c4 bump version to v0.9.0
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
2023-10-10 14:08:11 +02:00
51eac97260 update version to 0.8.4
Some checks failed
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is failing
2023-09-05 23:28:12 +02:00
cece1be1bb bump version to 0.8.3
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
2023-08-28 13:17:26 +02:00
c3d3b837eb bump k2v-client to v0.0.4
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2023-05-22 10:47:15 +02:00
e2ce5970c6 Add basic k2v_client integration tests 2023-05-22 10:45:06 +02:00
217d429937 fix clippy lint in format-table crate
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone/pr Build is passing
2023-05-17 13:06:37 +02:00
a1cec2cd60 Split format_table into separate crate and reduce k2v-client dependencies
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/tag Build was killed
continuous-integration/drone/pr Build was killed
2023-05-17 13:01:37 +02:00
75759a163c Allow to really disable sled feature
All checks were successful
continuous-integration/drone/pr Build is passing
2023-05-09 08:46:15 +00:00
d2deee0b8b Declare garage crates using workspace.dependencies
This will allow to really disable "sled" feature without declaring
`default-features = false` in every Cargo.toml where garage_db and
garage_model is used.

See https://doc.rust-lang.org/cargo/reference/workspaces.html#the-dependencies-table
2023-05-09 08:46:15 +00:00
db72812f01 Use the new cargo feature resolver "2"
Garage currently uses the legacy resolver "1". The new one is used
by default if the root package specifies 'edition = 2021', which
Garage does not (yet).

The problem with the legacy resolver is, among others, that features
enabled by dev-dependencies are propagated to normal dependencies.
This affects e.g. hyper - one of the dev-dependencies enables "http2"
feature that adds many extra dependencies. If we build garage without
opentelemetry-otlp (this is enabled in the following commit), there's
no normal dependency enabling "http2" feature.

See https://doc.rust-lang.org/cargo/reference/resolver.html#feature-resolver-version-2
2022-09-06 01:14:19 +02:00
b44d3fc796 Abstract database behind generic interface and implement alternative drivers (#322)
All checks were successful
continuous-integration/drone/push Build is passing
- [x] Design interface
- [x] Implement Sled backend
  - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
  - [x] `.updated()` trigger on Garage tables

Fix #284

**Bugs**

- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work

**Known issues for various back-ends**

- Sled:
  - Eats all my RAM and also all my disk space
  - `.len()` has to traverse the whole table
  - Is actually quite slow on some operations
  - And is actually pretty bad code...
- Sqlite:
  - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  - (adapter uses a bunch of unsafe code)
- Heed (LMDB):
  - Not suited for 32-bit machines as it has to map the whole DB in memory.
  - (adpater uses a tiny bit of unsafe code)

**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.

**Converting databases** is actually quite easy. For example from Sled to LMDB:

```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```

Then, just add this to your `config.toml`:

```toml
db_engine = "lmdb"
```

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-06-08 10:01:44 +02:00
382e74c798 First version of admin API (#298)
All checks were successful
continuous-integration/drone/push Build is passing
**Spec:**

- [x] Start writing
- [x] Specify all layout endpoints
- [x] Specify all endpoints for operations on keys
- [x] Specify all endpoints for operations on key/bucket permissions
- [x] Specify all endpoints for operations on buckets
- [x] Specify all endpoints for operations on bucket aliases

View rendered spec at <https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/admin-api/doc/drafts/admin-api.md>

**Code:**

- [x] Refactor code for admin api to use common api code that was created for K2V

**General endpoints:**

- [x] Metrics
- [x] GetClusterStatus
- [x] ConnectClusterNodes
- [x] GetClusterLayout
- [x] UpdateClusterLayout
- [x] ApplyClusterLayout
- [x] RevertClusterLayout

**Key-related endpoints:**

- [x] ListKeys
- [x] CreateKey
- [x] ImportKey
- [x] GetKeyInfo
- [x] UpdateKey
- [x] DeleteKey

**Bucket-related endpoints:**

- [x] ListBuckets
- [x] CreateBucket
- [x] GetBucketInfo
- [x] DeleteBucket
- [x] PutBucketWebsite
- [x] DeleteBucketWebsite

**Operations on key/bucket permissions:**

- [x] BucketAllowKey
- [x] BucketDenyKey

**Operations on bucket aliases:**

- [x] GlobalAliasBucket
- [x] GlobalUnaliasBucket
- [x] LocalAliasBucket
- [x] LocalUnaliasBucket

**And also:**

- [x] Separate error type for the admin API (this PR includes a quite big refactoring of error handling)
- [x] Add management of website access
- [ ] Check that nothing is missing wrt what can be done using the CLI
- [ ] Improve formatting of the spec
- [x] Make sure everyone is cool with the API design

Fix #231
Fix #295

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #298
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-05-24 12:16:39 +02:00
64c193e3db Add a K2V client library and CLI (#303)
All checks were successful
continuous-integration/drone/push Build is passing
lib.rs could use getting split in modules, but I'm not sure how exactly

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Reviewed-on: #303
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-05-18 22:24:09 +02:00
43945234ae
Add missing src/block to toplevel cargo.toml
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2022-03-23 10:26:10 +01:00
mricher
e349af13a7
Update dependencies and add admin module with metrics
- Global dependencies updated in Cargo.lock
- New module created in src/admin to host:
  - the (future) admin REST API
  - the metric collection
- add configuration block

No metrics implemented yet
2022-03-14 10:51:12 +01:00
cea871d944 Skeleton to the new web API 2020-11-02 15:48:39 +01:00
bec26a1312 Rename garage_core to garage_model 2020-07-07 13:59:22 +02:00
d8f5e643bc Split code for modular compilation 2020-04-24 10:10:01 +00:00
c0335ac690 Remove a few features in dependencies 2020-04-21 20:37:02 +00:00
cc4f2f1cfb Pretty logging 2020-04-21 12:54:55 +00:00