Commit graph

24 commits

Author SHA1 Message Date
a1cec2cd60 Split format_table into separate crate and reduce k2v-client dependencies
Some checks reported errors
continuous-integration/drone/push Build was killed
continuous-integration/drone/tag Build was killed
continuous-integration/drone/pr Build was killed
2023-05-17 13:01:37 +02:00
Jonathan Davies
ff70e09aa0 util/forwarded_headers.rs: Generalized handle_forwarded_for_headers()
here.
2023-03-03 19:17:40 +00:00
13c5549886
Remove token_bucket.rs 2023-01-04 11:47:56 +01:00
8d5505514f
Make it explicit when using nonversioned encoding
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2023-01-03 15:27:36 +01:00
cdb2a591e9
Refactor how things are migrated
Some checks failed
continuous-integration/drone/push Build is failing
continuous-integration/drone/pr Build is failing
2023-01-03 14:44:47 +01:00
ebc20a8798
Merge branch 'main' into lx-perf-improvements
All checks were successful
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build is passing
2022-08-29 16:44:13 +02:00
2c7bae935a
Configure structopt to report the right version
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone Build is passing
continuous-integration/drone/push Build is passing
By default, structopt reports the value provided by
the env var CARGO_PKG_VERSION, feeded by Cargo when reading
Cargo.toml. However for Garage we use a versioning based on git,
so we often report a version that is behind the real version.
In this commit, we create garage_util::version::garage() that
reports the right version and configure all structopt subcommands
to call this function instead of using the env var.
2022-08-11 10:21:45 +02:00
1b2e1296eb
Compute hashes on dedicated threads 2022-07-29 12:24:44 +02:00
4f38cadf6e Background task manager (#332)
All checks were successful
continuous-integration/drone/push Build is passing
- [x] New background worker trait
- [x] Adapt all current workers to use new API
- [x] Command to list currently running workers, and whether they are active, idle, or dead
- [x] Error reporting
- Optimizations
  - [x] Merkle updater: several items per iteration
  - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on
- scrub:
  - [x] have only one worker with a channel to start/pause/cancel
  - [x] automatic scrub
  - [x] ability to view and change tranquility from CLI
  - [x] persistence of a few info
- [ ] Testing

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #332
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-07-08 13:30:26 +02:00
b44d3fc796 Abstract database behind generic interface and implement alternative drivers (#322)
All checks were successful
continuous-integration/drone/push Build is passing
- [x] Design interface
- [x] Implement Sled backend
  - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
  - [x] `.updated()` trigger on Garage tables

Fix #284

**Bugs**

- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work

**Known issues for various back-ends**

- Sled:
  - Eats all my RAM and also all my disk space
  - `.len()` has to traverse the whole table
  - Is actually quite slow on some operations
  - And is actually pretty bad code...
- Sqlite:
  - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  - (adapter uses a bunch of unsafe code)
- Heed (LMDB):
  - Not suited for 32-bit machines as it has to map the whole DB in memory.
  - (adpater uses a tiny bit of unsafe code)

**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.

**Converting databases** is actually quite easy. For example from Sled to LMDB:

```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```

Then, just add this to your `config.toml`:

```toml
db_engine = "lmdb"
```

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: #322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
2022-06-08 10:01:44 +02:00
64c193e3db Add a K2V client library and CLI (#303)
All checks were successful
continuous-integration/drone/push Build is passing
lib.rs could use getting split in modules, but I'm not sure how exactly

Co-authored-by: trinity-1686a <trinity@deuxfleurs.fr>
Reviewed-on: #303
Co-authored-by: trinity-1686a <trinity.pointard@gmail.com>
Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>
2022-05-18 22:24:09 +02:00
2377a92f6b
Add wrapper over sled tree to count items (used for big queues) 2022-03-14 10:54:25 +01:00
818daa5c78
Refactor how durations are measured 2022-03-14 10:53:35 +01:00
8c2fb0c066
Add tracing integration with opentelemetry 2022-03-14 10:52:13 +01:00
c94406f428
Improve how node roles are assigned in Garage
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/tag Build is passing
continuous-integration/drone/push Build is passing
continuous-integration/drone Build is passing
- change the terminology: the network configuration becomes the role
  table, the configuration of a nodes becomes a node's role
- the modification of the role table takes place in two steps: first,
  changes are staged in a CRDT data structure. Then, once the user is
  happy with the changes, they can commit them all at once (or revert
  them).
- update documentation
- fix tests
- implement smarter partition assignation algorithm

This patch breaks the format of the network configuration: when
migrating, the cluster will be in a state where no roles are assigned.
All roles must be re-assigned and commited at once. This migration
should not pose an issue.
2021-11-16 16:05:53 +01:00
2090a6187f
Add tranquilizer mechanism to improve on token bucket mechanism
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-11-04 13:26:59 +01:00
6b47c294f5
Refactoring on repair commands
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-10-27 11:14:55 +02:00
e4b9e4e24d
rename types to CamelCase
All checks were successful
continuous-integration/drone/pr Build is passing
continuous-integration/drone/push Build is passing
2021-05-03 22:15:09 +02:00
f05bb111c2
fix clippy warnings on util and rpc 2021-05-03 22:11:41 +02:00
74373aebcf
make most requested changes 2021-04-27 16:47:08 +02:00
f9bd2d8fb7
document util crate 2021-04-27 16:37:10 +02:00
9ced9f78dc
Improve bootstraping: do it regularly; persist peer list 2021-04-27 16:37:08 +02:00
3bf2df622a Time and metadata improvements
All checks were successful
continuous-integration/drone/push Build is passing
2021-03-15 16:21:41 +01:00
d8f5e643bc Split code for modular compilation 2020-04-24 10:10:01 +00:00