rune/garage
1
0
Fork 0
forked from Deuxfleurs/garage
Code Issues Pull requests Projects Releases Wiki Activity
1579 commits 14 branches 0 tags 69 MiB
Commit graph

50 commits

Author SHA1 Message Date
Alex Auvolat
df24bb806d
layout/sync: fix bugs and add tracing 2023-11-11 12:44:27 +01:00
Alex Auvolat
ce89d1ddab
table sync: adapt to new layout history 2023-11-11 12:08:32 +01:00
Alex Auvolat
bfb1845fdc
layout: refactor to use a RwLock on LayoutHistory 2023-11-09 14:12:05 +01:00
Alex Auvolat
8a2b1dd422
wip: split out layout management from System into separate LayoutManager 2023-11-09 12:55:36 +01:00
Alex Auvolat
8dccee3ccf
cluster layout: adapt all uses of ClusterLayout to LayoutHistory 2023-11-08 19:28:36 +01:00
Alex Auvolat
4a9c94514f
avoid using layout_watch in System directly 2023-11-08 16:41:00 +01:00
Alex Auvolat
12d1dbfc6b
remove Ring and use ClusterLayout everywhere 2023-11-08 15:41:24 +01:00
Alex Auvolat
d6d571d512
cargo fmt 2023-01-03 15:30:21 +01:00
Alex Auvolat
a54b67740d
move debug_serialize to garage_util::encode 2023-01-03 15:29:29 +01:00
Alex Auvolat
8d5505514f
Make it explicit when using nonversioned encoding 2023-01-03 15:27:36 +01:00
Alex Auvolat
426d8784da
cleanup 2023-01-03 15:08:37 +01:00
Alex Auvolat
cdb2a591e9
Refactor how things are migrated 2023-01-03 14:44:47 +01:00
Alex Auvolat
dfc131850a
Simplified and more aggressive worker exit logic 2022-12-14 15:25:29 +01:00
Alex Auvolat
d56c472712
Refactor background runner and get rid of job worker 2022-12-14 12:51:42 +01:00
Alex Auvolat
2183518edc
Spawn all background workers in a separate step 2022-12-14 12:28:07 +01:00
Alex Auvolat
de9d6cddf7
Prettier worker list table; remove useless CLI log messages 2022-12-12 17:17:05 +01:00
Alex Auvolat
ad917ffd3f
Fix instant substractions that might have panicked 2022-09-29 15:53:54 +02:00
Alex Auvolat
357b72f4ff
Merge branch 'main' into configurable-timeouts 2022-09-20 15:19:58 +02:00
Alex Auvolat
1f7b050b7d
Change a warn! into a debug! 2022-09-20 11:49:48 +02:00
Alex Auvolat
56592e1853
RPC performance changes 
- configurable ping timeout
- single, much higher, configurable RPC timeout
- no more concurrency semaphore
 2022-09-19 20:31:00 +02:00
Alex Auvolat
df094bd807
Less strict timeouts 2022-09-01 16:30:44 +02:00
Alex
4f38cadf6e Background task manager (#332) 
- [x] New background worker trait
- [x] Adapt all current workers to use new API
- [x] Command to list currently running workers, and whether they are active, idle, or dead
- [x] Error reporting
- Optimizations
  - [x] Merkle updater: several items per iteration
  - [ ] Use `tokio::task::spawn_blocking` where appropriate so that CPU-intensive tasks don't block other things going on
- scrub:
  - [x] have only one worker with a channel to start/pause/cancel
  - [x] automatic scrub
  - [x] ability to view and change tranquility from CLI
  - [x] persistence of a few info
- [ ] Testing

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#332
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-07-08 13:30:26 +02:00
Alex
b44d3fc796 Abstract database behind generic interface and implement alternative drivers (#322) 
- [x] Design interface
- [x] Implement Sled backend
  - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction)
- [x] Convert Garage code to use generic interface
- [x] Proof-read converted Garage code
- [ ] Test everything well
- [x] Implement sqlite backend
- [x] Implement LMDB backend
- [ ] (Implement Persy backend?)
- [ ] (Implement other backends? (like RocksDB, ...))
- [x] Implement backend choice in config file and garage server module
- [x] Add CLI for converting between DB formats
- Exploit the new interface to put more things in transactions
  - [x] `.updated()` trigger on Garage tables

Fix #284

**Bugs**

- [x] When exporting sqlite, trees iterate empty??
- [x] LMDB doesn't work

**Known issues for various back-ends**

- Sled:
  - Eats all my RAM and also all my disk space
  - `.len()` has to traverse the whole table
  - Is actually quite slow on some operations
  - And is actually pretty bad code...
- Sqlite:
  - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  - (adapter uses a bunch of unsafe code)
- Heed (LMDB):
  - Not suited for 32-bit machines as it has to map the whole DB in memory.
  - (adpater uses a tiny bit of unsafe code)

**My recommendation:** avoid 32-bit machines and use LMDB as much as possible.

**Converting databases** is actually quite easy. For example from Sled to LMDB:

```bash
cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb
```

Then, just add this to your `config.toml`:

```toml
db_engine = "lmdb"
```

Co-authored-by: Alex Auvolat <alex@adnab.me>
Reviewed-on: Deuxfleurs/garage#322
Co-authored-by: Alex <alex@adnab.me>
Co-committed-by: Alex <alex@adnab.me>
 2022-06-08 10:01:44 +02:00
Alex Auvolat
2cab84b1fe
Add many metrics in table/ and rpc/ 2022-03-14 10:51:50 +01:00
Alex Auvolat
beeef4758e
Some movement of helper code and refactoring of error handling 2022-01-04 12:52:46 +01:00
Alex Auvolat
8f6026de5e
Make table name a const in trait 2021-12-15 15:39:10 +01:00
Alex Auvolat
ada7899b24
Fix clippy lints (fix #121) 2021-10-26 10:20:05 +02:00
Alex Auvolat
1b450c4b49
Improvements to CLI and various fixes for netapp version 
Discovery via consul, persist peer list to file
 2021-10-22 16:55:24 +02:00
Alex Auvolat
4067797d01
First port of Garage to Netapp 2021-10-22 15:55:18 +02:00
Trinity Pointard
e4b9e4e24d
rename types to CamelCase 2021-05-03 22:15:09 +02:00
Trinity Pointard
f5a0cf0414
fix clippy warnings on table 2021-05-03 22:11:41 +02:00
Alex Auvolat
f4346cc5f4 Update dependencies 2021-03-16 15:58:40 +01:00
Alex Auvolat
2a41b82384 Simpler Merkle & sync 2021-03-16 12:18:03 +01:00
Alex Auvolat
515029d026 Refactor code 2021-03-16 11:43:58 +01:00
Alex Auvolat
1d9961e411 Simplify replication logic 2021-03-16 11:14:27 +01:00
Alex Auvolat
6a8439fd13 Some improvements in background worker but we terminate late 2021-03-15 23:14:12 +01:00
Alex Auvolat
0cd5b2ae19 WIP migrate to tokio 1 2021-03-15 22:36:41 +01:00
Alex Auvolat
4d4117f2b4 Refactor block resync loop; make workers infaillible 2021-03-15 20:09:44 +01:00
Alex Auvolat
831eb35763 cargo fmt 2021-03-12 21:52:19 +01:00
Alex Auvolat
c475471e7a Implement table gc, currently for block_ref and version only 2021-03-12 19:57:37 +01:00
Alex Auvolat
cbe7e1a66a Move table rpc client out of tableaux 2021-03-12 15:07:23 +01:00
Alex Auvolat
8860aa19b8 Make syncer have its own rpc client/server 2021-03-12 15:05:26 +01:00
Alex Auvolat
1fea257291 Don't sync at beginning 2021-03-12 14:51:17 +01:00
Alex Auvolat
7fdaf7aef0 Fix merkle updater not being notified; improved logging 2021-03-12 14:37:46 +01:00
Alex Auvolat
1ec49980ec whoops 2021-03-11 19:30:24 +01:00
Alex Auvolat
3f7a496355 More security: don't delete stuff too easily 2021-03-11 19:06:27 +01:00
Alex Auvolat
f7c2cd1cd7 Add comment, and also whoops, this wasn't doing what we expected 2021-03-11 18:56:18 +01:00
Alex Auvolat
fae5104a2c Add a nice warning 2021-03-11 18:50:32 +01:00
Alex Auvolat
db7a9d4948 Tiny changes 2021-03-11 18:45:26 +01:00
Alex Auvolat
046b649bcc (not well tested) use merkle tree for sync 2021-03-11 18:28:27 +01:00