Abstract database behind generic interface and implement alternative drivers #322

Merged
lx merged 64 commits from db-abstraction into main 2022-06-08 08:01:56 +00:00
Owner
  • Design interface
  • Implement Sled backend
    • Re-implement the SledCountedTree hack on Sled backend on all backends (i.e. over the abstraction)
  • Convert Garage code to use generic interface
  • Proof-read converted Garage code
  • Test everything well
  • Implement sqlite backend
  • Implement LMDB backend
  • (Implement Persy backend?)
  • (Implement other backends? (like RocksDB, ...))
  • Implement backend choice in config file and garage server module
  • Add CLI for converting between DB formats
  • Exploit the new interface to put more things in transactions
    • .updated() trigger on Garage tables

Fix #284

Bugs

  • When exporting sqlite, trees iterate empty??
  • LMDB doesn't work

Known issues for various back-ends

  • Sled:
    • Eats all my RAM and also all my disk space
    • .len() has to traverse the whole table
    • Is actually quite slow on some operations
    • And is actually pretty bad code...
  • Sqlite:
    • Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with .iter(), and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a .iter() loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
    • (adapter uses a bunch of unsafe code)
  • Heed (LMDB):
    • Not suited for 32-bit machines as it has to map the whole DB in memory.
    • (adpater uses a tiny bit of unsafe code)

My recommendation: avoid 32-bit machines and use LMDB as much as possible.

Converting databases is actually quite easy. For example from Sled to LMDB:

cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb

Then, just add this to your config.toml:

db_engine = "lmdb"
- [x] Design interface - [x] Implement Sled backend - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction) - [x] Convert Garage code to use generic interface - [x] Proof-read converted Garage code - [ ] Test everything well - [x] Implement sqlite backend - [x] Implement LMDB backend - [ ] (Implement Persy backend?) - [ ] (Implement other backends? (like RocksDB, ...)) - [x] Implement backend choice in config file and garage server module - [x] Add CLI for converting between DB formats - Exploit the new interface to put more things in transactions - [x] `.updated()` trigger on Garage tables Fix #284 **Bugs** - [x] When exporting sqlite, trees iterate empty?? - [x] LMDB doesn't work **Known issues for various back-ends** - Sled: - Eats all my RAM and also all my disk space - `.len()` has to traverse the whole table - Is actually quite slow on some operations - And is actually pretty bad code... - Sqlite: - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason. - (adapter uses a bunch of unsafe code) - Heed (LMDB): - Not suited for 32-bit machines as it has to map the whole DB in memory. - (adpater uses a tiny bit of unsafe code) **My recommendation:** avoid 32-bit machines and use LMDB as much as possible. **Converting databases** is actually quite easy. For example from Sled to LMDB: ```bash cd src/db cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb ``` Then, just add this to your `config.toml`: ```toml db_engine = "lmdb" ```
lx added 4 commits 2022-06-02 15:04:14 +00:00
continuous-integration/drone/push Build is failing Details
fd8d5c37f7
First iteration of a generic DB layer
continuous-integration/drone/push Build is failing Details
04901093e7
Implement iter() and range() on db
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
6805e184e9
Do not put sled in garage's cargo.toml
lx added 1 commit 2022-06-02 15:39:16 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
fbd5b64ff3
Complete sled abstraction
lx added 1 commit 2022-06-02 16:11:52 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
f29b91232f
Use Cell instead of ArcSwap
lx added 1 commit 2022-06-02 17:59:56 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
43704afb29
Begin sqlite adapter
lx added 2 commits 2022-06-02 21:15:57 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
c439cb11a9
Sqlite iter with unsafe code
lx added 1 commit 2022-06-02 21:44:00 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
0e4f336887
Horrible implementation of range for sqlite
lx added 1 commit 2022-06-02 21:46:40 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
6ebb54cddb
Just refactor a bit
lx added 2 commits 2022-06-03 08:45:33 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
f25309e58f
Change value type to be a dyn thing
lx added 1 commit 2022-06-03 09:15:09 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
cc0d984118
Fix most clippy lints
lx added 1 commit 2022-06-03 09:45:07 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
df0877bbba
Conversion utility
lx added 1 commit 2022-06-03 10:12:57 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
bd9ff432d7
Garage works on sqlite, but it's a hack
lx added 2 commits 2022-06-03 10:53:06 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
295bc2741f
fix clipy lint
lx added 2 commits 2022-06-03 11:36:19 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
3d18c9e183
whoops
lx added 2 commits 2022-06-03 12:17:03 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
16e0a655d0
Update rmp-serde
lx added 1 commit 2022-06-03 13:31:48 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
4e72c713f1
Start LMDB adapter, with fixed semantics
lx added 1 commit 2022-06-03 13:53:44 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
bd2997a453
LMDB: use Heed wrapper (much more complete)
lx added 2 commits 2022-06-03 14:42:04 +00:00
continuous-integration/drone/push Build was killed Details
continuous-integration/drone/pr Build was killed Details
2d5541b0e7
Integrate LMDB with rest
lx added 2 commits 2022-06-06 10:02:59 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
4539a6c229
Fix more sqlite deadlocks
lx added 3 commits 2022-06-06 12:08:28 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
0ab21e780e
Small syntax changes
lx added 1 commit 2022-06-06 12:09:24 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
17918e2ef5
Whoops
lx added 1 commit 2022-06-06 13:07:43 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
a65049db8f
Simplify lmdb adapter (one less indirection)
lx added 2 commits 2022-06-06 13:46:18 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
1897815358
Table updated trigger now happens in transaction, this is waaaay better!
lx added 1 commit 2022-06-06 14:13:24 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
9238fda9b2
Update Cargo.nix
lx added 1 commit 2022-06-06 14:29:19 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
1dabd98330
Improve things in block manager & correctly propagate .len() errors
lx added 1 commit 2022-06-06 14:31:50 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
72e6419b1b
add comment
lx added 1 commit 2022-06-06 14:32:12 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
bbf68aa039
fix
lx added 1 commit 2022-06-06 14:37:53 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
c3c8d59c99
Fix more .unwrap()'s
lx added 1 commit 2022-06-06 14:45:07 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
fd2fe18de3
in db convert: map size up to 1TB on 64-bit systems
lx added 1 commit 2022-06-06 14:51:12 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
72a87c8686
Add inline marker on a bunch of db functions
lx added 1 commit 2022-06-06 15:02:00 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
1f9a371b9a
Fix clippy lint
lx added 1 commit 2022-06-06 15:06:31 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
e42c7bed38
Remove useless clones in sqlite
lx added 1 commit 2022-06-06 20:02:43 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is failing Details
35544cac39
Add nix overrides to include lmdb and sqlite3 libraries
lx added 1 commit 2022-06-06 20:35:56 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
3de9320a0e
bundled sqlite
lx added 1 commit 2022-06-06 21:00:27 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
5dbc79b77e
Try once again to fix sqlite bundling in nix
lx added 1 commit 2022-06-07 14:53:34 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
a9e79f848b
Bring back the counted tree hack for Sled (with caveat)
caveat: it's not only for sled
lx added 1 commit 2022-06-07 15:05:35 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
a3a01141ec
db abstraction: make .insert() and .remove() return the old value
lx added 1 commit 2022-06-07 15:10:30 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
1d408d52b9
Simplify
lx added 1 commit 2022-06-07 15:12:04 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
cdc03da0b8
make things more like before
lx added 1 commit 2022-06-07 15:14:25 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is passing Details
8c6f690fa5
less verbose code
lx added 3 commits 2022-06-07 15:50:29 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
0543cb3453
Cleaner error management (less error-prone api)
lx added 1 commit 2022-06-07 15:56:24 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
c103ca8bf2
Correct error type for (unimplemented) iterators in transactions
lx added 1 commit 2022-06-07 15:56:36 +00:00
continuous-integration/drone/push Build is failing Details
continuous-integration/drone/pr Build is failing Details
daec7995c3
cargo fmt
lx added 1 commit 2022-06-07 16:06:46 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
d41a67c4ee
simplify & fix db tests
lx added 1 commit 2022-06-07 16:21:05 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
845c344231
Inform user of what DB engine Garage is running on
lx added 1 commit 2022-06-07 16:22:37 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
cc4f1aca97
sqlite can actually tell us its version
lx added 1 commit 2022-06-07 16:26:36 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
48928d2d70
Simplify sqlite
lx added 1 commit 2022-06-07 16:36:21 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
0c0a02ad03
Remove useless function
lx changed title from WIP: Abstract database behind generic interface and implement alternative drivers to Abstract database behind generic interface and implement alternative drivers 2022-06-07 17:02:26 +00:00
lx added 2 commits 2022-06-07 17:11:27 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
e682478d75
Create meta directory in server (or else sqlite fails)
lx merged commit b44d3fc796 into main 2022-06-08 08:01:56 +00:00
Sign in to join this conversation.
No description provided.