Abstract database behind generic interface and implement alternative drivers #322

Merged

lx merged 64 commits from db-abstraction into main 2022-06-08 08:01:56 +00:00

Conversation 0 Commits 64 Files changed 42 +3086 -641

lx commented 2022-06-02 15:03:58 +00:00

Owner

Copy link

• Design interface
• Implement Sled backend
  • Re-implement the SledCountedTree hack on Sled backend on all backends (i.e. over the abstraction)
• Convert Garage code to use generic interface
• Proof-read converted Garage code
• Test everything well
• Implement sqlite backend
• Implement LMDB backend
• (Implement Persy backend?)
• (Implement other backends? (like RocksDB, ...))
• Implement backend choice in config file and garage server module
• Add CLI for converting between DB formats
• Exploit the new interface to put more things in transactions
  • .updated() trigger on Garage tables

Fix #284

Bugs

• When exporting sqlite, trees iterate empty??
• LMDB doesn't work

Known issues for various back-ends

• Sled:
  • Eats all my RAM and also all my disk space
  • .len() has to traverse the whole table
  • Is actually quite slow on some operations
  • And is actually pretty bad code...
• Sqlite:
  • Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with .iter(), and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a .iter() loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason.
  • (adapter uses a bunch of unsafe code)
• Heed (LMDB):
  • Not suited for 32-bit machines as it has to map the whole DB in memory.
  • (adpater uses a tiny bit of unsafe code)

My recommendation: avoid 32-bit machines and use LMDB as much as possible.

Converting databases is actually quite easy. For example from Sled to LMDB:

cd src/db
cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb

Then, just add this to your config.toml:

db_engine = "lmdb"

- [x] Design interface - [x] Implement Sled backend - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction) - [x] Convert Garage code to use generic interface - [x] Proof-read converted Garage code - [ ] Test everything well - [x] Implement sqlite backend - [x] Implement LMDB backend - [ ] (Implement Persy backend?) - [ ] (Implement other backends? (like RocksDB, ...)) - [x] Implement backend choice in config file and garage server module - [x] Add CLI for converting between DB formats - Exploit the new interface to put more things in transactions - [x] `.updated()` trigger on Garage tables Fix #284 **Bugs** - [x] When exporting sqlite, trees iterate empty?? - [x] LMDB doesn't work **Known issues for various back-ends** - Sled: - Eats all my RAM and also all my disk space - `.len()` has to traverse the whole table - Is actually quite slow on some operations - And is actually pretty bad code... - Sqlite: - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason. - (adapter uses a bunch of unsafe code) - Heed (LMDB): - Not suited for 32-bit machines as it has to map the whole DB in memory. - (adpater uses a tiny bit of unsafe code) **My recommendation:** avoid 32-bit machines and use LMDB as much as possible. **Converting databases** is actually quite easy. For example from Sled to LMDB: ```bash cd src/db cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb ``` Then, just add this to your `config.toml`: ```toml db_engine = "lmdb" ```

lx added 4 commits 2022-06-02 15:04:14 +00:00

First iteration of a generic DB layer fd8d5c37f7

Implement iter() and range() on db 04901093e7

Adapt Garage to use new DB abstraction 9f0f5b2e37

Do not put sled in garage's cargo.toml 6805e184e9

lx added 1 commit 2022-06-02 15:39:16 +00:00

Complete sled abstraction fbd5b64ff3

lx added 1 commit 2022-06-02 16:11:52 +00:00

Use Cell instead of ArcSwap f29b91232f

lx added 1 commit 2022-06-02 17:59:56 +00:00

Begin sqlite adapter 43704afb29

lx added 2 commits 2022-06-02 21:15:57 +00:00

Add back Send 364061453c

Sqlite iter with unsafe code c439cb11a9

lx added 1 commit 2022-06-02 21:44:00 +00:00

Horrible implementation of range for sqlite 0e4f336887

lx added 1 commit 2022-06-02 21:46:40 +00:00

Just refactor a bit 6ebb54cddb

lx added 2 commits 2022-06-03 08:45:33 +00:00

Drop later 7b79906350

Change value type to be a dyn thing f25309e58f

lx added 1 commit 2022-06-03 09:15:09 +00:00

Fix most clippy lints cc0d984118

lx added 1 commit 2022-06-03 09:45:07 +00:00

Conversion utility df0877bbba

lx added 1 commit 2022-06-03 10:12:57 +00:00

Garage works on sqlite, but it's a hack bd9ff432d7

lx added 2 commits 2022-06-03 10:53:06 +00:00

Slightly prettier code 4bf706b170

fix clipy lint 295bc2741f

lx added 2 commits 2022-06-03 11:36:19 +00:00

Fix block repair to not deadlock with sqlite 9bb58638f0

whoops 3d18c9e183

lx added 2 commits 2022-06-03 12:17:03 +00:00

small fixes d3d7df098b

Update rmp-serde 16e0a655d0

lx added 1 commit 2022-06-03 13:31:48 +00:00

Start LMDB adapter, with fixed semantics 4e72c713f1

lx added 1 commit 2022-06-03 13:53:44 +00:00

LMDB: use Heed wrapper (much more complete) bd2997a453

lx added 2 commits 2022-06-03 14:42:04 +00:00

Implement iterator for LMDB f7a1c70089

Integrate LMDB with rest 2d5541b0e7

lx added 2 commits 2022-06-06 10:02:59 +00:00

fixes 4f5d17d464

Fix more sqlite deadlocks 4539a6c229

lx added 3 commits 2022-06-06 12:08:28 +00:00

Safe choice: return Vec<u8> and not some fancy zero-copy type 7f2cf0b809

Safe choice: revert "Update rmp-serde" ... a77efd7ca6

This reverts commit 16e0a655d0.

Small syntax changes 0ab21e780e

lx added 1 commit 2022-06-06 12:09:24 +00:00

Whoops 17918e2ef5

lx added 1 commit 2022-06-06 13:07:43 +00:00

Simplify lmdb adapter (one less indirection) a65049db8f

lx added 2 commits 2022-06-06 13:46:18 +00:00

Fix clippy c56d858834

Table updated trigger now happens in transaction, this is waaaay better! 1897815358

lx added 1 commit 2022-06-06 14:13:24 +00:00

Update Cargo.nix 9238fda9b2

lx added 1 commit 2022-06-06 14:29:19 +00:00

Improve things in block manager & correctly propagate .len() errors 1dabd98330

lx added 1 commit 2022-06-06 14:31:50 +00:00

add comment 72e6419b1b

lx added 1 commit 2022-06-06 14:32:12 +00:00

fix bbf68aa039

lx added 1 commit 2022-06-06 14:37:53 +00:00

Fix more .unwrap()'s c3c8d59c99

lx added 1 commit 2022-06-06 14:45:07 +00:00

in db convert: map size up to 1TB on 64-bit systems fd2fe18de3

lx added 1 commit 2022-06-06 14:51:12 +00:00

Add inline marker on a bunch of db functions 72a87c8686

lx added 1 commit 2022-06-06 15:02:00 +00:00

Fix clippy lint 1f9a371b9a

lx added 1 commit 2022-06-06 15:06:31 +00:00

Remove useless clones in sqlite e42c7bed38

lx added 1 commit 2022-06-06 20:02:43 +00:00

Add nix overrides to include lmdb and sqlite3 libraries 35544cac39

lx added 1 commit 2022-06-06 20:35:56 +00:00

bundled sqlite 3de9320a0e

lx added 1 commit 2022-06-06 21:00:27 +00:00

Try once again to fix sqlite bundling in nix 5dbc79b77e

lx added 1 commit 2022-06-07 14:53:34 +00:00

Bring back the counted tree hack for Sled (with caveat) ... a9e79f848b

caveat: it's not only for sled

lx added 1 commit 2022-06-07 15:05:35 +00:00

db abstraction: make .insert() and .remove() return the old value a3a01141ec

lx added 1 commit 2022-06-07 15:10:30 +00:00

Simplify 1d408d52b9

lx added 1 commit 2022-06-07 15:12:04 +00:00

make things more like before cdc03da0b8

lx added 1 commit 2022-06-07 15:14:25 +00:00

less verbose code 8c6f690fa5

lx added 3 commits 2022-06-07 15:50:29 +00:00

prettier code 6f3d43b75b

less pub(crate) 1bbe0794f3

Cleaner error management (less error-prone api) 0543cb3453

lx added 1 commit 2022-06-07 15:56:24 +00:00

Correct error type for (unimplemented) iterators in transactions c103ca8bf2

lx added 1 commit 2022-06-07 15:56:36 +00:00

cargo fmt daec7995c3

lx added 1 commit 2022-06-07 16:06:46 +00:00

simplify & fix db tests d41a67c4ee

lx added 1 commit 2022-06-07 16:21:05 +00:00

Inform user of what DB engine Garage is running on 845c344231

lx added 1 commit 2022-06-07 16:22:37 +00:00

sqlite can actually tell us its version cc4f1aca97

lx added 1 commit 2022-06-07 16:26:36 +00:00

Simplify sqlite 48928d2d70

lx added 1 commit 2022-06-07 16:36:21 +00:00

Remove useless function 0c0a02ad03

lx changed title from WIP: Abstract database behind generic interface and implement alternative drivers to Abstract database behind generic interface and implement alternative drivers 2022-06-07 17:02:26 +00:00

lx added 2 commits 2022-06-07 17:11:27 +00:00

remove useless clippy allow 1e4814568b

Create meta directory in server (or else sqlite fails) e682478d75

lx merged commit b44d3fc796 into main 2022-06-08 08:01:56 +00:00

lx referenced this pull request from a commit 2022-06-08 08:01:57 +00:00

Abstract database behind generic interface and implement alternative drivers (#322)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

action

check-aws

The behaviour should be checked against AWS S3

  

action

discussion-needed

We need to discuss if the feature is desirable and/or better specify the envisioned behavior or technical implementation

  

action

for-external-contributors

Core team does not plan working on this in the near future however an external contribution will be reviewed & merged

  

action

for-newcomers

An issue that should be newcomer friendly.

  

action

more-info-needed

More information is needed to investigate this issue

  

action

need-funding

This issue represents lots of work, we need to find money to pay our core team work

  

action

triage-required

It's not very clear what we want to do with this issue yet, more thoughts is required

  

kind

correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

kind

ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

kind

improvement

Feature requests that do not fall in other categories

  

kind

performance

Deceptive performance observed OR idea to improve performances

  

kind

testing

Functional tests, integration tests, Jepsen, etc.

  

kind

usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

  

kind

wrong-behavior

Unintended behavior

  

prio

critical

Could impact integrity or availability of production clusters

  

prio

low

For issues that we don't want to spend time on until made absolutely necessary

  

scope

admin-api

Issues concerning the administration API

  

scope

background-healing

Background healing covers repair tasks, anti-entropy, merkle trees, etc. ; everything related to replicating data appropriately.

  

scope

build

Issues with Nix, Drone, Rust Compiler, etc.

  

scope

documentation

  

scope

k8s

Kubernetes related, including Helm Charts

  

scope

layout

Linked to the cluster layout and storage management

  

scope

metadata

Metadata Engine (LMDB, sqlite, etc.)

  

scope

ops

Operations related to operations/maintenance issue, tooling related (like CLI missing info ; impossible, too dangerous, too complicated migration, etc.)

  

scope

rpc

Linked to RPC

  

scope

s3-api

Related to the S3 API (especially compatibility issues).

  

scope

security

Cryptography-related, auth-related, authorization-related issues

  

scope

telemetry

OpenTelemetry (traces, metrics, etc.)

No labels

action

check-aws

action

discussion-needed

action

for-external-contributors

action

for-newcomers

action

more-info-needed

action

need-funding

action

triage-required

kind

correctness

kind

ideas

kind

improvement

kind

performance

kind

testing

kind

usability

kind

wrong-behavior

prio

critical

prio

low

scope

admin-api

scope

background-healing

scope

build

scope

documentation

scope

k8s

scope

layout

scope

metadata

scope

ops

scope

rpc

scope

s3-api

scope

security

scope

telemetry

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#322

No description provided.