use a real merkle tree for sync #34

New Issue

Closed

opened 2021-03-10 16:52:58 +00:00 by lx · 1 comment

lx commented

2021-03-10 16:52:58 +00:00

Owner

Store a Merkle tree of the dataset in Sled, next to the actual data.

Sled tables:

table_data
table_merkle_todo, updated with transactions when table_data is updated
table_merkle, that contains the actual merkle tree. Updated by a single thread using Sled batches.
for the gc (#32), a table table_gc_todo

Threads that need to run:

a thread that processes table_merkle_todo and updates table_merkle
the sync thread, which can do full sync much more often since it does not need to calculate merkle hashes anymore
the gc thread that runs independently (causes the merkle stuff to be updated via the merkle_todo table, as if it were regular updates)

Store a Merkle tree of the dataset in Sled, next to the actual data. Sled tables: - `table_data` - `table_merkle_todo`, updated with transactions when `table_data` is updated - `table_merkle`, that contains the actual merkle tree. Updated by a single thread using Sled batches. - for the gc (#32), a table `table_gc_todo` Threads that need to run: - a thread that processes `table_merkle_todo` and updates `table_merkle` - the sync thread, which can do full sync much more often since it does not need to calculate merkle hashes anymore - the gc thread that runs independently (causes the merkle stuff to be updated via the merkle_todo table, as if it were regular updates)