48 Commits

Author	SHA1	Message	Date
Alex	0a1ddcf630	Prepare for v0.8.2	2023-03-13 18:46:31 +01:00
Jonathan Davies	20c1cdf662	Cargo.toml: Loosen tracing dependency to just 0.1.	2023-01-26 11:13:11 +00:00
Jonathan Davies	f952e37ba7	{model,util}/Cargo.toml: Updated blake2 from 0.9 to 0.10.	2023-01-26 11:13:11 +00:00
Jonathan Davies	36944f1839	Cargo.toml: Updated base64 from 0.13 to 0.21.	2023-01-26 11:13:07 +00:00
Jonathan Davies	5c3075fe01	Cargo.toml: Updated zstd from 0.9 to 0.12.	2023-01-23 18:08:14 +00:00
Alex	8d5505514f	Make it explicit when using nonversioned encoding	2023-01-03 15:27:36 +01:00
Alex	6775569525	Bump everything to v0.8.1	2023-01-02 14:15:33 +01:00
Alex	c1fb65194c	Add sled default in garage_model also	2022-11-21 14:25:54 +01:00
Alex	28a4af73ca	Use netapp 0.5 published from crates.io	2022-09-13 13:11:44 +02:00
Alex	7f54706b95	Merge branch 'lx-perf-improvements' into netapp-stream-body	2022-09-08 15:50:56 +02:00
Alex	ceb1f0229a	Move version back into util	2022-09-07 18:36:46 +02:00
Alex	28d86e7602	Report build features in garage --help	2022-09-07 17:05:21 +02:00
Alex	db61f41030	Move GIT_VERSION injection later in build chain to reduce build times	2022-09-07 11:59:56 +02:00
Alex	0f5689c169	Include code from v0.5.1 directly to remove dependencies	2022-09-06 17:52:50 +02:00
Alex	b886c75450	Make all DB engines optional build features	2022-09-06 17:09:43 +02:00
Alex	48ffaaadfc	Bump versions to 0.8.0 (compatibility is broken already)	2022-09-06 16:47:56 +02:00
Alex	a35d4da721	update netapp to 0.5	2022-07-29 12:25:02 +02:00
Alex	8e7e680afe	First adaptation to WIP netapp with streaming body	2022-07-29 12:25:02 +02:00
Alex	b44d3fc796	Abstract database behind generic interface and implement alternative drivers (#322) ... - [x] Design interface - [x] Implement Sled backend - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction) - [x] Convert Garage code to use generic interface - [x] Proof-read converted Garage code - [ ] Test everything well - [x] Implement sqlite backend - [x] Implement LMDB backend - [ ] (Implement Persy backend?) - [ ] (Implement other backends? (like RocksDB, ...)) - [x] Implement backend choice in config file and garage server module - [x] Add CLI for converting between DB formats - Exploit the new interface to put more things in transactions - [x] `.updated()` trigger on Garage tables Fix #284 **Bugs** - [x] When exporting sqlite, trees iterate empty?? - [x] LMDB doesn't work **Known issues for various back-ends** - Sled: - Eats all my RAM and also all my disk space - `.len()` has to traverse the whole table - Is actually quite slow on some operations - And is actually pretty bad code... - Sqlite: - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason. - (adapter uses a bunch of unsafe code) - Heed (LMDB): - Not suited for 32-bit machines as it has to map the whole DB in memory. - (adpater uses a tiny bit of unsafe code) **My recommendation:** avoid 32-bit machines and use LMDB as much as possible. **Converting databases** is actually quite easy. For example from Sled to LMDB: ```bash cd src/db cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb ``` Then, just add this to your `config.toml`: ```toml db_engine = "lmdb" ``` Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #322 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-08 10:01:44 +02:00
Alex	5768bf3622	First implementation of K2V (#293) ... **Specification:** View spec at [this URL](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/k2v/doc/drafts/k2v-spec.md) - [x] Specify the structure of K2V triples - [x] Specify the DVVS format used for causality detection - [x] Specify the K2V index (just a counter of number of values per partition key) - [x] Specify single-item endpoints: ReadItem, InsertItem, DeleteItem - [x] Specify index endpoint: ReadIndex - [x] Specify multi-item endpoints: InsertBatch, ReadBatch, DeleteBatch - [x] Move to JSON objects instead of tuples - [x] Specify endpoints for polling for updates on single values (PollItem) **Implementation:** - [x] Table for K2V items, causal contexts - [x] Indexing mechanism and table for K2V index - [x] Make API handlers a bit more generic - [x] K2V API endpoint - [x] K2V API router - [x] ReadItem - [x] InsertItem - [x] DeleteItem - [x] PollItem - [x] ReadIndex - [x] InsertBatch - [x] ReadBatch - [x] DeleteBatch **Testing:** - [x] Just a simple Python script that does some requests to check visually that things are going right (does not contain parsing of results or assertions on returned values) - [x] Actual tests: - [x] Adapt testing framework - [x] Simple test with InsertItem + ReadItem - [x] Test with several Insert/Read/DeleteItem + ReadIndex - [x] Test all combinations of return formats for ReadItem - [x] Test with ReadBatch, InsertBatch, DeleteBatch - [x] Test with PollItem - [x] Test error codes - [ ] Fix most broken stuff - [x] test PollItem broken randomly - [x] when invalid causality tokens are given, errors should be 4xx not 5xx **Improvements:** - [x] Descending range queries - [x] Specify - [x] Implement - [x] Add test - [x] Batch updates to index counter - [x] Put K2V behind `k2v` feature flag Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #293 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-05-10 13:16:57 +02:00
Alex	94f1e48fff	Update to netapp 0.4.2 (a tiny fix)	2022-04-07 11:50:03 +02:00
Alex	c1d9854d2c	Move block manager to separate module	2022-03-23 10:25:15 +01:00
Alex	db46cdef79	Update netapp to v0.4.1	2022-03-15 17:09:57 +01:00
Alex	203e8d2c34	Bump version to 0.7 because of incompatible Netapp	2022-03-14 10:54:24 +01:00
Alex	bb04d94fa9	Update to Netapp 0.4 which supports distributed tracing	2022-03-14 10:52:30 +01:00
Alex	8c2fb0c066	Add tracing integration with opentelemetry	2022-03-14 10:52:13 +01:00
Alex	2cab84b1fe	Add many metrics in table/ and rpc/	2022-03-14 10:51:50 +01:00
Alex	beeef4758e	Some movement of helper code and refactoring of error handling	2022-01-04 12:52:46 +01:00
Alex	4d30e62db4	New buckets for 0.6.0: migration code and build files	2022-01-04 12:46:13 +01:00
Alex	5b1117e582	New model for buckets	2022-01-04 12:45:46 +01:00
trinity-1686a	1eb972b1ac	Add compression using zstd (#173) ... fix #27 Co-authored-by: Trinity Pointard <trinity.pointard@gmail.com> Reviewed-on: #173 Co-authored-by: trinity-1686a <trinity.pointard@gmail.com> Co-committed-by: trinity-1686a <trinity.pointard@gmail.com>	2021-12-15 11:26:43 +01:00
Alex	c94406f428	Improve how node roles are assigned in Garage ... - change the terminology: the network configuration becomes the role table, the configuration of a nodes becomes a node's role - the modification of the role table takes place in two steps: first, changes are staged in a CRDT data structure. Then, once the user is happy with the changes, they can commit them all at once (or revert them). - update documentation - fix tests - implement smarter partition assignation algorithm This patch breaks the format of the network configuration: when migrating, the cluster will be in a state where no roles are assigned. All roles must be re-assigned and commited at once. This migration should not pose an issue.	2021-11-16 16:05:53 +01:00
Alex	43e13a501d	Use published netapp crate instead of git repo	2021-10-26 10:36:57 +02:00
Alex	4067797d01	First port of Garage to Netapp	2021-10-22 15:55:18 +02:00
Alex	b9127dd6f8	Prepare for v0.3.0 and add migration path from v0.2.1.x	2021-05-28 15:29:58 +02:00
Alex	f859d15062	update to v0.2.1	2021-03-19 13:39:18 +01:00
Alex	4c26a0b9c1	Update Cargo.toml files with AGPL license info	2021-03-18 21:59:17 +01:00
Alex	dead945c8f	Prepare for release 0.2	2021-03-18 19:33:15 +01:00
Alex	f4346cc5f4	Update dependencies	2021-03-16 15:58:40 +01:00
Alex	0cd5b2ae19	WIP migrate to tokio 1	2021-03-15 22:36:41 +01:00
Alex	445912dc6a	Remove migration paths from 0.1 branch	2021-03-10 16:38:31 +01:00
Alex	20e6e9fa20	Update sled & try to debug deadlock (but its in sled...)	2021-02-23 21:27:28 +01:00
Alex	ceeb0732a2	Use 0.1.0b instead of 0.1.0 (for compatibility with new Error type)	2021-01-15 18:27:58 +01:00
Alex	1d1d497e2b	Bump everything to 0.1.1	2021-01-15 17:54:48 +01:00
Alex	86bf4dedac	Add support for model migrations	2020-07-08 16:10:53 +02:00
Alex	3b0b11085e	Add versions to dependencies	2020-07-07 14:18:47 +02:00
Alex	cc65cdc0fe	Add license, description and repository to .toml files	2020-07-07 14:14:58 +02:00
Alex	bec26a1312	Rename garage_core to garage_model	2020-07-07 13:59:22 +02:00

Renamed from src/core/Cargo.toml (Browse further)