785 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Alex Auvolat	08cd5f2f1d	cargo fmt	2022-07-08 10:45:20 +02:00
Alex Auvolat	18d014dcc5	Merge branch 'main' into background-task-manager	2022-07-08 10:43:23 +02:00
Alex Auvolat	aab34bfe54	add delays in k2v test_items_and_indices	2022-07-08 10:41:57 +02:00
Alex Auvolat	d1cf1a0fa6	Rename WorkerStatus to WorkerState ... because it's a state in a state machine	2022-07-08 10:39:41 +02:00
Alex Auvolat	0f660b086c	fix comments	2022-07-08 10:32:41 +02:00
Alex Auvolat	4312623930	whoops and small refactoring	2022-07-08 10:16:46 +02:00
Alex Auvolat	4cc9a648ab	Small refactoring	2022-07-08 10:10:30 +02:00
Alex Auvolat	0200eae679	cleanup	2022-07-08 09:55:27 +02:00
Alex	fe3fa83de7	Publish k2v-client crate to crates.io (#337) ... Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #337 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-07-04 18:27:25 +02:00
Alex Auvolat	b6d59ec19a	Fix poll item when item didn't change	2022-07-04 14:00:02 +02:00
Alex Auvolat	5ee9cb7768	add delays in k2v test_items_and_indices	2022-07-04 12:53:47 +02:00
Alex Auvolat	17d0254c7c	Merge branch 'main' into background-task-manager	2022-07-04 12:46:44 +02:00
Alex	0850bac874	Add poll command to k2v-cli (#335) ... Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #335 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-07-04 12:45:32 +02:00
Alex	b74b533b7b	Fix typo	2022-06-29 11:50:51 +02:00
Alex Auvolat	c4be56704d	remove useless info!()	2022-06-28 17:43:56 +02:00
Alex Auvolat	368414a261	Small things	2022-06-28 17:42:54 +02:00
Alex Auvolat	fc2bc8b0ca	cargo fmt	2022-06-28 17:31:23 +02:00
Alex Auvolat	b0a181e17e	Count corruptions in scrub worker	2022-06-28 17:10:59 +02:00
Alex Auvolat	b053fc0518	Persist scrub worker thing in Persister	2022-06-28 16:59:19 +02:00
Alex Auvolat	f1c972289d	update cargo.nix	2022-06-28 16:09:26 +02:00
Alex Auvolat	247dbcd598	Only one scrub worker (wip)	2022-06-27 17:57:48 +02:00
Alex Auvolat	0e5175abee	Report progress of scrub and block repair	2022-06-27 16:52:46 +02:00
Alex Auvolat	fc50724256	Update cargo.nix	2022-06-27 12:06:17 +02:00
Alex Auvolat	fdfe7dd60d	Fix clippy lint	2022-06-27 12:01:23 +02:00
Alex Auvolat	928394cc32	(makefile with run release)	2022-06-27 11:58:14 +02:00
Alex Auvolat	e7810e9cb3	Smaller batches for index counter propagation	2022-06-24 11:04:55 +02:00
Alex Auvolat	0837b3dacd	Slightly improve blocking code, add info to resync worker	2022-06-24 10:49:52 +02:00
Alex Auvolat	95ffba343f	Error reporting	2022-06-24 10:31:11 +02:00
Alex Auvolat	59b43914d4	(to test) error reporting and throttling at higher layer	2022-06-23 17:05:11 +02:00
Alex	996f2a6d58	Slides for talk at IMT Atlantique / STACK on 2022-06-23 (#333) ... Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #333 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-23 14:28:40 +02:00
Alex Auvolat	f82cf164f5	Update Cargo.nix	2022-06-21 18:06:59 +02:00
Alex Auvolat	8fb8569912	fix clippy	2022-06-21 18:04:03 +02:00
Alex Auvolat	708dab6e0f	fix clippy	2022-06-21 17:22:20 +02:00
Alex Auvolat	10c886111e	Rename things, garage worker list cmd	2022-06-21 17:18:16 +02:00
Alex Auvolat	14337d2a56	rename things	2022-06-21 16:27:12 +02:00
Alex Auvolat	ba1ace6cf6	Block repair with new worker semantics	2022-06-21 16:00:08 +02:00
Alex Auvolat	a855c54bdb	Online repair new workers, except blocks and scrub	2022-06-21 15:27:58 +02:00
Alex Auvolat	b8338dea56	New worker for index counter propagator	2022-06-21 14:23:54 +02:00
Alex Auvolat	269f996fd0	Block manager with new worker	2022-06-21 14:07:10 +02:00
Alex Auvolat	3119ea59b0	New worker semantics applied to garage_table	2022-06-21 13:50:55 +02:00
Alex Auvolat	e12bc3b595	First try on background worker manager	2022-06-21 12:37:52 +02:00
Alex	77e3fd6db2	improve internal item counter mechanisms and implement bucket quotas (#326) ... - [x] Refactoring of internal counting API - [x] Repair procedure for counters (it's an offline procedure!!!) - [x] New counter for objects in buckets - [x] Add quotas to buckets struct - [x] Add CLI to manage bucket quotas - [x] Add admin API to manage bucket quotas - [x] Apply quotas by adding checks on put operations - [x] Proof-read Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #326 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-15 20:20:28 +02:00
Quentin Dufour	d544a0e0e0	Send CORS headers for all requests	2022-06-13 10:19:52 +02:00
Alex Auvolat	138e13071b	Fix garage_db build on 32-bit systems	2022-06-09 14:55:20 +02:00
Alex	b44d3fc796	Abstract database behind generic interface and implement alternative drivers (#322) ... - [x] Design interface - [x] Implement Sled backend - [x] Re-implement the SledCountedTree hack ~~on Sled backend~~ on all backends (i.e. over the abstraction) - [x] Convert Garage code to use generic interface - [x] Proof-read converted Garage code - [ ] Test everything well - [x] Implement sqlite backend - [x] Implement LMDB backend - [ ] (Implement Persy backend?) - [ ] (Implement other backends? (like RocksDB, ...)) - [x] Implement backend choice in config file and garage server module - [x] Add CLI for converting between DB formats - Exploit the new interface to put more things in transactions - [x] `.updated()` trigger on Garage tables Fix #284 **Bugs** - [x] When exporting sqlite, trees iterate empty?? - [x] LMDB doesn't work **Known issues for various back-ends** - Sled: - Eats all my RAM and also all my disk space - `.len()` has to traverse the whole table - Is actually quite slow on some operations - And is actually pretty bad code... - Sqlite: - Requires a lock to be taken on all operations. The lock is also taken when iterating on a table with `.iter()`, and the lock isn't released until the iterator is dropped. This means that we must be VERY carefull to not do anything else inside a `.iter()` loop or else we will have a deadlock! Most such cases have been eliminated from the Garage codebase, but there might still be some that remain. If your Garage-over-Sqlite seems to hang/freeze, this is the reason. - (adapter uses a bunch of unsafe code) - Heed (LMDB): - Not suited for 32-bit machines as it has to map the whole DB in memory. - (adpater uses a tiny bit of unsafe code) **My recommendation:** avoid 32-bit machines and use LMDB as much as possible. **Converting databases** is actually quite easy. For example from Sled to LMDB: ```bash cd src/db cargo run --features cli --bin convert -- -i path/to/garage/meta/db -a sled -o path/to/garage/meta/db.lmdb -b lmdb ``` Then, just add this to your `config.toml`: ```toml db_engine = "lmdb" ``` Co-authored-by: Alex Auvolat <alex@adnab.me> Reviewed-on: #322 Co-authored-by: Alex <alex@adnab.me> Co-committed-by: Alex <alex@adnab.me>	2022-06-08 10:01:44 +02:00
Simon C	7eed3ceda9	docs: Add Trafik reverse proxy documentation	2022-06-07 16:16:52 +02:00
Simon C	4b8f48f3c5	docs: Fix title level	2022-06-07 13:32:52 +02:00
Simon C	7d3b5585f1	docs: Add link to facilitate navigation in the documentation	2022-06-07 09:38:59 +02:00
Quentin Dufour	a1abed0378	Remove useless MC_REGION env variable	2022-06-02 12:50:11 +02:00
Alex Auvolat	b54a938724	Fix garage_version() now that GIT_VERSION is read in crate garage_rpc	2022-06-02 12:00:10 +02:00