Garage v1.0 #683

Merged

lx merged 119 commits from next-0.10 into main 2024-04-10 15:23:13 +00:00

Conversation 0 Commits 119 Files changed 124 +7724 -3830

119 commits

Author	SHA1	Message	Date
Alex	ff093ddbb8	Merge branch 'main' into next-0.10	2024-04-10 14:38:14 +02:00
Alex	90e3c2af91	[next-0.10] small updates to mention Garage v0.9.4	2024-04-10 14:35:30 +02:00
Alex	50669b3e76	[next-0.10] bump helm chart version	2024-04-03 14:19:59 +02:00
Alex	554437254e	[next-0.10] Add migration guide for v1.0	2024-03-28 18:45:06 +01:00
Alex	afad62939e	[next-0.10] bump version number to 1.0	2024-03-28 15:19:44 +01:00
Alex	8bfc16ba7d	Merge branch 'main' into next-0.10	2024-03-28 15:01:05 +01:00
Alex	25c196f34d	[next-0.10] admin api: fix logic in get cluster status	2024-03-27 13:55:49 +01:00
Alex	4eba32f29f	[next-0.10] layout helper: rename & clarify updates to update trackers	2024-03-27 13:47:06 +01:00
Alex	32f1786f9f	[next-0.10] cache layout check result	2024-03-27 13:37:20 +01:00
Alex	01a0bd5410	[next-0.10] remove impl Deref for LayoutHelper	2024-03-27 13:32:13 +01:00
Alex	c0eeb0b0f3	[next-0.10] fixes to k2v rpc + comment fixes	2024-03-27 10:44:03 +01:00
Alex	51d11b4b26	[next-0.10] doc: 2 changes ... - rewrite section on encryption to mention SSE-C - change to real-world to make it closer to main branch	2024-03-27 10:10:45 +01:00
Alex	f7cd4eb600	Merge branch 'main' into next-0.10	2024-03-26 16:34:40 +01:00
Alex	cdde0f19ee	Merge pull request 'checksum algorithms' (#787) from s3-checksum into next-0.10 ... Reviewed-on: #787	2024-03-26 14:24:58 +00:00
Alex	74949c69cb	[s3-checksum] implement x-amz-checksum-* headers	2024-03-26 15:01:34 +01:00
Alex	7e0107c47d	Merge pull request 'Fixes to garage_net peering manager' (#786) from net-fixes into next-0.10 ... Reviewed-on: #786	2024-03-21 10:26:36 +00:00
Alex	3844110cd0	[net-fixes] netapp peer exchange: send only currently connected address	2024-03-21 10:50:44 +01:00
Alex	961b4f9af3	[net-fixes] fix issues with local peer address (fix #761)	2024-03-21 10:45:34 +01:00
Alex	5225a81dee	[net-fixes] peering: only count node IDs and not addresses in hash	2024-03-21 09:47:04 +01:00
Alex	ce69dc302c	Merge branch 'main' into next-0.10	2024-03-19 17:17:46 +01:00
Alex	65853a4863	Merge pull request 'block refcount repair' (#782) from block-ref-repair into next-0.10 ... Reviewed-on: #782	2024-03-19 15:59:19 +00:00
Alex	3eab639c14	[block-ref-repair] mention garage block repair-rc in documentation	2024-03-19 16:24:34 +01:00
Alex	3165ab926c	[block-ref-repair] rename rc's rc field to rc_table	2024-03-19 16:20:22 +01:00
Alex	dc0b78cdb8	[block-ref-repair] Block refcount recalculation and repair ... - We always recalculate the reference count of a block before deleting it locally, to make sure that it is indeed zero. - If we had to fetch a remote block but we were not able to get it, check that refcount is indeed > 0. - Repair procedure that checks everything	2024-03-19 16:20:22 +01:00
Alex	0038ca8a78	Merge branch 'main' into next-0.10	2024-03-18 20:19:30 +01:00
Alex	81191d2d92	Merge pull request 'Remove Sled' (#767) from rm-sled into next-0.10 ... Reviewed-on: #767	2024-03-12 10:45:57 +00:00
Alex	2795b53b8b	[rm-sled] factorize some code in sqlite backend	2024-03-12 11:15:26 +01:00
Alex	32aa246300	[rm-sled] Make proper use of pinning in LMDB adapter + comment unsafe	2024-03-08 17:39:17 +01:00
Alex	b942949940	[rm-sled] Implement iterators in sqlite & lmdb transactions ... with way too much unsafe code	2024-03-08 16:38:01 +01:00
Alex	66c23890c1	[rm-sled] Implement some missing functionality in garage_db	2024-03-08 16:02:58 +01:00
Alex	05c92204ec	[rm-sled] Remove counted_tree_hack	2024-03-08 15:09:57 +01:00
Alex	2128b5febd	Merge pull request 'Remove migration path from Garage v0.5' (#766) from rm-migration into next-0.10 ... Reviewed-on: #766	2024-03-08 13:43:42 +00:00
Alex	44454aac01	[rm-sled] Remove the Sled database engine	2024-03-08 14:11:02 +01:00
Alex	1ace34adbb	Merge branch 'main' into next-0.10	2024-03-08 13:57:10 +01:00
Alex	f537f76681	[rm-migration] Remove migration path from Garage v0.5	2024-03-08 13:24:47 +01:00
Alex	20c0b4ffb2	Merge pull request 'ReplicationMode -> ConsistencyMode+ReplicationFactor' (#750) from yuka/garage:split-consistency-mode into next-0.10 ... Reviewed-on: #750	2024-03-07 16:32:52 +00:00
Alex	2fd13c7d13	Merge pull request 'SSE-C encryption' (#730) from sse-c into next-0.10 ... Reviewed-on: #730	2024-03-07 15:21:37 +00:00
Alex	3fcb54e3cf	[sse-c] Remove special case for Content-Type header	2024-03-07 15:43:48 +01:00
Alex	e3333f2ac5	[sse-c] Documentation for SSE-C	2024-03-07 15:43:48 +01:00
Alex	fa4878bad6	[sse-c] Testing for SSE-C encryption	2024-03-07 15:43:48 +01:00
Alex	57acc60082	[sse-c] Implement SSE-C encryption	2024-03-07 15:43:47 +01:00
Alex	fe2dc5d51c	Merge branch 'main' into next-0.10	2024-03-07 14:00:34 +01:00
Yureka	c1769bbe69	ReplicationMode -> ConsistencyMode+ReplicationFactor	2024-03-07 12:45:33 +01:00
Yureka	8f86af52ed	adjust docs for replication factor	2024-03-05 22:57:08 +01:00
Alex	603604cdfc	Merge pull request 'refactor: remove max_write_errors and max_faults' (#760) from yuka/garage:remove-max-write-errors into next-0.10 ... Reviewed-on: #760	2024-03-05 21:56:17 +00:00
Yureka	6760895926	refactor: remove max_write_errors and max_faults	2024-03-04 18:39:56 +01:00
Alex	bbde9bc912	Merge branch 'main' into next-0.10	2024-03-04 15:56:10 +01:00
Alex	d0d95fd53f	[next-0.10] woodpecker: run debug pipeline on manual trigger	2024-02-27 10:13:09 +01:00
Alex	4b978b7533	Merge branch 'main' into next-0.10	2024-02-26 18:55:24 +01:00
Alex	3692af7052	Merge branch 'main' into next-0.10	2024-02-23 18:28:05 +01:00
Alex	916c67ccf4	Merge branch 'main' into next-0.10	2024-02-23 16:50:34 +01:00
Alex	81cebdd124	[next-0.10] fix build	2024-02-22 15:53:47 +01:00
Alex	59f61c966a	Merge branch 'main' into next-0.10	2024-02-22 15:45:45 +01:00
Alex	75e591727d	[next-0.10] cluster node status metrics: report nodes of all active layout versions	2024-02-20 17:08:31 +01:00
Alex	643d1aabd8	Merge branch 'main' into next-0.10	2024-02-20 17:02:44 +01:00
Alex	eb4a6ce106	Merge branch 'main' into next-0.10	2024-02-15 14:06:34 +01:00
Alex	cf2af186fc	Merge branch 'main' into next-0.10	2024-02-13 11:36:28 +01:00
Alex	db48dd3d6c	bump crate versions to 0.10.0	2024-01-11 12:05:51 +01:00
Alex	8a6ec1d611	Merge pull request 'NLnet task 3' (#667) from nlnet-task3 into next-0.10 ... Reviewed-on: #667	2024-01-11 10:58:08 +00:00
Alex	0041b013a4	layout: refactoring and fix in layout helper	2023-12-11 16:09:22 +01:00
Alex	adccce1145	layout: refactor/fix bad while loop	2023-12-11 15:45:14 +01:00
Alex	85b5a6bcd1	fix some clippy lints	2023-12-11 15:31:47 +01:00
Alex	e4f493b481	table: remove redundant tracing in insert_many	2023-12-11 14:57:42 +01:00
Alex	f8df90b79b	table: fix insert_many to not send duplicates	2023-12-08 14:54:11 +01:00
Alex	4dbf254512	layout: refactoring, merge two files	2023-12-08 14:15:52 +01:00
Alex	64a6e557a4	rpc helper: small refactorings	2023-12-08 12:18:12 +01:00
Alex	5dd200c015	layout: move block_read_nodes_of to rpc_helper to avoid double-locking ... (in theory, this could have caused a deadlock)	2023-12-08 12:02:24 +01:00
Alex	063294dd56	layout version: refactor get_node_zone	2023-12-08 11:50:58 +01:00
Alex	7f2541101f	cli: improvements to the layout commands when multiple layouts are live	2023-12-08 11:24:23 +01:00
Alex	91b874c4ef	rpc: fix system::health	2023-12-08 10:36:37 +01:00
Alex	431b28e0cf	fix build with discovery features	2023-12-07 15:15:59 +01:00
Alex	9cecea64d4	layout: allow sync update tracker to progress with only quorums	2023-12-07 14:51:20 +01:00
Alex	aa59059a91	layout cli: safer skip-dead-nodes command	2023-12-07 11:56:14 +01:00
Alex	d90de365b3	table sync: use write quorums to report global success or failure of sync	2023-12-07 11:16:10 +01:00
Alex	95eb13eb08	rpc: refactor result tracking for quorum sets	2023-12-07 10:57:21 +01:00
Alex	c8356a91d9	layout updates: fix the set of nodes among which minima are calculated	2023-12-07 10:30:26 +01:00
Alex	c04dd8788a	admin: more info in admin GetClusterStatus	2023-11-28 14:25:04 +01:00
Alex	539af6eac4	rpc helper: write comments + small refactoring of tracing	2023-11-28 11:12:39 +01:00
Alex	c539077d30	cli: remove historic layout info from status	2023-11-27 16:22:27 +01:00
Alex	11e6fef93c	cli: add layout history and layout assume-sync commands	2023-11-27 16:22:25 +01:00
Alex	539a920313	cli: show when nodes are draining metadata	2023-11-27 13:18:59 +01:00
Alex	78362140f5	rpc: update system::health to take into account write sets for all partitions	2023-11-27 12:10:21 +01:00
Alex	d6d239fc79	block manager: read_block using old layout versions if necessary	2023-11-27 11:52:57 +01:00
Alex	3ecd14b9f6	table: implement write sets for insert_many	2023-11-16 16:41:45 +01:00
Alex	22f38808e7	rpc_helper: don't use tokio::spawn for individual requests	2023-11-16 16:34:01 +01:00
Alex	707442f5de	layout: refactor digests and add "!=" assertions before epidemic bcast	2023-11-16 13:51:40 +01:00
Alex	ad5c6f779f	layout: split helper in separate file; more precise difference tracking	2023-11-16 13:26:43 +01:00
Alex	d4df03424f	layout: fix test	2023-11-15 15:56:57 +01:00
Alex	33c8a489b0	layou: implement ack locking	2023-11-15 15:40:44 +01:00
Alex	393c4d4515	layout: add helper for cached/external values to centralize recomputation	2023-11-15 14:20:50 +01:00
Alex	65066c7064	layout: wip cache global mins	2023-11-15 13:28:30 +01:00
Alex	acd49de9f9	rpc: fix write set quorums	2023-11-15 13:07:42 +01:00
Alex	46007bf01d	integration test: print stdout and stderr on subcommand crash	2023-11-15 12:56:52 +01:00
Alex	b3e729f4b8	layout history merge: rm invalid versions when valid versions are added	2023-11-15 12:15:58 +01:00
Alex	7ef2c23120	layout: fix test	2023-11-14 15:45:01 +01:00
Alex	90e1619b1e	table: take into account multiple write sets in inserts	2023-11-14 15:40:46 +01:00
Alex	3b361d2959	layout: prepare for write sets	2023-11-14 14:28:16 +01:00
Alex	866196750f	system: add todo wrt new layout	2023-11-14 13:36:58 +01:00
Alex	83a11374ca	layout: fixes in schema	2023-11-14 13:29:26 +01:00
Alex	1aab1f4e68	layout: refactoring of all_nodes	2023-11-14 13:12:32 +01:00
Alex	8e292e06b3	layout: some refactoring of nongateway nodes	2023-11-14 12:48:38 +01:00
Alex	9a491fa137	layout: fix test	2023-11-11 13:10:59 +01:00
Alex	df24bb806d	layout/sync: fix bugs and add tracing	2023-11-11 12:44:27 +01:00
Alex	ce89d1ddab	table sync: adapt to new layout history	2023-11-11 12:08:32 +01:00
Alex	df36cf3099	layout: add helpers to LayoutHistory and prepare integration with Table	2023-11-09 16:32:31 +01:00
Alex	9d95f6f704	layout: fix tracker bugs	2023-11-09 15:52:45 +01:00
Alex	bad7cc812e	layout admin: add missing calls to update_hash	2023-11-09 15:42:10 +01:00
Alex	03ebf18830	layout: begin managing the update tracker values	2023-11-09 15:31:59 +01:00
Alex	94caf9c0c1	layout: separate code path for synchronizing update trackers only	2023-11-09 14:53:34 +01:00
Alex	bfb1845fdc	layout: refactor to use a RwLock on LayoutHistory	2023-11-09 14:12:05 +01:00
Alex	19ef1ec8e7	layout: more refactoring	2023-11-09 13:34:14 +01:00
Alex	8a2b1dd422	wip: split out layout management from System into separate LayoutManager	2023-11-09 12:55:36 +01:00
Alex	523d2ecb95	layout: use separate CRDT for staged layout changes	2023-11-09 11:19:43 +01:00
Alex	1da0a5676e	bump garage protocol version tag to 0x000A (0.10)	2023-11-08 19:30:58 +01:00
Alex	8dccee3ccf	cluster layout: adapt all uses of ClusterLayout to LayoutHistory	2023-11-08 19:28:36 +01:00
Alex	fe9af1dcaa	WIP: garage_rpc: store layout version history	2023-11-08 17:49:06 +01:00
Alex	4a9c94514f	avoid using layout_watch in System directly	2023-11-08 16:41:00 +01:00
Alex	12d1dbfc6b	remove Ring and use ClusterLayout everywhere	2023-11-08 15:41:24 +01:00
Alex	0962313ebd	garage_rpc: reorder functions in layout.rs	2023-11-08 13:13:04 +01:00