WIP: Garage v2 #948

Draft

lx wants to merge 125 commits from next-v2 into main

pull from: next-v2

merge into: Deuxfleurs:main

Deuxfleurs:main

Deuxfleurs:fix-bucket-aliases

Deuxfleurs:add-unsigned-payload-trailer-test

Deuxfleurs:v1.99-internal

Deuxfleurs:feat-metrics-metadata-engine

Deuxfleurs:main-0.9.x

Deuxfleurs:db-no-unsafe

Deuxfleurs:main-0.8.x

Deuxfleurs:k2v/shared_http_client

Deuxfleurs:feat/website-redir

Deuxfleurs:smithy2

Deuxfleurs:test/compilation

Deuxfleurs:poc/nomad-ci

Deuxfleurs:test-ci-wait

Deuxfleurs:pnet_datalink-0.33.0

Deuxfleurs:k2v-watch-range

Deuxfleurs:db-debug-log

Deuxfleurs:build-convert-db

Deuxfleurs:demonstrate-cargo2nix-bug

Deuxfleurs:dangerous/no-fsync

Deuxfleurs:dev0.7.3

Deuxfleurs:doc/benchmarks

Deuxfleurs:bug/check_static

Deuxfleurs:bucket-cors-more

Deuxfleurs:storage-optimizations

Deuxfleurs:old-main

Conversation 0 Commits 125 Files changed 113 +14595 -4879

125 commits

Author	SHA1	Message	Date
Alex	2d1c073d2f	Merge pull request 'admin api: implement InspectObject (fix #892)' (#1005) from inspect-object into next-v2 ... Reviewed-on: #1005	2025-04-06 12:33:11 +00:00
Alex Auvolat	5e7307cbf3	admin api: add comments for InspectObject	2025-04-06 14:21:54 +02:00
Alex Auvolat	fd0e23e984	admin api: implement InspectObject (fix #892)	2025-04-06 13:23:39 +02:00
Alex	d7506b282c	Merge pull request 'admin api: always return latest bucket info' (#1004) from fix-bucket-info-api into next-v2 ... Reviewed-on: #1004	2025-04-06 09:43:41 +00:00
Alex Auvolat	6bbdca2e48	admin api: always return latest bucket info	2025-04-06 11:14:42 +02:00
Alex	c6d6cc1fc3	Merge pull request 'read/write quorums on admin operations' (#997) from admin-quorums into next-v2 ... Reviewed-on: #997	2025-03-25 16:09:06 +00:00
Alex Auvolat	5fa6df6ee3	improve comments in bucket helper	2025-03-25 16:58:12 +01:00
Alex Auvolat	c6bed26347	relocalize logic into block manager	2025-03-25 16:41:19 +01:00
Alex Auvolat	d25e631a4a	relocalize logic for write_sets	2025-03-25 16:35:56 +01:00
Alex Auvolat	514eb29874	use a WriteLock for write operations on fullcopy tables	2025-03-25 13:26:23 +01:00
Alex Auvolat	8ba6454e21	reduce anti-entropy interval for fullcopy tables	2025-03-25 13:11:11 +01:00
Alex Auvolat	9dcc5232a6	admin api: use fast local reads for token verification	2025-03-25 13:07:45 +01:00
Alex Auvolat	1e13a66b42	rework bucket helper functions to use local access where relevant	2025-03-25 13:00:48 +01:00
Alex Auvolat	2c9e849bbf	remove dependency from garage_block to garage_table	2025-03-25 11:43:41 +01:00
Alex Auvolat	34baade499	fullcopy replication: quorum reads and writes	2025-03-25 11:40:09 +01:00
Alex Auvolat	2f2a96b51d	layout & replication mode refactoring	2025-03-25 11:05:00 +01:00
Alex Auvolat	c9156f6828	Merge branch 'main' into next-v2	2025-03-25 10:02:41 +01:00
Alex	4629ee25f7	Merge pull request 'add crc64nvme checksumming algorithm (fix #963)' (#991) from add-crc64nvme into next-v2 ... Reviewed-on: #991	2025-03-20 12:28:35 +00:00
Alex Auvolat	a826c361a9	add crc64nvme checksumming algorithm (fix #963)	2025-03-19 15:51:06 +01:00
Alex	fb6db494cc	Merge pull request 'sse-c: use different object encryption key for each object (fix #848)' (#990) from fix-ssec-crypto into next-v2 ... Reviewed-on: #990	2025-03-19 13:12:24 +00:00
Alex Auvolat	97e2fa5b8b	add upgrade test for sse-c	2025-03-18 19:09:38 +01:00
Alex Auvolat	cfd259190f	sse-c: use different object encryption key for each object	2025-03-18 16:23:37 +01:00
Alex	48e0436f29	Merge pull request 'cli: uniformize output and add some infos' (#984) from uniformize-cli into next-v2 ... Reviewed-on: #984	2025-03-12 15:51:11 +00:00
Alex Auvolat	9c745548c4	test-upgrade with v1 -> v2	2025-03-12 16:06:28 +01:00
Alex Auvolat	f7d9c2b383	cli: add garage json-api command and fix cargo tests	2025-03-12 15:55:52 +01:00
Alex Auvolat	e6862c5d3d	cli: uniformize output and add some infos	2025-03-12 15:05:37 +01:00
Alex	d032e2017c	Merge pull request 'admin api improvements' (#983) from admin-api-improvements into next-v2 ... Reviewed-on: #983	2025-03-12 11:00:24 +00:00
Alex Auvolat	0b12debf6c	admin api: generate params from struct	2025-03-12 11:07:12 +01:00
Alex Auvolat	795b4a41b7	admin api: add special endpoints to openapi spec	2025-03-12 10:52:58 +01:00
Alex	fd2472d488	Merge pull request 'support for multiple admin API token' (#982) from multi-admin-token into next-v2 ... Reviewed-on: #982	2025-03-12 09:30:19 +00:00
Alex Auvolat	d2a064bb1b	cli: add and remove scopes using --scope=+Scope or --scope=-Scope	2025-03-12 10:20:08 +01:00
Alex Auvolat	88b4623bf1	add creation date to admin api tokens	2025-03-12 09:57:32 +01:00
Alex Auvolat	325f79012c	admin_token_table: implement is_tombstone()	2025-03-12 09:29:54 +01:00
Alex Auvolat	eb40475f1e	move bucket search logic from helper to admin api	2025-03-12 09:21:53 +01:00
Alex Auvolat	22c0420607	admin api: specify date-time format in openapi spec	2025-03-11 18:25:46 +01:00
Alex Auvolat	1bd7689301	cli: add functions to manage admin api tokens	2025-03-11 18:09:24 +01:00
Alex Auvolat	ec0da3b644	admin api: mention admin_token and metrics_token in ListAdminTokensResponse	2025-03-11 15:57:29 +01:00
Alex Auvolat	9511b20153	admin api: add openapi spec for admin token management functions	2025-03-11 15:38:38 +01:00
Alex Auvolat	d067a40b3f	admin api: add functions to manage admin api tokens	2025-03-11 15:17:31 +01:00
Alex Auvolat	ff6ec62d54	admin api: add metrics_require_token config option and update doc	2025-03-11 14:17:22 +01:00
Alex Auvolat	004eb94e14	admin api: verify tokens using the new admin api token table	2025-03-11 13:51:19 +01:00
Alex Auvolat	46f620119b	add model for admin key table	2025-03-11 13:51:19 +01:00
Alex Auvolat	576d0d950e	admin api: move functions to their correct location	2025-03-11 13:09:35 +01:00
Alex	85a07c87d7	Merge pull request 'admin refactoring, step 4' (#980) from refactor-admin into next-v2 ... Reviewed-on: #980	2025-03-11 09:19:12 +00:00
Alex Auvolat	1f645830a4	layout: make optional version mandatory in apply layout changes	2025-03-11 10:05:02 +01:00
Alex Auvolat	5f308bd688	move zone redundancy parsing/formatting to cli	2025-03-11 10:00:37 +01:00
Alex Auvolat	df758e8e0d	cli v2: simplify	2025-03-11 09:54:05 +01:00
Alex Auvolat	e83864af24	layout: better encapsulation	2025-03-11 09:29:38 +01:00
Alex Auvolat	3b49dd9e63	admin api: small refactor + add comments to layout-related calls	2025-03-11 09:19:20 +01:00
Alex Auvolat	cef8d75983	admin api: avoid overwriting redirect rules in UpdateBucket	2025-03-07 16:25:55 +01:00
Alex Auvolat	cd0728cd20	cli: move files around	2025-03-06 18:54:40 +01:00
Alex Auvolat	0951b5db75	admin api: implement ClusterLayoutSkipDeadNodes and use it in CLI	2025-03-06 18:49:56 +01:00
Alex Auvolat	3d94eb8d4b	admin api: implement GetClusterLayoutHistory and use it in CLI	2025-03-06 18:33:05 +01:00
Alex Auvolat	004866caac	admin api, cliv2: implement garage layout show using api functions	2025-03-06 17:56:22 +01:00
Alex Auvolat	913e6da41b	admin api: implement PreviewClusterLayoutChanges	2025-03-06 17:27:23 +01:00
Alex Auvolat	e4881e62f1	admin api: management of layout parameters through admin api	2025-03-06 17:12:52 +01:00
Alex	7ccbfda26d	Merge pull request 'admin api: generate openapi spec using utoipa' (#979) from refactor-admin into next-v2 ... Reviewed-on: #979	2025-03-06 15:09:00 +00:00
Alex Auvolat	6b19d7628e	admin api: small fixes and reordering	2025-03-06 14:21:25 +01:00
Alex Auvolat	411f1d495c	admin api: add all missing endpoints to openapi spec	2025-03-06 14:01:39 +01:00
Alex Auvolat	ba68506c36	admin api: generate openapi spec using utoipa (wip)	2025-03-06 12:26:14 +01:00
Alex	21c83ab311	Merge pull request 'admi api: remove info about local node from GetClusterStatus and add specific GetNodeInfo endpoint' (#978) from refactor-admin into next-v2 ... Reviewed-on: #978	2025-03-06 09:50:22 +00:00
Alex Auvolat	2e03d90585	admi api: remove info about local node from GetClusterStatus and add specific GetNodeInfo endpoint	2025-03-06 10:26:01 +01:00
Alex Auvolat	29ce490dd6	Merge branch 'main' into next-v2	2025-03-05 14:50:22 +01:00
Alex	c3e8e5e38c	Merge pull request 'support redirection on s3 endpoint' (#910) from trinity-1686a/garage:1686a/s3-redirects into next-v2 ... Reviewed-on: #910	2025-02-14 12:58:48 +00:00
Alex Auvolat	62a3003cca	rename Condition into RedirectCondition in internal model	2025-02-14 13:45:32 +01:00
Alex Auvolat	3151695011	Merge branch 'next-v2' into 1686a/s3-redirects	2025-02-14 13:11:33 +01:00
trinity-1686a	f034e834fa	Merge branch 'main' into 1686a/s3-redirects	2025-02-08 16:01:11 +01:00
trinity-1686a	bf0f792418	add redirect_all to WebsiteConfig model	2025-02-08 15:36:48 +01:00
Alex Auvolat	61f3de6496	Merge branch 'main' into next-v2	2025-02-05 21:00:19 +01:00
Alex	71655c1e89	Merge pull request 'Admin API refactoring: convert existing commands to API requests (step 3)' (#945) from refactor-admin into next-v2 ... Reviewed-on: #945	2025-02-05 19:54:42 +00:00
Alex Auvolat	7c8fc04b96	massively speed up compilation of garage_api_admin by not using async_trait	2025-02-05 19:37:38 +01:00
Alex Auvolat	f914db057a	cli_v2: implement LaunchRepairOperation and remove old stuff	2025-02-05 15:36:47 +01:00
Alex Auvolat	406b6da163	cli_v2: implement Get{Node,Cluster}Statistics	2025-02-05 15:06:10 +01:00
Alex Auvolat	9f468b4439	cli_v2: implement CreateMetadataSnapshot	2025-02-05 14:22:10 +01:00
Alex Auvolat	97be7b38fa	admin api: reorder things	2025-02-03 18:54:51 +01:00
Alex Auvolat	6a1079c412	admin api: impl RequestHandler for MetricsRequest	2025-02-03 18:54:51 +01:00
Alex Auvolat	b1629dd355	cli_v2: implement RetryBlockResync and PurgeBlocks	2025-02-03 18:54:51 +01:00
Alex Auvolat	d405a9f839	cli_v2: implement ListBlockErrors and GetBlockInfo	2025-02-03 18:54:51 +01:00
Alex Auvolat	7b9c047b11	cli_v2: add local_api_request with crazy type bound	2025-02-03 18:54:51 +01:00
Alex Auvolat	10bbb26b30	cli_v2: implement ListWorkers and GetWorkerInfo	2025-02-03 18:54:51 +01:00
Alex Auvolat	89ff9f5576	admin api: base infrastructure for local endpoints ... admin api: rename EndpointHandler into RequestHandler to avoid confusion with RPC wip: infrastructure for local api calls admin api: fix things admin api: first local endpoint to work with new scheme admin api: implement SetWorkerVariable	2025-02-03 18:54:51 +01:00
Alex Auvolat	bdaf55ab3f	cli_v2: migrate cleanupincompleteuploads to Admin API ... admin api: add CleanupIncompleteUploads spec	2025-02-03 18:54:51 +01:00
Alex Auvolat	e96014ca60	Merge branch 'main' into next-v2	2025-02-03 18:53:17 +01:00
Alex Auvolat	568c4954e9	Merge branch 'main' into next-v2	2025-02-01 19:58:36 +01:00
Alex Auvolat	fe937c2901	Merge branch 'main' into next-v2	2025-02-01 19:07:17 +01:00
Alex	3192088aac	Merge pull request 'admin refactoring: refactor CLI to use Admin API requests (step 2)' (#943) from refactor-admin into next-v2 ... Reviewed-on: #943	2025-01-30 16:18:38 +00:00
Alex Auvolat	5a89350b38	cli_v2: fix garage status	2025-01-30 16:45:59 +01:00
Alex Auvolat	3caea5fc06	cli_v2: merge util.rs into layout.rs	2025-01-30 16:24:55 +01:00
Alex Auvolat	ebc0e9319e	cli_v2: error messages	2025-01-30 16:17:35 +01:00
Alex Auvolat	f8c6a8373d	convert cli key operations to admin rpc	2025-01-30 16:12:16 +01:00
Alex Auvolat	076ce04fe5	fix garage status output	2025-01-30 15:38:22 +01:00
Alex Auvolat	f37d5d2b08	admin api: convert most bucket operations	2025-01-30 13:36:25 +01:00
Alex Auvolat	819f4f0050	cli: migrate layout remove, apply, revert	2025-01-30 12:19:23 +01:00
Alex Auvolat	69ddaafc60	wip: migrate garage status and garage layout assign	2025-01-30 12:07:12 +01:00
Alex Auvolat	145130481e	wip: proxy admin api requests through admin rpc, prepare new cli	2025-01-30 10:44:08 +01:00
Alex	6ed78abb5c	Merge pull request 'admin API refactoring (step 1)' (#939) from refactor-admin into next-v2 ... Reviewed-on: #939	2025-01-29 20:42:56 +00:00
Alex Auvolat	19454c1679	admin api: remove log message	2025-01-29 19:47:37 +01:00
Alex Auvolat	1c03941b19	admin api: fix panic on GetKeyInfo with no args	2025-01-29 19:26:16 +01:00
Alex Auvolat	4f0b923c4f	admin api: small fixes	2025-01-29 19:26:16 +01:00
Alex Auvolat	420bbc162d	admin api: clearer syntax for AddBucketAlias and RemoveBucketAlias	2025-01-29 19:26:16 +01:00
Alex Auvolat	12ea4cda5f	admin api: merge calls to manage global/local aliases	2025-01-29 19:26:16 +01:00
Alex Auvolat	5fefbd94e9	admin api: rename allow/deny api calls in api v2	2025-01-29 19:26:16 +01:00
Alex Auvolat	ba810b2e81	admin api: rename bucket aliasing operations	2025-01-29 19:26:16 +01:00
Alex Auvolat	f8ed3fdbc4	fix test_website_check_domain	2025-01-29 19:26:16 +01:00
Alex Auvolat	2daeb89834	admin api: fixes to openapi v2 spec	2025-01-29 19:26:16 +01:00
Alex Auvolat	4cb45bd398	admin api: fix CORS to work in browser	2025-01-29 19:26:16 +01:00
Alex Auvolat	d5ad797ad7	admin api: update v2 openapi spec	2025-01-29 19:26:16 +01:00
Alex Auvolat	a99925e0ed	admin api: initialize v2 openapi spec from v1	2025-01-29 19:26:16 +01:00
Alex Auvolat	f538dc34d3	admin api: make all requests and responses (de)serializable	2025-01-29 19:26:16 +01:00
Alex Auvolat	ed58f8b0fe	admin api: update semantics of some endpoints, and update doc	2025-01-29 19:26:16 +01:00
Alex Auvolat	5037b97dd4	admin api: add compatibility from v1/ to v2/	2025-01-29 19:26:16 +01:00
Alex Auvolat	af1a530834	admin api: refactor using macro	2025-01-29 19:26:16 +01:00
Alex Auvolat	c99bfe69ea	admin api: new router_v2 with unified path syntax	2025-01-29 19:26:16 +01:00
Alex Auvolat	831f2b0207	admin api: make all handlers impls of a single trait	2025-01-29 19:26:16 +01:00
Alex Auvolat	c1eb1610ba	admin api: create structs for all requests/responess in src/api/admin/api.rs	2025-01-29 19:26:16 +01:00
Quentin Dufour	5560a963e0	decrease write quorum	2025-01-29 19:25:58 +01:00
Alex Auvolat	2aaba39ddc	refactor web_server.rs	2025-01-04 20:11:54 +01:00
Alex Auvolat	47467df83e	avoid handling status_code-related logic in api/s3/get.rs	2025-01-04 19:52:14 +01:00
Alex Auvolat	9b7fea4cb0	put bucket website: improve error message for redirectallrequests	2025-01-04 19:16:24 +01:00
Alex Auvolat	44ce6ae5b4	properly implement new bucket model using a migration	2025-01-04 18:50:49 +01:00
Alex Auvolat	22487ceddf	move Redirect::compute_target to standalone function in web_server.rs	2025-01-04 18:22:42 +01:00
Alex Auvolat	6ccfbb2986	remove obsolete RedirectAll struct	2025-01-04 17:04:17 +01:00
trinity-1686a	c939d2a936	clippy	2024-12-22 15:26:06 +01:00
trinity-1686a	65e9dde8c9	add tests	2024-12-22 15:20:09 +01:00
trinity-1686a	c9b733a4a6	support redirection on s3 endpoint	2024-12-14 17:46:27 +01:00