WIP: Garage v2 #948

Draft
lx wants to merge 125 commits from next-v2 into main
Owner
  • Refactor admin api with clearer URL syntax
  • Proxy admin api commands through RPC and use that in the CLI instead of redundant admin RPC protocol
  • Convert any functionnality that was only in the CLI into now-standardized admin API calls
    • admin rpc
    • standard rpc
      • Management of layout parameters in GetClusterLayout / UpdateClusterLayout
      • PreviewClusterLayoutChanges
      • GetClusterLayoutHistory
      • ForceClusterLayoutSkipDeadNodes
  • Automatically generate OpenAPI spec from code using utoipa
    • add special endpoints to openapi spec
  • More uniform use of chrono datetime in api not for time intervals, it's better to let the server compute time intervals since they are relative to server time
  • Uniformize CLI outputs
  • Multiple admin API tokens

For v2.1:

  • Make the CLI able to call the admin API through HTTP as well

TODO changelog

- [x] Refactor admin api with clearer URL syntax - [x] Proxy admin api commands through RPC and use that in the CLI instead of redundant admin RPC protocol - [x] Convert any functionnality that was only in the CLI into now-standardized admin API calls - [x] admin rpc - [x] standard rpc - [x] Management of layout parameters in GetClusterLayout / UpdateClusterLayout - [x] PreviewClusterLayoutChanges - [x] GetClusterLayoutHistory - [x] ForceClusterLayoutSkipDeadNodes - [x] Automatically generate OpenAPI spec from code using utoipa - [x] add special endpoints to openapi spec - [ ] ~More uniform use of chrono datetime in api~ not for time intervals, it's better to let the server compute time intervals since they are relative to server time - [x] Uniformize CLI outputs - [x] Multiple admin API tokens --- For v2.1: - [ ] Make the CLI able to call the admin API through HTTP as well --- TODO changelog
lx added 32 commits 2025-02-01 18:27:20 +00:00
decrease write quorum
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
5560a963e0
admin api: fix panic on GetKeyInfo with no args
Some checks are pending
ci/woodpecker/push/debug Pipeline is running
ci/woodpecker/pr/debug Pipeline was successful
1c03941b19
admin api: remove log message
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
19454c1679
Merge pull request 'admin API refactoring (step 1)' (#939) from refactor-admin into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
6ed78abb5c
Reviewed-on: #939
wip: proxy admin api requests through admin rpc, prepare new cli
Some checks failed
ci/woodpecker/push/debug Pipeline failed
145130481e
wip: migrate garage status and garage layout assign
Some checks failed
ci/woodpecker/push/debug Pipeline failed
ci/woodpecker/pr/debug Pipeline failed
69ddaafc60
cli: migrate layout remove, apply, revert
Some checks failed
ci/woodpecker/push/debug Pipeline failed
ci/woodpecker/pr/debug Pipeline failed
819f4f0050
admin api: convert most bucket operations
Some checks failed
ci/woodpecker/push/debug Pipeline failed
ci/woodpecker/pr/debug Pipeline failed
f37d5d2b08
fix garage status output
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
076ce04fe5
convert cli key operations to admin rpc
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
f8c6a8373d
cli_v2: error messages
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
ebc0e9319e
cli_v2: fix garage status
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
5a89350b38
Reviewed-on: #943
Merge branch 'main' into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
fe937c2901
lx added 2 commits 2025-02-03 17:50:03 +00:00
Merge branch 'main' into next-v2
Some checks failed
ci/woodpecker/push/debug Pipeline failed
ci/woodpecker/pr/debug Pipeline failed
77ad0600b4
lx force-pushed next-v2 from 77ad0600b4 to e96014ca60 2025-02-03 17:53:22 +00:00 Compare
lx added 13 commits 2025-02-05 19:54:44 +00:00
admin api: add CleanupIncompleteUploads spec
admin api: rename EndpointHandler into RequestHandler to avoid confusion with RPC

wip: infrastructure for local api calls

admin api: fix things

admin api: first local endpoint to work with new scheme

admin api: implement SetWorkerVariable
admin api: reorder things
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
97be7b38fa
cli_v2: implement CreateMetadataSnapshot
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
9f468b4439
cli_v2: implement Get{Node,Cluster}Statistics
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
406b6da163
cli_v2: implement LaunchRepairOperation and remove old stuff
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
f914db057a
massively speed up compilation of garage_api_admin by not using async_trait
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
7c8fc04b96
Merge pull request 'Admin API refactoring: convert existing commands to API requests (step 3)' (#945) from refactor-admin into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
71655c1e89
Reviewed-on: #945
lx added 1 commit 2025-02-05 20:00:23 +00:00
Merge branch 'main' into next-v2
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
61f3de6496
lx added 14 commits 2025-02-14 12:58:50 +00:00
support redirection on s3 endpoint
Some checks failed
ci/woodpecker/pr/debug Pipeline failed
c9b733a4a6
clippy
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
c939d2a936
remove obsolete RedirectAll struct
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
6ccfbb2986
move Redirect::compute_target to standalone function in web_server.rs
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
22487ceddf
properly implement new bucket model using a migration
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
44ce6ae5b4
put bucket website: improve error message for redirectallrequests
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
9b7fea4cb0
avoid handling status_code-related logic in api/s3/get.rs
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
47467df83e
refactor web_server.rs
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
2aaba39ddc
Merge branch 'main' into 1686a/s3-redirects
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
f034e834fa
Merge branch 'next-v2' into 1686a/s3-redirects
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
3151695011
rename Condition into RedirectCondition in internal model
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
62a3003cca
Merge pull request 'support redirection on s3 endpoint' (#910) from trinity-1686a/garage:1686a/s3-redirects into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
c3e8e5e38c
Reviewed-on: #910
lx added this to the 0.2 milestone 2025-02-14 13:03:18 +00:00
lx modified the milestone from 0.2 to v2.0 2025-02-14 13:03:21 +00:00
lx added 1 commit 2025-03-05 13:50:29 +00:00
Merge branch 'main' into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
29ce490dd6
lx added 2 commits 2025-03-06 09:50:25 +00:00
admi api: remove info about local node from GetClusterStatus and add specific GetNodeInfo endpoint
Some checks are pending
ci/woodpecker/push/debug Pipeline is running
ci/woodpecker/pr/debug Pipeline was successful
2e03d90585
Merge pull request 'admi api: remove info about local node from GetClusterStatus and add specific GetNodeInfo endpoint' (#978) from refactor-admin into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
21c83ab311
Reviewed-on: #978
lx added 4 commits 2025-03-06 15:09:03 +00:00
admin api: generate openapi spec using utoipa (wip)
Some checks failed
ci/woodpecker/pr/debug Pipeline failed
ci/woodpecker/push/debug Pipeline failed
ba68506c36
admin api: add all missing endpoints to openapi spec
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
411f1d495c
admin api: small fixes and reordering
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
6b19d7628e
Merge pull request 'admin api: generate openapi spec using utoipa' (#979) from refactor-admin into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
7ccbfda26d
Reviewed-on: #979
lx added 1 commit 2025-03-07 15:26:04 +00:00
admin api: avoid overwriting redirect rules in UpdateBucket
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
cef8d75983
lx added 12 commits 2025-03-11 09:19:13 +00:00
admin api: management of layout parameters through admin api
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
e4881e62f1
admin api: implement PreviewClusterLayoutChanges
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
913e6da41b
admin api, cliv2: implement garage layout show using api functions
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
004866caac
admin api: implement GetClusterLayoutHistory and use it in CLI
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
3d94eb8d4b
admin api: implement ClusterLayoutSkipDeadNodes and use it in CLI
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
0951b5db75
cli: move files around
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
cd0728cd20
cli v2: simplify
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
df758e8e0d
move zone redundancy parsing/formatting to cli
Some checks failed
ci/woodpecker/pr/debug Pipeline failed
ci/woodpecker/push/debug Pipeline failed
5f308bd688
layout: make optional version mandatory in apply layout changes
Some checks are pending
ci/woodpecker/push/debug Pipeline is running
ci/woodpecker/pr/debug Pipeline is pending
1f645830a4
Merge pull request 'admin refactoring, step 4' (#980) from refactor-admin into next-v2
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
85a07c87d7
Reviewed-on: #980
lx added 1 commit 2025-03-11 09:27:53 +00:00
admin api: move functions to their correct location
Some checks failed
ci/woodpecker/push/debug Pipeline failed
ci/woodpecker/pr/debug Pipeline failed
6b420ff71d
lx force-pushed next-v2 from 6b420ff71d to 576d0d950e 2025-03-11 12:09:40 +00:00 Compare
lx added 13 commits 2025-03-12 09:30:22 +00:00
admin api: verify tokens using the new admin api token table
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
004eb94e14
admin api: add metrics_require_token config option and update doc
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
ff6ec62d54
admin api: add functions to manage admin api tokens
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
d067a40b3f
admin api: add openapi spec for admin token management functions
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
9511b20153
admin api: mention admin_token and metrics_token in ListAdminTokensResponse
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
ec0da3b644
cli: add functions to manage admin api tokens
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
1bd7689301
admin api: specify date-time format in openapi spec
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
22c0420607
move bucket search logic from helper to admin api
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
eb40475f1e
admin_token_table: implement is_tombstone()
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
325f79012c
add creation date to admin api tokens
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
88b4623bf1
cli: add and remove scopes using --scope=+Scope or --scope=-Scope
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
d2a064bb1b
Merge pull request 'support for multiple admin API token' (#982) from multi-admin-token into next-v2
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
fd2472d488
Reviewed-on: #982
lx added 3 commits 2025-03-12 11:00:26 +00:00
admin api: generate params from struct
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
0b12debf6c
Merge pull request 'admin api improvements' (#983) from admin-api-improvements into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
d032e2017c
Reviewed-on: #983
lx added 4 commits 2025-03-12 15:51:13 +00:00
cli: uniformize output and add some infos
Some checks failed
ci/woodpecker/push/debug Pipeline failed
ci/woodpecker/pr/debug Pipeline failed
e6862c5d3d
cli: add garage json-api command and fix cargo tests
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
f7d9c2b383
test-upgrade with v1 -> v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/deployment/debug Pipeline was successful
ci/woodpecker/deployment/release/3 Pipeline was successful
ci/woodpecker/deployment/release/2 Pipeline was successful
ci/woodpecker/deployment/release/1 Pipeline was successful
ci/woodpecker/deployment/release/4 Pipeline was successful
ci/woodpecker/deployment/publish Pipeline was successful
9c745548c4
Merge pull request 'cli: uniformize output and add some infos' (#984) from uniformize-cli into next-v2
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
48e0436f29
Reviewed-on: #984
arnaudlevy reviewed 2025-03-14 14:08:11 +00:00
@ -2,3 +2,3 @@
<html>
<head>
<title>Garage Adminstration API v0</title>
<title>Garage adminstration API v1</title>
First-time contributor

administration

administration
arnaudlevy reviewed 2025-03-14 14:08:25 +00:00
@ -2,3 +2,3 @@
<html>
<head>
<title>Garage Adminstration API v0</title>
<title>Garage adminstration API v0</title>
First-time contributor

administration

administration
lx added 3 commits 2025-03-19 13:12:26 +00:00
sse-c: use different object encryption key for each object
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
cfd259190f
add upgrade test for sse-c
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/deployment/debug Pipeline was successful
ci/woodpecker/deployment/release/2 Pipeline was successful
ci/woodpecker/deployment/release/1 Pipeline was successful
ci/woodpecker/deployment/release/3 Pipeline was successful
ci/woodpecker/deployment/release/4 Pipeline was successful
ci/woodpecker/deployment/publish Pipeline was successful
97e2fa5b8b
Merge pull request 'sse-c: use different object encryption key for each object (fix #848)' (#990) from fix-ssec-crypto into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
fb6db494cc
Reviewed-on: #990
lx added 2 commits 2025-03-20 12:28:37 +00:00
add crc64nvme checksumming algorithm (fix #963)
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
a826c361a9
Merge pull request 'add crc64nvme checksumming algorithm (fix #963)' (#991) from add-crc64nvme into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
4629ee25f7
Reviewed-on: #991
lx added 1 commit 2025-03-25 09:02:49 +00:00
Merge branch 'main' into next-v2
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
c9156f6828
lx added 11 commits 2025-03-25 16:09:08 +00:00
layout & replication mode refactoring
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
2f2a96b51d
fullcopy replication: quorum reads and writes
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
34baade499
remove dependency from garage_block to garage_table
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
2c9e849bbf
rework bucket helper functions to use local access where relevant
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
1e13a66b42
admin api: use fast local reads for token verification
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
9dcc5232a6
reduce anti-entropy interval for fullcopy tables
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
8ba6454e21
use a WriteLock for write operations on fullcopy tables
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
514eb29874
relocalize logic for write_sets
Some checks failed
ci/woodpecker/push/debug Pipeline failed
ci/woodpecker/pr/debug Pipeline failed
d25e631a4a
relocalize logic into block manager
Some checks failed
ci/woodpecker/push/debug Pipeline failed
ci/woodpecker/pr/debug Pipeline failed
c6bed26347
improve comments in bucket helper
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
5fa6df6ee3
Merge pull request 'read/write quorums on admin operations' (#997) from admin-quorums into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
c6d6cc1fc3
Reviewed-on: #997
lx added 2 commits 2025-04-06 09:43:42 +00:00
admin api: always return latest bucket info
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
6bbdca2e48
Merge pull request 'admin api: always return latest bucket info' (#1004) from fix-bucket-info-api into next-v2
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
d7506b282c
Reviewed-on: #1004
lx added 3 commits 2025-04-06 12:33:14 +00:00
admin api: implement InspectObject (fix #892)
All checks were successful
ci/woodpecker/pr/debug Pipeline was successful
ci/woodpecker/push/debug Pipeline was successful
fd0e23e984
admin api: add comments for InspectObject
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
5e7307cbf3
Merge pull request 'admin api: implement InspectObject (fix #892)' (#1005) from inspect-object into next-v2
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
2d1c073d2f
Reviewed-on: #1005
All checks were successful
ci/woodpecker/push/debug Pipeline was successful
ci/woodpecker/pr/debug Pipeline was successful
Required
Details
This pull request is marked as a work in progress.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin next-v2:next-v2
git checkout next-v2
Sign in to join this conversation.
No reviewers
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#948
No description provided.