Garage v0.9 #473

Merged

lx merged 175 commits from next into main 2023-10-10 13:28:29 +00:00

Conversation 0 Commits 175 Files changed 120 +27828 -2367

lx commented 2023-01-11 16:50:08 +00:00

Owner

Copy link

2023-10-03: Garage v0.9 is entering feature freeze, it will no longer be merged with the main branch. We will only be doing tests and debuging, for a release ASAP (this has lasted too long!)

Preliminary changelog

BREAKING CHANGES:

• LMDB is now the default metadata db engine and Sled is deprecated. Make sure you are using the correct value of db_engine in your config file before updating. Sled will be removed in Garage v1.0.

• Capacity values are now in actual byte units. The translation from the old layout will assign 1 capacity = 1Gb by default, which might be wrong for your cluster. This does not cause any data to be moved around, but users might want to re-assign correct capacities to all their nodes to be future-proof.

• Multipart uploads that were started in Garage v0.8 will not be visible in Garage v0.9 and will have to be restarted from scratch.

• Updated admin API endpoints, prefixed by v1/. For endpoints with incompatible syntax, the v0/ endpoints have been disabled; for others the v0/ URLs are translated automatically to their v1/ counterpart. Endpoints under v0/ will be removed in Garage v1.0.

  • affected admin endpoints: GetClusterStatus, GetClusterHealth, GetClusterLayout, UpdateClusterLayout, ApplyClusterLayout, GetKeyInfo, UpdateKey, CreateKey, ImportKey

• Changes to the garage key CLI:

  • garage key new is renamed to garage key create to be the same as garage bucket create

  • garage key import will not work without the --yes flag and will reject invalid keys (keys that are not generated by Garage)

  • garage key info will no longer display the secret key by default (fix #506)

CAUTION:

• rmp_serde was updated from 0.15 to 1.1.2, this seems to not break any previously existing data, but more testing is required

CHANELOG:

• Full multipart upload semantics (#204, #553)

• Support for multiple HDDs per server (#218, #625)

• Optimal layout assignation algorithm (#296)

• Support for basic lifecycle configurations (#309, #620)

• Uniformize serialization conventions in admin API (#377, #381)

• Add checks to garage key import (#278), rename garage key new into garage key create (#459)

• s/assignation/assignment (#465)

• many smaller changes not tracked by issues or PRs

---

NLnet tracking (task 2)

• Address issue #309 - Support for some basic form of lifecycle configuration to allow automatically cleaning up unfinished multipart upload

  • → See PR #620

• Address issue #218 - Handle multiple hard drives per server

  • → See PR #625

• Testing and validation of the new layout calculation algorithm and its migration path, addressing issues found

  • → See PR #634

2023-10-03: Garage v0.9 is entering **feature freeze**, it will no longer be merged with the main branch. We will only be doing tests and debuging, for a release ASAP (this has lasted too long!) ## Preliminary changelog BREAKING CHANGES: - LMDB is now the default metadata db engine and Sled is deprecated. Make sure you are using the correct value of `db_engine` in your config file before updating. **Sled will be removed in Garage v1.0**. - Capacity values are now in actual byte units. The translation from the old layout will assign 1 capacity = 1Gb by default, which might be wrong for your cluster. This does not cause any data to be moved around, but users might want to re-assign correct capacities to all their nodes to be future-proof. - Multipart uploads that were started in Garage v0.8 will not be visible in Garage v0.9 and will have to be restarted from scratch. - Updated admin API endpoints, prefixed by `v1/`. For endpoints with incompatible syntax, the `v0/` endpoints have been disabled; for others the `v0/` URLs are translated automatically to their `v1/` counterpart. **Endpoints under `v0/` will be removed in Garage v1.0.** - affected admin endpoints: GetClusterStatus, GetClusterHealth, GetClusterLayout, UpdateClusterLayout, ApplyClusterLayout, GetKeyInfo, UpdateKey, CreateKey, ImportKey - Changes to the `garage key` CLI: - `garage key new` is renamed to `garage key create` to be the same as `garage bucket create` - `garage key import` will not work without the `--yes` flag and will reject invalid keys (keys that are not generated by Garage) - `garage key info` will no longer display the secret key by default (fix #506) CAUTION: - `rmp_serde` was updated from 0.15 to 1.1.2, this seems to not break any previously existing data, but more testing is required CHANELOG: - Full multipart upload semantics (#204, #553) - Support for multiple HDDs per server (#218, #625) - Optimal layout assignation algorithm (#296) - Support for basic lifecycle configurations (#309, #620) - Uniformize serialization conventions in admin API (#377, #381) - Add checks to `garage key import` (#278), rename `garage key new` into `garage key create` (#459) - s/assignation/assignment (#465) - many smaller changes not tracked by issues or PRs ----- ## NLnet tracking (task 2) - [x] Address issue #309 - Support for some basic form of lifecycle configuration to allow automatically cleaning up unfinished multipart upload - → See PR #620 - [x] Address issue #218 - Handle multiple hard drives per server - → See PR #625 - [x] Testing and validation of the new layout calculation algorithm and its migration path, addressing issues found - → See PR #634

lx added this to the v0.9 milestone 2023-01-11 16:50:08 +00:00

lx added 40 commits 2023-01-11 16:50:09 +00:00

Change the way new layout assignations are computed. ... c1d1646c4d

The function now computes an optimal assignation (with respect to partition size) that minimizes the distance to the former assignation, using flow algorithms.

This commit was written by Mendes Oulamara <mendes.oulamara@pm.me>

Apply cargo fmt 2aeaddd5e2

updated cargo.lock 3ba2c5b424

Corrected the warnings and errors issued by cargo clippy 948ff93cf1

Correct small formatting issue 617f28bfa4

Added the latex report on the optimal layout algorithm 03e3a1bd15

Merge branch 'optimal-layout' of https://git.deuxfleurs.fr/Deuxfleurs/garage into optimal-layout 7b2c065c82

Added a first draft version of the algorithm and analysis for the non-strict mode. 81083dd415

ignore log files in commit d38fb6c250

Added the section with description proofs of the parametric assignment computation in the optimal layout report c4adbeed51

New version of the algorithm that calculate the layout. ... 7f3249a237

It takes as paramters the replication factor and the zone redundancy, computes the
largest partition size reachable with these constraints, and among the possible
assignation with this partition size, it computes the one that moves the least number
of partitions compared to the previous assignation.
This computation uses graph algorithms defined in graph_algo.rs

Correction of a few bugs in the tests, modification of ClusterLayout::check bd842e1388

deleted zone_redundancy from System struct 99f96b9564

Merge remote-tracking branch 'origin/main' into optimal-layout 829f815a89

modifications in several files to : ... ceac3713d6

- have consistent error return types
- store the zone redundancy in a Lww
- print the error and message in the CLI (TODO: for the server Api, should msg be returned in the body response?)

Added a CLI command to update the parameters for the layout computation (for now, only the zone redundancy) a951b6c452

Corrected two bugs: ... 9407df60cc

- self.node_id_vec was not properly updated when the previous ring was empty
- ClusterLayout::merge was not considering changes in the layout parameters

corrected warnings of cargo clippy 911eb17bd9

Tests written in layout.rs ... fcf9ac674a

added staged_parameters to ClusterLayout
removed the serde(default) -> will need a migration function

cargo fmt 4abab246f1

Improved the statistics displayed in layout show ... e5664c9822

corrected a few bugs

Added some comment bcdd1e0c33

rm .gitattributes 3039bb5d43

Merge branch 'main' into optimal-layout 28d7a49f63

Style improvements ea5afc2511

Ensure .sort() is called before counting unique items fd5bc142b5

Use bytes as capacity units 73a4ca8b15

Return more info when layout's .check() fails, fix compilation, fix test d75b37b018

Fix integration test fc2729cd81

Fix HTTP return code 217abdca18

Slightly simplify code at places ec12d6c8dd

itertools .unique() doesn't require sorted items 9d83364ad9

Merge commit 'ec12d6c' into next 2c2e65ad8b

Merge pull request 'Optimal layout assignation algorithm' (#296) from optimal-layout into next ... 6e44369cbc

Reviewed-on: #296

Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time. 8be862aa19

Merge branch 'main' into next 570e5e5bbb

Merge pull request 'Changed all instances of 'key new' to 'key create' to make it the same as the bucket commands.' (#459) from jpds/garage:key-create-standardize into next ... 80e4abb98d

Reviewed-on: #459

Changed all instances of assignation to assignment. cb07e6145c

Merge pull request 'Changed all instances of assignation to assignment' (#465) from jpds/garage:assignments-correction into next ... 4f409f73dc

Reviewed-on: #465

Migration of cluster layout from v0.8 to v0.9 84b4a868e3

lx added 3 commits 2023-01-11 16:50:48 +00:00

First draft of t a preprint describing the layout computation algorithm e3cc7a89b0

change in gitignore 597d64b31a

Merge branch 'optimal-layout' into next 12a4e1f303

lx added 1 commit 2023-01-26 14:46:40 +00:00

util/Cargo.toml: Updated rmp-serde from 0.15 to 1.1. db56d4658f

lx added 1 commit 2023-01-26 14:51:09 +00:00

Update Cargo.nix 654999e254

lx added 1 commit 2023-04-25 10:34:33 +00:00

Merge branch 'main' into next fa78d806e3

lx added 1 commit 2023-04-27 20:58:31 +00:00

Merge branch 'main' into next a1fcf1b175

lx added 1 commit 2023-05-09 10:40:14 +00:00

Merge branch 'main' into next 351d734e6c

lx added 1 commit 2023-05-17 12:31:14 +00:00

Mark sled as deprecated, make lmdb default, and improve sqlite and lmdb defaults 19639705e6

lx added 1 commit 2023-06-06 13:39:28 +00:00

Split garage/admin.rs into smaller files 8a74e1c2bd

lx added 1 commit 2023-06-09 10:04:27 +00:00

Integration tests: print logs when fails 906fe78b24

lx added 1 commit 2023-06-09 10:04:51 +00:00

Add LMDB to debug builds 865f0c7d0c

lx force-pushed next from b3ff9498fe to 1e466b11eb 2023-06-09 11:23:36 +00:00 Compare

lx added 1 commit 2023-06-09 14:23:32 +00:00

Make fsync an option for meta and data e7e164a280

lx added 21 commits 2023-06-09 15:34:11 +00:00

Minio tests for multipart upload behaviour: ... ea3bfd2ab1

- upload part renumbering test
- part skipping test

Use Cow<[u8]> for sort keys 6005491cd8

New multipart upload table layout 38d6ac4295

Adapt S3 API code to use new multipart upload models ... 82e75c0e29

- Create and PutPart
- completemultipartupload
- upload part copy
- list_parts

updaet block admin for new multipartupload models 87be8eeb93

move git-version dependency to main crate to reduce rebuilds c1e1764f17

cargo fmt bb176ebcb8

fix online repair 75a0e01372

fix s3 list test 7ad7dae5d4

fix test; simplify code 8644376ac2

refactor repair workers with a trait 058518c22b

Add multipart upload repair 4ea53dc759

fix mpu counter (add missing workers) and report info at appropriate places 511e07ecd4

comments and clippy lint fixes 412ab77b08

undo sort_key() returning Cow 53bf2f070c

Add test for multipart uploads and fix part renumbering c14d3735e5

UploadPart: automatic cleanup of version (and reference blocked) when interrupted a6cc563bdd

Add multipart upload using aws s3api 58563ed700

smoke test: add multipart upload test with part re-upload e645bbd3ce

properly delete multipart uploads after completion 3d477906d4

Merge pull request 'Fix #204 (full Multipart Uploads semantics)' (#553) from nlnet-task1 into next ... 0a06fda0da

Reviewed-on: #553

lx added 1 commit 2023-06-13 08:48:31 +00:00

multipart uploads: save timestamp 942c1f1bfe

lx added 1 commit 2023-06-13 13:59:18 +00:00

garage key import: add checks and --yes CLI flag (fix #278) 7126f3e1d1

lx added 1 commit 2023-06-13 14:16:17 +00:00

admin API: add missing camelCase conversions (fix #381) bf19a44fd9

lx force-pushed next from fa7fa2a544 to ce62c3a5f4 2023-06-13 15:02:50 +00:00 Compare

lx force-pushed next from ce62c3a5f4 to 90b2d43eb4 2023-06-13 15:14:18 +00:00 Compare

lx added 1 commit 2023-06-13 15:35:01 +00:00

update netapp's rmp-serde dependency to v1.1 cda957b4b1

lx added 1 commit 2023-06-13 21:11:00 +00:00

multipartupload in test: add forgotten timestamp 5670367126

lx added 1 commit 2023-06-14 11:02:52 +00:00

Merge branch 'main' into next 187240e539

lx added 1 commit 2023-06-14 11:45:59 +00:00

admin api: change cluster status/layout to use lists and not maps (fix #377) 52376d47ca

lx added 1 commit 2023-06-14 11:53:41 +00:00

admin api: switch GetClusterHealth to camelcase (fix #381 again) 35c108b85d

lx added 2 commits 2023-06-14 11:56:57 +00:00

admin api: fix doc in drafts 2c83006608

admin api: make name optionnal for CreateKey 28cc9f178a

lx added 1 commit 2023-06-14 12:16:07 +00:00

admin api: move all endpoints to v1/ by default (v0/ still supported) 4a82f6380e

lx added 1 commit 2023-06-14 14:56:33 +00:00

admin and cli: hide secret keys unless asked 7895f99d3a

lx added 1 commit 2023-06-14 15:18:36 +00:00

admin: uniformize layout api and improve code a83a092c03

lx force-pushed next from b189653322 to 8ef42c9609 2023-06-14 15:19:32 +00:00 Compare

lx added 1 commit 2023-08-29 09:32:48 +00:00

Merge branch 'main' into next 2e90e1c124

lx added 26 commits 2023-09-04 09:45:12 +00:00

bucket_table: data model for lifecycle configuration 0b83e0558e

s3: add xml structures to serialize/deserialize lifecycle configs 8041d9a827

lifecycle: implement validation into garage's internal data structure abf011c290

use a NaiveDate in data model, it serializes to string (iso 8601 format) f7b409f114

lifecycle: skeleton for lifecycle worker a2e0e34db5

lifecycle worker: implement main functionality 2996dc875f

lifecycle worker: skip entire bucket when no lifecycle config is set da8b224e24

lifecycle worker: launch with the rest of Garage 0f1849e1ac

lifecycle worker: add logging 7200954318

lifecycle config: store date as given, try to debug 75ccc5a95c

lifecycle config: add missing line in merge() and remove tracing d2e94e36d6

bucket_table: bucketparams::new doesn't need to be pub a1d57283c0

reference manual: document support for lifecycle configuration 5c923d48d7

reference manual: remove obsolete caveat about multipart uploads d94f1c9178

s3 bucket apis: remove redundant call f0a395e2e5

lifecycle worker: avoid building chrono's serde feature 01c327a07a

update cargo.nix 5fad4c4658

lifecycle worker: take into account disabled rules b2f679675e

s3api: remove redundant serde rename attribute be03a4610f

lifecycle worker: mitigate potential bugs + refactoring 1cfcc61de8

lifecycle worker: use queue_insert and process objects in batches adbf5925de

lifecycle worker: add log message when starting a00a52633f

lifecycle worker: fix potential inifinite loop f579d6d9b4

lifecycle worker: don't get stuck on non-existent bucket 1cdc321e28

lifecycle worker: correct small clippy lints 8e0c020bb9

Merge pull request 'object lifecycles (fix #309)' (#620) from bucket-lifecycle into next ... 3f461d8891

Reviewed-on: #620

lx added 1 commit 2023-09-06 14:35:02 +00:00

lifecycle: fix SkipBucket bug 4b4f2000f4

lx added 27 commits 2023-09-11 10:52:03 +00:00

block manager: skeleton for multi-hdd support 71c0188055

block manager: multi-directory layout computation 6c420c0880

block manager: use data paths from layout 887b3233f4

block manager: move blocks in write_block if necessary a09f86729c

update cargo.nix 3199cab4c8

block manager: get rid of check_block_status 1b8c265c14

table queue: increase batch size fd00a47ddc

block manager: refactoring 93114a9747

block manager: fix dir_not_empty 3a74844df0

block manager: refactoring & increase max worker count to 8 a44f486931

block manager: fixes in layout 55c514999e

block manager: scrub checkpointing e30865984a

block manager: avoid incorrect data_dir configs and avoid losing files f38a31b330

block manager: refactor and fix monitoring/statistics 99ed18350f

doc: update page on upgradin clusters bca347a1e8

Document multi-hdd support 6595efd82f

block manager: add rebalance operation to rebalance multi-hdd setups 6b008b5bd3

doc: documentation of rebalance repair 6a067e30ee

correct free data space accounting for multiple data dirs on same fs 2f112ac682

doc: update multi-hdd section eb972a8422

block manager: fix bugs 2657b5c1b9

block manager: fix bug where rebalance didn't delete old copies be91ef6294

block manager: fix indentation (why not detected by cargo fmt?) de5d792181

block manager: remove data_dir field 7f9ba49c71

scrub: clear saved checkpoint when canceling scrub 9526328d38

block repair: simpler/more robust iterator progress calculation ba7ac52c19

Merge pull request 'multi-hdd support (fix #218)' (#625) from multihdd into next ... 7228fbfd4f

Reviewed-on: #625

lx added 1 commit 2023-09-11 11:14:24 +00:00

Merge branch 'main' into next ad6b1cc0be

lx added 1 commit 2023-09-11 18:00:08 +00:00

Merge branch 'main' into next 51abbb02d8

lx added 1 commit 2023-09-11 21:09:26 +00:00

Merge branch 'main' into next fd7d8fec59

lx added 3 commits 2023-09-21 14:03:37 +00:00

garage_db: refactor transactions and add on_commit mechanism f97168f805

garage_table/queue_insert: delay worker notification to after transaction commit (fix #583) 0635250b2b

Merge pull request 'Refactor db transactions and add on_commit for table.queue_insert' (#637) from k2v-indices-lmdb into next ... 1d986bd889

Reviewed-on: #637

lx added 11 commits 2023-09-27 09:04:33 +00:00

new layout: improve output display 2e229d4430

new layout: make zone_redundancy optionnal (if not set, is maximum) 015ccb39aa

new layout: improve display and fix comments 749b4865d0

new layout: fix clippy lints 0088599f52

update cargo.nix 013b026d56

new layout: use deterministic randomness for reproducible results 3ecc17f8c5

new layout doc: add examples of unexpected layout, to explain b4a0e636d8

layout doc: update old text 405aa42b7d

layout doc: write explanations for bizarre scenarios 8d07888fa2

layout doc: reformulate 0e5925fff6

Merge pull request 'New layout: fixes and UX improvements' (#634) from new-layout-ux into next ... aa7eadc799

Reviewed-on: #634

lx added 2 commits 2023-09-27 11:13:33 +00:00

Merge branch 'main' into next ad82035b98

actually update rmp-serde to 1.1.2 for both garage and netapp dependency (fix #629) 897cbf2c27

lx force-pushed next from 638741c951 to cc80eac633 2023-09-27 11:57:49 +00:00 Compare

lx force-pushed next from cc80eac633 to 21f6327bad 2023-09-27 12:19:31 +00:00 Compare

lx force-pushed next from 21f6327bad to 4e11ba4ff5 2023-09-27 12:35:42 +00:00 Compare

lx force-pushed next from 4e11ba4ff5 to 57c6e9e9c5 2023-09-27 12:39:20 +00:00 Compare

lx force-pushed next from 57c6e9e9c5 to 9ac1d5be0e 2023-09-27 12:57:43 +00:00 Compare

lx added 1 commit 2023-10-03 16:40:43 +00:00

Merge branch 'main' into next 2e656b541b

lx added 1 commit 2023-10-04 08:44:55 +00:00

cli: more precise doc comment 920dec393a

lx added 1 commit 2023-10-05 08:42:22 +00:00

move upgrade test to release build 9ccc1d6f4a

lx added 1 commit 2023-10-05 13:20:58 +00:00

Add migration to v0.9 guide 6790e24f5a

lx added 1 commit 2023-10-05 13:30:05 +00:00

upgrade doc: fixes and precisions 2448eb7713

lx added 1 commit 2023-10-05 14:27:37 +00:00

admin api: remove broken GET /v0/key router rule 1c13135f25

lx added 1 commit 2023-10-05 14:56:30 +00:00

admin api: increased compatibility for v0/ endpoints 0c431b0c03

lx added 1 commit 2023-10-10 11:57:58 +00:00

Merge branch 'main' into next d4932c31ea

lx added 2 commits 2023-10-10 12:08:21 +00:00

convert_db: fix build 3d7892477d

bump version to v0.9.0 952c9570c4

lx changed title from WIP: Garage v0.9 to Garage v0.9 2023-10-10 13:16:55 +00:00

lx referenced this pull request from a commit 2023-10-10 13:28:29 +00:00

Merge pull request 'Garage v0.9' (#473) from next into main

lx merged commit e6df7089a1 into main 2023-10-10 13:28:29 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

action

check-aws

The behaviour should be checked against AWS S3

  

action

discussion-needed

We need to discuss if the feature is desirable and/or better specify the envisioned behavior or technical implementation

  

action

for-external-contributors

Core team does not plan working on this in the near future however an external contribution will be reviewed & merged

  

action

for-newcomers

An issue that should be newcomer friendly.

  

action

more-info-needed

More information is needed to investigate this issue

  

action

need-funding

This issue represents lots of work, we need to find money to pay our core team work

  

action

triage-required

It's not very clear what we want to do with this issue yet, more thoughts is required

  

kind

correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

kind

ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

kind

improvement

Feature requests that do not fall in other categories

  

kind

performance

Deceptive performance observed OR idea to improve performances

  

kind

testing

Functional tests, integration tests, Jepsen, etc.

  

kind

usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

  

kind

wrong-behavior

Unintended behavior

  

prio

critical

Could impact integrity or availability of production clusters

  

prio

low

For issues that we don't want to spend time on until made absolutely necessary

  

scope

admin-api

Issues concerning the administration API

  

scope

background-healing

Background healing covers repair tasks, anti-entropy, merkle trees, etc. ; everything related to replicating data appropriately.

  

scope

build

Issues with Nix, Drone, Rust Compiler, etc.

  

scope

documentation

  

scope

k8s

Kubernetes related, including Helm Charts

  

scope

layout

Linked to the cluster layout and storage management

  

scope

metadata

Metadata Engine (LMDB, sqlite, etc.)

  

scope

ops

Operations related to operations/maintenance issue, tooling related (like CLI missing info ; impossible, too dangerous, too complicated migration, etc.)

  

scope

rpc

Linked to RPC

  

scope

s3-api

Related to the S3 API (especially compatibility issues).

  

scope

security

Cryptography-related, auth-related, authorization-related issues

  

scope

telemetry

OpenTelemetry (traces, metrics, etc.)

No labels

action

check-aws

action

discussion-needed

action

for-external-contributors

action

for-newcomers

action

more-info-needed

action

need-funding

action

triage-required

kind

correctness

kind

ideas

kind

improvement

kind

performance

kind

testing

kind

usability

kind

wrong-behavior

prio

critical

prio

low

scope

admin-api

scope

background-healing

scope

build

scope

documentation

scope

k8s

scope

layout

scope

metadata

scope

ops

scope

rpc

scope

s3-api

scope

security

scope

telemetry

Milestone

Clear milestone

No items

No milestone

v0.9

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#473

No description provided.