Garage v0.9 #473

Merged

lx merged 175 commits from next into main 2023-10-10 13:28:29 +00:00

Conversation 0 Commits 175 Files Changed 120 +27828 -2367

lx commented 2023-01-11 16:50:08 +00:00

Owner

Copy Link

2023-10-03: Garage v0.9 is entering feature freeze, it will no longer be merged with the main branch. We will only be doing tests and debuging, for a release ASAP (this has lasted too long!)

Preliminary changelog

BREAKING CHANGES:

• LMDB is now the default metadata db engine and Sled is deprecated. Make sure you are using the correct value of db_engine in your config file before updating. Sled will be removed in Garage v1.0.

• Capacity values are now in actual byte units. The translation from the old layout will assign 1 capacity = 1Gb by default, which might be wrong for your cluster. This does not cause any data to be moved around, but users might want to re-assign correct capacities to all their nodes to be future-proof.

• Multipart uploads that were started in Garage v0.8 will not be visible in Garage v0.9 and will have to be restarted from scratch.

• Updated admin API endpoints, prefixed by v1/. For endpoints with incompatible syntax, the v0/ endpoints have been disabled; for others the v0/ URLs are translated automatically to their v1/ counterpart. Endpoints under v0/ will be removed in Garage v1.0.

  • affected admin endpoints: GetClusterStatus, GetClusterHealth, GetClusterLayout, UpdateClusterLayout, ApplyClusterLayout, GetKeyInfo, UpdateKey, CreateKey, ImportKey

• Changes to the garage key CLI:

  • garage key new is renamed to garage key create to be the same as garage bucket create

  • garage key import will not work without the --yes flag and will reject invalid keys (keys that are not generated by Garage)

  • garage key info will no longer display the secret key by default (fix #506)

CAUTION:

• rmp_serde was updated from 0.15 to 1.1.2, this seems to not break any previously existing data, but more testing is required

CHANELOG:

• Full multipart upload semantics (#204, #553)

• Support for multiple HDDs per server (#218, #625)

• Optimal layout assignation algorithm (#296)

• Support for basic lifecycle configurations (#309, #620)

• Uniformize serialization conventions in admin API (#377, #381)

• Add checks to garage key import (#278), rename garage key new into garage key create (#459)

• s/assignation/assignment (#465)

• many smaller changes not tracked by issues or PRs

---

NLnet tracking (task 2)

• Address issue #309 - Support for some basic form of lifecycle configuration to allow automatically cleaning up unfinished multipart upload

  • → See PR #620

• Address issue #218 - Handle multiple hard drives per server

  • → See PR #625

• Testing and validation of the new layout calculation algorithm and its migration path, addressing issues found

  • → See PR #634

2023-10-03: Garage v0.9 is entering **feature freeze**, it will no longer be merged with the main branch. We will only be doing tests and debuging, for a release ASAP (this has lasted too long!) ## Preliminary changelog BREAKING CHANGES: - LMDB is now the default metadata db engine and Sled is deprecated. Make sure you are using the correct value of `db_engine` in your config file before updating. **Sled will be removed in Garage v1.0**. - Capacity values are now in actual byte units. The translation from the old layout will assign 1 capacity = 1Gb by default, which might be wrong for your cluster. This does not cause any data to be moved around, but users might want to re-assign correct capacities to all their nodes to be future-proof. - Multipart uploads that were started in Garage v0.8 will not be visible in Garage v0.9 and will have to be restarted from scratch. - Updated admin API endpoints, prefixed by `v1/`. For endpoints with incompatible syntax, the `v0/` endpoints have been disabled; for others the `v0/` URLs are translated automatically to their `v1/` counterpart. **Endpoints under `v0/` will be removed in Garage v1.0.** - affected admin endpoints: GetClusterStatus, GetClusterHealth, GetClusterLayout, UpdateClusterLayout, ApplyClusterLayout, GetKeyInfo, UpdateKey, CreateKey, ImportKey - Changes to the `garage key` CLI: - `garage key new` is renamed to `garage key create` to be the same as `garage bucket create` - `garage key import` will not work without the `--yes` flag and will reject invalid keys (keys that are not generated by Garage) - `garage key info` will no longer display the secret key by default (fix #506) CAUTION: - `rmp_serde` was updated from 0.15 to 1.1.2, this seems to not break any previously existing data, but more testing is required CHANELOG: - Full multipart upload semantics (#204, #553) - Support for multiple HDDs per server (#218, #625) - Optimal layout assignation algorithm (#296) - Support for basic lifecycle configurations (#309, #620) - Uniformize serialization conventions in admin API (#377, #381) - Add checks to `garage key import` (#278), rename `garage key new` into `garage key create` (#459) - s/assignation/assignment (#465) - many smaller changes not tracked by issues or PRs ----- ## NLnet tracking (task 2) - [x] Address issue #309 - Support for some basic form of lifecycle configuration to allow automatically cleaning up unfinished multipart upload - → See PR #620 - [x] Address issue #218 - Handle multiple hard drives per server - → See PR #625 - [x] Testing and validation of the new layout calculation algorithm and its migration path, addressing issues found - → See PR #634

lx added this to the v0.9 milestone 2023-01-11 16:50:08 +00:00

lx added 40 commits 2023-01-11 16:50:09 +00:00

c1d1646c4d

Change the way new layout assignations are computed. ...

The function now computes an optimal assignation (with respect to partition size) that minimizes the distance to the former assignation, using flow algorithms.

This commit was written by Mendes Oulamara <mendes.oulamara@pm.me>

2aeaddd5e2

Apply cargo fmt

3ba2c5b424

updated cargo.lock

948ff93cf1 Corrected the warnings and errors issued by cargo clippy

617f28bfa4

Correct small formatting issue

03e3a1bd15 Added the latex report on the optimal layout algorithm

7b2c065c82 Merge branch 'optimal-layout' of https://git.deuxfleurs.fr/Deuxfleurs/garage into optimal-layout

81083dd415 Added a first draft version of the algorithm and analysis for the non-strict mode.

d38fb6c250 ignore log files in commit

c4adbeed51 Added the section with description proofs of the parametric assignment computation in the optimal layout report

7f3249a237 New version of the algorithm that calculate the layout. ...

It takes as paramters the replication factor and the zone redundancy, computes the
largest partition size reachable with these constraints, and among the possible
assignation with this partition size, it computes the one that moves the least number
of partitions compared to the previous assignation.
This computation uses graph algorithms defined in graph_algo.rs

bd842e1388 Correction of a few bugs in the tests, modification of ClusterLayout::check

99f96b9564 deleted zone_redundancy from System struct

829f815a89 Merge remote-tracking branch 'origin/main' into optimal-layout

ceac3713d6 modifications in several files to : ...

- have consistent error return types
- store the zone redundancy in a Lww
- print the error and message in the CLI (TODO: for the server Api, should msg be returned in the body response?)

a951b6c452 Added a CLI command to update the parameters for the layout computation (for now, only the zone redundancy)

9407df60cc Corrected two bugs: ...

- self.node_id_vec was not properly updated when the previous ring was empty
- ClusterLayout::merge was not considering changes in the layout parameters

911eb17bd9 corrected warnings of cargo clippy

fcf9ac674a Tests written in layout.rs ...

added staged_parameters to ClusterLayout
removed the serde(default) -> will need a migration function

4abab246f1 cargo fmt

e5664c9822 Improved the statistics displayed in layout show ...

corrected a few bugs

bcdd1e0c33 Added some comment

3039bb5d43

rm .gitattributes

28d7a49f63

Merge branch 'main' into optimal-layout

ea5afc2511

Style improvements

fd5bc142b5

Ensure .sort() is called before counting unique items

73a4ca8b15

Use bytes as capacity units

d75b37b018

Return more info when layout's .check() fails, fix compilation, fix test

fc2729cd81

Fix integration test

217abdca18

Fix HTTP return code

ec12d6c8dd

Slightly simplify code at places

9d83364ad9

itertools .unique() doesn't require sorted items

2c2e65ad8b

Merge commit 'ec12d6c' into next

6e44369cbc Merge pull request 'Optimal layout assignation algorithm' (#296) from optimal-layout into next ...

Reviewed-on: #296

8be862aa19 Changed all instances of 'key new' to 'key create' to make it consistent as bucket commands issued normally around the same time.

570e5e5bbb

Merge branch 'main' into next

80e4abb98d Merge pull request 'Changed all instances of 'key new' to 'key create' to make it the same as the bucket commands.' (#459) from jpds/garage:key-create-standardize into next ...

Reviewed-on: #459

cb07e6145c Changed all instances of assignation to assignment.

4f409f73dc Merge pull request 'Changed all instances of assignation to assignment' (#465) from jpds/garage:assignments-correction into next ...

Reviewed-on: #465

84b4a868e3

Migration of cluster layout from v0.8 to v0.9

lx added 3 commits 2023-01-11 16:50:48 +00:00

e3cc7a89b0 First draft of t a preprint describing the layout computation algorithm

597d64b31a change in gitignore

12a4e1f303

Merge branch 'optimal-layout' into next

lx added 1 commit 2023-01-26 14:46:40 +00:00

db56d4658f util/Cargo.toml: Updated rmp-serde from 0.15 to 1.1.

lx added 1 commit 2023-01-26 14:51:09 +00:00

654999e254 Update Cargo.nix

lx added 1 commit 2023-04-25 10:34:33 +00:00

fa78d806e3 Merge branch 'main' into next

lx added 1 commit 2023-04-27 20:58:31 +00:00

a1fcf1b175 Merge branch 'main' into next

lx added 1 commit 2023-05-09 10:40:14 +00:00

351d734e6c Merge branch 'main' into next

lx added 1 commit 2023-05-17 12:31:14 +00:00

19639705e6 Mark sled as deprecated, make lmdb default, and improve sqlite and lmdb defaults

lx added 1 commit 2023-06-06 13:39:28 +00:00

8a74e1c2bd Split garage/admin.rs into smaller files

lx added 1 commit 2023-06-09 10:04:27 +00:00

906fe78b24 Integration tests: print logs when fails

lx added 1 commit 2023-06-09 10:04:51 +00:00

865f0c7d0c Add LMDB to debug builds

lx force-pushed next from b3ff9498fe to 1e466b11eb 2023-06-09 11:23:36 +00:00 Compare

lx added 1 commit 2023-06-09 14:23:32 +00:00

e7e164a280 Make fsync an option for meta and data

lx added 21 commits 2023-06-09 15:34:11 +00:00

ea3bfd2ab1 Minio tests for multipart upload behaviour: ...

- upload part renumbering test
- part skipping test

6005491cd8 Use Cow<[u8]> for sort keys

38d6ac4295 New multipart upload table layout

82e75c0e29 Adapt S3 API code to use new multipart upload models ...

- Create and PutPart
- completemultipartupload
- upload part copy
- list_parts

87be8eeb93 updaet block admin for new multipartupload models

c1e1764f17 move git-version dependency to main crate to reduce rebuilds

bb176ebcb8 cargo fmt

75a0e01372 fix online repair

7ad7dae5d4 fix s3 list test

8644376ac2 fix test; simplify code

058518c22b refactor repair workers with a trait

4ea53dc759 Add multipart upload repair

511e07ecd4 fix mpu counter (add missing workers) and report info at appropriate places

412ab77b08 comments and clippy lint fixes

53bf2f070c undo sort_key() returning Cow

c14d3735e5 Add test for multipart uploads and fix part renumbering

a6cc563bdd UploadPart: automatic cleanup of version (and reference blocked) when interrupted

58563ed700 Add multipart upload using aws s3api

e645bbd3ce smoke test: add multipart upload test with part re-upload

3d477906d4 properly delete multipart uploads after completion

0a06fda0da Merge pull request 'Fix #204 (full Multipart Uploads semantics)' (#553) from nlnet-task1 into next ...

Reviewed-on: #553

lx added 1 commit 2023-06-13 08:48:31 +00:00

942c1f1bfe multipart uploads: save timestamp

lx added 1 commit 2023-06-13 13:59:18 +00:00

7126f3e1d1 garage key import: add checks and `--yes` CLI flag (fix #278)

lx added 1 commit 2023-06-13 14:16:17 +00:00

bf19a44fd9 admin API: add missing camelCase conversions (fix #381)

lx force-pushed next from fa7fa2a544 to ce62c3a5f4 2023-06-13 15:02:50 +00:00 Compare

lx force-pushed next from ce62c3a5f4 to 90b2d43eb4 2023-06-13 15:14:18 +00:00 Compare

lx added 1 commit 2023-06-13 15:35:01 +00:00

cda957b4b1 update netapp's rmp-serde dependency to v1.1

lx added 1 commit 2023-06-13 21:11:00 +00:00

5670367126 multipartupload in test: add forgotten timestamp

lx added 1 commit 2023-06-14 11:02:52 +00:00

187240e539 Merge branch 'main' into next

lx added 1 commit 2023-06-14 11:45:59 +00:00

52376d47ca admin api: change cluster status/layout to use lists and not maps (fix #377)

lx added 1 commit 2023-06-14 11:53:41 +00:00

35c108b85d admin api: switch GetClusterHealth to camelcase (fix #381 again)

lx added 2 commits 2023-06-14 11:56:57 +00:00

2c83006608 admin api: fix doc in drafts

28cc9f178a admin api: make name optionnal for CreateKey

lx added 1 commit 2023-06-14 12:16:07 +00:00

4a82f6380e admin api: move all endpoints to v1/ by default (v0/ still supported)

lx added 1 commit 2023-06-14 14:56:33 +00:00

7895f99d3a admin and cli: hide secret keys unless asked

lx added 1 commit 2023-06-14 15:18:36 +00:00

a83a092c03 admin: uniformize layout api and improve code

lx force-pushed next from b189653322 to 8ef42c9609 2023-06-14 15:19:32 +00:00 Compare

lx added 1 commit 2023-08-29 09:32:48 +00:00

2e90e1c124 Merge branch 'main' into next

lx added 26 commits 2023-09-04 09:45:12 +00:00

0b83e0558e bucket_table: data model for lifecycle configuration

8041d9a827 s3: add xml structures to serialize/deserialize lifecycle configs

abf011c290 lifecycle: implement validation into garage's internal data structure

f7b409f114 use a NaiveDate in data model, it serializes to string (iso 8601 format)

a2e0e34db5 lifecycle: skeleton for lifecycle worker

2996dc875f lifecycle worker: implement main functionality

da8b224e24 lifecycle worker: skip entire bucket when no lifecycle config is set

0f1849e1ac lifecycle worker: launch with the rest of Garage

7200954318 lifecycle worker: add logging

75ccc5a95c lifecycle config: store date as given, try to debug

d2e94e36d6 lifecycle config: add missing line in merge() and remove tracing

a1d57283c0 bucket_table: bucketparams::new doesn't need to be pub

5c923d48d7 reference manual: document support for lifecycle configuration

d94f1c9178 reference manual: remove obsolete caveat about multipart uploads

f0a395e2e5 s3 bucket apis: remove redundant call

01c327a07a lifecycle worker: avoid building chrono's serde feature

5fad4c4658 update cargo.nix

b2f679675e lifecycle worker: take into account disabled rules

be03a4610f s3api: remove redundant serde rename attribute

1cfcc61de8 lifecycle worker: mitigate potential bugs + refactoring

adbf5925de lifecycle worker: use queue_insert and process objects in batches

a00a52633f lifecycle worker: add log message when starting

f579d6d9b4 lifecycle worker: fix potential inifinite loop

1cdc321e28 lifecycle worker: don't get stuck on non-existent bucket

8e0c020bb9 lifecycle worker: correct small clippy lints

3f461d8891 Merge pull request 'object lifecycles (fix #309)' (#620) from bucket-lifecycle into next ...

Reviewed-on: #620

lx added 1 commit 2023-09-06 14:35:02 +00:00

4b4f2000f4 lifecycle: fix SkipBucket bug

lx added 27 commits 2023-09-11 10:52:03 +00:00

71c0188055 block manager: skeleton for multi-hdd support

6c420c0880 block manager: multi-directory layout computation

887b3233f4 block manager: use data paths from layout

a09f86729c block manager: move blocks in write_block if necessary

3199cab4c8 update cargo.nix

1b8c265c14 block manager: get rid of check_block_status

fd00a47ddc table queue: increase batch size

93114a9747 block manager: refactoring

3a74844df0 block manager: fix dir_not_empty

a44f486931 block manager: refactoring & increase max worker count to 8

55c514999e block manager: fixes in layout

e30865984a block manager: scrub checkpointing

f38a31b330 block manager: avoid incorrect data_dir configs and avoid losing files

99ed18350f block manager: refactor and fix monitoring/statistics

bca347a1e8 doc: update page on upgradin clusters

6595efd82f Document multi-hdd support

6b008b5bd3 block manager: add rebalance operation to rebalance multi-hdd setups

6a067e30ee doc: documentation of rebalance repair

2f112ac682 correct free data space accounting for multiple data dirs on same fs

eb972a8422 doc: update multi-hdd section

2657b5c1b9 block manager: fix bugs

be91ef6294 block manager: fix bug where rebalance didn't delete old copies

de5d792181 block manager: fix indentation (why not detected by cargo fmt?)

7f9ba49c71 block manager: remove data_dir field

9526328d38 scrub: clear saved checkpoint when canceling scrub

ba7ac52c19 block repair: simpler/more robust iterator progress calculation

7228fbfd4f Merge pull request 'multi-hdd support (fix #218)' (#625) from multihdd into next ...

Reviewed-on: #625

lx added 1 commit 2023-09-11 11:14:24 +00:00

ad6b1cc0be Merge branch 'main' into next

lx added 1 commit 2023-09-11 18:00:08 +00:00

51abbb02d8 Merge branch 'main' into next

lx added 1 commit 2023-09-11 21:09:26 +00:00

fd7d8fec59 Merge branch 'main' into next

lx added 3 commits 2023-09-21 14:03:37 +00:00

f97168f805 garage_db: refactor transactions and add on_commit mechanism

0635250b2b garage_table/queue_insert: delay worker notification to after transaction commit (fix #583)

1d986bd889 Merge pull request 'Refactor db transactions and add on_commit for table.queue_insert' (#637) from k2v-indices-lmdb into next ...

Reviewed-on: #637

lx added 11 commits 2023-09-27 09:04:33 +00:00

2e229d4430 new layout: improve output display

015ccb39aa new layout: make zone_redundancy optionnal (if not set, is maximum)

749b4865d0 new layout: improve display and fix comments

0088599f52 new layout: fix clippy lints

013b026d56 update cargo.nix

3ecc17f8c5 new layout: use deterministic randomness for reproducible results

b4a0e636d8 new layout doc: add examples of unexpected layout, to explain

405aa42b7d layout doc: update old text

8d07888fa2 layout doc: write explanations for bizarre scenarios

0e5925fff6 layout doc: reformulate

aa7eadc799 Merge pull request 'New layout: fixes and UX improvements' (#634) from new-layout-ux into next ...

Reviewed-on: #634

lx added 2 commits 2023-09-27 11:13:33 +00:00

ad82035b98 Merge branch 'main' into next

897cbf2c27 actually update rmp-serde to 1.1.2 for both garage and netapp dependency (fix #629)

lx force-pushed next from 638741c951 to cc80eac633 2023-09-27 11:57:49 +00:00 Compare

lx force-pushed next from cc80eac633 to 21f6327bad 2023-09-27 12:19:31 +00:00 Compare

lx force-pushed next from 21f6327bad to 4e11ba4ff5 2023-09-27 12:35:42 +00:00 Compare

lx force-pushed next from 4e11ba4ff5 to 57c6e9e9c5 2023-09-27 12:39:20 +00:00 Compare

lx force-pushed next from 57c6e9e9c5 to 9ac1d5be0e 2023-09-27 12:57:43 +00:00 Compare

lx added 1 commit 2023-10-03 16:40:43 +00:00

2e656b541b Merge branch 'main' into next

lx added 1 commit 2023-10-04 08:44:55 +00:00

920dec393a cli: more precise doc comment

lx added 1 commit 2023-10-05 08:42:22 +00:00

9ccc1d6f4a move upgrade test to release build

lx added 1 commit 2023-10-05 13:20:58 +00:00

6790e24f5a Add migration to v0.9 guide

lx added 1 commit 2023-10-05 13:30:05 +00:00

2448eb7713 upgrade doc: fixes and precisions

lx added 1 commit 2023-10-05 14:27:37 +00:00

1c13135f25 admin api: remove broken GET /v0/key router rule

lx added 1 commit 2023-10-05 14:56:30 +00:00

0c431b0c03 admin api: increased compatibility for v0/ endpoints

lx added 1 commit 2023-10-10 11:57:58 +00:00

d4932c31ea Merge branch 'main' into next

lx added 2 commits 2023-10-10 12:08:21 +00:00

3d7892477d convert_db: fix build

952c9570c4 bump version to v0.9.0

lx changed title from WIP: Garage v0.9 to Garage v0.9 2023-10-10 13:16:55 +00:00

lx referenced this issue from a commit 2023-10-10 13:28:29 +00:00

Merge pull request 'Garage v0.9' (#473) from next into main

lx merged commit e6df7089a1 into main 2023-10-10 13:28:29 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

AdminAPI

Issues concerning the administration API

  

Bug

  

Check AWS

The behaviour should be checked against AWS S3

  

CI

Issues with Nix, Drone, Rust Compiler, etc.

  

Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

Critical

  

Documentation

  

Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

Improvement

  

Low priority

For issues that we don't want to spend time on until made absolutely necessary

  

Newcomer

A bug that should be newcomer friendly.

  

Performance

  

S3 Compatibility

  

Testing

  

Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No Label

AdminAPI

Bug

Check AWS

CI

Correctness

Critical

Documentation

Ideas

Improvement

Low priority

Newcomer

Performance

S3 Compatibility

Testing

Usability

Milestone

Clear milestone

No items

No Milestone

v0.9

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#473

No description provided.