Garage v1.0 #683

Merged

lx merged 119 commits from next-0.10 into main 2024-04-10 15:23:13 +00:00

Conversation 0 Commits 119 Files changed 124 +7724 -3830

lx commented 2024-01-11 11:03:29 +00:00

Owner

Copy link

Preliminary changelog

• This new Garage version fixes consistency issues that could arise when cluster composition was updated (adding or removing nodes) and data was being rebalanced between nodes. This release is the result of work done for the 3rd task of the Garage NLnet project n°2022-12-018 (see #667). The fixed version has been extensively tested using Jepsen (see #544). This version is a major version since the internal format of the cluster layout has changed, and nodes running version 0.10 are not able to communicate with nodes running version 0.9.x, however from an end-user perspective, no breaking changes are introduced by this version and the upgrade process is transparent.

• Implementation of SSE-C encryption (#602, #730)

• Implementation of checksum algorithms (#347, #787)

• The replication_mode configuration parameter is now dividied in replication_factor and consistency_mode (#750)

• Add a repair procedure for block reference counters (#782)

• Fixes and refactoring in peering manager (#761, #786)

• Remove migration path from Garage v0.5 (#766)

• Remove the Sled database engine (#767)

Internals:

• Remove max_write_errors and max_faults (#760)

Thanks: Yureka

Other changes from the main branch

• add snapshotting of metadata db, manual with garage meta snapshot and automatic with the metadata_auto_snapshot_interval configuration parameter (#775)

• add a disable_scrub configuration option (#774)

• CLI: allow manipulating buckets using a prefix of their full hex id (#783)

• add support for logging to syslog (#368, #784)

• add marker file in data directory that is checked upon startup (#601, #785)

• avoid unbounded buffering in RAM, add config option block_ram_buffer_max (#788, #792)

• db engines: refactoring, improved defaults, connection pooling for sqlite backend, testing of multiple backends (#660, #765, #779, #793)

• allow utf-8 in headers such as object metadata (#763)

• refactoring: add request helper (#751)

• disable unreliable K2V tests (#789, #791)

• documentation updates (#773, #781)

Thanks: jirutka, stefano, Yureka

### Preliminary changelog - This new Garage version fixes consistency issues that could arise when cluster composition was updated (adding or removing nodes) and data was being rebalanced between nodes. This release is the result of work done for the 3rd task of the Garage NLnet project n°2022-12-018 (see #667). The fixed version has been extensively tested using Jepsen (see #544). This version is a major version since the internal format of the cluster layout has changed, and nodes running version 0.10 are not able to communicate with nodes running version 0.9.x, however from an end-user perspective, no breaking changes are introduced by this version and the upgrade process is transparent. - Implementation of SSE-C encryption (#602, #730) - Implementation of checksum algorithms (#347, #787) - The `replication_mode` configuration parameter is now dividied in `replication_factor` and `consistency_mode` (#750) - Add a repair procedure for block reference counters (#782) - Fixes and refactoring in peering manager (#761, #786) - Remove migration path from Garage v0.5 (#766) - Remove the Sled database engine (#767) Internals: - Remove `max_write_errors` and `max_faults` (#760) Thanks: Yureka **Other changes from the main branch** - add snapshotting of metadata db, manual with `garage meta snapshot` and automatic with the `metadata_auto_snapshot_interval` configuration parameter (#775) - add a `disable_scrub` configuration option (#774) - CLI: allow manipulating buckets using a prefix of their full hex id (#783) - add support for logging to syslog (#368, #784) - add marker file in data directory that is checked upon startup (#601, #785) - avoid unbounded buffering in RAM, add config option `block_ram_buffer_max` (#788, #792) - db engines: refactoring, improved defaults, connection pooling for sqlite backend, testing of multiple backends (#660, #765, #779, #793) - allow utf-8 in headers such as object metadata (#763) - refactoring: add request helper (#751) - disable unreliable K2V tests (#789, #791) - documentation updates (#773, #781) Thanks: jirutka, stefano, Yureka

lx added 61 commits 2024-01-11 11:03:30 +00:00

garage_rpc: reorder functions in layout.rs 0962313ebd

remove Ring and use ClusterLayout everywhere 12d1dbfc6b

avoid using layout_watch in System directly 4a9c94514f

WIP: garage_rpc: store layout version history fe9af1dcaa

cluster layout: adapt all uses of ClusterLayout to LayoutHistory 8dccee3ccf

bump garage protocol version tag to 0x000A (0.10) 1da0a5676e

layout: use separate CRDT for staged layout changes 523d2ecb95

wip: split out layout management from System into separate LayoutManager 8a2b1dd422

layout: more refactoring 19ef1ec8e7

layout: refactor to use a RwLock on LayoutHistory bfb1845fdc

layout: separate code path for synchronizing update trackers only 94caf9c0c1

layout: begin managing the update tracker values 03ebf18830

layout admin: add missing calls to update_hash bad7cc812e

layout: fix tracker bugs 9d95f6f704

layout: add helpers to LayoutHistory and prepare integration with Table df36cf3099

table sync: adapt to new layout history ce89d1ddab

layout/sync: fix bugs and add tracing df24bb806d

layout: fix test 9a491fa137

layout: some refactoring of nongateway nodes 8e292e06b3

layout: refactoring of all_nodes 1aab1f4e68

layout: fixes in schema 83a11374ca

system: add todo wrt new layout 866196750f

layout: prepare for write sets 3b361d2959

table: take into account multiple write sets in inserts 90e1619b1e

layout: fix test 7ef2c23120

layout history merge: rm invalid versions when valid versions are added b3e729f4b8

integration test: print stdout and stderr on subcommand crash 46007bf01d

rpc: fix write set quorums acd49de9f9

layout: wip cache global mins 65066c7064

layout: add helper for cached/external values to centralize recomputation 393c4d4515

layou: implement ack locking 33c8a489b0

layout: fix test d4df03424f

layout: split helper in separate file; more precise difference tracking ad5c6f779f

layout: refactor digests and add "!=" assertions before epidemic bcast 707442f5de

rpc_helper: don't use tokio::spawn for individual requests 22f38808e7

table: implement write sets for insert_many 3ecd14b9f6

block manager: read_block using old layout versions if necessary d6d239fc79

rpc: update system::health to take into account write sets for all partitions 78362140f5

cli: show when nodes are draining metadata 539a920313

cli: add layout history and layout assume-sync commands 11e6fef93c

cli: remove historic layout info from status c539077d30

rpc helper: write comments + small refactoring of tracing 539af6eac4

admin: more info in admin GetClusterStatus c04dd8788a

layout updates: fix the set of nodes among which minima are calculated c8356a91d9

rpc: refactor result tracking for quorum sets 95eb13eb08

table sync: use write quorums to report global success or failure of sync d90de365b3

layout cli: safer skip-dead-nodes command aa59059a91

layout: allow sync update tracker to progress with only quorums 9cecea64d4

fix build with discovery features 431b28e0cf

rpc: fix system::health 91b874c4ef

cli: improvements to the layout commands when multiple layouts are live 7f2541101f

layout version: refactor get_node_zone 063294dd56

layout: move block_read_nodes_of to rpc_helper to avoid double-locking ... 5dd200c015

(in theory, this could have caused a deadlock)

rpc helper: small refactorings 64a6e557a4

layout: refactoring, merge two files 4dbf254512

table: fix insert_many to not send duplicates f8df90b79b

table: remove redundant tracing in insert_many e4f493b481

fix some clippy lints 85b5a6bcd1

layout: refactor/fix bad while loop adccce1145

layout: refactoring and fix in layout helper 0041b013a4

Merge pull request 'NLnet task 3' (#667) from nlnet-task3 into next-0.10 ... 8a6ec1d611

Reviewed-on: #667

lx added 1 commit 2024-01-11 11:06:17 +00:00

bump crate versions to 0.10.0 db48dd3d6c

lx force-pushed next-0.10 from 31a5547569 to cf2af186fc 2024-02-13 10:36:39 +00:00 Compare

lx force-pushed next-0.10 from 7a9f1fadad to eb4a6ce106 2024-02-15 13:07:03 +00:00 Compare

lx added 1 commit 2024-02-20 16:03:17 +00:00

Merge branch 'main' into next-0.10 643d1aabd8

lx added 1 commit 2024-02-20 16:09:17 +00:00

[next-0.10] cluster node status metrics: report nodes of all active layout versions 75e591727d

lx added 1 commit 2024-02-22 14:46:13 +00:00

Merge branch 'main' into next-0.10 59f61c966a

lx added 1 commit 2024-02-22 14:54:13 +00:00

[next-0.10] fix build 81cebdd124

lx added 1 commit 2024-02-23 15:51:00 +00:00

Merge branch 'main' into next-0.10 916c67ccf4

lx added 1 commit 2024-02-23 17:28:41 +00:00

Merge branch 'main' into next-0.10 3692af7052

lx added 1 commit 2024-02-26 17:55:59 +00:00

Merge branch 'main' into next-0.10 4b978b7533

lx added 1 commit 2024-02-27 09:13:53 +00:00

[next-0.10] woodpecker: run debug pipeline on manual trigger d0d95fd53f

lx added 1 commit 2024-03-04 14:56:23 +00:00

Merge branch 'main' into next-0.10 bbde9bc912

lx added 2 commits 2024-03-05 21:56:20 +00:00

refactor: remove max_write_errors and max_faults 6760895926

Merge pull request 'refactor: remove max_write_errors and max_faults' (#760) from yuka/garage:remove-max-write-errors into next-0.10 ... 603604cdfc

Reviewed-on: #760

lx added 1 commit 2024-03-07 13:00:43 +00:00

Merge branch 'main' into next-0.10 fe2dc5d51c

lx added 5 commits 2024-03-07 15:21:39 +00:00

[sse-c] Implement SSE-C encryption 57acc60082

[sse-c] Testing for SSE-C encryption fa4878bad6

[sse-c] Documentation for SSE-C e3333f2ac5

[sse-c] Remove special case for Content-Type header 3fcb54e3cf

Merge pull request 'SSE-C encryption' (#730) from sse-c into next-0.10 ... 2fd13c7d13

Reviewed-on: #730

lx added 3 commits 2024-03-07 16:32:54 +00:00

adjust docs for replication factor 8f86af52ed

ReplicationMode -> ConsistencyMode+ReplicationFactor c1769bbe69

Merge pull request 'ReplicationMode -> ConsistencyMode+ReplicationFactor' (#750) from yuka/garage:split-consistency-mode into next-0.10 ... 20c0b4ffb2

Reviewed-on: #750

lx added 1 commit 2024-03-08 12:57:22 +00:00

Merge branch 'main' into next-0.10 1ace34adbb

lx added 2 commits 2024-03-08 13:43:43 +00:00

[rm-migration] Remove migration path from Garage v0.5 f537f76681

Merge pull request 'Remove migration path from Garage v0.5' (#766) from rm-migration into next-0.10 ... 2128b5febd

Reviewed-on: #766

lx added 7 commits 2024-03-12 10:46:00 +00:00

[rm-sled] Remove the Sled database engine 44454aac01

[rm-sled] Remove counted_tree_hack 05c92204ec

[rm-sled] Implement some missing functionality in garage_db 66c23890c1

[rm-sled] Implement iterators in sqlite & lmdb transactions ... b942949940

with way too much unsafe code

[rm-sled] Make proper use of pinning in LMDB adapter + comment unsafe 32aa246300

[rm-sled] factorize some code in sqlite backend 2795b53b8b

Merge pull request 'Remove Sled' (#767) from rm-sled into next-0.10 ... 81191d2d92

Reviewed-on: #767

lx force-pushed next-0.10 from 4901ae7616 to 0038ca8a78 2024-03-18 19:19:37 +00:00 Compare

lx added 4 commits 2024-03-19 15:59:22 +00:00

[block-ref-repair] Block refcount recalculation and repair ... dc0b78cdb8

- We always recalculate the reference count of a block before deleting
  it locally, to make sure that it is indeed zero.

- If we had to fetch a remote block but we were not able to get it,
  check that refcount is indeed > 0.

- Repair procedure that checks everything

[block-ref-repair] rename rc's rc field to rc_table 3165ab926c

[block-ref-repair] mention garage block repair-rc in documentation 3eab639c14

Merge pull request 'block refcount repair' (#782) from block-ref-repair into next-0.10 ... 65853a4863

Reviewed-on: #782

lx added 1 commit 2024-03-19 16:17:57 +00:00

Merge branch 'main' into next-0.10 ce69dc302c

lx added 4 commits 2024-03-21 10:26:38 +00:00

[net-fixes] peering: only count node IDs and not addresses in hash 5225a81dee

[net-fixes] fix issues with local peer address (fix #761) 961b4f9af3

[net-fixes] netapp peer exchange: send only currently connected address 3844110cd0

Merge pull request 'Fixes to garage_net peering manager' (#786) from net-fixes into next-0.10 ... 7e0107c47d

Reviewed-on: #786

lx added 2 commits 2024-03-26 14:25:00 +00:00

[s3-checksum] implement x-amz-checksum-* headers 74949c69cb

Merge pull request 'checksum algorithms' (#787) from s3-checksum into next-0.10 ... cdde0f19ee

Reviewed-on: #787

lx changed title from WIP: Garage v0.10 to WIP: Garage v1.0 2024-03-26 14:58:23 +00:00

lx changed title from WIP: Garage v1.0 to WIP: Garage v0.10 2024-03-26 14:58:33 +00:00

lx added 1 commit 2024-03-26 15:34:46 +00:00

Merge branch 'main' into next-0.10 f7cd4eb600

lx force-pushed next-0.10 from 666a965b0c to 51d11b4b26 2024-03-27 09:11:08 +00:00 Compare

lx added 1 commit 2024-03-27 09:44:23 +00:00

[next-0.10] fixes to k2v rpc + comment fixes c0eeb0b0f3

lx added 1 commit 2024-03-27 12:32:29 +00:00

[next-0.10] remove impl Deref for LayoutHelper 01a0bd5410

lx added 1 commit 2024-03-27 12:37:37 +00:00

[next-0.10] cache layout check result 32f1786f9f

lx added 1 commit 2024-03-27 12:47:24 +00:00

[next-0.10] layout helper: rename & clarify updates to update trackers 4eba32f29f

lx added 1 commit 2024-03-27 12:56:03 +00:00

[next-0.10] admin api: fix logic in get cluster status 25c196f34d

lx added 1 commit 2024-03-28 14:01:16 +00:00

Merge branch 'main' into next-0.10 8bfc16ba7d

lx added 1 commit 2024-03-28 14:20:10 +00:00

[next-0.10] bump version number to 1.0 afad62939e

lx changed title from WIP: Garage v0.10 to WIP: Garage v1.0 2024-03-28 14:21:27 +00:00

lx force-pushed next-0.10 from a5ff5f3ce8 to 554437254e 2024-03-28 17:45:12 +00:00 Compare

lx added 1 commit 2024-04-03 12:20:09 +00:00

[next-0.10] bump helm chart version 50669b3e76

lx added 1 commit 2024-04-10 12:35:47 +00:00

[next-0.10] small updates to mention Garage v0.9.4 90e3c2af91

lx added 1 commit 2024-04-10 12:38:27 +00:00

Merge branch 'main' into next-0.10 ff093ddbb8

lx changed title from WIP: Garage v1.0 to Garage v1.0 2024-04-10 15:23:01 +00:00

lx merged commit 1779fd40c0 into main 2024-04-10 15:23:13 +00:00

lx referenced this pull request from a commit 2024-04-10 15:23:13 +00:00

Merge pull request 'Garage v1.0' (#683) from next-0.10 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

action

check-aws

The behaviour should be checked against AWS S3

  

action

discussion-needed

We need to discuss if the feature is desirable and/or better specify the envisioned behavior or technical implementation

  

action

for-external-contributors

Core team does not plan working on this in the near future however an external contribution will be reviewed & merged

  

action

for-newcomers

An issue that should be newcomer friendly.

  

action

more-info-needed

More information is needed to investigate this issue

  

action

need-funding

This issue represents lots of work, we need to find money to pay our core team work

  

action

triage-required

It's not very clear what we want to do with this issue yet, more thoughts is required

  

kind

correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

kind

ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

kind

improvement

Feature requests that do not fall in other categories

  

kind

performance

Deceptive performance observed OR idea to improve performances

  

kind

testing

Functional tests, integration tests, Jepsen, etc.

  

kind

usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

  

kind

wrong-behavior

Unintended behavior

  

prio

critical

Could impact integrity or availability of production clusters

  

prio

low

For issues that we don't want to spend time on until made absolutely necessary

  

scope

admin-api

Issues concerning the administration API

  

scope

background-healing

Background healing covers repair tasks, anti-entropy, merkle trees, etc. ; everything related to replicating data appropriately.

  

scope

build

Issues with Nix, Drone, Rust Compiler, etc.

  

scope

documentation

  

scope

k8s

Kubernetes related, including Helm Charts

  

scope

layout

Linked to the cluster layout and storage management

  

scope

metadata

Metadata Engine (LMDB, sqlite, etc.)

  

scope

ops

Operations related to operations/maintenance issue, tooling related (like CLI missing info ; impossible, too dangerous, too complicated migration, etc.)

  

scope

rpc

Linked to RPC

  

scope

s3-api

Related to the S3 API (especially compatibility issues).

  

scope

security

Cryptography-related, auth-related, authorization-related issues

  

scope

telemetry

OpenTelemetry (traces, metrics, etc.)

No labels

action

check-aws

action

discussion-needed

action

for-external-contributors

action

for-newcomers

action

more-info-needed

action

need-funding

action

triage-required

kind

correctness

kind

ideas

kind

improvement

kind

performance

kind

testing

kind

usability

kind

wrong-behavior

prio

critical

prio

low

scope

admin-api

scope

background-healing

scope

build

scope

documentation

scope

k8s

scope

layout

scope

metadata

scope

ops

scope

rpc

scope

s3-api

scope

security

scope

telemetry

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#683

No description provided.