Garage v1.0 #683

Merged

lx merged 119 commits from next-0.10 into main 2024-04-10 15:23:13 +00:00

Conversation 0 Commits 119 Files Changed 124 +7724 -3830

lx commented 2024-01-11 11:03:29 +00:00

Owner

Copy Link

Preliminary changelog

• This new Garage version fixes consistency issues that could arise when cluster composition was updated (adding or removing nodes) and data was being rebalanced between nodes. This release is the result of work done for the 3rd task of the Garage NLnet project n°2022-12-018 (see #667). The fixed version has been extensively tested using Jepsen (see #544). This version is a major version since the internal format of the cluster layout has changed, and nodes running version 0.10 are not able to communicate with nodes running version 0.9.x, however from an end-user perspective, no breaking changes are introduced by this version and the upgrade process is transparent.

• Implementation of SSE-C encryption (#602, #730)

• Implementation of checksum algorithms (#347, #787)

• The replication_mode configuration parameter is now dividied in replication_factor and consistency_mode (#750)

• Add a repair procedure for block reference counters (#782)

• Fixes and refactoring in peering manager (#761, #786)

• Remove migration path from Garage v0.5 (#766)

• Remove the Sled database engine (#767)

Internals:

• Remove max_write_errors and max_faults (#760)

Thanks: Yureka

Other changes from the main branch

• add snapshotting of metadata db, manual with garage meta snapshot and automatic with the metadata_auto_snapshot_interval configuration parameter (#775)

• add a disable_scrub configuration option (#774)

• CLI: allow manipulating buckets using a prefix of their full hex id (#783)

• add support for logging to syslog (#368, #784)

• add marker file in data directory that is checked upon startup (#601, #785)

• avoid unbounded buffering in RAM, add config option block_ram_buffer_max (#788, #792)

• db engines: refactoring, improved defaults, connection pooling for sqlite backend, testing of multiple backends (#660, #765, #779, #793)

• allow utf-8 in headers such as object metadata (#763)

• refactoring: add request helper (#751)

• disable unreliable K2V tests (#789, #791)

• documentation updates (#773, #781)

Thanks: jirutka, stefano, Yureka

### Preliminary changelog - This new Garage version fixes consistency issues that could arise when cluster composition was updated (adding or removing nodes) and data was being rebalanced between nodes. This release is the result of work done for the 3rd task of the Garage NLnet project n°2022-12-018 (see #667). The fixed version has been extensively tested using Jepsen (see #544). This version is a major version since the internal format of the cluster layout has changed, and nodes running version 0.10 are not able to communicate with nodes running version 0.9.x, however from an end-user perspective, no breaking changes are introduced by this version and the upgrade process is transparent. - Implementation of SSE-C encryption (#602, #730) - Implementation of checksum algorithms (#347, #787) - The `replication_mode` configuration parameter is now dividied in `replication_factor` and `consistency_mode` (#750) - Add a repair procedure for block reference counters (#782) - Fixes and refactoring in peering manager (#761, #786) - Remove migration path from Garage v0.5 (#766) - Remove the Sled database engine (#767) Internals: - Remove `max_write_errors` and `max_faults` (#760) Thanks: Yureka **Other changes from the main branch** - add snapshotting of metadata db, manual with `garage meta snapshot` and automatic with the `metadata_auto_snapshot_interval` configuration parameter (#775) - add a `disable_scrub` configuration option (#774) - CLI: allow manipulating buckets using a prefix of their full hex id (#783) - add support for logging to syslog (#368, #784) - add marker file in data directory that is checked upon startup (#601, #785) - avoid unbounded buffering in RAM, add config option `block_ram_buffer_max` (#788, #792) - db engines: refactoring, improved defaults, connection pooling for sqlite backend, testing of multiple backends (#660, #765, #779, #793) - allow utf-8 in headers such as object metadata (#763) - refactoring: add request helper (#751) - disable unreliable K2V tests (#789, #791) - documentation updates (#773, #781) Thanks: jirutka, stefano, Yureka

lx added 61 commits 2024-01-11 11:03:30 +00:00

0962313ebd

garage_rpc: reorder functions in layout.rs

12d1dbfc6b

remove Ring and use ClusterLayout everywhere

4a9c94514f

avoid using layout_watch in System directly

fe9af1dcaa

WIP: garage_rpc: store layout version history

8dccee3ccf

cluster layout: adapt all uses of ClusterLayout to LayoutHistory

1da0a5676e

bump garage protocol version tag to 0x000A (0.10)

523d2ecb95

layout: use separate CRDT for staged layout changes

8a2b1dd422

wip: split out layout management from System into separate LayoutManager

19ef1ec8e7

layout: more refactoring

bfb1845fdc

layout: refactor to use a RwLock on LayoutHistory

94caf9c0c1

layout: separate code path for synchronizing update trackers only

03ebf18830

layout: begin managing the update tracker values

bad7cc812e

layout admin: add missing calls to update_hash

9d95f6f704

layout: fix tracker bugs

df36cf3099

layout: add helpers to LayoutHistory and prepare integration with Table

ce89d1ddab

table sync: adapt to new layout history

df24bb806d

layout/sync: fix bugs and add tracing

9a491fa137

layout: fix test

8e292e06b3

layout: some refactoring of nongateway nodes

1aab1f4e68

layout: refactoring of all_nodes

83a11374ca

layout: fixes in schema

866196750f

system: add todo wrt new layout

3b361d2959

layout: prepare for write sets

90e1619b1e

table: take into account multiple write sets in inserts

7ef2c23120

layout: fix test

b3e729f4b8

layout history merge: rm invalid versions when valid versions are added

46007bf01d

integration test: print stdout and stderr on subcommand crash

acd49de9f9

rpc: fix write set quorums

65066c7064

layout: wip cache global mins

393c4d4515

layout: add helper for cached/external values to centralize recomputation

33c8a489b0

layou: implement ack locking

d4df03424f

layout: fix test

ad5c6f779f

layout: split helper in separate file; more precise difference tracking

707442f5de

layout: refactor digests and add "!=" assertions before epidemic bcast

22f38808e7

rpc_helper: don't use tokio::spawn for individual requests

3ecd14b9f6

table: implement write sets for insert_many

d6d239fc79

block manager: read_block using old layout versions if necessary

78362140f5

rpc: update system::health to take into account write sets for all partitions

539a920313

cli: show when nodes are draining metadata

11e6fef93c

cli: add layout history and layout assume-sync commands

c539077d30

cli: remove historic layout info from status

539af6eac4

rpc helper: write comments + small refactoring of tracing

c04dd8788a

admin: more info in admin GetClusterStatus

c8356a91d9

layout updates: fix the set of nodes among which minima are calculated

95eb13eb08

rpc: refactor result tracking for quorum sets

d90de365b3

table sync: use write quorums to report global success or failure of sync

aa59059a91

layout cli: safer skip-dead-nodes command

9cecea64d4

layout: allow sync update tracker to progress with only quorums

431b28e0cf

fix build with discovery features

91b874c4ef

rpc: fix system::health

7f2541101f

cli: improvements to the layout commands when multiple layouts are live

063294dd56

layout version: refactor get_node_zone

5dd200c015

layout: move block_read_nodes_of to rpc_helper to avoid double-locking ...

(in theory, this could have caused a deadlock)

64a6e557a4

rpc helper: small refactorings

4dbf254512

layout: refactoring, merge two files

f8df90b79b

table: fix insert_many to not send duplicates

e4f493b481

table: remove redundant tracing in insert_many

85b5a6bcd1

fix some clippy lints

adccce1145

layout: refactor/fix bad while loop

0041b013a4

layout: refactoring and fix in layout helper

8a6ec1d611 Merge pull request 'NLnet task 3' (#667) from nlnet-task3 into next-0.10 ...

Reviewed-on: #667

lx added 1 commit 2024-01-11 11:06:17 +00:00

db48dd3d6c

bump crate versions to 0.10.0

lx force-pushed next-0.10 from 31a5547569 to cf2af186fc 2024-02-13 10:36:39 +00:00 Compare

lx force-pushed next-0.10 from 7a9f1fadad to eb4a6ce106 2024-02-15 13:07:03 +00:00 Compare

lx added 1 commit 2024-02-20 16:03:17 +00:00

643d1aabd8

Merge branch 'main' into next-0.10

lx added 1 commit 2024-02-20 16:09:17 +00:00

75e591727d

[next-0.10] cluster node status metrics: report nodes of all active layout versions

lx added 1 commit 2024-02-22 14:46:13 +00:00

59f61c966a

Merge branch 'main' into next-0.10

lx added 1 commit 2024-02-22 14:54:13 +00:00

81cebdd124

[next-0.10] fix build

lx added 1 commit 2024-02-23 15:51:00 +00:00

916c67ccf4

Merge branch 'main' into next-0.10

lx added 1 commit 2024-02-23 17:28:41 +00:00

3692af7052

Merge branch 'main' into next-0.10

lx added 1 commit 2024-02-26 17:55:59 +00:00

4b978b7533

Merge branch 'main' into next-0.10

lx added 1 commit 2024-02-27 09:13:53 +00:00

d0d95fd53f

[next-0.10] woodpecker: run debug pipeline on manual trigger

lx added 1 commit 2024-03-04 14:56:23 +00:00

bbde9bc912

Merge branch 'main' into next-0.10

lx added 2 commits 2024-03-05 21:56:20 +00:00

6760895926 refactor: remove max_write_errors and max_faults

603604cdfc Merge pull request 'refactor: remove max_write_errors and max_faults' (#760) from yuka/garage:remove-max-write-errors into next-0.10 ...

Reviewed-on: #760

lx added 1 commit 2024-03-07 13:00:43 +00:00

fe2dc5d51c

Merge branch 'main' into next-0.10

lx added 5 commits 2024-03-07 15:21:39 +00:00

57acc60082

[sse-c] Implement SSE-C encryption

fa4878bad6

[sse-c] Testing for SSE-C encryption

e3333f2ac5

[sse-c] Documentation for SSE-C

3fcb54e3cf

[sse-c] Remove special case for Content-Type header

2fd13c7d13 Merge pull request 'SSE-C encryption' (#730) from sse-c into next-0.10 ...

Reviewed-on: #730

lx added 3 commits 2024-03-07 16:32:54 +00:00

8f86af52ed adjust docs for replication factor

c1769bbe69 ReplicationMode -> ConsistencyMode+ReplicationFactor

20c0b4ffb2 Merge pull request 'ReplicationMode -> ConsistencyMode+ReplicationFactor' (#750) from yuka/garage:split-consistency-mode into next-0.10 ...

Reviewed-on: #750

lx added 1 commit 2024-03-08 12:57:22 +00:00

1ace34adbb

Merge branch 'main' into next-0.10

lx added 2 commits 2024-03-08 13:43:43 +00:00

f537f76681

[rm-migration] Remove migration path from Garage v0.5

2128b5febd Merge pull request 'Remove migration path from Garage v0.5' (#766) from rm-migration into next-0.10 ...

Reviewed-on: #766

lx added 7 commits 2024-03-12 10:46:00 +00:00

44454aac01

[rm-sled] Remove the Sled database engine

05c92204ec

[rm-sled] Remove counted_tree_hack

66c23890c1

[rm-sled] Implement some missing functionality in garage_db

b942949940

[rm-sled] Implement iterators in sqlite & lmdb transactions ...

with way too much unsafe code

32aa246300

[rm-sled] Make proper use of pinning in LMDB adapter + comment unsafe

2795b53b8b

[rm-sled] factorize some code in sqlite backend

81191d2d92 Merge pull request 'Remove Sled' (#767) from rm-sled into next-0.10 ...

Reviewed-on: #767

lx added 1 commit 2024-03-18 19:18:09 +00:00

4901ae7616

Merge branch 'main' into next-0.10

lx force-pushed next-0.10 from 4901ae7616 to 0038ca8a78 2024-03-18 19:19:37 +00:00 Compare

lx added 4 commits 2024-03-19 15:59:22 +00:00

dc0b78cdb8

[block-ref-repair] Block refcount recalculation and repair ...

- We always recalculate the reference count of a block before deleting
  it locally, to make sure that it is indeed zero.

- If we had to fetch a remote block but we were not able to get it,
  check that refcount is indeed > 0.

- Repair procedure that checks everything

3165ab926c

[block-ref-repair] rename rc's rc field to rc_table

3eab639c14

[block-ref-repair] mention `garage block repair-rc` in documentation

65853a4863 Merge pull request 'block refcount repair' (#782) from block-ref-repair into next-0.10 ...

Reviewed-on: #782

lx added 1 commit 2024-03-19 16:17:57 +00:00

ce69dc302c

Merge branch 'main' into next-0.10

lx added 4 commits 2024-03-21 10:26:38 +00:00

5225a81dee

[net-fixes] peering: only count node IDs and not addresses in hash

961b4f9af3

[net-fixes] fix issues with local peer address (fix #761)

3844110cd0

[net-fixes] netapp peer exchange: send only currently connected address

7e0107c47d Merge pull request 'Fixes to garage_net peering manager' (#786) from net-fixes into next-0.10 ...

Reviewed-on: #786

lx added 2 commits 2024-03-26 14:25:00 +00:00

74949c69cb

[s3-checksum] implement x-amz-checksum-* headers

cdde0f19ee Merge pull request 'checksum algorithms' (#787) from s3-checksum into next-0.10 ...

Reviewed-on: #787

lx changed title from WIP: Garage v0.10 to WIP: Garage v1.0 2024-03-26 14:58:23 +00:00

lx changed title from WIP: Garage v1.0 to WIP: Garage v0.10 2024-03-26 14:58:33 +00:00

lx added 1 commit 2024-03-26 15:34:46 +00:00

f7cd4eb600

Merge branch 'main' into next-0.10

lx added 1 commit 2024-03-27 09:04:54 +00:00

666a965b0c

[next-0.10] doc: rewrite section on encryption to mention SSE-C

lx force-pushed next-0.10 from 666a965b0c to 51d11b4b26 2024-03-27 09:11:08 +00:00 Compare

lx added 1 commit 2024-03-27 09:44:23 +00:00

c0eeb0b0f3

[next-0.10] fixes to k2v rpc + comment fixes

lx added 1 commit 2024-03-27 12:32:29 +00:00

01a0bd5410

[next-0.10] remove impl Deref for LayoutHelper

lx added 1 commit 2024-03-27 12:37:37 +00:00

32f1786f9f

[next-0.10] cache layout check result

lx added 1 commit 2024-03-27 12:47:24 +00:00

4eba32f29f

[next-0.10] layout helper: rename & clarify updates to update trackers

lx added 1 commit 2024-03-27 12:56:03 +00:00

25c196f34d

[next-0.10] admin api: fix logic in get cluster status

lx added 1 commit 2024-03-28 14:01:16 +00:00

8bfc16ba7d

Merge branch 'main' into next-0.10

lx added 1 commit 2024-03-28 14:20:10 +00:00

afad62939e

[next-0.10] bump version number to 1.0

lx changed title from WIP: Garage v0.10 to WIP: Garage v1.0 2024-03-28 14:21:27 +00:00

lx added 1 commit 2024-03-28 17:38:51 +00:00

a5ff5f3ce8

[next-0.10] Add migration guide for v1.0

lx force-pushed next-0.10 from a5ff5f3ce8 to 554437254e 2024-03-28 17:45:12 +00:00 Compare

lx added 1 commit 2024-04-03 12:20:09 +00:00

50669b3e76

[next-0.10] bump helm chart version

lx added 1 commit 2024-04-10 12:35:47 +00:00

90e3c2af91

[next-0.10] small updates to mention Garage v0.9.4

lx added 1 commit 2024-04-10 12:38:27 +00:00

ff093ddbb8

Merge branch 'main' into next-0.10

lx changed title from WIP: Garage v1.0 to Garage v1.0 2024-04-10 15:23:01 +00:00

lx merged commit 1779fd40c0 into main 2024-04-10 15:23:13 +00:00

lx referenced this issue from a commit 2024-04-10 15:23:13 +00:00

Merge pull request 'Garage v1.0' (#683) from next-0.10 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

AdminAPI

Issues concerning the administration API

  

Bug

  

Check AWS

The behaviour should be checked against AWS S3

  

CI

Issues with Nix, Drone, Rust Compiler, etc.

  

Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

Critical

  

Documentation

  

Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

Improvement

  

Low priority

For issues that we don't want to spend time on until made absolutely necessary

  

Newcomer

A bug that should be newcomer friendly.

  

Performance

  

S3 Compatibility

  

Testing

  

Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No Label

AdminAPI

Bug

Check AWS

CI

Correctness

Critical

Documentation

Ideas

Improvement

Low priority

Newcomer

Performance

S3 Compatibility

Testing

Usability

Milestone

Clear milestone

No items

No Milestone

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#683

No description provided.