object lifecycles (fix #309) #620

Merged

lx merged 25 commits from bucket-lifecycle into next 2023-09-04 09:45:11 +00:00

Conversation 0 Commits 25 Files Changed 14 +945 -53

lx commented 2023-08-29 15:03:17 +00:00

Owner

Copy Link

This PR tracks the progress of NLnet task 2 (release garage v0.9), subtask 1 (implement object lifecycles).

• Add data model to bucket table for lifecycle configuration
• Implement S3 API calls to put, get and delete a bucket's lifecycle configuration
• Implement a worker that scans all objects at midnight UTC to apply the configuration
• Optimize the object scan to skip buckets with no lifecycle config
• Proof-reading
• Testing

This PR tracks the progress of NLnet task 2 (release garage v0.9), subtask 1 (implement object lifecycles). - [x] Add data model to bucket table for lifecycle configuration - [x] Implement S3 API calls to put, get and delete a bucket's lifecycle configuration - [x] Implement a worker that scans all objects at midnight UTC to apply the configuration - [x] Optimize the object scan to skip buckets with no lifecycle config - [x] Proof-reading - [x] Testing

lx added 1 commit 2023-08-29 15:03:18 +00:00

0b83e0558e bucket_table: data model for lifecycle configuration

lx added this to the v0.9 milestone 2023-08-29 15:03:55 +00:00

lx referenced this pull request 2023-08-29 15:04:31 +00:00

Garage v0.9 #473

lx added 1 commit 2023-08-29 15:44:41 +00:00

8041d9a827 s3: add xml structures to serialize/deserialize lifecycle configs

lx added 1 commit 2023-08-29 16:22:19 +00:00

abf011c290 lifecycle: implement validation into garage's internal data structure

lx added 1 commit 2023-08-30 09:24:33 +00:00

f7b409f114 use a NaiveDate in data model, it serializes to string (iso 8601 format)

lx added 1 commit 2023-08-30 10:41:24 +00:00

a2e0e34db5 lifecycle: skeleton for lifecycle worker

lx added 1 commit 2023-08-30 12:29:18 +00:00

2996dc875f lifecycle worker: implement main functionality

lx added 2 commits 2023-08-30 12:51:20 +00:00

da8b224e24 lifecycle worker: skip entire bucket when no lifecycle config is set

0f1849e1ac lifecycle worker: launch with the rest of Garage

lx added 1 commit 2023-08-30 12:55:01 +00:00

7200954318 lifecycle worker: add logging

lx added 1 commit 2023-08-30 18:02:23 +00:00

75ccc5a95c lifecycle config: store date as given, try to debug

lx added 1 commit 2023-08-30 18:06:05 +00:00

d2e94e36d6 lifecycle config: add missing line in merge() and remove tracing

lx added 1 commit 2023-08-30 18:07:27 +00:00

a1d57283c0 bucket_table: bucketparams::new doesn't need to be pub

lx added 1 commit 2023-08-30 21:24:40 +00:00

5c923d48d7 reference manual: document support for lifecycle configuration

lx added 1 commit 2023-08-30 21:27:18 +00:00

d94f1c9178 reference manual: remove obsolete caveat about multipart uploads

lx added 1 commit 2023-08-30 21:39:49 +00:00

f0a395e2e5 s3 bucket apis: remove redundant call

lx added 1 commit 2023-08-30 21:46:28 +00:00

01c327a07a lifecycle worker: avoid building chrono's serde feature

lx added 1 commit 2023-08-30 21:47:49 +00:00

5fad4c4658 update cargo.nix

lx added 1 commit 2023-08-30 21:52:21 +00:00

b2f679675e lifecycle worker: take into account disabled rules

lx added 1 commit 2023-08-30 22:00:37 +00:00

be03a4610f s3api: remove redundant serde rename attribute

lx added 1 commit 2023-08-30 22:28:50 +00:00

1cfcc61de8 lifecycle worker: mitigate potential bugs + refactoring

lx added 1 commit 2023-08-31 09:19:41 +00:00

adbf5925de lifecycle worker: use queue_insert and process objects in batches

lx added 1 commit 2023-08-31 09:25:26 +00:00

a00a52633f lifecycle worker: add log message when starting

lx added 1 commit 2023-08-31 09:30:06 +00:00

f579d6d9b4 lifecycle worker: fix potential inifinite loop

lx added 1 commit 2023-08-31 09:36:44 +00:00

1cdc321e28 lifecycle worker: don't get stuck on non-existent bucket

lx force-pushed bucket-lifecycle from 4bdd73e974 to 8e0c020bb9 2023-09-04 09:33:50 +00:00 Compare

lx changed title from WIP: object lifecycles (fix #309) to object lifecycles (fix #309) 2023-09-04 09:34:09 +00:00

lx changed title from object lifecycles (fix #309) to object lifecycles (fix #309) 2023-09-04 09:34:09 +00:00

lx merged commit 3f461d8891 into next 2023-09-04 09:45:11 +00:00

lx referenced this issue from a commit 2023-09-04 09:45:11 +00:00

Merge pull request 'object lifecycles (fix #309)' (#620) from bucket-lifecycle into next

lx deleted branch bucket-lifecycle 2023-09-04 09:45:11 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

AdminAPI

Issues concerning the administration API

  

Bug

  

Check AWS

The behaviour should be checked against AWS S3

  

CI

Issues with Nix, Drone, Rust Compiler, etc.

  

Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  

Critical

  

Documentation

  

Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  

Improvement

  

Low priority

For issues that we don't want to spend time on until made absolutely necessary

  

Newcomer

A bug that should be newcomer friendly.

  

Performance

  

S3 Compatibility

  

Testing

  

Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No Label

AdminAPI

Bug

Check AWS

CI

Correctness

Critical

Documentation

Ideas

Improvement

Low priority

Newcomer

Performance

S3 Compatibility

Testing

Usability

Milestone

Clear milestone

No items

No Milestone

v0.9

Assignees

Clear assignees

No Assignees

1 Participants

Notifications

Subscribe

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#620

No description provided.