Deuxfleurs
/
garage
42
36
Fork 55
Code Issues 64 Pull Requests 4 Releases 29 Wiki Activity

Fix #204 (full Multipart Uploads semantics) #553

Merged
lx merged 20 commits from nlnet-task1 into next 2023-06-09 15:34:10 +00:00
Conversation 0 Commits 20 Files Changed 33 +2057 -877
lx commented 2023-04-25 10:38:49 +00:00
Owner
Copy Link

Tis PR tracks progress of Task 1 of the NLnet Garage project started in April 2023, which consists in fixing Issue #204.

  • Check some basic facts against Minio

    • Parts are renumbered using sequential numbers after CompleteMultipartUpload. If I upload parts 1, 4, 5, and 6, I need to use ?partNumber=4 on GetObject to get the part that was originally uploaded with number 6.
    • ListParts fails once a multipart upload is completed
    • A part that was uploaded using UploadPart can be skipped when calling CompleteMultipartUpload, in which case it is simply not included in the final object
    • In case of skipped parts, part renumbering only counts actually included parts

  • Design changes required to Garage's internal datastructures to properly implement multipart uploads

    • When uploading a multipart upload, instead of saving all blocks in the same Version, a temporary MultipartUpload entry is created to store all uploaded parts. The MultipartUpload table is a new table added in the data model by this PR.
    • A separate Version is created by each call to UploadPart, which contains only the data uploaded for this part. The MultipartUpload contains the identifiers of all of these versions, with some metadata (part number, timestamp of upload, size, etag).
    • Crucially, the MultipartUpload can contain several version identifiers for the same part number, if they have different upload timestamp. This is what allows an already uploaded part to be reuploaded in order to fix #204. When completing the multipart object, only the Version with the latest upload timestamp is taken into account.
    • All of the Versions referenced in the MultipartUpload are short-lived, they are deleted at the same time than the MultipartUpload when it is completed or aborted.
    • When the multipart upload is completed, the Version for each part are combined into a single big Version for the entire object, which is constructed in the same way as how multipart uploads were done before. This means that no changes need to be made to the GetObject call.
    • As a consequence, entries in the Version table can now be owned by a finalized object but also by a multipart upload in progress. In terms of data model, this means that the "owner" refernce that links from a version back to its owner has to be changed to be of either of two kinds (link to object or link to MPU), and all code that uses this link has to be updated accordingly.
    • The temporary Versions will, by definition, reference the same data blocks as final object Versions. This means that when an upload is completed and the final version is created, the reference counter for each block will be decremented once (for the temporary versions that are deleted) and incremented once (for the final version that is created). The decrement is done asynchronously during cleanup, after the CompleteMultipartUpload and therefore after the increment, preventing the reference counters from going to zero.
    • Since the MultipartUpload object is temporary, so are the part numbers that it contains. This allows part renumbering (and skipping uploaded parts) to be done in CompleteMultipartUpload when the final Version is created.

  • Implement these changes and the correct multipart upload tracking

    • Data model/CRDT structures
    • Fix CreateMultipartUpload
    • Fix UploadPart
      • Delete version if part upload is interrupted
    • Fix UploadPartCopy
    • Fix CompleteMultipartUpload
    • Fix AbortMultipartUpload
    • Fix ListParts
    • Fix repair operations / new repair operations
    • Any necessary changes to other parts of Garage

  • Testing and validation

    • Multiparts uploads now work as expected
    • No regressions
    • Data is safely migrated from older Garage versions

This PR has been merged, marking the completion of Task 1

**Tis PR tracks progress of Task 1 of the NLnet Garage project started in April 2023, which consists in fixing Issue #204.** - [x] Check some basic facts against Minio - [x] Parts are renumbered using sequential numbers after CompleteMultipartUpload. If I upload parts 1, 4, 5, and 6, I need to use `?partNumber=4` on GetObject to get the part that was originally uploaded with number 6. - [x] ListParts fails once a multipart upload is completed - [x] A part that was uploaded using UploadPart can be skipped when calling CompleteMultipartUpload, in which case it is simply not included in the final object - [x] In case of skipped parts, part renumbering only counts actually included parts - [x] Design changes required to Garage's internal datastructures to properly implement multipart uploads - When uploading a multipart upload, instead of saving all blocks in the same `Version`, a temporary `MultipartUpload` entry is created to store all uploaded parts. The `MultipartUpload` table is a new table added in the data model by this PR. - A separate `Version` is created by each call to `UploadPart`, which contains only the data uploaded for this part. The `MultipartUpload` contains the identifiers of all of these versions, with some metadata (part number, timestamp of upload, size, etag). - Crucially, the `MultipartUpload` can contain several version identifiers for the same part number, if they have different upload timestamp. This is what allows an already uploaded part to be reuploaded in order to fix #204. When completing the multipart object, only the `Version` with the latest upload timestamp is taken into account. - All of the `Version`s referenced in the `MultipartUpload` are short-lived, they are deleted at the same time than the `MultipartUpload` when it is completed or aborted. - When the multipart upload is completed, the `Version` for each part are combined into a single big `Version` for the entire object, which is constructed in the same way as how multipart uploads were done before. This means that no changes need to be made to the GetObject call. - As a consequence, entries in the Version table can now be owned by a finalized object but also by a multipart upload in progress. In terms of data model, this means that the "owner" refernce that links from a version back to its owner has to be changed to be of either of two kinds (link to object or link to MPU), and all code that uses this link has to be updated accordingly. - The temporary `Version`s will, by definition, reference the same data blocks as final object `Version`s. This means that when an upload is completed and the final version is created, the reference counter for each block will be decremented once (for the temporary versions that are deleted) and incremented once (for the final version that is created). The decrement is done asynchronously during cleanup, after the CompleteMultipartUpload and therefore after the increment, preventing the reference counters from going to zero. - Since the `MultipartUpload` object is temporary, so are the part numbers that it contains. This allows part renumbering (and skipping uploaded parts) to be done in CompleteMultipartUpload when the final `Version` is created. - [x] Implement these changes and the correct multipart upload tracking - [x] Data model/CRDT structures - [x] Fix CreateMultipartUpload - [x] Fix UploadPart - [x] Delete version if part upload is interrupted - [x] Fix UploadPartCopy - [x] Fix CompleteMultipartUpload - [x] Fix AbortMultipartUpload - [x] Fix ListParts - [x] Fix repair operations / new repair operations - [x] Any necessary changes to other parts of Garage - [x] Testing and validation - [x] Multiparts uploads now work as expected - [x] No regressions - [x] Data is safely migrated from older Garage versions **This PR has been merged, marking the completion of Task 1**
lx force-pushed nlnet-task1 from 05e0848a92 to 9d464ccc96 2023-04-25 10:55:42 +00:00 Compare
lx force-pushed nlnet-task1 from 9d464ccc96 to 90303983fa 2023-04-25 10:56:21 +00:00 Compare
lx force-pushed nlnet-task1 from 90303983fa to 325aa02593 2023-04-25 15:02:18 +00:00 Compare
lx force-pushed nlnet-task1 from a80b463f6e to b41e71126a 2023-05-03 13:56:53 +00:00 Compare
lx force-pushed nlnet-task1 from 99183df1b8 to d0022b83bd 2023-05-03 17:22:11 +00:00 Compare
lx force-pushed nlnet-task1 from 10fb503cdb to daf4620eac 2023-05-04 08:10:47 +00:00 Compare
lx force-pushed nlnet-task1 from e13268c03b to ff5b1f47fe 2023-05-04 08:53:47 +00:00 Compare
lx force-pushed nlnet-task1 from ff5b1f47fe to 936d145767 2023-05-04 09:07:50 +00:00 Compare
lx force-pushed nlnet-task1 from 620623d3a6 to 911cc74469 2023-05-09 10:41:09 +00:00 Compare
lx force-pushed nlnet-task1 from 93f902729f to 9f1d1d4366 2023-05-09 11:06:51 +00:00 Compare
lx added this to the v0.9 milestone 2023-06-05 10:16:45 +00:00
lx added the
S3 Compatibility
 label 2023-06-05 10:18:21 +00:00
lx force-pushed nlnet-task1 from 9f1d1d4366 to 5cce28c84a 2023-06-06 12:36:25 +00:00 Compare
lx force-pushed nlnet-task1 from 4b75f70ddf to cc6b96eee2 2023-06-06 13:42:11 +00:00 Compare
lx force-pushed nlnet-task1 from cc6b96eee2 to 0781b35038 2023-06-06 13:43:20 +00:00 Compare
lx force-pushed nlnet-task1 from dca883efcd to 459964c3be 2023-06-09 09:54:17 +00:00 Compare
lx force-pushed nlnet-task1 from 459964c3be to c3920b97ac 2023-06-09 09:59:42 +00:00 Compare
lx force-pushed nlnet-task1 from c3920b97ac to 700b68f590 2023-06-09 10:22:50 +00:00 Compare
lx force-pushed nlnet-task1 from 700b68f590 to 5f3d4bf78c 2023-06-09 11:36:21 +00:00 Compare
lx force-pushed nlnet-task1 from 5f3d4bf78c to e645bbd3ce 2023-06-09 14:23:45 +00:00 Compare
lx added 1 commit 2023-06-09 15:13:39 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
3d477906d4 properly delete multipart uploads after completion
lx changed title from WIP: Fix #204 (full Multipart Uploads semantics) to Fix #204 (full Multipart Uploads semantics) 2023-06-09 15:33:02 +00:00
lx merged commit 0a06fda0da into next 2023-06-09 15:34:10 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
AdminAPI

Issues concerning the administration API

  
Bug

   
Check AWS

The behaviour should be checked against AWS S3

  
CI

Issues with Nix, Drone, Rust Compiler, etc.

  
Correctness

Issues of theoretical correctness that may impact edge-case scenarios

  
Critical

   
Documentation

   
Ideas

For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  
Improvement

   
Low priority

For issues that we don't want to spend time on until made absolutely necessary

  
Newcomer

A bug that should be newcomer friendly.

  
Performance

   
S3 Compatibility

   
Testing

   
Usability

Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

No Label
AdminAPI
Bug
Check AWS
CI
Correctness
Critical
Documentation
Ideas
Improvement
Low priority
Newcomer
Performance
S3 Compatibility
Testing
Usability
Milestone
Clear milestone
No items
No Milestone
v0.9
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#553
No description provided.
Delete Branch "nlnet-task1"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?