Fix #204 (full Multipart Uploads semantics) #553

Merged

lx merged 20 commits from nlnet-task1 into next

2023-06-09 15:34:10 +00:00

Conversation 0 Commits 20 Files changed 33 +2057 -877

lx commented

2023-04-25 10:38:49 +00:00

Owner

Copy link

Tis PR tracks progress of Task 1 of the NLnet Garage project started in April 2023, which consists in fixing Issue #204.

Check some basic facts against Minio
- Parts are renumbered using sequential numbers after CompleteMultipartUpload. If I upload parts 1, 4, 5, and 6, I need to use ?partNumber=4 on GetObject to get the part that was originally uploaded with number 6.
- ListParts fails once a multipart upload is completed
- A part that was uploaded using UploadPart can be skipped when calling CompleteMultipartUpload, in which case it is simply not included in the final object
- In case of skipped parts, part renumbering only counts actually included parts
Design changes required to Garage's internal datastructures to properly implement multipart uploads
- When uploading a multipart upload, instead of saving all blocks in the same Version, a temporary MultipartUpload entry is created to store all uploaded parts. The MultipartUpload table is a new table added in the data model by this PR.
- A separate Version is created by each call to UploadPart, which contains only the data uploaded for this part. The MultipartUpload contains the identifiers of all of these versions, with some metadata (part number, timestamp of upload, size, etag).
- Crucially, the MultipartUpload can contain several version identifiers for the same part number, if they have different upload timestamp. This is what allows an already uploaded part to be reuploaded in order to fix #204. When completing the multipart object, only the Version with the latest upload timestamp is taken into account.
- All of the Versions referenced in the MultipartUpload are short-lived, they are deleted at the same time than the MultipartUpload when it is completed or aborted.
- When the multipart upload is completed, the Version for each part are combined into a single big Version for the entire object, which is constructed in the same way as how multipart uploads were done before. This means that no changes need to be made to the GetObject call.
- As a consequence, entries in the Version table can now be owned by a finalized object but also by a multipart upload in progress. In terms of data model, this means that the "owner" refernce that links from a version back to its owner has to be changed to be of either of two kinds (link to object or link to MPU), and all code that uses this link has to be updated accordingly.
- The temporary Versions will, by definition, reference the same data blocks as final object Versions. This means that when an upload is completed and the final version is created, the reference counter for each block will be decremented once (for the temporary versions that are deleted) and incremented once (for the final version that is created). The decrement is done asynchronously during cleanup, after the CompleteMultipartUpload and therefore after the increment, preventing the reference counters from going to zero.
- Since the MultipartUpload object is temporary, so are the part numbers that it contains. This allows part renumbering (and skipping uploaded parts) to be done in CompleteMultipartUpload when the final Version is created.
Implement these changes and the correct multipart upload tracking
- Data model/CRDT structures
- Fix CreateMultipartUpload
- Fix UploadPart
  - Delete version if part upload is interrupted
- Fix UploadPartCopy
- Fix CompleteMultipartUpload
- Fix AbortMultipartUpload
- Fix ListParts
- Fix repair operations / new repair operations
- Any necessary changes to other parts of Garage
Testing and validation
- Multiparts uploads now work as expected
- No regressions
- Data is safely migrated from older Garage versions

This PR has been merged, marking the completion of Task 1

**Tis PR tracks progress of Task 1 of the NLnet Garage project started in April 2023, which consists in fixing Issue #204.** - [x] Check some basic facts against Minio - [x] Parts are renumbered using sequential numbers after CompleteMultipartUpload. If I upload parts 1, 4, 5, and 6, I need to use `?partNumber=4` on GetObject to get the part that was originally uploaded with number 6. - [x] ListParts fails once a multipart upload is completed - [x] A part that was uploaded using UploadPart can be skipped when calling CompleteMultipartUpload, in which case it is simply not included in the final object - [x] In case of skipped parts, part renumbering only counts actually included parts - [x] Design changes required to Garage's internal datastructures to properly implement multipart uploads - When uploading a multipart upload, instead of saving all blocks in the same `Version`, a temporary `MultipartUpload` entry is created to store all uploaded parts. The `MultipartUpload` table is a new table added in the data model by this PR. - A separate `Version` is created by each call to `UploadPart`, which contains only the data uploaded for this part. The `MultipartUpload` contains the identifiers of all of these versions, with some metadata (part number, timestamp of upload, size, etag). - Crucially, the `MultipartUpload` can contain several version identifiers for the same part number, if they have different upload timestamp. This is what allows an already uploaded part to be reuploaded in order to fix #204. When completing the multipart object, only the `Version` with the latest upload timestamp is taken into account. - All of the `Version`s referenced in the `MultipartUpload` are short-lived, they are deleted at the same time than the `MultipartUpload` when it is completed or aborted. - When the multipart upload is completed, the `Version` for each part are combined into a single big `Version` for the entire object, which is constructed in the same way as how multipart uploads were done before. This means that no changes need to be made to the GetObject call. - As a consequence, entries in the Version table can now be owned by a finalized object but also by a multipart upload in progress. In terms of data model, this means that the "owner" refernce that links from a version back to its owner has to be changed to be of either of two kinds (link to object or link to MPU), and all code that uses this link has to be updated accordingly. - The temporary `Version`s will, by definition, reference the same data blocks as final object `Version`s. This means that when an upload is completed and the final version is created, the reference counter for each block will be decremented once (for the temporary versions that are deleted) and incremented once (for the final version that is created). The decrement is done asynchronously during cleanup, after the CompleteMultipartUpload and therefore after the increment, preventing the reference counters from going to zero. - Since the `MultipartUpload` object is temporary, so are the part numbers that it contains. This allows part renumbering (and skipping uploaded parts) to be done in CompleteMultipartUpload when the final `Version` is created. - [x] Implement these changes and the correct multipart upload tracking - [x] Data model/CRDT structures - [x] Fix CreateMultipartUpload - [x] Fix UploadPart - [x] Delete version if part upload is interrupted - [x] Fix UploadPartCopy - [x] Fix CompleteMultipartUpload - [x] Fix AbortMultipartUpload - [x] Fix ListParts - [x] Fix repair operations / new repair operations - [x] Any necessary changes to other parts of Garage - [x] Testing and validation - [x] Multiparts uploads now work as expected - [x] No regressions - [x] Data is safely migrated from older Garage versions **This PR has been merged, marking the completion of Task 1**

lx referenced this pull request

2023-04-25 10:39:11 +00:00

Full compatibility with S3 multipart uploads #204

lx force-pushed nlnet-task1 from 05e0848a92 to 9d464ccc96

2023-04-25 10:55:42 +00:00

Compare

lx force-pushed nlnet-task1 from 9d464ccc96 to 90303983fa

2023-04-25 10:56:21 +00:00

Compare

lx force-pushed nlnet-task1 from 90303983fa to 325aa02593

2023-04-25 15:02:18 +00:00

Compare

lx force-pushed nlnet-task1 from a80b463f6e to b41e71126a

2023-05-03 13:56:53 +00:00

Compare

lx force-pushed nlnet-task1 from 99183df1b8 to d0022b83bd

2023-05-03 17:22:11 +00:00

Compare

lx force-pushed nlnet-task1 from 10fb503cdb to daf4620eac

2023-05-04 08:10:47 +00:00

Compare

lx force-pushed nlnet-task1 from e13268c03b to ff5b1f47fe

2023-05-04 08:53:47 +00:00

Compare

lx force-pushed nlnet-task1 from ff5b1f47fe to 936d145767

2023-05-04 09:07:50 +00:00

Compare

lx force-pushed nlnet-task1 from 620623d3a6 to 911cc74469

2023-05-09 10:41:09 +00:00

Compare

lx force-pushed nlnet-task1 from 93f902729f to 9f1d1d4366

2023-05-09 11:06:51 +00:00

Compare

lx added this to the v0.9 milestone 2023-06-05 10:16:45 +00:00

lx added the