Deuxfleurs/garage

Fork 73

Full compatibility with S3 multipart uploads #204

New issue

Closed

opened 2022-01-25 11:22:25 +00:00 by lx · 2 comments

lx commented

2022-01-25 11:22:25 +00:00

Owner

Task 1 of the NLnet Garage project started in April 2023 consists in fixing this issue. Progress on this task is tracked in #553.

See #197 and #198.

We are missing the following features:

Ability to re-upload a part that has been uploaded, by re-calling UploadPart. It should be silently overwritten:

If you upload a new part using the same part number that was used with a previous part, the previously uploaded part is overwritten.

https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html
Ability to drop some of the uploaded parts by not including them in the CompleteMultipartUpload call
On CompleteMultipartUpload, renumbering of parts with consecutive numbers starting at 1 (necessary so that the partNumber parameter in GetObject and HeadObject works correctly).

These features can only be implemented by fundamentally reworking the data model of Garage, meaning it's kind of a big change we aren't going to do just now.

For now, since renumbering is not supported, and to ensure that partNumber works correctly, the following restriction is implemented:

The uploaded parts must have consecutive numbers starting at 1.

Moreover, the following restrictions apply:

All uploaded parts must be used in CompleteMultipartUpload.
In case one of the parts upload went wrong, you have to abort the multipart upload and start over.

**Task 1 of the NLnet Garage project started in April 2023 consists in fixing this issue. Progress on this task is tracked in #553.** See #197 and #198. We are missing the following features: - Ability to re-upload a part that has been uploaded, by re-calling UploadPart. It should be silently overwritten: > If you upload a new part using the same part number that was used with a previous part, the previously uploaded part is overwritten. https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html - Ability to drop some of the uploaded parts by not including them in the CompleteMultipartUpload call - On CompleteMultipartUpload, renumbering of parts with consecutive numbers starting at 1 (necessary so that the `partNumber` parameter in GetObject and HeadObject works correctly). These features can only be implemented by fundamentally reworking the data model of Garage, meaning it's kind of a big change we aren't going to do just now. For now, since renumbering is not supported, and to ensure that `partNumber` works correctly, the following restriction is implemented: - The uploaded parts must have consecutive numbers starting at 1. Moreover, the following restrictions apply: - All uploaded parts must be used in CompleteMultipartUpload. - In case one of the parts upload went wrong, you have to abort the multipart upload and start over.

lx added the

scope

s3-api

label 2022-01-25 11:25:59 +00:00

lx referenced this issue

2022-01-25 11:26:12 +00:00

Incorrect behavior on UploadPart when a part already exists #197

lx referenced this issue

2022-01-25 11:26:24 +00:00

Incorrect behavior on CompleteMultipartUpload when some parts are unused #198

trinity-1686a commented

2022-01-25 13:20:13 +00:00

Owner

In case one of the parts upload went wrong, you have to abort the multipart upload and start over.

I think this is the most annoying part of this ticket. It's one of the main selling point of multipart upload with higher bandwidth usage. From Amazon documentation :

If you're uploading over a spotty network, use multipart upload to increase resiliency to network errors by avoiding upload restarts. When using multipart upload, you need to retry uploading only parts that are interrupted during the upload. You don't need to restart uploading your object from the beginning.

> - In case one of the parts upload went wrong, you have to abort the multipart upload and start over. I think this is the most annoying part of this ticket. It's one of the main selling point of multipart upload with higher bandwidth usage. From [Amazon documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html) : > If you're uploading over a spotty network, use multipart upload to increase resiliency to network errors by avoiding upload restarts. When using multipart upload, you need to retry uploading only parts that are interrupted during the upload. You don't need to restart uploading your object from the beginning.

👍 2

quentin referenced this issue

2022-08-03 14:15:09 +00:00

x-amz-checksum-algorithm for multipart uploads #347

quentin commented

2022-08-12 08:09:43 +00:00

Owner

We observed one of the undesired behaviour in our CI during a nightly build.
Some artefacts failed to upload as a part upload was interrupted and couldn't be resumed by Garage.

$ nix-shell --attr release --run "to_s3"

upload failed: result/bin/garage to s3://garagehq.deuxfleurs.fr/_releases/8cd02639dc688dcb736b5c36dae822706862fac1/x86_64-unknown-linux-musl/garage An error occurred (InvalidRequest) when calling the UploadPart operation: Bad request: Part number 4 has already been uploaded

We observed one of the undesired behaviour in our CI during a nightly build. Some artefacts failed to upload as a part upload was interrupted and couldn't be resumed by Garage. ``` $ nix-shell --attr release --run "to_s3" upload failed: result/bin/garage to s3://garagehq.deuxfleurs.fr/_releases/8cd02639dc688dcb736b5c36dae822706862fac1/x86_64-unknown-linux-musl/garage An error occurred (InvalidRequest) when calling the UploadPart operation: Bad request: Part number 4 has already been uploaded ```

lx added this to the v1.0 milestone 2022-10-16 19:09:57 +00:00

lx modified the milestone from v1.0 to v0.9

2022-11-10 10:45:14 +00:00

lx self-assigned this 2022-12-11 17:45:27 +00:00

lx referenced this issue from a pull request that will close it,

2023-04-25 10:38:49 +00:00

Fix #204 (full Multipart Uploads semantics) #553

lx closed this issue

2023-06-09 15:34:10 +00:00

lx referenced this issue from a commit

2023-06-09 15:34:10 +00:00

Merge pull request 'Fix #204 (full Multipart Uploads semantics)' (#553) from nlnet-task1 into next