Deuxfleurs/garage
44
56
Fork 73
Code Issues 100 Pull requests 14 Releases 30 Wiki Activity

WIP: feat: x-amz-website-redirect-location support #872

Draft
raitobezarius wants to merge 5 commits from raitobezarius/garage:redirect-meta into main
pull from: raitobezarius/garage:redirect-meta
merge into: Deuxfleurs:main
Conversation 1 Commits 5 Files changed 11 +230 -15
raitobezarius commented 2024-09-03 13:48:01 +00:00
First-time contributor
Copy link

Each commit contains a piece of an implementation.

Fixes #869.

Open questions

  • Do we want to validate the header field?
  • What is the behavior of the flattening in case of an HTTP(S) redirect instead of a bucket local redirect?
  • How to handle infinite loops? Disallow them? Bound the number of flattens?

GET

This is an implementation of the GET handler for the
x-amz-website-redirect-location.

This implements the semantics described in
https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-page-redirect.html
for the S3 API and the web API.

PUT

Just propagate the header.

PUT Copy

In case the metadata directive is REPLACE, we reuse the PUT propagated header.
In case the metadata directive is COPY, we probably need to check if the header is set and actually replace the header.

TODO:

  • add an integration test for PUT copy.

Multipart

TODO:

  • add an integration test for 2 multipart uploads, one redirecting the other.
  • add an integration test for 2 multipart upload copy, one redirecting the other.

Signed-off-by: Raito Bezarius masterancpp@gmail.com

Each commit contains a piece of an implementation. Fixes https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/869. ## Open questions - Do we want to validate the header field? - Amazon write in https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html that the length is 2 KB at most and needs to start by `/`, `http://` or `https://` - Should we validate if the target of redirection exist or is it OK to create speculative redirections and leave them as 404? - What is the behavior of the flattening in case of an HTTP(S) redirect instead of a bucket local redirect? - How to handle infinite loops? Disallow them? Bound the number of flattens? ### GET This is an implementation of the GET handler for the `x-amz-website-redirect-location`. This implements the semantics described in https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-page-redirect.html for the S3 API and the web API. ### PUT Just propagate the header. ### PUT Copy In case the metadata directive is REPLACE, we reuse the PUT propagated header. In case the metadata directive is COPY, we probably need to check if the header is set and actually *replace* the header. TODO: - [ ] add an integration test for PUT copy. ### Multipart TODO: - [ ] add an integration test for 2 multipart uploads, one redirecting the other. - [ ] add an integration test for 2 multipart upload copy, one redirecting the other. Signed-off-by: Raito Bezarius <masterancpp@gmail.com>
raitobezarius force-pushed redirect-meta from bd44f702f2 to 5275ae60b0 2024-09-03 15:00:28 +00:00 Compare
lx requested changes 2024-09-05 14:14:57 +00:00
lx left a comment
Owner
Copy link

I have a feeling that this does not implement the expected semantics of x-amz-website-redirect-location. As I understand by reading the AWS docs, when the header is present:

  • Fetching the object through the S3 API is not affected, the object is always returned as it exists. Its x-amz-website-redirect-location header is simply added to the HTTP headers of the response

  • Accessing the object as a web page through the web endpoint would yield an HTTP 301 with the Location header set to the value of x-amz-website-redirect-location

I think that there is no case in which S3 internally interprets the redirection, and returns the targeted object instead of the original one. In particular, this means that there is no case where multiple layers of redirection are flattened internally by S3 (and we can get rid of async_recursion).

This is not certain but it's only my interpretation of the AWS documentation, which is very badly written. What we need to do here is to test against AWS S3 to see how it behaves, and then emulate that behavior.

I have a feeling that this does not implement the expected semantics of `x-amz-website-redirect-location`. As I understand by reading the AWS docs, when the header is present: - Fetching the object through the S3 API *is not affected*, the object is always returned as it exists. Its `x-amz-website-redirect-location` header is simply added to the HTTP headers of the response - Accessing the object as a web page through the web endpoint would yield an HTTP 301 with the `Location` header set to the value of `x-amz-website-redirect-location` I think that there is no case in which S3 internally interprets the redirection, and returns the targeted object instead of the original one. In particular, this means that there is no case where multiple layers of redirection are flattened internally by S3 (and we can get rid of async_recursion). This is not certain but it's only my interpretation of the AWS documentation, which is very badly written. What we need to do here is to test against AWS S3 to see how it behaves, and then emulate that behavior.
src/api/s3/get.rs
@ -329,6 +332,30 @@ pub async fn handle_get_without_ctx(
let checksum_mode = checksum_mode(&req);
if let Some(redirect_hdr) = headers
lx commented 2024-09-05 14:07:00 +00:00
Owner
Copy link

We want this code path to trigger only for the web endpoint, x-amz-website-redirect-location should be ignored (i.e. simply returned as a normal object header) on the S3 endpoint.

We want this code path to trigger only for the web endpoint, `x-amz-website-redirect-location` should be ignored (i.e. simply returned as a normal object header) on the S3 endpoint.
src/api/s3/get.rs
@ -332,0 +335,4 @@
if let Some(redirect_hdr) = headers
.headers
.iter()
.find(|(k, _)| k == "x-amz-website-redirect-location")
lx commented 2024-09-05 13:58:13 +00:00
Owner
Copy link

Could we define a constant as follows and use that instead?

pub const X_AMZ_WEBSITE_REDIRECT_LOCATION: HeaderName = HeaderName::from_static("x-amz-website-redirect-location");
Could we define a constant as follows and use that instead? ```rust pub const X_AMZ_WEBSITE_REDIRECT_LOCATION: HeaderName = HeaderName::from_static("x-amz-website-redirect-location"); ```
src/api/s3/put.rs
@ -623,1 +623,3 @@
if name.as_str().starts_with("x-amz-meta-") {
if name.as_str().starts_with("x-amz-meta-")
|| name.as_str() == "x-amz-website-redirect-location"
{
lx commented 2024-09-05 13:59:15 +00:00
Owner
Copy link

The value of the header should be validated here to ensure it is syntactically correct, according to S3 rules

The value of the header should be validated here to ensure it is syntactically correct, according to S3 rules
lx commented 2024-09-05 14:17:48 +00:00
Owner
Copy link

Amazon write in https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html that the length is 2 KB at most and needs to start by /, http:// or https://

I think we should validate this as well.

Should we validate if the target of redirection exist or is it OK to create speculative redirections and leave them as 404?

I would expect that Amazon does not validate this, so I don't think we should validate it. We can test this on AWS if we want to be sure.

> Amazon write in https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOST.html that the length is 2 KB at most and needs to start by /, http:// or https:// I think we should validate this as well. > Should we validate if the target of redirection exist or is it OK to create speculative redirections and leave them as 404? I would expect that Amazon does not validate this, so I don't think we should validate it. We can test this on AWS if we want to be sure.
Some checks failed
ci/woodpecker/pr/debug Pipeline failed
Required
Details
This pull request has changes conflicting with the target branch.
  • Cargo.nix
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u redirect-meta:raitobezarius-redirect-meta
git checkout raitobezarius-redirect-meta
Sign in to join this conversation.
Reviewers
No reviewers
lx
Labels
Clear labels   
action
check-aws
The behaviour should be checked against AWS S3

  
action
discussion-needed
We need to discuss if the feature is desirable and/or better specify the envisioned behavior or technical implementation

  
action
for-external-contributors
Core team does not plan working on this in the near future however an external contribution will be reviewed & merged

  
action
for-newcomers
An issue that should be newcomer friendly.

  
action
more-info-needed
More information is needed to investigate this issue

  
action
need-funding
This issue represents lots of work, we need to find money to pay our core team work

  
action
triage-required
It's not very clear what we want to do with this issue yet, more thoughts is required

  
kind
correctness
Issues of theoretical correctness that may impact edge-case scenarios

  
kind
ideas
For abstract ideas/proposal to make Garage better, to better communicate our not yet achieved aims

  
kind
improvement
Feature requests that do not fall in other categories

  
kind
performance
Deceptive performance observed OR idea to improve performances

  
kind
testing
Functional tests, integration tests, Jepsen, etc.

  
kind
usability
Propositions to make Garage easier to operate, by improving error reporting, CLI, etc.

  
kind
wrong-behavior
Unintended behavior

  
prio
critical
Could impact integrity or availability of production clusters

  
prio
low
For issues that we don't want to spend time on until made absolutely necessary

  
scope
admin-api
Issues concerning the administration API

  
scope
background-healing
Background healing covers repair tasks, anti-entropy, merkle trees, etc. ; everything related to replicating data appropriately.

  
scope
build
Issues with Nix, Drone, Rust Compiler, etc.

  
scope
documentation

  
scope
k8s
Kubernetes related, including Helm Charts

  
scope
layout
Linked to the cluster layout and storage management

  
scope
metadata
Metadata Engine (LMDB, sqlite, etc.)

  
scope
ops
Operations related to operations/maintenance issue, tooling related (like CLI missing info ; impossible, too dangerous, too complicated migration, etc.)

  
scope
rpc
Linked to RPC

  
scope
s3-api
Related to the S3 API (especially compatibility issues).

  
scope
security
Cryptography-related, auth-related, authorization-related issues

  
scope
telemetry
OpenTelemetry (traces, metrics, etc.)

No labels
action
check-aws
action
discussion-needed
action
for-external-contributors
action
for-newcomers
action
more-info-needed
action
need-funding
action
triage-required
kind
correctness
kind
ideas
kind
improvement
kind
performance
kind
testing
kind
usability
kind
wrong-behavior
prio
critical
prio
low
scope
admin-api
scope
background-healing
scope
build
scope
documentation
scope
k8s
scope
layout
scope
metadata
scope
ops
scope
rpc
scope
s3-api
scope
security
scope
telemetry
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#872
No description provided.
Delete branch "raitobezarius/garage:redirect-meta"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?