Rclone says AccessDenied: Forbidden: Invalid signature

Promotion1877 commented

2024-11-01 22:15:51 +00:00

Hi, I read all the issues, for example #700 but i am not understanding what's going on

server: 1.0.1

Created new s3 user via api
then created new bucket
assigned permissions to bucket/allow

{
  "bucketId": "xxx",
  "accessKeyId": "yyy",
  "permissions": {
    "read": true,
    "write": true,
    "owner": true
  }
}

created rclone config:

[garage]
type = s3
provider = Other
env_auth = false
access_key_id = yyy
secret_access_key = zzz
endpoint = s3.example.com
force_path_style = true
acl = private
bucket_acl = private
region = garage

but I get AccessDenied: Forbidden: Invalid signature

If i change region to something else, the error is different.

I tried to also add those extra parameters but they don't have effect:

v2_auth = false
signature_version = v4
location_constraint = garage
no_check_bucket = true

Tried also to force on v2 auth, but it replies unsupported authorization method.

With other clients everything works fine. I don't understand if I'm doing something wrong and missed something obvious in the configuration, if it's a bug of rclone v1.68.1, or if it's a bug of garage

Hi, I read all the issues, for example #700 but i am not understanding what's going on server: 1.0.1 - Created new s3 user via api - then created new bucket - assigned permissions to bucket/allow ``` { "bucketId": "xxx", "accessKeyId": "yyy", "permissions": { "read": true, "write": true, "owner": true } } ``` created rclone config: ``` [garage] type = s3 provider = Other env_auth = false access_key_id = yyy secret_access_key = zzz endpoint = s3.example.com force_path_style = true acl = private bucket_acl = private region = garage ``` but I get AccessDenied: Forbidden: Invalid signature If i change region to something else, the error is different. I tried to also add those extra parameters but they don't have effect: ``` v2_auth = false signature_version = v4 location_constraint = garage no_check_bucket = true ``` Tried also to force on v2 auth, but it replies unsupported authorization method. With other clients everything works fine. I don't understand if I'm doing something wrong and missed something obvious in the configuration, if it's a bug of rclone v1.68.1, or if it's a bug of garage

maximilien added the

action

more-info-needed

label 2024-11-02 23:08:24 +00:00

maximilien commented

2024-11-02 23:11:09 +00:00

Owner

@Promotion1877 can you give some data of what garage is logging in response to there requests?

Promotion1877 commented

2024-11-06 12:19:24 +00:00

Author

sorry, missed the notification.

Requesting

~ $ rclone ls garage: -vv
2024/11/06 12:16:20 DEBUG : rclone: Version "v1.68.1-termux" starting with parameters ["rclone" "ls" "garage:" "-vv"]
2024/11/06 12:16:20 DEBUG : Creating backend with remote "garage:"
2024/11/06 12:16:20 DEBUG : Using config file from "/data/data/com.termux/files/home/.config/rclone/rclone.conf"
2024/11/06 12:16:20 DEBUG : 5 go routines active2024/11/06 12:16:20 NOTICE: Failed to ls: operation error S3: ListBuckets, https response error StatusCode: 403, RequestID: , HostID: , api error AccessDenied: Forbidden: Invalid signature
~ $

(in this case i used rclone on my phone but it's the same error when i use it on the desktop with the same config file)

i get in the logs:

2024-11-06T12:16:19.448631Z  INFO garage_api::generic_server: x.x.x.x (via [::ffff:172.19.0.5]:41940) GET /?x-id=ListBuckets
2024-11-06T12:16:19.452231Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature

sorry, missed the notification. Requesting ``` ~ $ rclone ls garage: -vv 2024/11/06 12:16:20 DEBUG : rclone: Version "v1.68.1-termux" starting with parameters ["rclone" "ls" "garage:" "-vv"] 2024/11/06 12:16:20 DEBUG : Creating backend with remote "garage:" 2024/11/06 12:16:20 DEBUG : Using config file from "/data/data/com.termux/files/home/.config/rclone/rclone.conf" 2024/11/06 12:16:20 DEBUG : 5 go routines active2024/11/06 12:16:20 NOTICE: Failed to ls: operation error S3: ListBuckets, https response error StatusCode: 403, RequestID: , HostID: , api error AccessDenied: Forbidden: Invalid signature ~ $ ``` (in this case i used rclone on my phone but it's the same error when i use it on the desktop with the same config file) i get in the logs: ``` 2024-11-06T12:16:19.448631Z INFO garage_api::generic_server: x.x.x.x (via [::ffff:172.19.0.5]:41940) GET /?x-id=ListBuckets 2024-11-06T12:16:19.452231Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature ```

vk commented

2024-11-06 17:11:47 +00:00

Shouldn't the rclone command line specify the bucket name? Ie. rclone ls garage:xxx -vv ?

Shouldn't the rclone command line specify the bucket name? Ie. `rclone ls garage:xxx -vv` ?

Promotion1877 commented

2024-11-07 09:37:19 +00:00

Author

I get the same result with the bucket name specified

from x.x.x.x:3903/v1/bucket?list

[
  {
    "id": "af9dd92d6xxxxxxxx",
    "globalAliases": [
      "pixel7"
    ],
    "localAliases": []
  }
]

command rclone ls garage:pixel7 -vv

in the logs:

2024-11-07T09:34:01.209567Z  INFO garage_api::generic_server: x.x.x.x (via [::ffff:172.19.0.5]:60116) GET /pixel7?delimiter=&max-keys=1000&prefix=
2024-11-07T09:34:01.209759Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature

edit: also tried to use the id instead of alias but i get

2024-11-07T09:40:56.430784Z  INFO garage_api::generic_server: x.x.x.x (via [::ffff:172.19.0.5]:32786) GET /af9dd92d6xxxxxxxx?delimiter=&max-keys=1000&prefix=
2024-11-07T09:40:56.431115Z  INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature

I get the same result with the bucket name specified from `x.x.x.x:3903/v1/bucket?list` ``` [ { "id": "af9dd92d6xxxxxxxx", "globalAliases": [ "pixel7" ], "localAliases": [] } ] ``` command `rclone ls garage:pixel7 -vv` in the logs: ``` 2024-11-07T09:34:01.209567Z INFO garage_api::generic_server: x.x.x.x (via [::ffff:172.19.0.5]:60116) GET /pixel7?delimiter=&max-keys=1000&prefix= 2024-11-07T09:34:01.209759Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature ``` edit: also tried to use the id instead of alias but i get ``` 2024-11-07T09:40:56.430784Z INFO garage_api::generic_server: x.x.x.x (via [::ffff:172.19.0.5]:32786) GET /af9dd92d6xxxxxxxx?delimiter=&max-keys=1000&prefix= 2024-11-07T09:40:56.431115Z INFO garage_api::generic_server: Response: error 403 Forbidden, Forbidden: Invalid signature ```

vk commented

2024-11-10 10:08:13 +00:00

This looks like a misconfiguration somewhere. What does garage key info <keyname> say? Does it have access to the pixel7 bucket?

This looks like a misconfiguration somewhere. What does `garage key info <keyname>` say? Does it have access to the `pixel7` bucket?

Promotion1877 commented

2024-11-13 10:51:48 +00:00

Author

yes it looks like it has access. With winscp i can login to this bucket using the same key with no issue

:~# docker exec garage-1 /garage key info pixel7
2024-11-13T10:49:32.089437Z  INFO garage_net::netapp: Connected to 127.0.0.1:3901, negotiating handshake...    
2024-11-13T10:49:32.132023Z  INFO garage_net::netapp: Connection established to xxx    
Key name: pixel7
Key ID: GKxxxx
Secret key: (redacted)
Can create buckets: false

Key-specific bucket aliases:

Authorized buckets:
  RWO  pixel7    af9dd92d6xxxxxx

yes it looks like it has access. With winscp i can login to this bucket using the same key with no issue ``` :~# docker exec garage-1 /garage key info pixel7 2024-11-13T10:49:32.089437Z INFO garage_net::netapp: Connected to 127.0.0.1:3901, negotiating handshake... 2024-11-13T10:49:32.132023Z INFO garage_net::netapp: Connection established to xxx Key name: pixel7 Key ID: GKxxxx Secret key: (redacted) Can create buckets: false Key-specific bucket aliases: Authorized buckets: RWO pixel7 af9dd92d6xxxxxx ```

vk commented

2024-11-14 12:47:17 +00:00

I just noticed this in one of your logs above:

2024/11/06 12:16:20 DEBUG : 5 go routines active2024/11/06 12:16:20 NOTICE: Failed to ls: operation error S3: ListBuckets, https response error StatusCode: 403, RequestID: , HostID: , api error AccessDenied: Forbidden: Invalid signature

Are you using an https proxy (this could explain signature failure, as the http host header is part of it)?

I'd suggest that you paste all the configurations here (rclone, proxy if any, eventually garage.toml as well), as unmodified/unobfuscated as possible.

I just noticed this in one of your logs above: > 2024/11/06 12:16:20 DEBUG : 5 go routines active2024/11/06 12:16:20 NOTICE: Failed to ls: operation error S3: ListBuckets, **https** response error StatusCode: 403, RequestID: , HostID: , api error AccessDenied: Forbidden: Invalid signature Are you using an https proxy (this could explain signature failure, as the http `host` header is part of it)? I'd suggest that you paste all the configurations here (rclone, proxy if any, eventually `garage.toml` as well), as unmodified/unobfuscated as possible.

👍 1

Promotion1877 commented

2024-11-23 18:06:54 +00:00

Author

Hi, I investigated more after you said: I directly connected to port 3900 http (no SSL) and rclone commands worked without this signature problem.

So it might be the issue of the proxy server? I'm using cloudflared in the docker compose as a proxy server

But with aws cli, winscp and cyberduck i don't get any problem with the same config

Hi, I investigated more after you said: I directly connected to port 3900 http (no SSL) and rclone commands worked without this signature problem. So it might be the issue of the proxy server? I'm using cloudflared in the docker compose as a proxy server ![image](/attachments/1074b156-0bc0-4eeb-8c49-440fd964fd57) But with aws cli, winscp and cyberduck i don't get any problem with the same config

image.png

41 KiB

vk commented

2024-11-24 10:58:05 +00:00

The signature uses the (http) hostname.

rclone computes the signature using the hostname in the endpoint setting ("s3.example.com" in your rclone.conf above). If your proxy server provides a different hostname to garage (~~which seems to be "garage" in the above screenshot~~ actually, your screenshot seems to indicate you're explicitly setting the "HTTP Host Header"), then garage computes a different signature, resulting in a signature mismatch.

The signature uses the (http) hostname. `rclone` computes the signature using the hostname in the `endpoint` setting ("s3.example.com" in your rclone.conf above). If your proxy server provides a different hostname to `garage` (~~which seems to be "garage" in the above screenshot~~ actually, your screenshot seems to indicate you're explicitly setting the "HTTP Host Header"), then `garage` computes a different signature, resulting in a signature mismatch.

❤️ 1

Promotion1877 commented

2024-12-03 00:35:41 +00:00

Author

I changed to use caddy as a reverse proxy and still get the signature error

Caddyfile (I copy/pasted and used it verbatim from documentation):

s3.example.com {
        reverse_proxy garage:3900 {
        health_uri       /health
        health_port      3903
        #health_interval 15s
        #health_timeout  5s
    }

garage.toml

[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
root_domain = "s3.example.com"

I'm not understanding what's wrong

I changed to use caddy as a reverse proxy and still get the signature error Caddyfile (I copy/pasted and used it verbatim from documentation): ``` s3.example.com { reverse_proxy garage:3900 { health_uri /health health_port 3903 #health_interval 15s #health_timeout 5s } ``` garage.toml ``` [s3_api] s3_region = "garage" api_bind_addr = "[::]:3900" root_domain = "s3.example.com" ``` I'm not understanding what's wrong

vk commented

2024-12-05 18:06:38 +00:00

Try adding header_up Host hostname to your caddy configuration (within reverse_proxy {} block), where hostname matches the hostname used in rclone configuration endpoint directive.

Try adding `header_up Host hostname` to your caddy configuration (within `reverse_proxy {}` block), where `hostname` matches the hostname used in rclone configuration `endpoint` directive.

Rclone says AccessDenied: Forbidden: Invalid signature #895