Unclear how to troubleshoot connection to Docker image #801

Open
opened 2024-04-09 19:23:16 +00:00 by DougReeder · 5 comments

As my dev machine runs MacOS rather than linux, I tried following the instructions at https://garagehq.deuxfleurs.fr/documentation/quick-start/ but substituting the Docker image for a binary.

The last config file I tried is

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"

replication_mode = "none"

rpc_bind_addr = "127.0.0.1:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "xxxxxxx"

[s3_api]
s3_region = "garage"
api_bind_addr = "127.0.0.1:3900"
root_domain = ".s3.garage.localhost"

[s3_web]
bind_addr = "127.0.0.1:3902"
root_domain = ".web.garage.localhost"
index = "index.html"

[k2v_api]
api_bind_addr = "[::]:3904"

[admin]
api_bind_addr = "127.0.0.1:3903"
admin_token = "xxxxx"
metrics_token = "xxxxxxx"

I tried several variations on network addresses, but they all suffer the problem below.

/etc/garage.toml, /var/lib/garage/meta and /var/lib/garage/data on my dev machine are all owned by me.

I can create a container so:

docker run \
  -d \
  --name garaged \
  --restart always \
  --network host \
  -v /etc/garage.toml:/etc/garage.toml \
  -v /var/lib/garage/meta:/var/lib/garage/meta \
  -v /var/lib/garage/data:/var/lib/garage/data \
  dxflrs/garage:v0.9.4

The image starts up and the log includes

2024-04-09 14:36:39 2024-04-09T18:36:39.513469Z  INFO garage::server: Initializing background runner...
2024-04-09 14:36:39 2024-04-09T18:36:39.513504Z  INFO garage::server: Spawning Garage workers...
2024-04-09 14:36:39 2024-04-09T18:36:39.513777Z  INFO garage::server: Initialize Admin API server and metrics collector...
2024-04-09 14:36:39 2024-04-09T18:36:39.581100Z  INFO garage::server: Launching internal Garage cluster communications...
2024-04-09 14:36:39 2024-04-09T18:36:39.581188Z  INFO garage::server: Create admin RPC handler...
2024-04-09 14:36:39 2024-04-09T18:36:39.581202Z  INFO garage::server: Initializing S3 API server...
2024-04-09 14:36:39 2024-04-09T18:36:39.581205Z  INFO garage::server: Initializing K2V API server...
2024-04-09 14:36:39 2024-04-09T18:36:39.581208Z  INFO garage::server: Initializing web server...
2024-04-09 14:36:39 2024-04-09T18:36:39.581218Z  INFO garage::server: Launching Admin API server...
2024-04-09 14:36:39 2024-04-09T18:36:39.581287Z  INFO garage_net::netapp: Listening on 127.0.0.1:3901    
2024-04-09 14:36:39 2024-04-09T18:36:39.581316Z  INFO garage_api::generic_server: S3 API server listening on http://127.0.0.1:3900
2024-04-09 14:36:39 2024-04-09T18:36:39.581381Z  INFO garage_web::web_server: Web server listening on http://127.0.0.1:3902
2024-04-09 14:36:39 2024-04-09T18:36:39.581586Z  INFO garage_api::generic_server: K2V API server listening on http://[::]:3904
2024-04-09 14:36:39 2024-04-09T18:36:39.581836Z  INFO garage_api::generic_server: Admin API server listening on http://127.0.0.1:3903

so it appears to running properly.

I create an alias so: alias garage="docker exec -ti garaged /garage"

and garage status returns

2024-04-09T19:08:18.467817Z  INFO garage_net::netapp: Connected to 127.0.0.1:3901, negotiating handshake...    
2024-04-09T19:08:18.509152Z  INFO garage_net::netapp: Connection established to d08e1c7b9177d649    
==== HEALTHY NODES ====
ID                Hostname        Address         Tags  Zone  Capacity   DataAvail
d08e1c7b9177d649  docker-desktop  127.0.0.1:3901  []    dc1   1000.0 MB  3.2 TB (79.5%)

I can create and apply a cluster layout:

$ garage layout apply --version 2
2024-04-09T19:14:36.866461Z  INFO garage_net::netapp: Connected to 127.0.0.1:3901, negotiating handshake...    
2024-04-09T19:14:36.908111Z  INFO garage_net::netapp: Connection established to d08e1c7b9177d649    
==== COMPUTATION OF A NEW PARTITION ASSIGNATION ====

Partitions are replicated 1 times on at least 1 distinct zones.

Optimal partition size:                     3.9 MB (3.9 MB in previous layout)
Usable capacity / total cluster capacity:   1000.0 MB / 1000.0 MB (100.0 %)
Effective capacity (replication factor 1):  1000.0 MB

A total of 0 new copies of partitions need to be transferred.

dc1                 Tags  Partitions        Capacity   Usable capacity
  d08e1c7b9177d649        256 (0 new)       1000.0 MB  1000.0 MB (100.0%)
  TOTAL                   256 (256 unique)  1000.0 MB  1000.0 MB (100.0%)


New cluster layout with updated role assignment has been applied in cluster.
Data will now be moved around between nodes accordingly.

I can use the command line to create a bucket and API key, and give the key permission to use the bucket:

$ garage bucket info cl-first
2024-04-09T19:10:00.595340Z  INFO garage_net::netapp: Connected to 127.0.0.1:3901, negotiating handshake...    
2024-04-09T19:10:00.636526Z  INFO garage_net::netapp: Connection established to d08e1c7b9177d649    
Bucket: 6bd35158ce746f19bdde4e00d57d7ae00c74cfe6f5590b1d05ec264ade49c034

Size: 0 B (0 B)
Objects: 0
Unfinished uploads (multipart and non-multipart): 0
Unfinished multipart uploads: 0
Size of unfinished multipart uploads: 0 B (0 B)

Website access: false

Global aliases:
  cl-first

Key-specific aliases:

Authorized keys:
  RWO  GK5bec374936673b0b236b1618  cl-second

I can set environment variables for AWSCLI:

AWS_ENDPOINT_URL=http://localhost:3900
AWS_DEFAULT_REGION=garage
AWS_SECRET_ACCESS_KEY=xxxxx
AWS_ACCESS_KEY_ID=xxxxx

But AWSCLI can't access Garage via the S3 API:

$ aws s3 ls

Could not connect to the endpoint URL: "http://localhost:3900/"

With appropriate configuration, I can use AWSCLI to access a Min.IO server running locally.

Also, I have a Node.JS app, which, when properly configured, can access the Min.IO server. When I use the AWS configuration above for Garage, I get:

Error: connect ECONNREFUSED 127.0.0.1:3900
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16)

I've been through all this, using localhost or http://[::]:3900 as the endpoint, but all fail with the above errors.

Is there a guide to troubleshooting connection issues?

As my dev machine runs MacOS rather than linux, I tried following the instructions at https://garagehq.deuxfleurs.fr/documentation/quick-start/ but substituting the Docker image for a binary. The last config file I tried is ``` metadata_dir = "/var/lib/garage/meta" data_dir = "/var/lib/garage/data" db_engine = "sqlite" replication_mode = "none" rpc_bind_addr = "127.0.0.1:3901" rpc_public_addr = "127.0.0.1:3901" rpc_secret = "xxxxxxx" [s3_api] s3_region = "garage" api_bind_addr = "127.0.0.1:3900" root_domain = ".s3.garage.localhost" [s3_web] bind_addr = "127.0.0.1:3902" root_domain = ".web.garage.localhost" index = "index.html" [k2v_api] api_bind_addr = "[::]:3904" [admin] api_bind_addr = "127.0.0.1:3903" admin_token = "xxxxx" metrics_token = "xxxxxxx" ``` I tried several variations on network addresses, but they all suffer the problem below. /etc/garage.toml, /var/lib/garage/meta and /var/lib/garage/data on my dev machine are all owned by me. I can create a container so: ``` docker run \ -d \ --name garaged \ --restart always \ --network host \ -v /etc/garage.toml:/etc/garage.toml \ -v /var/lib/garage/meta:/var/lib/garage/meta \ -v /var/lib/garage/data:/var/lib/garage/data \ dxflrs/garage:v0.9.4 ``` The image starts up and the log includes ``` 2024-04-09 14:36:39 2024-04-09T18:36:39.513469Z INFO garage::server: Initializing background runner... 2024-04-09 14:36:39 2024-04-09T18:36:39.513504Z INFO garage::server: Spawning Garage workers... 2024-04-09 14:36:39 2024-04-09T18:36:39.513777Z INFO garage::server: Initialize Admin API server and metrics collector... 2024-04-09 14:36:39 2024-04-09T18:36:39.581100Z INFO garage::server: Launching internal Garage cluster communications... 2024-04-09 14:36:39 2024-04-09T18:36:39.581188Z INFO garage::server: Create admin RPC handler... 2024-04-09 14:36:39 2024-04-09T18:36:39.581202Z INFO garage::server: Initializing S3 API server... 2024-04-09 14:36:39 2024-04-09T18:36:39.581205Z INFO garage::server: Initializing K2V API server... 2024-04-09 14:36:39 2024-04-09T18:36:39.581208Z INFO garage::server: Initializing web server... 2024-04-09 14:36:39 2024-04-09T18:36:39.581218Z INFO garage::server: Launching Admin API server... 2024-04-09 14:36:39 2024-04-09T18:36:39.581287Z INFO garage_net::netapp: Listening on 127.0.0.1:3901 2024-04-09 14:36:39 2024-04-09T18:36:39.581316Z INFO garage_api::generic_server: S3 API server listening on http://127.0.0.1:3900 2024-04-09 14:36:39 2024-04-09T18:36:39.581381Z INFO garage_web::web_server: Web server listening on http://127.0.0.1:3902 2024-04-09 14:36:39 2024-04-09T18:36:39.581586Z INFO garage_api::generic_server: K2V API server listening on http://[::]:3904 2024-04-09 14:36:39 2024-04-09T18:36:39.581836Z INFO garage_api::generic_server: Admin API server listening on http://127.0.0.1:3903 ``` so it appears to running properly. I create an alias so: `alias garage="docker exec -ti garaged /garage"` and `garage status` returns ``` 2024-04-09T19:08:18.467817Z INFO garage_net::netapp: Connected to 127.0.0.1:3901, negotiating handshake... 2024-04-09T19:08:18.509152Z INFO garage_net::netapp: Connection established to d08e1c7b9177d649 ==== HEALTHY NODES ==== ID Hostname Address Tags Zone Capacity DataAvail d08e1c7b9177d649 docker-desktop 127.0.0.1:3901 [] dc1 1000.0 MB 3.2 TB (79.5%) ``` I can create and apply a cluster layout: ``` $ garage layout apply --version 2 2024-04-09T19:14:36.866461Z INFO garage_net::netapp: Connected to 127.0.0.1:3901, negotiating handshake... 2024-04-09T19:14:36.908111Z INFO garage_net::netapp: Connection established to d08e1c7b9177d649 ==== COMPUTATION OF A NEW PARTITION ASSIGNATION ==== Partitions are replicated 1 times on at least 1 distinct zones. Optimal partition size: 3.9 MB (3.9 MB in previous layout) Usable capacity / total cluster capacity: 1000.0 MB / 1000.0 MB (100.0 %) Effective capacity (replication factor 1): 1000.0 MB A total of 0 new copies of partitions need to be transferred. dc1 Tags Partitions Capacity Usable capacity d08e1c7b9177d649 256 (0 new) 1000.0 MB 1000.0 MB (100.0%) TOTAL 256 (256 unique) 1000.0 MB 1000.0 MB (100.0%) New cluster layout with updated role assignment has been applied in cluster. Data will now be moved around between nodes accordingly. ``` I can use the command line to create a bucket and API key, and give the key permission to use the bucket: ``` $ garage bucket info cl-first 2024-04-09T19:10:00.595340Z INFO garage_net::netapp: Connected to 127.0.0.1:3901, negotiating handshake... 2024-04-09T19:10:00.636526Z INFO garage_net::netapp: Connection established to d08e1c7b9177d649 Bucket: 6bd35158ce746f19bdde4e00d57d7ae00c74cfe6f5590b1d05ec264ade49c034 Size: 0 B (0 B) Objects: 0 Unfinished uploads (multipart and non-multipart): 0 Unfinished multipart uploads: 0 Size of unfinished multipart uploads: 0 B (0 B) Website access: false Global aliases: cl-first Key-specific aliases: Authorized keys: RWO GK5bec374936673b0b236b1618 cl-second ``` I can set environment variables for AWSCLI: ``` AWS_ENDPOINT_URL=http://localhost:3900 AWS_DEFAULT_REGION=garage AWS_SECRET_ACCESS_KEY=xxxxx AWS_ACCESS_KEY_ID=xxxxx ``` But AWSCLI can't access Garage via the S3 API: ``` $ aws s3 ls Could not connect to the endpoint URL: "http://localhost:3900/" ``` With appropriate configuration, I can use AWSCLI to access a Min.IO server running locally. Also, I have a Node.JS app, which, when properly configured, can access the Min.IO server. When I use the AWS configuration above for Garage, I get: ``` Error: connect ECONNREFUSED 127.0.0.1:3900 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1595:16) ``` I've been through all this, using localhost or http://[::]:3900 as the endpoint, but all fail with the above errors. Is there a guide to troubleshooting connection issues?
Owner

Hello, I think your issue is that you are using --network host, which is not supported under mac os. You should instead try forwarding ports individually using the --port argument.

Hello, I think your issue is that you are using `--network host`, which is not supported under mac os. You should instead try forwarding ports individually using the `--port` argument.
Author

Thanks, but creating the container with

docker run \
>   -d \
>   --name garaged \
>   --restart always \
>   -p 3900:3900 \
>   -v /etc/garage.toml:/etc/garage.toml \
>   -v /var/lib/garage/meta:/var/lib/garage/meta \
>   -v /var/lib/garage/data:/var/lib/garage/data \
>   dxflrs/garage:v0.9.4

only results in different error messages. AWSCLI returns

Connection was closed before we received a valid response from endpoint URL: "http://localhost:3900/"

and my Node.JS app gets

Error: read ECONNRESET at TCP.onStreamRead (node:internal/stream_base_commons:217:20)

Thanks, but creating the container with ``` docker run \ > -d \ > --name garaged \ > --restart always \ > -p 3900:3900 \ > -v /etc/garage.toml:/etc/garage.toml \ > -v /var/lib/garage/meta:/var/lib/garage/meta \ > -v /var/lib/garage/data:/var/lib/garage/data \ > dxflrs/garage:v0.9.4 ``` only results in different error messages. AWSCLI returns `Connection was closed before we received a valid response from endpoint URL: "http://localhost:3900/"` and my Node.JS app gets `Error: read ECONNRESET at TCP.onStreamRead (node:internal/stream_base_commons:217:20)`
Author

FWIW, when I use alias garage="docker exec -ti garaged /garage" to run garage bucket list, I see some messages in the log, starting with INFO garage_net::netapp: Accepted connection from 75b6268c2b8f3587 at 127.0.0.1:49370

When I try to use AWSCLI to connect, I don't see anything whatsoever in the log, even when I set RUST_LOG=garage=trace

Do I need to rebuild the image, after editing the Dockerfile to assert that the image uses the various ports?

FWIW, when I use `alias garage="docker exec -ti garaged /garage"` to run `garage bucket list`, I see some messages in the log, starting with `INFO garage_net::netapp: Accepted connection from 75b6268c2b8f3587 at 127.0.0.1:49370` When I try to use AWSCLI to connect, I don't see anything whatsoever in the log, even when I set `RUST_LOG=garage=trace` Do I need to rebuild the image, after editing the Dockerfile to assert that the image uses the various ports?
Owner

You might need to set api_bind_addr = "[::]:3900" in your config file to allow connctions arriving through the Docker port forward.

You might need to set `api_bind_addr = "[::]:3900"` in your config file to allow connctions arriving through the Docker port forward.
Author

Yep, one needs to leave the network addresses as [::]:3900 but change --network host to -p 3900:3900 -p 3901:3901 -p 3902:3902 -p 3903:3903.

I've submitted a doc PR for this: #803

Yep, one needs to leave the network addresses as `[::]:3900` but change `--network host` to `-p 3900:3900 -p 3901:3901 -p 3902:3902 -p 3903:3903`. I've submitted a doc PR for this: https://git.deuxfleurs.fr/Deuxfleurs/garage/pulls/803
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#801
No description provided.