Internal error when no s3 binding #613

New issue

Closed

opened 2023-08-10 15:19:54 +00:00 by huetremy · 2 comments

huetremy commented

2023-08-10 15:19:54 +00:00

Hi, first thank you for this amazing tool!

I am planning to deploy it at Rhizome FAI (student FFDN FAI in Compiègne) for our backup.

I just installed garage on a fresh Debian 12 install, following systemd tutorial.

My /etc/garage.toml is as follows:

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "lmdb"

replication_mode = "3"
compression_level = 2

rpc_bind_addr = "0.0.0.0:3901"
rpc_public_addr = "80.67.182.87:3901"
rpc_secret = "<...>"

[s3_api]
s3_region = "garage"

Note that I did not put the [s3_api] section in a first place, as I did not want to expose it directly (but do it in a gateway node behind a reverse proxy). However, to start the server, I had to add this section and s3_region, the configuration being not valid without.

With this file, the service starts correctly. However, when trying to garage status, I have the following error:

Error: Internal error: Netapp error: Remote error (Other): netapp error: No handler / shutting down

with following systemctl status garage:

août 10 16:52:57 g1 garage[1856]: 2023-08-10T14:52:57.564450Z  INFO netapp::netapp: Incoming connection from 80.67.182.87:56838, negotiating handshake...
août 10 16:52:57 g1 garage[1856]: 2023-08-10T14:52:57.565449Z  INFO netapp::netapp: Accepted connection from b0b2753d36a3f8c4 at 80.67.182.87:56838
août 10 16:52:57 g1 garage[1856]: 2023-08-10T14:52:57.607182Z  INFO netapp::netapp: Connection from b0b2753d36a3f8c4 closed

By adding api_bind_addr = "127.0.0.1:3900" to the garage config file, everything is fine ad I get the following expected return:

==== HEALTHY NODES ====
ID                Hostname  Address            Tags              Zone  Capacity  DataAvail
922a5f24e93fb23b  g1        80.67.182.87:3901  NO ROLE ASSIGNED

Questions:

Please correct me if I am miss-understanding something

Why do I need to expose this API endpoint to get a working garage status?
May nodes do the job between them without this endpoint? (of course, one should be set to communicate with clients, but what about intra-cluster coms?)

With my understanding of the docs, I plan to do the following:

Several Storage Nodes that do not expose S3 API endpoint and just communicate between each other
One or several Gateway Nodes that do not store anything, but listen on localhost:3900 with a reverse proxy forwarding requests from :443 to :3900 (with working ACME of course)

Does it seems to be a working idea?

Thank you again !

Hi, first thank you for this amazing tool! I am planning to deploy it at Rhizome FAI (student FFDN FAI in Compiègne) for our backup. I just installed garage on a fresh Debian 12 install, following [systemd tutorial](https://garagehq.deuxfleurs.fr/documentation/cookbook/systemd/). My `/etc/garage.toml` is as follows: ```toml metadata_dir = "/var/lib/garage/meta" data_dir = "/var/lib/garage/data" db_engine = "lmdb" replication_mode = "3" compression_level = 2 rpc_bind_addr = "0.0.0.0:3901" rpc_public_addr = "80.67.182.87:3901" rpc_secret = "<...>" [s3_api] s3_region = "garage" ``` Note that I did not put the `[s3_api]` section in a first place, as I did not want to expose it directly (but do it in a gateway node behind a reverse proxy). However, to start the server, I had to add this section and `s3_region`, the configuration being not valid without. With this file, the service starts correctly. However, when trying to `garage status`, I have the following error: ``` Error: Internal error: Netapp error: Remote error (Other): netapp error: No handler / shutting down ``` with following `systemctl status garage`: ``` août 10 16:52:57 g1 garage[1856]: 2023-08-10T14:52:57.564450Z INFO netapp::netapp: Incoming connection from 80.67.182.87:56838, negotiating handshake... août 10 16:52:57 g1 garage[1856]: 2023-08-10T14:52:57.565449Z INFO netapp::netapp: Accepted connection from b0b2753d36a3f8c4 at 80.67.182.87:56838 août 10 16:52:57 g1 garage[1856]: 2023-08-10T14:52:57.607182Z INFO netapp::netapp: Connection from b0b2753d36a3f8c4 closed ``` By adding `api_bind_addr = "127.0.0.1:3900"` to the garage config file, everything is fine ad I get the following expected return: ``` ==== HEALTHY NODES ==== ID Hostname Address Tags Zone Capacity DataAvail 922a5f24e93fb23b g1 80.67.182.87:3901 NO ROLE ASSIGNED ``` ## Questions: *Please correct me if I am miss-understanding something* - Why do I need to expose this API endpoint to get a working `garage status`? - May nodes do the job between them without this endpoint? (of course, one should be set to communicate with *clients*, but what about intra-cluster coms?) With my understanding of the docs, I plan to do the following: - Several *Storage Nodes* that do not expose S3 API endpoint and just communicate between each other - One or several *Gateway Nodes* that do not store anything, but listen on localhost:3900 with a reverse proxy forwarding requests from :443 to :3900 (with working ACME of course) Does it seems to be a working idea? Thank you again !

jpds commented

2023-08-16 12:33:57 +00:00

Contributor

Why do I need to expose this API endpoint to get a working garage status?

There is no real distinction in the codebase between a storage node and a gateway - this only really exists at the garage layout layer. And noone has gotten round to making the API endpoint optional.

May nodes do the job between them without this endpoint? (of course, one should be set to communicate with clients, but what about intra-cluster coms?)

They only use the RPC endpoint to communicate.

With my understanding of the docs, I plan to do the following:

Several Storage Nodes that do not expose S3 API endpoint and just communicate between each other
One or several Gateway Nodes that do not store anything, but listen on localhost:3900 with a reverse proxy forwarding requests from :443 to :3900 (with working ACME of course)

This is all perfectly doable - just use nftables with a default deny policy for your inbound firewall to block access to the other ports.

> Why do I need to expose this API endpoint to get a working garage status? There is no real distinction in the codebase between a storage node and a gateway - this only really exists at the garage layout layer. And noone has gotten round to making the API endpoint optional. > May nodes do the job between them without this endpoint? (of course, one should be set to communicate with clients, but what about intra-cluster coms?) They only use the RPC endpoint to communicate. > With my understanding of the docs, I plan to do the following: > > Several Storage Nodes that do not expose S3 API endpoint and just communicate between each other > One or several Gateway Nodes that do not store anything, but listen on localhost:3900 with a reverse proxy forwarding requests from :443 to :3900 (with working ACME of course) This is all perfectly doable - just use nftables with a default deny policy for your inbound firewall to block access to the other ports.

lx commented

2023-09-05 15:10:40 +00:00

Owner

The error you are getting when starting without binding the S3 API port is not normal, it's probably a bug. Thanks for the detailed report, I'll look into it soon.