[k8s] Error: Internal error: Remote error: Could not reach quorum of 1

zerored commented

2023-01-13 14:47:47 +00:00

Hi, after installing garage via helm chart - it started succesfully and, for example, garage status - works fine, but when i want to create bucket or key - it drops such error:

Error: Internal error: Remote error: Could not reach quorum of 1. 0 of 0 request succeeded, others returned errors: []

and gives me warns:

WARN  garage > Trying to contact Garage node at default address 127.0.0.1:3901
WARN  garage > If this doesn't work, consider adding rpc_public_addr in your config file or specifying the -h command line parameter.

is there's any additional steps to work in k8s?

Hi, after installing garage via helm chart - it started succesfully and, for example, garage status - works fine, but when i want to create bucket or key - it drops such error: ``` Error: Internal error: Remote error: Could not reach quorum of 1. 0 of 0 request succeeded, others returned errors: [] ``` and gives me warns: ``` WARN garage > Trying to contact Garage node at default address 127.0.0.1:3901 WARN garage > If this doesn't work, consider adding rpc_public_addr in your config file or specifying the -h command line parameter. ``` is there's any additional steps to work in k8s?

zerored commented

2023-01-13 20:37:47 +00:00

Author

So, the problem was in

replicaCount: 1

seems like there's should be more than 1 copy of garage

i'll leave this issue open until fix for single-node cluster

So, the problem was in `replicaCount: 1` seems like there's should be more than 1 copy of garage i'll leave this issue open until fix for single-node cluster

lx commented

2023-01-14 18:11:38 +00:00

Owner

I am not the author of the helm chart, and I don't know how Kubernetes does things, but can you confirm that your node has a zone and capacity assigned in garage status ? I.e. that a layout has been created and applied, even with just one node

zerored commented

2023-01-14 20:25:50 +00:00

Author

@lx

No, it wasn't, also, garage didn't offer me to create layout with single node, but after changing replicaCount to 2 - garage offered me creation of layout which successfully fixed that issue

@lx No, it wasn't, also, garage didn't offer me to create layout with single node, but after changing `replicaCount` to 2 - garage offered me creation of layout which successfully fixed that issue

zerored closed this issue

2023-01-14 20:25:52 +00:00

maximilien commented

2023-01-14 22:08:07 +00:00

Owner

So, the problem was in

replicaCount: 1

seems like there's should be more than 1 copy of garage

i'll leave this issue open until fix for single-node cluster

The default configuration for the garage helm chart specify 3 replicas for the pods, and 3 for the replication mode as well. If you change the number of replicas, you have to adjust the config provided to garage to change the replication_mode to something compatible with the number of pods that you are running.

> So, the problem was in > > `replicaCount: 1` > > > seems like there's should be more than 1 copy of garage > > i'll leave this issue open until fix for single-node cluster The default configuration for the garage helm chart [specify 3 replicas for the pods](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/script/k8s/daemon.yaml#L10), and [3 for the replication mode as well](https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/script/k8s/config.yaml#L11). If you change the number of replicas, you have to adjust the config provided to garage to change the `replication_mode` to something compatible with the number of pods that you are running.

zerored commented

2023-01-15 16:39:05 +00:00

Author

@maximilien yeah, but I set replicaCount and replication_mode to 1 from the first launch for helm chart

@maximilien yeah, but I set `replicaCount` and `replication_mode` to 1 from the first launch for helm chart

zerored commented

2023-01-15 22:45:24 +00:00

Author

WARN garage_rpc::ring: Could not build ring: network role assignation data has invalid length
WARN garage_rpc::system: Using autodetected rpc_public_addr: 10.42.1.14:3901. Consider specifying it explicitly in configuration file if possible.

fresh install single replica, replication_mode = 1

``` WARN garage_rpc::ring: Could not build ring: network role assignation data has invalid length WARN garage_rpc::system: Using autodetected rpc_public_addr: 10.42.1.14:3901. Consider specifying it explicitly in configuration file if possible. ``` fresh install single replica, replication_mode = 1

zerored reopened this issue

2023-01-15 22:45:28 +00:00

lx commented

2023-01-19 12:26:20 +00:00

Owner

I don't understand this issue. Are the helm chart files wrong? If so, can you fix them and make a PR?

maximilien commented

2023-01-20 09:50:19 +00:00

Owner

WARN garage_rpc::ring: Could not build ring: network role assignation data has invalid length
WARN garage_rpc::system: Using autodetected rpc_public_addr: 10.42.1.14:3901. Consider specifying it explicitly in configuration file if possible.

fresh install single replica, replication_mode = 1

Thanks, I'll see if I can reproduce this on my side.

> ``` > WARN garage_rpc::ring: Could not build ring: network role assignation data has invalid length > WARN garage_rpc::system: Using autodetected rpc_public_addr: 10.42.1.14:3901. Consider specifying it explicitly in configuration file if possible. > ``` > > fresh install single replica, replication_mode = 1 Thanks, I'll see if I can reproduce this on my side.

lx commented

2023-10-16 09:58:21 +00:00

Owner

Closing for inactivity

lx closed this issue

2023-10-16 09:58:21 +00:00

[k8s] Error: Internal error: Remote error: Could not reach quorum of 1 #478