There's a regression in v0.7.2, in cluster setup. #319

New Issue

Closed

opened 2022-05-29 13:16:37 +00:00 by bugzbunny · 4 comments

bugzbunny commented

2022-05-29 13:16:37 +00:00

Hi Team,

There seems to be a regression in v0.7.2 in regards to cluster setup. Whenever a client attempts to store an object, the server throws an Internal Server Error. Reverting to v0.7.1 fixes it and #317 might be related. I don't have the time to git bisect, sorry.

WARN  garage_api::generic_server > Response: error 503 Service Unavailable, Internal error: Could not reach quorum of 1. 0 of 1 request succeeded, others returned errors: ["Netapp error: Remote error 20: No handler / shutting down"]

NOTE: I might have jumped the gun to early here. The error disappeared for a short while, but returned.

garage status
 INFO  netapp::netapp > Connected to <redacted>:3901, negotiating handshake...
 INFO  netapp::netapp > Connection established to e0fc9e6d27bc1fb6
==== HEALTHY NODES ====
ID                Hostname            Address                      Tags  Zone  Capacity
e0fc9e6d27bc1fb6  <redacted>  <redacted>:3901          []    NJ    19
e08219eea7ca6b19  s3pa-server.vm      <redacted>:3901  []    PA    6
8714d77ba0c61816  <redacted>  <redacted>:3901  []    TX    10

Regards

Hi Team, There seems to be a regression in v0.7.2 in regards to cluster setup. Whenever a client attempts to store an object, the server throws an `Internal Server Error`. Reverting to v0.7.1 fixes it and #317 might be related. I don't have the time to `git bisect`, sorry. ``` WARN garage_api::generic_server > Response: error 503 Service Unavailable, Internal error: Could not reach quorum of 1. 0 of 1 request succeeded, others returned errors: ["Netapp error: Remote error 20: No handler / shutting down"] ``` **NOTE:** I might have jumped the gun to early here. The error disappeared for a short while, but returned. ``` garage status INFO netapp::netapp > Connected to <redacted>:3901, negotiating handshake... INFO netapp::netapp > Connection established to e0fc9e6d27bc1fb6 ==== HEALTHY NODES ==== ID Hostname Address Tags Zone Capacity e0fc9e6d27bc1fb6 <redacted> <redacted>:3901 [] NJ 19 e08219eea7ca6b19 s3pa-server.vm <redacted>:3901 [] PA 6 8714d77ba0c61816 <redacted> <redacted>:3901 [] TX 10 ``` Regards

lx commented

2022-05-29 13:37:09 +00:00

Owner

Just a quick check: are all your nodes running the same build of Garage ?

Also, can you paste the logs from your nodes ?

Just a quick check: are all your nodes running the same build of Garage ? Also, can you paste the logs from your nodes ?

bugzbunny commented

2022-05-29 14:18:42 +00:00

Author

I thought there were and they were not. I've just verified all nodes are running the same version now. Now I'm running the same tests again. Also, both v0.7.1 and v0.7.2 is reporting as 0.7.0 in garage -V.

I thought there were and they were not. I've just verified all nodes are running the same version now. Now I'm running the same tests again. Also, both `v0.7.1` and `v0.7.2` is reporting as `0.7.0` in `garage -V`.

bugzbunny commented

2022-05-29 14:23:47 +00:00

Author

So far, it's working fine now. I update this issue if I run into the same problem again in the next few days.