"Messagepack decode error: wrong msgpack marker FixPos(0)" #897

Closed
opened 2024-11-06 11:17:36 +00:00 by raucao · 1 comment

Hi,

On one of our cluster nodes, I'm seeing this error pop up in various WARN and ERROR messages (see below). I think it started after the VM host had a drive be unavailable temporarily and was power-cycled then.

The various repair commands don't seem to be able to fix them.

WARN garage_table::sync: (block_ref) Sync error: Messagepack decode error: wrong msgpack marker FixPos(0)
ERROR garage_util::background::worker: Error in worker version Merkle (TID 26): Messagepack decode error: wrong msgpack marker FixPos(0)
ERROR garage_util::background::worker: Error in worker block_ref Merkle (TID 30): Messagepack decode error: wrong msgpack marker FixPos(0)
ERROR garage_util::background::worker: Error in worker block_ref sync (TID 31): Sync failed with too many nodes (should have been: [4e10df0c1eded2ed]).

From what I found online, this is coming from rmp-serde, but my knowledge of Rust is not enough to debug it myself. I'm wondering if there is a way to go around this somehow, like deleting and re-sycing the metadata directory or something like that. Or even just replacing the node entirely. Any ideas?

Edit: we're still on 0.8.4 by the way. So the other question would be if we should upgrade regardless of the errors.

Hi, On one of our cluster nodes, I'm seeing this error pop up in various WARN and ERROR messages (see below). I think it started after the VM host had a drive be unavailable temporarily and was power-cycled then. The various repair commands don't seem to be able to fix them. ``` WARN garage_table::sync: (block_ref) Sync error: Messagepack decode error: wrong msgpack marker FixPos(0) ``` ``` ERROR garage_util::background::worker: Error in worker version Merkle (TID 26): Messagepack decode error: wrong msgpack marker FixPos(0) ERROR garage_util::background::worker: Error in worker block_ref Merkle (TID 30): Messagepack decode error: wrong msgpack marker FixPos(0) ``` ``` ERROR garage_util::background::worker: Error in worker block_ref sync (TID 31): Sync failed with too many nodes (should have been: [4e10df0c1eded2ed]). ``` From what I found online, this is coming from rmp-serde, but my knowledge of Rust is not enough to debug it myself. I'm wondering if there is a way to go around this somehow, like deleting and re-sycing the metadata directory or something like that. Or even just replacing the node entirely. Any ideas? Edit: we're still on 0.8.4 by the way. So the other question would be if we should upgrade regardless of the errors.
Author

Deleting the database on that node and doing a table re-sync seems to have fixed the issue.

Deleting the database on that node and doing a table re-sync seems to have fixed the issue.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Deuxfleurs/garage#897
No description provided.