Unable to run on ARM 7L #371

New issue

Closed

opened 2022-09-04 09:14:26 +00:00 by rob-air · 3 comments

rob-air commented

2022-09-04 09:14:26 +00:00

Hi there and thank you for your work! \o/

I just tried running Garage version 0.7.3 on a synology NAS without much success, details included below.

Disclaimer: I'll try my best but be advised: I have no experience at all running stuff on ARM arch specifically or debugging binaries in general. I might include useless garbage and/or exclude important information.

I'm noticing just now "armv6l" in the download link, the machine i'm trying to use being 7L i'm guessing that has something to do with the binary not running... No idea what to do from there though.

Machine info

Synology DS218J
CPU Marvell Armada 385 88F6820

rob@toaster:~$ uname -a
Linux toaster 3.10.108 #42661 SMP Mon Jun 27 15:07:25 CST 2022 armv7l GNU/Linux synology_armada38x_ds218j

rob@toaster:~$ cat /proc/cpuinfo 
processor	: 0
model name	: ARMv7 Processor rev 1 (v7l)
BogoMIPS	: 2655.84
Features	: swp half thumb fastmult vfp edsp neon vfpv3 tls 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x4
CPU part	: 0xc09
CPU revision	: 1

processor	: 1
model name	: ARMv7 Processor rev 1 (v7l)
BogoMIPS	: 2655.84
Features	: swp half thumb fastmult vfp edsp neon vfpv3 tls 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x4
CPU part	: 0xc09
CPU revision	: 1

Hardware	: Marvell Armada 380/381/382/383/384/385/388 (Device Tree)
Revision	: 0000
Serial		: 0000000000000000

Running and crashing

Using Garage 0.7.3 linux/arm build downloaded from https://garagehq.deuxfleurs.fr/_releases/v0.7.3/armv6l-unknown-linux-musleabihf/garage

rob@toaster:~$ export RUST_BACKTRACE=full
rob@toaster:~$ ./garage
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }', library/std/src/sys/unix/time.rs:371:62
stack backtrace:
   0:  0x19941a8 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hee884e3990ff8322
   1:  0x19d4228 - core::fmt::write::hbb685d6aaf00f62b
   2:  0x19829a0 - std::io::Write::write_fmt::h4c2542c941c382eb
   3:  0x1983fcc - std::panicking::default_hook::{{closure}}::hca633ca93cea8e3c
   4:  0x19839d0 - std::panicking::default_hook::h215ec60d1b785086
   5:  0x1984618 - std::panicking::rust_panic_with_hook::h067cd60966377806
   6:  0x199477c - std::panicking::begin_panic_handler::{{closure}}::hc1c1618c36aca724
   7:  0x199430c - std::sys_common::backtrace::__rust_end_short_backtrace::h134b3e6545ab5098
   8:  0x19840d0 - rust_begin_unwind
   9:    0x64068 - core::panicking::panic_fmt::h41fd2817f7ef0934
  10:    0x6413c - core::result::unwrap_failed::h593ab467b5d45c78
  11:  0x197de40 - std::time::Instant::now::h0018b0a7a3d37fb0
  12:  0x1920d8c - tokio::time::driver::Driver<P>::new::h04581e33ebd3acf2
  13:  0x192d1e8 - tokio::runtime::driver::Driver::new::h9e108753dec5de1b
  14:  0x19339f0 - tokio::runtime::builder::Builder::build::hfd3ff40d6ad3f204
  15:   0x1bb7c4 - garage::main::hea6a4247dfc4dacd
  16:   0x32985c - std::sys_common::backtrace::__rust_begin_short_backtrace::hc91e4a8ad94c5cbe
  17:   0x3e2984 - std::rt::lang_start::{{closure}}::h2c9516dba341cd2a
  18:  0x1987c68 - std::rt::lang_start_internal::h267aa73a12004922
  19:   0x1bb974 - main

Hi there and thank you for your work! \o/ I just tried running Garage version 0.7.3 on a synology NAS without much success, details included below. *Disclaimer: I'll try my best but be advised: I have no experience at all running stuff on ARM arch specifically or debugging binaries in general. I might include useless garbage and/or exclude important information.* I'm noticing just now "armv6l" in the download link, the machine i'm trying to use being 7L i'm guessing that has something to do with the binary not running... No idea what to do from there though. ### Machine info Synology DS218J CPU Marvell Armada 385 88F6820 ``` rob@toaster:~$ uname -a Linux toaster 3.10.108 #42661 SMP Mon Jun 27 15:07:25 CST 2022 armv7l GNU/Linux synology_armada38x_ds218j ``` ``` rob@toaster:~$ cat /proc/cpuinfo processor : 0 model name : ARMv7 Processor rev 1 (v7l) BogoMIPS : 2655.84 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x4 CPU part : 0xc09 CPU revision : 1 processor : 1 model name : ARMv7 Processor rev 1 (v7l) BogoMIPS : 2655.84 Features : swp half thumb fastmult vfp edsp neon vfpv3 tls CPU implementer : 0x41 CPU architecture: 7 CPU variant : 0x4 CPU part : 0xc09 CPU revision : 1 Hardware : Marvell Armada 380/381/382/383/384/385/388 (Device Tree) Revision : 0000 Serial : 0000000000000000 ``` ### Running and crashing Using Garage 0.7.3 linux/arm build downloaded from https://garagehq.deuxfleurs.fr/_releases/v0.7.3/armv6l-unknown-linux-musleabihf/garage ``` rob@toaster:~$ export RUST_BACKTRACE=full rob@toaster:~$ ./garage thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }', library/std/src/sys/unix/time.rs:371:62 stack backtrace: 0: 0x19941a8 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hee884e3990ff8322 1: 0x19d4228 - core::fmt::write::hbb685d6aaf00f62b 2: 0x19829a0 - std::io::Write::write_fmt::h4c2542c941c382eb 3: 0x1983fcc - std::panicking::default_hook::{{closure}}::hca633ca93cea8e3c 4: 0x19839d0 - std::panicking::default_hook::h215ec60d1b785086 5: 0x1984618 - std::panicking::rust_panic_with_hook::h067cd60966377806 6: 0x199477c - std::panicking::begin_panic_handler::{{closure}}::hc1c1618c36aca724 7: 0x199430c - std::sys_common::backtrace::__rust_end_short_backtrace::h134b3e6545ab5098 8: 0x19840d0 - rust_begin_unwind 9: 0x64068 - core::panicking::panic_fmt::h41fd2817f7ef0934 10: 0x6413c - core::result::unwrap_failed::h593ab467b5d45c78 11: 0x197de40 - std::time::Instant::now::h0018b0a7a3d37fb0 12: 0x1920d8c - tokio::time::driver::Driver<P>::new::h04581e33ebd3acf2 13: 0x192d1e8 - tokio::runtime::driver::Driver::new::h9e108753dec5de1b 14: 0x19339f0 - tokio::runtime::builder::Builder::build::hfd3ff40d6ad3f204 15: 0x1bb7c4 - garage::main::hea6a4247dfc4dacd 16: 0x32985c - std::sys_common::backtrace::__rust_begin_short_backtrace::hc91e4a8ad94c5cbe 17: 0x3e2984 - std::rt::lang_start::{{closure}}::h2c9516dba341cd2a 18: 0x1987c68 - std::rt::lang_start_internal::h267aa73a12004922 19: 0x1bb974 - main ```

lx commented

2022-09-05 08:45:17 +00:00

Owner

Hi @rob-air, thanks for your detailed report on this issue.

Looking at your logs, I think the issue is not about armv6l vs. armv7l, but more about your kernel being too old. Indeed the binary itself seems to run correctly, but the issue showed up when Instant::now was called. This function is a time-related function that tries to grab a kernel timer so that time measurements can be done in the application. Apparently when Garage tries to grab such a timer, the kernel returns an "invalid argument" error, which I suspect is because the Rust standard library is using some more recent call semantics that your kernel does not support (FYI, Linux 3.10 is from 2013).

Do you have an option to try updating your Synology NAS to run a more recent kernel and see if that fixes the issue?

Another option would be to try to build Garage yourself using cargo build, so as to produce a build that is able to call into your system Glibc instead of using Musl. This might help produce a build that knows the capabilities of your kernel and doesn't try to call functions it doesn't have, but it means doing the build on the Synology NAS itself, which will probably extremely long (and also have a significant chance of failure).

Hi @rob-air, thanks for your detailed report on this issue. Looking at your logs, I think the issue is not about armv6l vs. armv7l, but more about your kernel being too old. Indeed the binary itself seems to run correctly, but the issue showed up when `Instant::now` was called. This function is a time-related function that tries to grab a kernel timer so that time measurements can be done in the application. Apparently when Garage tries to grab such a timer, the kernel returns an "invalid argument" error, which I suspect is because the Rust standard library is using some more recent call semantics that your kernel does not support (FYI, Linux 3.10 is from 2013). Do you have an option to try updating your Synology NAS to run a more recent kernel and see if that fixes the issue? Another option would be to try to build Garage yourself using `cargo build`, so as to produce a build that is able to call into your system Glibc instead of using Musl. This might help produce a build that knows the capabilities of your kernel and doesn't try to call functions it doesn't have, but it means doing the build on the Synology NAS itself, which will probably extremely long (and also have a significant chance of failure).

rob-air commented

2022-09-06 09:44:15 +00:00

Author

Hi @lx, thanks for the quick reply and insights.

Yes, this kernel is a very dusty one... but unfortunately I'm pretty reluctant to update it, not to mention quite scared of the task. This kernel is maintained by synology and they (obviously) don't follow upstream updates very assiduously but they release security update patches when needed. I have no idea how the poor thing would handle a forcibly updated kernel, so this might be an issue big enough for me to postpone it into eternity.

So... I looked around and ended up trying to cross-compile garage from my laptop (Debian 11) targeting the NAS arch; without much success... yet. Not giving up that early but I have to admit this is a pretty deep rabbit hole for a first-timer.
I followed this for the cross-compilation https://github.com/japaric/rust-cross, and it's working as expected, but I end up with a missing glibc library when executing the binary on the NAS.
If you've got some good quality guides or docs at hand regarding this topic, I'll gladly take it.

I also tried compiling from the NAS itself but that seems to be an even darker hole, too many missing tools on the thing and too shady procedures to even try to figure out how to put them there.

I'll keep trying, and report here if I have something new to tell.

Hi @lx, thanks for the quick reply and insights. Yes, this kernel is a very dusty one... but unfortunately I'm pretty reluctant to update it, not to mention quite scared of the task. This kernel is maintained by synology and they (obviously) don't follow upstream updates very assiduously but they release security update patches when needed. I have no idea how the poor thing would handle a forcibly updated kernel, so this might be an issue big enough for me to postpone it into eternity. So... I looked around and ended up trying to cross-compile garage from my laptop (Debian 11) targeting the NAS arch; without much success... yet. Not giving up that early but I have to admit this is a pretty deep rabbit hole for a first-timer. I followed this for the cross-compilation https://github.com/japaric/rust-cross, and it's working as expected, but I end up with a missing glibc library when executing the binary on the NAS. If you've got some good quality guides or docs at hand regarding this topic, I'll gladly take it. I also tried compiling from the NAS itself but that seems to be an even darker hole, too many missing tools on the thing and too shady procedures to even try to figure out how to put them there. I'll keep trying, and report here if I have something new to tell.

lx commented

2022-09-06 21:47:33 +00:00

Owner

I'm closing this issue for now as I don't think there is much we can do on our side. But please do keep us updated if you manage to compile a working binary for your platform!