Arm docker image fails on armv7 with error no such file or directory

runasleep commented

2022-02-16 08:21:58 +00:00

Raspberry pi 3 with RaspiOS, kernel 5.10.63-v7+ armv7l

$ docker run --rm -it dxflrs/arm_garage:v0.6.0
Unable to find image 'dxflrs/arm_garage:v0.6.0' locally
v0.6.0: Pulling from dxflrs/arm_garage
01d54abdf6a9: Pull complete 
Digest: sha256:0010c989c6170911499a3cbb7af1255f568a58c155b54a1d3574f86abd6d3529
Status: Downloaded newer image for dxflrs/arm_garage:v0.6.0
standard_init_linux.go:228: exec user process caused: no such file or directory

I also tried to create a small Dockerfile to extract the garage binary:

$ cat Dockerfile 
FROM dxflrs/arm_garage:v0.6.0 as garage

FROM alpine

COPY --from=garage /garage /bin/garage

~ $ docker build - < Dockerfile
Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM dxflrs/arm_garage:v0.6.0 as garage
 ---> a4e69364b088
Step 2/3 : FROM alpine
latest: Pulling from library/alpine
5480d2ca1740: Pull complete 
Digest: sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300
Status: Downloaded newer image for alpine:latest
 ---> 78571b13081b
Step 3/3 : COPY --from=garage /garage /bin/garage
 ---> 9178586088e5
Successfully built 9178586088e5

~ $ docker run --rm -it 9178586088e5
/ # /bin/garage help
/bin/sh: /bin/garage: not found
/ # ls -ald /bin/garage
-r-xr-xr-x    1 root     root      22156876 Feb  5 11:02 /bin/garage

It's NOT caused by missing config:

root@4c85bdb19abd:/# /bin/garage 
bash: /bin/garage: No such file or directory

root@4c85bdb19abd:/# ls -ald /bin/garage
-r-xr-xr-x 1 root root 22156876 Feb  5 11:02 /bin/garage

root@4c85bdb19abd:/# mkdir /etc/garage    
root@4c85bdb19abd:/# touch /etc/garage/garage.toml
root@4c85bdb19abd:/# /bin/garage 
bash: /bin/garage: No such file or directory

root@4c85bdb19abd:/# touch /etc/garage.toml
root@4c85bdb19abd:/# /bin/garage 
bash: /bin/garage: No such file or directory

Raspberry pi 3 with RaspiOS, kernel `5.10.63-v7+ armv7l` ``` $ docker run --rm -it dxflrs/arm_garage:v0.6.0 Unable to find image 'dxflrs/arm_garage:v0.6.0' locally v0.6.0: Pulling from dxflrs/arm_garage 01d54abdf6a9: Pull complete Digest: sha256:0010c989c6170911499a3cbb7af1255f568a58c155b54a1d3574f86abd6d3529 Status: Downloaded newer image for dxflrs/arm_garage:v0.6.0 standard_init_linux.go:228: exec user process caused: no such file or directory ``` I also tried to create a small Dockerfile to extract the garage binary: ``` $ cat Dockerfile FROM dxflrs/arm_garage:v0.6.0 as garage FROM alpine COPY --from=garage /garage /bin/garage ~ $ docker build - < Dockerfile Sending build context to Docker daemon 2.048kB Step 1/3 : FROM dxflrs/arm_garage:v0.6.0 as garage ---> a4e69364b088 Step 2/3 : FROM alpine latest: Pulling from library/alpine 5480d2ca1740: Pull complete Digest: sha256:21a3deaa0d32a8057914f36584b5288d2e5ecc984380bc0118285c70fa8c9300 Status: Downloaded newer image for alpine:latest ---> 78571b13081b Step 3/3 : COPY --from=garage /garage /bin/garage ---> 9178586088e5 Successfully built 9178586088e5 ~ $ docker run --rm -it 9178586088e5 / # /bin/garage help /bin/sh: /bin/garage: not found / # ls -ald /bin/garage -r-xr-xr-x 1 root root 22156876 Feb 5 11:02 /bin/garage ``` It's NOT caused by missing config: ``` root@4c85bdb19abd:/# /bin/garage bash: /bin/garage: No such file or directory root@4c85bdb19abd:/# ls -ald /bin/garage -r-xr-xr-x 1 root root 22156876 Feb 5 11:02 /bin/garage root@4c85bdb19abd:/# mkdir /etc/garage root@4c85bdb19abd:/# touch /etc/garage/garage.toml root@4c85bdb19abd:/# /bin/garage bash: /bin/garage: No such file or directory root@4c85bdb19abd:/# touch /etc/garage.toml root@4c85bdb19abd:/# /bin/garage bash: /bin/garage: No such file or directory ```

runasleep commented

2022-02-16 08:39:47 +00:00

Author

linux/arm binaries downloaded from https://garagehq.deuxfleurs.fr/_releases.html fail with the same error

`linux/arm` binaries downloaded from `https://garagehq.deuxfleurs.fr/_releases.html` fail with the same error

lx commented

2022-02-16 11:03:08 +00:00

Owner

Can you strace your invokation of Garage to see what file it is trying to access?

Something like:

strace -yy /bin/garage

Can you `strace` your invokation of Garage to see what file it is trying to access? Something like: ``` strace -yy /bin/garage ```

maximilien commented

2022-02-16 11:37:40 +00:00

Owner

This is what I have on arm64 with the binary from https://garagehq.deuxfleurs.fr/_releases/v0.6.0/armv6l-unknown-linux-musleabihf/garage

$ strace -yy /bin/garage
execve("/bin/garage", ["/bin/garage"], 0xfffffb74c0a8 /* 32 vars */) = -1 ENOENT (No such file or directory)
strace: exec: No such file or directory
+++ exited with 1 +++

Target

Linux helios64 5.15.18-rockchip64 #trunk.0023 SMP PREEMPT Sat Jan 29 16:01:11 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

This is what I have on arm64 with the binary from https://garagehq.deuxfleurs.fr/_releases/v0.6.0/armv6l-unknown-linux-musleabihf/garage ``` $ strace -yy /bin/garage execve("/bin/garage", ["/bin/garage"], 0xfffffb74c0a8 /* 32 vars */) = -1 ENOENT (No such file or directory) strace: exec: No such file or directory +++ exited with 1 +++ ``` Target ``` Linux helios64 5.15.18-rockchip64 #trunk.0023 SMP PREEMPT Sat Jan 29 16:01:11 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux ```

runasleep commented

2022-02-16 13:51:57 +00:00

Author

Yes, I'm getting the same strace on the Pi.

Could this be relevant?

# file /bin/garage 
/bin/garage: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld.so.1, not stripped

# ls -ald /usr/lib/ld.so.1    
ls: cannot access '/usr/lib/ld.so.1': No such file or directory

# ls -ald /lib/ld-linux-armhf.so.3 
lrwxrwxrwx 1 root root 30 Oct  2 12:47 /lib/ld-linux-armhf.so.3 -> arm-linux-gnueabihf/ld-2.31.so

Yes, I'm getting the same strace on the Pi. Could this be relevant? ``` # file /bin/garage /bin/garage: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld.so.1, not stripped # ls -ald /usr/lib/ld.so.1 ls: cannot access '/usr/lib/ld.so.1': No such file or directory # ls -ald /lib/ld-linux-armhf.so.3 lrwxrwxrwx 1 root root 30 Oct 2 12:47 /lib/ld-linux-armhf.so.3 -> arm-linux-gnueabihf/ld-2.31.so ```

runasleep commented

2022-02-16 14:02:44 +00:00

Author

Oh yeah, ln -s /lib/ld-linux-armhf.so.3 /usr/lib/ld.so.1 fixed it, but it doesn't feel right, lol

Oh yeah, `ln -s /lib/ld-linux-armhf.so.3 /usr/lib/ld.so.1` fixed it, but it doesn't feel right, lol

lx commented

2022-02-16 15:22:23 +00:00

Owner

No, that definitely shouldn't be the solution! Theoretically all our builds are statically linked so I'm surprised they still depend on ld.so. Maybe @quentin could shed some light on the issue as he is the one who designed the build system?

No, that definitely shouldn't be the solution! Theoretically all our builds are statically linked so I'm surprised they still depend on `ld.so`. Maybe @quentin could shed some light on the issue as he is the one who designed the build system?

quentin commented

2022-02-16 16:46:35 +00:00

Owner

I am using Nix toolchains to get statically compiled binaries with musl. For armv6+armv7, I am using this toolchain: armv6l-unknown-linux-musleabihf. We can compare it to amd64 where we have: x86_64-unknown-linux-musl.

But when we compare the ouputs, it seems that arm is dynamically linked while amd64 is not, despite the only difference between them being the target above.

$ file garage_amd64
garage_amd64: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, not stripped

$ file garage_arm
garage_arm: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld.so.1, not stripped

$ file garage_arm_0.4.0
garage_arm_0.4.0: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld.so.1, not stripped

I will get a more in-depth look at this issue next week and fix it. I will probably add a check to ensure that the generated build is, indeed, statically compiled, for all architectures.

For now the only thing I can confirm is that's a bug on our side and we will try to fix it as soon as possible. Thanks for reporting it.

I am using Nix toolchains to get statically compiled binaries with musl. For armv6+armv7, I am using this toolchain: `armv6l-unknown-linux-musleabihf`. We can compare it to amd64 where we have: `x86_64-unknown-linux-musl`. But when we compare the ouputs, it seems that arm is dynamically linked while amd64 is not, despite the only difference between them being the target above. ``` $ file garage_amd64 garage_amd64: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, not stripped $ file garage_arm garage_arm: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld.so.1, not stripped $ file garage_arm_0.4.0 garage_arm_0.4.0: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /usr/lib/ld.so.1, not stripped ``` I will get a more in-depth look at this issue next week and fix it. I will probably add a check to ensure that the generated build is, indeed, statically compiled, for all architectures. For now the only thing I can confirm is that's a bug on our side and we will try to fix it as soon as possible. Thanks for reporting it.

quentin added the

CI

label 2022-02-23 11:24:57 +00:00

quentin referenced a pull request that will close this issue

2022-02-23 11:25:17 +00:00

Force static builds for all platforms #250 #250

lx closed this issue

2022-02-24 17:00:21 +00:00

Arm docker image fails on armv7 with error no such file or directory #245