OutOfMemory error from LMDB when starting a fresh installation #439

New Issue

Closed

opened 2022-12-05 00:26:01 +00:00 by nehu · 2 comments

nehu commented

2022-12-05 00:26:01 +00:00

First of all, thanks a lot for this amazing project!

I encountered an issue where the application would panic right after starting. It seems to come from LMDB.

The only workaround known to me is to switch the DB engine to another one:
I went with sqlite, but it is definitely, sensibly slower.

I tested it in three different environments, but I can't tell what's causing the panic.

In each environment, I use docker and docker-compose to spin up Garage.

I used the same docker-compose.yml and garage.toml files (see below), with one exception:
On the MacBook, I used an own dockerfile to statically copy the garage.toml file - somehow it would mount the file as a directory inside the container...

First environment: VPS running Arch Linux VPS

✅ Works

Kernel: Linux neptune.internal 6.0.10-arch2-1 #1 SMP PREEMPT_DYNAMIC Sat, 26 Nov 2022 16:51:18 +0000 x86_64 GNU/Linux
Docker: Docker version 20.10.21, build baeda1f82a

Second environment: Raspberry Pi 4 running Void Linux musl

❌ Does not work

Kernel: Linux m-pi 5.15.72_1 #1 SMP PREEMPT Sun Oct 16 14:46:40 UTC 2022 aarch64 GNU/Linux
Docker: Docker version 20.10.21, build tag v20.10.21

No matter where the mounted directories are located (in the same directory as the compose file, or on another HDD),
and no matter what the permissions and ownership (root or not) are, it just won't work.

At the time of running, this is the resource usage:

CPU load: 0.21 / 0.12 / 0.17
RAM: 821M/3.71G
Storage: 18G/119G (root partition, f2fs, here lies the compose file) & 650G/932G (extra HDD, ext4)

Third environment: MacBook Pro running Docker in Minikube

⚠️ Works sometimes

OS: Catalina 10.15.7, Intel Core i5 (x86_64)
Minikube: minikube version: v1.28.0 commit: 986b1ebd987211ed16f8cc10aed7d2c42fc8392f
Docker: Docker version 20.10.21, build baeda1f82a

And, truth be told, I can't reproduce the panic very well.

I know it does not work when I start a clean container with empty mounts as a user (in this case, user: "510:80" which is my user and the "admin" group on Mac).

But I also had the panic starting as root, and I can't reproduce that anymore.

Stack trace

storage  | 2022-12-04T23:34:54.631587Z  INFO garage::server: Loading configuration...
storage  | 2022-12-04T23:34:54.631732Z  INFO garage::server: Loading configuration...
storage  | 2022-12-04T23:34:54.633931Z  INFO garage::server: Initializing background runner...
storage  | 2022-12-04T23:34:54.634020Z  INFO garage::server: Initializing background runner...
storage  | 2022-12-04T23:34:54.634231Z  INFO garage::server: Initializing Garage main data store...
storage  | 2022-12-04T23:34:54.634272Z  INFO garage::server: Initializing Garage main data store...
storage  | 2022-12-04T23:34:54.634558Z  INFO garage_model::garage: Opening database...
storage  | 2022-12-04T23:34:54.634604Z  INFO garage_model::garage: Opening database...
storage  | 2022-12-04T23:34:54.634758Z  INFO garage_model::garage: Opening LMDB database at: /opt/metadata/db.lmdb
storage  | 2022-12-04T23:34:54.634800Z  INFO garage_model::garage: Opening LMDB database at: /opt/metadata/db.lmdb
storage  | ======== PANIC (internal Garage error) ========
storage  | panicked at 'Unable to open LMDB DB: Io(Os { code: 12, kind: OutOfMemory, message: "Out of memory" })', garage.rs:135:53
storage  |
storage  | Panics are internal errors that Garage is unable to handle on its own.
storage  | They can be caused by bugs in Garage's code, or by corrupted data in
storage  | the node's storage. If you feel that this error is likely to be a bug
storage  | in Garage, please report it on our issue tracker a the following address:
storage  |
storage  |         https://git.deuxfleurs.fr/Deuxfleurs/garage/issues
storage  |
storage  | Please include the last log messages and the the full backtrace below in
storage  | your bug report, as well as any relevant information on the context in
storage  | which Garage was running when this error occurred.
storage  |
storage  | GARAGE VERSION: v0.8.0 [features: k2v, sled, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs]
storage  |
storage  | BACKTRACE:
storage  |    0: garage::main::{{closure}}::{{closure}}
storage  |    1: std::panicking::rust_panic_with_hook
storage  |    2: std::panicking::begin_panic_handler::{{closure}}
storage  |    3: std::sys_common::backtrace::__rust_end_short_backtrace
storage  |    4: rust_begin_unwind
storage  |    5: core::panicking::panic_fmt
storage  |    6: core::result::unwrap_failed
storage  |    7: garage_model::garage::Garage::new
storage  |    8: garage::server::run_server::{{closure}}
storage  |    9: garage::main::{{closure}}
storage  |   10: std::thread::local::LocalKey<T>::with
storage  |   11: tokio::park::thread::CachedParkThread::block_on
storage  |   12: tokio::runtime::thread_pool::ThreadPool::block_on
storage  |   13: garage::main
storage  |   14: std::sys_common::backtrace::__rust_begin_short_backtrace
storage  |   15: std::rt::lang_start::{{closure}}
storage  |   16: std::rt::lang_start_internal
storage  |   17: main
storage  |
storage exited with code 139

docker-compose.yml

version: "3.7"

services:
  garage:
    image: dxflrs/garage:v0.8.0
    #build: .
    container_name: storage
    environment:
      RUST_LOG: "debug"
    volumes:
      - ./garage.toml:/etc/garage.toml
      - ./data:/opt/data
      - ./metadata:/opt/metadata
    ports:
      - "3900:3900"
      - "3902:3902"
    restart: unless-stopped

generate-config.sh

#!/bin/sh

set -e

S3_API_DOMAIN="neptune.internal"
S3_API_PORT=3900
S3_WEB_DOMAIN="neptune.internal"
S3_WEB_PORT=3902

RPC_PORT=3901
ADMIN_API_PORT=3903
K2V_API_PORT=3904

REGION="home"

cat > garage.toml <<EOF
metadata_dir = "/opt/metadata"
data_dir = "/opt/data"
db_engine = "lmdb"

replication_mode = "none"

rpc_bind_addr = "[::]:$RPC_PORT"
rpc_public_addr = "127.0.0.1:$RPC_PORT"
rpc_secret = "$(openssl rand -hex 32)"

[s3_api]
s3_region = "$REGION"
api_bind_addr = "[::]:$S3_API_PORT"
root_domain = "$S3_API_DOMAIN"

[s3_web]
bind_addr = "[::]:$S3_WEB_PORT"
root_domain = "$S3_WEB_DOMAIN"
index = "index.html"

[k2v_api]
api_bind_addr = "[::]:$3904"

[admin]
api_bind_addr = "0.0.0.0:$ADMIN_API_PORT"
admin_token = "$(openssl rand -base64 32)"
EOF

First of all, thanks a lot for this amazing project! I encountered an issue where the application would panic right after starting. It seems to come from LMDB. _The only workaround known to me is to switch the DB engine to another one: I went with sqlite, but it is definitely, sensibly slower._ I tested it in three different environments, but I can't tell what's causing the panic. In each environment, I use docker and docker-compose to spin up Garage. I used the same `docker-compose.yml` and `garage.toml` files (see below), with one exception: On the MacBook, I used an own `dockerfile` to statically copy the `garage.toml` file - somehow it would mount the file as a directory inside the container... ### First environment: VPS running Arch Linux VPS ✅ **Works** Kernel: `Linux neptune.internal 6.0.10-arch2-1 #1 SMP PREEMPT_DYNAMIC Sat, 26 Nov 2022 16:51:18 +0000 x86_64 GNU/Linux` Docker: `Docker version 20.10.21, build baeda1f82a` ### Second environment: Raspberry Pi 4 running Void Linux musl ❌ **Does not work** Kernel: `Linux m-pi 5.15.72_1 #1 SMP PREEMPT Sun Oct 16 14:46:40 UTC 2022 aarch64 GNU/Linux` Docker: `Docker version 20.10.21, build tag v20.10.21` No matter where the mounted directories are located (in the same directory as the compose file, or on another HDD), and no matter what the permissions and ownership (root or not) are, it just won't work. At the time of running, this is the resource usage: - CPU load: 0.21 / 0.12 / 0.17 - RAM: 821M/3.71G - Storage: 18G/119G (root partition, f2fs, here lies the compose file) & 650G/932G (extra HDD, ext4) ### Third environment: MacBook Pro running Docker in Minikube ⚠️ **Works sometimes** OS: Catalina 10.15.7, Intel Core i5 (x86_64) Minikube: `minikube version: v1.28.0 commit: 986b1ebd987211ed16f8cc10aed7d2c42fc8392f` Docker: `Docker version 20.10.21, build baeda1f82a` And, truth be told, I can't reproduce the panic very well. I know it does **not** work when I start a clean container with empty mounts _as a user_ (in this case, `user: "510:80"` which is my user and the "admin" group on Mac). But I also had the panic starting as root, and I can't reproduce that anymore. ### Stack trace ``` storage | 2022-12-04T23:34:54.631587Z INFO garage::server: Loading configuration... storage | 2022-12-04T23:34:54.631732Z INFO garage::server: Loading configuration... storage | 2022-12-04T23:34:54.633931Z INFO garage::server: Initializing background runner... storage | 2022-12-04T23:34:54.634020Z INFO garage::server: Initializing background runner... storage | 2022-12-04T23:34:54.634231Z INFO garage::server: Initializing Garage main data store... storage | 2022-12-04T23:34:54.634272Z INFO garage::server: Initializing Garage main data store... storage | 2022-12-04T23:34:54.634558Z INFO garage_model::garage: Opening database... storage | 2022-12-04T23:34:54.634604Z INFO garage_model::garage: Opening database... storage | 2022-12-04T23:34:54.634758Z INFO garage_model::garage: Opening LMDB database at: /opt/metadata/db.lmdb storage | 2022-12-04T23:34:54.634800Z INFO garage_model::garage: Opening LMDB database at: /opt/metadata/db.lmdb storage | ======== PANIC (internal Garage error) ======== storage | panicked at 'Unable to open LMDB DB: Io(Os { code: 12, kind: OutOfMemory, message: "Out of memory" })', garage.rs:135:53 storage | storage | Panics are internal errors that Garage is unable to handle on its own. storage | They can be caused by bugs in Garage's code, or by corrupted data in storage | the node's storage. If you feel that this error is likely to be a bug storage | in Garage, please report it on our issue tracker a the following address: storage | storage | https://git.deuxfleurs.fr/Deuxfleurs/garage/issues storage | storage | Please include the last log messages and the the full backtrace below in storage | your bug report, as well as any relevant information on the context in storage | which Garage was running when this error occurred. storage | storage | GARAGE VERSION: v0.8.0 [features: k2v, sled, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs] storage | storage | BACKTRACE: storage | 0: garage::main::{{closure}}::{{closure}} storage | 1: std::panicking::rust_panic_with_hook storage | 2: std::panicking::begin_panic_handler::{{closure}} storage | 3: std::sys_common::backtrace::__rust_end_short_backtrace storage | 4: rust_begin_unwind storage | 5: core::panicking::panic_fmt storage | 6: core::result::unwrap_failed storage | 7: garage_model::garage::Garage::new storage | 8: garage::server::run_server::{{closure}} storage | 9: garage::main::{{closure}} storage | 10: std::thread::local::LocalKey<T>::with storage | 11: tokio::park::thread::CachedParkThread::block_on storage | 12: tokio::runtime::thread_pool::ThreadPool::block_on storage | 13: garage::main storage | 14: std::sys_common::backtrace::__rust_begin_short_backtrace storage | 15: std::rt::lang_start::{{closure}} storage | 16: std::rt::lang_start_internal storage | 17: main storage | storage exited with code 139 ``` ### docker-compose.yml ```yaml version: "3.7" services: garage: image: dxflrs/garage:v0.8.0 #build: . container_name: storage environment: RUST_LOG: "debug" volumes: - ./garage.toml:/etc/garage.toml - ./data:/opt/data - ./metadata:/opt/metadata ports: - "3900:3900" - "3902:3902" restart: unless-stopped ``` ### generate-config.sh ```bash #!/bin/sh set -e S3_API_DOMAIN="neptune.internal" S3_API_PORT=3900 S3_WEB_DOMAIN="neptune.internal" S3_WEB_PORT=3902 RPC_PORT=3901 ADMIN_API_PORT=3903 K2V_API_PORT=3904 REGION="home" cat > garage.toml <<EOF metadata_dir = "/opt/metadata" data_dir = "/opt/data" db_engine = "lmdb" replication_mode = "none" rpc_bind_addr = "[::]:$RPC_PORT" rpc_public_addr = "127.0.0.1:$RPC_PORT" rpc_secret = "$(openssl rand -hex 32)" [s3_api] s3_region = "$REGION" api_bind_addr = "[::]:$S3_API_PORT" root_domain = "$S3_API_DOMAIN" [s3_web] bind_addr = "[::]:$S3_WEB_PORT" root_domain = "$S3_WEB_DOMAIN" index = "index.html" [k2v_api] api_bind_addr = "[::]:$3904" [admin] api_bind_addr = "0.0.0.0:$ADMIN_API_PORT" admin_token = "$(openssl rand -base64 32)" EOF ```

lx commented

2022-12-05 13:07:07 +00:00

Owner

In theory there is no problem with mapping a huge portion of address space, as only small pieces of it are actually filled with data from the database file. The rest is not actually assigned to any physical memory on your computer, and it's the job of the OS Kernel to read the data from disk and put it at the correct location in RAM every time LMDB tries to read from it.

I think the out-of-memory error here is most likely due to a parameter in the configuration of your OS kernel that limits the size of such chunks of memory that can be allocated. On your OSX machine, this would seem consistent with the fact that mapping this portion of memory is allowed as root, but forbidden as non-root user. You can check whether a limitation on virtual memory is set using the ulimit -v command (make sure to run it in the same conditions as your Garage binary, so that you will see the limitations applied to it). You can also check /proc/<pid>/limits at the "Max address space" line.

Thank you for this detailed report. LMDB works in a bit of a special way, as it requires a large section of virtual memory address space to be reserved so that it can have a view into the entire database file (it doesn't have to read it all from disk into RAM, but it kind of pretends it did). Since we don't know in advance how big the LMDB database will be, we have to use a really big chunk of virtual address space for this, to be sure that we won't run into issues. In theory there is no problem with mapping a huge portion of address space, as only small pieces of it are actually filled with data from the database file. The rest is not actually assigned to any physical memory on your computer, and it's the job of the OS Kernel to read the data from disk and put it at the correct location in RAM every time LMDB tries to read from it. I think the out-of-memory error here is most likely due to a parameter in the configuration of your OS kernel that limits the size of such chunks of memory that can be allocated. On your OSX machine, this would seem consistent with the fact that mapping this portion of memory is allowed as root, but forbidden as non-root user. You can check whether a limitation on virtual memory is set using the `ulimit -v` command (make sure to run it in the same conditions as your Garage binary, so that you will see the limitations applied to it). You can also check `/proc/<pid>/limits` at the "Max address space" line.

nehu commented

2022-12-10 13:42:59 +00:00

Author

Thanks a lot for the info, @lx, and sorry for the delayed reply.

The theory regarding limits makes a lot of sense.

I had a look at the limits (in running containers, just to be sure), and they are indeed different:

Arch Linux VPS

core file size (blocks)         (-c) unlimited
data seg size (kb)              (-d) unlimited
scheduling priority             (-e) 0
file size (blocks)              (-f) unlimited
pending signals                 (-i) 15623
max locked memory (kb)          (-l) 8192
max memory size (kb)            (-m) unlimited
open files                      (-n) 1048576
POSIX message queues (bytes)    (-q) 819200
real-time priority              (-r) 0
stack size (kb)                 (-s) 8192
cpu time (seconds)              (-t) unlimited
max user processes              (-u) unlimited
virtual memory (kb)             (-v) unlimited
file locks                      (-x) unlimited

Void Linux Raspberry Pi

core file size (blocks)         (-c) 0
data seg size (kb)              (-d) unlimited
scheduling priority             (-e) 0
file size (blocks)              (-f) unlimited
pending signals                 (-i) 13114
max locked memory (kb)          (-l) 64
max memory size (kb)            (-m) unlimited
open files                      (-n) 4096
POSIX message queues (bytes)    (-q) 819200
real-time priority              (-r) 0
stack size (kb)                 (-s) 8192
cpu time (seconds)              (-t) unlimited
max user processes              (-u) 13114
virtual memory (kb)             (-v) unlimited
file locks                      (-x) unlimited

But even though I increased the number of open files to 1048576 and set the max locked memory to unlimited, it doesn't seem to make a difference for LMDB.

I'm not quite sure where to go from here. If the problem is limited to RPi installations (whatever may be different there), using SQLite isn't a huge dealbreaker.

Moreover, FWIW it might make more sense for me to open an issue in their issue tracker.

Either way, I guess I'll close the issue here: This problem doesn't seem to be specific to garage.

Thanks a lot for the info, @lx, and sorry for the delayed reply. The theory regarding limits makes a lot of sense. I had a look at the limits (in running containers, just to be sure), and they are indeed different: **Arch Linux VPS** ``` core file size (blocks) (-c) unlimited data seg size (kb) (-d) unlimited scheduling priority (-e) 0 file size (blocks) (-f) unlimited pending signals (-i) 15623 max locked memory (kb) (-l) 8192 max memory size (kb) (-m) unlimited open files (-n) 1048576 POSIX message queues (bytes) (-q) 819200 real-time priority (-r) 0 stack size (kb) (-s) 8192 cpu time (seconds) (-t) unlimited max user processes (-u) unlimited virtual memory (kb) (-v) unlimited file locks (-x) unlimited ``` **Void Linux Raspberry Pi** ``` core file size (blocks) (-c) 0 data seg size (kb) (-d) unlimited scheduling priority (-e) 0 file size (blocks) (-f) unlimited pending signals (-i) 13114 max locked memory (kb) (-l) 64 max memory size (kb) (-m) unlimited open files (-n) 4096 POSIX message queues (bytes) (-q) 819200 real-time priority (-r) 0 stack size (kb) (-s) 8192 cpu time (seconds) (-t) unlimited max user processes (-u) 13114 virtual memory (kb) (-v) unlimited file locks (-x) unlimited ``` But even though I increased the number of open files to 1048576 and set the max locked memory to unlimited, it doesn't seem to make a difference for LMDB. I'm not quite sure where to go from here. If the problem is limited to RPi installations (whatever may be different there), using SQLite isn't a huge dealbreaker. Moreover, FWIW it might make more sense for me to open an issue in their issue tracker. Either way, I guess I'll close the issue here: This problem doesn't seem to be specific to garage.