OutOfMemory error #509

New Issue

Closed

opened 2023-02-23 11:35:31 +00:00 by oiseauroch · 3 comments

oiseauroch commented

2023-02-23 11:35:31 +00:00

I tried to launch garage on Raspberry pi 4 with lmdb DB. Launching the command server with a basic configuration result to panic with the following error :

2023-02-23T11:24:21.146736Z  INFO garage::server: Loading configuration...
2023-02-23T11:24:21.147161Z  INFO garage::server: Initializing background runner...
2023-02-23T11:24:21.147244Z  INFO garage::server: Initializing Garage main data store...
2023-02-23T11:24:21.147365Z  INFO garage_model::garage: Opening database...
2023-02-23T11:24:21.147392Z  INFO garage_model::garage: Opening LMDB database at: garage_data/meta/db.lmdb
======== PANIC (internal Garage error) ========
panicked at 'Unable to open LMDB DB: Io(Os { code: 12, kind: OutOfMemory, message: "Out of memory" })', garage.rs:138:53

Panics are internal errors that Garage is unable to handle on its own.
They can be caused by bugs in Garage's code, or by corrupted data in
the node's storage. If you feel that this error is likely to be a bug
in Garage, please report it on our issue tracker a the following address:

        https://git.deuxfleurs.fr/Deuxfleurs/garage/issues

Please include the last log messages and the the full backtrace below in
your bug report, as well as any relevant information on the context in
which Garage was running when this error occurred.

GARAGE VERSION: v0.8.1 [features: k2v, sled, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs]

BACKTRACE:
   0: garage::main::{{closure}}::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic_handler::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: core::result::unwrap_failed
   7: core::result::Result<T,E>::expect
   8: garage_model::garage::Garage::new
   9: garage::server::run_server::{{closure}}
  10: garage::main::{{closure}}
  11: std::thread::local::LocalKey<T>::with
  12: tokio::park::thread::CachedParkThread::block_on
  13: tokio::runtime::thread_pool::ThreadPool::block_on
  14: garage::main
  15: std::sys_common::backtrace::__rust_begin_short_backtrace
  16: std::rt::lang_start::{{closure}}
  17: std::rt::lang_start_internal
  18: main

Abandon

I used the following configuration :

metadata_dir = "garage_data/meta"
data_dir = "/garage_data/data" 
db_engine = "lmdb"

block_size = 1048576
block_manager_background_tranquility = 2

replication_mode = "3"

compression_level = 2

# openssl rand -hex 32
rpc_secret = "XXXXXXXXXXXXXXX"
rpc_bind_addr = "[::]:3901"
# Le port peut être différent (eg NAT) mais doit rediriger sur le
# port de rpc_bind_addr
rpc_public_addr = "garage.example.tld:3901"


[s3_api]
# Ne supporte pas TLS → reverse proxy obligatoire
# pour avoir de la sécurité
api_bind_addr = "[::]:3900"
s3_region = "garage"

Weird things are using the same configuration file and same binary from several platforms, I get the following results :

device	lmdb	without lmdb
rasp4 debian 10.16	panic	works
rasp4 debian 11.6	panic	works
Rockpro64 debian 11.6	works	works

Version of garage : 8.1

I tried to launch garage on Raspberry pi 4 with lmdb DB. Launching the command server with a basic configuration result to panic with the following error : ``` 2023-02-23T11:24:21.146736Z INFO garage::server: Loading configuration... 2023-02-23T11:24:21.147161Z INFO garage::server: Initializing background runner... 2023-02-23T11:24:21.147244Z INFO garage::server: Initializing Garage main data store... 2023-02-23T11:24:21.147365Z INFO garage_model::garage: Opening database... 2023-02-23T11:24:21.147392Z INFO garage_model::garage: Opening LMDB database at: garage_data/meta/db.lmdb ======== PANIC (internal Garage error) ======== panicked at 'Unable to open LMDB DB: Io(Os { code: 12, kind: OutOfMemory, message: "Out of memory" })', garage.rs:138:53 Panics are internal errors that Garage is unable to handle on its own. They can be caused by bugs in Garage's code, or by corrupted data in the node's storage. If you feel that this error is likely to be a bug in Garage, please report it on our issue tracker a the following address: https://git.deuxfleurs.fr/Deuxfleurs/garage/issues Please include the last log messages and the the full backtrace below in your bug report, as well as any relevant information on the context in which Garage was running when this error occurred. GARAGE VERSION: v0.8.1 [features: k2v, sled, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs] BACKTRACE: 0: garage::main::{{closure}}::{{closure}} 1: std::panicking::rust_panic_with_hook 2: std::panicking::begin_panic_handler::{{closure}} 3: std::sys_common::backtrace::__rust_end_short_backtrace 4: rust_begin_unwind 5: core::panicking::panic_fmt 6: core::result::unwrap_failed 7: core::result::Result<T,E>::expect 8: garage_model::garage::Garage::new 9: garage::server::run_server::{{closure}} 10: garage::main::{{closure}} 11: std::thread::local::LocalKey<T>::with 12: tokio::park::thread::CachedParkThread::block_on 13: tokio::runtime::thread_pool::ThreadPool::block_on 14: garage::main 15: std::sys_common::backtrace::__rust_begin_short_backtrace 16: std::rt::lang_start::{{closure}} 17: std::rt::lang_start_internal 18: main Abandon ``` I used the following configuration : ```toml metadata_dir = "garage_data/meta" data_dir = "/garage_data/data" db_engine = "lmdb" block_size = 1048576 block_manager_background_tranquility = 2 replication_mode = "3" compression_level = 2 # openssl rand -hex 32 rpc_secret = "XXXXXXXXXXXXXXX" rpc_bind_addr = "[::]:3901" # Le port peut être différent (eg NAT) mais doit rediriger sur le # port de rpc_bind_addr rpc_public_addr = "garage.example.tld:3901" [s3_api] # Ne supporte pas TLS → reverse proxy obligatoire # pour avoir de la sécurité api_bind_addr = "[::]:3900" s3_region = "garage" ``` Weird things are using the same configuration file and same binary from several platforms, I get the following results : | device | lmdb | without lmdb | | -------- | -------- | -------- | | rasp4 debian 10.16 | panic | works | | rasp4 debian 11.6 | panic | works | | Rockpro64 debian 11.6 | works | works | Version of garage : 8.1

w8emv commented

2023-02-25 05:49:02 +00:00

I'm seeing the same stack trace on my Pi 4, which is running Debian 10.

lx commented

2023-02-25 10:16:17 +00:00

Owner

Duplicate of #439

Duplicate of https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/439

lx closed this issue

2023-02-25 10:16:18 +00:00

lx commented

2023-10-14 11:39:46 +00:00

Owner

Ping @oiseauroch.

This seems to be a true issue with the raspi platform :( (see discussion in #439)

Since v0.9.0 the sqlite backend in garage doesn't suffer from such bad performance anymore as the fsync behavior is disabled by default, so it is a viable alternative for platforms which cannot run LMDB.

Ping @oiseauroch. This seems to be a true issue with the raspi platform :( (see discussion in #439) Since v0.9.0 the sqlite backend in garage doesn't suffer from such bad performance anymore as the fsync behavior is disabled by default, so it is a viable alternative for platforms which cannot run LMDB.