Error: Could not save peer list to file: IO error: No file descriptors available (os error 24) #481

New issue

Closed

opened 2023-01-17 10:45:03 +00:00 by TJHui · 2 comments

TJHui commented

2023-01-17 10:45:03 +00:00

WARN garage_rpc::system: Could not save peer list to file: IO error: No file descriptors available (os error 24)

Garage: 0.8.0 (linux/amd64)
Server OS: Debian 11

My knowledge of OS internals is quite limited, I have seen descriptions of raising the number of allowed file descriptors, but that seems like a temporary solution at best. Is this a configuration issue? Or a memory leak?

Help is appreciated!

Kind regards.

Hi there, We're currently running Garage (0.8.0) on our server, as of yet one node only, and after about a day the garage service has to be restarted because of the following error, which completely fills the logs: ``` WARN garage_rpc::system: Could not save peer list to file: IO error: No file descriptors available (os error 24) ``` Garage: 0.8.0 (linux/amd64) Server OS: Debian 11 My knowledge of OS internals is quite limited, I have seen descriptions of raising the number of allowed file descriptors, but that seems like a temporary solution at best. Is this a configuration issue? Or a memory leak? Help is appreciated! Kind regards.

TJHui commented

2023-01-17 10:47:25 +00:00

Author

I just noticed a new version 0.8.1 is available. Could that solve the issue? Is that a drop in replacement, or should I follow the migration guide here too?

Also, our garage.toml:

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "lmdb"

replication_mode = "none"

rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "...."

[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:2606"
root_domain = ".something.something.tld"

[admin]
api_bind_addr = "0.0.0.0:3903"
admin_token = "...."

I just noticed a new version 0.8.1 is available. Could that solve the issue? Is that a drop in replacement, or should I follow the [migration guide](https://garagehq.deuxfleurs.fr/documentation/working-documents/migration-08/) here too? Also, our `garage.toml`: ``` metadata_dir = "/var/lib/garage/meta" data_dir = "/var/lib/garage/data" db_engine = "lmdb" replication_mode = "none" rpc_bind_addr = "[::]:3901" rpc_public_addr = "127.0.0.1:3901" rpc_secret = "...." [s3_api] s3_region = "garage" api_bind_addr = "[::]:2606" root_domain = ".something.something.tld" [admin] api_bind_addr = "0.0.0.0:3903" admin_token = "...." ```

lx commented

2023-01-17 10:58:28 +00:00

Owner

You do need to raise your file descriptor limit. It's not a temporary solution, there are some usage scenarios where these limits by default are simply too restrictive, especially for server software. See these two pages for Debian: https://wiki.debian.org/Limits, https://manpages.debian.org/bullseye/libpam-modules/limits.conf.5.en.html. In your case, the nofile limit has to be raised in /etc/security/limits.conf

Concerning upgrading to v0.8.1 from v0.8.0, there is nothing to do, just restart your garage daemon with the new version.

You do need to raise your file descriptor limit. It's not a temporary solution, there are some usage scenarios where these limits by default are simply too restrictive, especially for server software. See these two pages for Debian: https://wiki.debian.org/Limits, https://manpages.debian.org/bullseye/libpam-modules/limits.conf.5.en.html. In your case, the `nofile` limit has to be raised in `/etc/security/limits.conf` Concerning upgrading to v0.8.1 from v0.8.0, there is nothing to do, just restart your garage daemon with the new version.

👍 1