Better doc on how systemd DynamicUser=True works

kristof.p commented

2022-11-23 14:42:05 +00:00

Hello all,

I have garage working, starting from CLI "garage server",
but when try to start with systemd script from https://garagehq.deuxfleurs.fr/documentation/cookbook/systemd/ it is failing.

server:~ # journalctl -u garage
Nov 23 15:29:52 server systemd[1]: Started Garage Data Store.
Nov 23 15:29:52 server systemd[1]: garage.service: Main process exited, code=exited, status=238/STATE_DIRECTORY
Nov 23 15:29:52 server systemd[1]: garage.service: Failed with result 'exit-code'.

My configuration is:

server:~ # whoami
root

server:~ # garage -V
garage v0.8.0 [features: k2v, sled, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs]

server:~ # whereis garage
garage: /etc/garage.toml /usr/local/bin/garage

server:~ # grep dir /etc/garage.toml 
metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"

server:~ # ls -al /var/lib/ | grep garage
drwxr-xr-x 1 root    root     16 Nov 23 15:13 garage

server:~ # ls -al /var/lib/garage/
total 0
drwxr-xr-x 1 root root  16 Nov 23 15:13 .
drwxr-xr-x 1 root root 590 Nov 23 15:13 ..
drwxr-xr-x 1 root root   0 Nov 23 15:13 data
drwxr-xr-x 1 root root  92 Nov 23 15:13 meta

OS is linux openSuse Leap 15.3

Thank you for any help.

Hello all, I have garage working, starting from CLI "garage server", but when try to start with systemd script from https://garagehq.deuxfleurs.fr/documentation/cookbook/systemd/ it is failing. ``` server:~ # journalctl -u garage Nov 23 15:29:52 server systemd[1]: Started Garage Data Store. Nov 23 15:29:52 server systemd[1]: garage.service: Main process exited, code=exited, status=238/STATE_DIRECTORY Nov 23 15:29:52 server systemd[1]: garage.service: Failed with result 'exit-code'. ``` My configuration is: ``` server:~ # whoami root server:~ # garage -V garage v0.8.0 [features: k2v, sled, lmdb, sqlite, consul-discovery, kubernetes-discovery, metrics, telemetry-otlp, bundled-libs] server:~ # whereis garage garage: /etc/garage.toml /usr/local/bin/garage server:~ # grep dir /etc/garage.toml metadata_dir = "/var/lib/garage/meta" data_dir = "/var/lib/garage/data" server:~ # ls -al /var/lib/ | grep garage drwxr-xr-x 1 root root 16 Nov 23 15:13 garage server:~ # ls -al /var/lib/garage/ total 0 drwxr-xr-x 1 root root 16 Nov 23 15:13 . drwxr-xr-x 1 root root 590 Nov 23 15:13 .. drwxr-xr-x 1 root root 0 Nov 23 15:13 data drwxr-xr-x 1 root root 92 Nov 23 15:13 meta ``` OS is linux openSuse Leap 15.3 Thank you for any help.

kristof.p commented

2022-11-23 15:44:00 +00:00

Author

I think the cause of the problem is the same as in the bug #359.
Will continue to troubleshoot on tomorow. Stay tuned.

I think the cause of the problem is the same as in the bug #359. Will continue to troubleshoot on tomorow. Stay tuned.

quentin commented

2022-11-23 23:19:52 +00:00

Owner

I think the problem is due to the fact that you launched Garage outside of systemd once, it created these 2 folders: /var/lib/garage/{meta,data}. But our systemd service has many hardening, including DynamicUser=True, which triggers some magic in systemd that with create a folder named /var/lib/private/garage and symlink /var/lib/garage to it. But systemd cannot create this symlink as you already created a folder at this path.

2 solutions, either:

Remove DynamicUser=True for the service at the cost of a less sandboxed Garage process, run systemctl daemon-reload and systemctl restart garage
If you have no important data, run rm -rf /var/lib/garage and restart the service with systemctl restart garage

I think the problem is due to the fact that you launched Garage outside of systemd once, it created these 2 folders: /var/lib/garage/{meta,data}. But our systemd service has many hardening, including `DynamicUser=True`, which triggers some magic in systemd that with create a folder named `/var/lib/private/garage` and symlink `/var/lib/garage` to it. But systemd cannot create this symlink as you already created a folder at this path. 2 solutions, either: - Remove `DynamicUser=True` for the service at the cost of a less sandboxed Garage process, run `systemctl daemon-reload` and `systemctl restart garage` - If you have no important data, run `rm -rf /var/lib/garage` and restart the service with `systemctl restart garage`

kristof.p commented

2022-11-24 13:48:57 +00:00

Author

Hello Quentin,

you are right. I removed /var/lib/garage and everything starts OK with systemd script.

Can you add some info to documentation https://garagehq.deuxfleurs.fr/documentation/cookbook/systemd/ ?

Something like this:

Your garage.toml must be set with metadata_dir=/var/lib/garage/meta and data_dir=/var/lib/garage/data. This is mandatory to use systemd hardening feature Dynamic User, what autocreate these directories as virtual mapping. If directory /var/lib/garage exists, the systemd service doesnt start correctly. Note that in your host filesystem, Garage data will be held in /var/lib/private/garage.

Thank you very much for your help.

Hello Quentin, you are right. I removed /var/lib/garage and everything starts OK with systemd script. Can you add some info to documentation https://garagehq.deuxfleurs.fr/documentation/cookbook/systemd/ ? Something like this: Your garage.toml must be set with metadata_dir=/var/lib/garage/meta and data_dir=/var/lib/garage/data. This is mandatory to use systemd hardening feature Dynamic User, **what autocreate these directories as virtual mapping. If directory /var/lib/garage exists, the systemd service doesnt start correctly.** Note that in your host filesystem, Garage data will be held in /var/lib/private/garage. Thank you very much for your help.