Garage reports wrong disk available size when run in systemd with DynamicUser=True #611

New Issue

Closed

opened 2023-08-08 11:02:07 +00:00 by quentin · 0 comments

quentin commented

2023-08-08 11:02:07 +00:00

Owner

Bug

[root@garage-got:~]# garage stats
[...]
  ID                Hostname    Zone  Capacity  Part.  DataAvail                  MetaAvail
  5582e579c00e76be  garage-got  got   100       256    206.4 GB/213.2 GB (96.8%)  206.4 GB/213.2 GB (96.8%)
[...]
[root@garage-got:~]# df -h /var/lib/garage /var/lib/garage/data/
Filesystem         Size  Used Avail Use% Mounted on
madeye-root/local  199G  6.4G  193G   4% /
madeye/e/garage    2.6T   30G  2.6T   2% /var/lib/private/garage/data

In this example, Garage reports 213GB instead of 2.6TB.
The daemon is started by Systemd with DynamicUser=True set.

Diagnostic

Garage uses the systemstat crate to fetch available space on a disk: https://github.com/valpackett/systemstat

And calls its mount() function: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/rpc/system.rs#L894-L895
Then find the longest mount point that matches the beginning of the meta/data folders.

I think the mount points are not rewritten/scoped in systemd: if garage sees the /var/lib/garage/data folder, when it lists the mount points, it gets /var/lib/private/garage/data.

Possible solution

(quoting Linux Hackerman on Garage Matrix channel) ah yeah that seems wrong
it should probably statfs the data dir. I don't know what the portability requirements are on garage, but https://docs.rs/nix/latest/nix/sys/statvfs/index.html looks appropriate in principle

## Bug ``` [root@garage-got:~]# garage stats [...] ID Hostname Zone Capacity Part. DataAvail MetaAvail 5582e579c00e76be garage-got got 100 256 206.4 GB/213.2 GB (96.8%) 206.4 GB/213.2 GB (96.8%) [...] [root@garage-got:~]# df -h /var/lib/garage /var/lib/garage/data/ Filesystem Size Used Avail Use% Mounted on madeye-root/local 199G 6.4G 193G 4% / madeye/e/garage 2.6T 30G 2.6T 2% /var/lib/private/garage/data ``` In this example, Garage reports 213GB instead of 2.6TB. The daemon is started by Systemd with `DynamicUser=True` set. ## Diagnostic Garage uses the systemstat crate to fetch available space on a disk: https://github.com/valpackett/systemstat And calls its mount() function: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main/src/rpc/system.rs#L894-L895 Then find the longest mount point that matches the beginning of the meta/data folders. I think the mount points are not rewritten/scoped in systemd: if garage sees the /var/lib/garage/data folder, when it lists the mount points, it gets /var/lib/private/garage/data. ## Possible solution (quoting Linux Hackerman on Garage Matrix channel) ah yeah that seems wrong it should probably statfs the data dir. I don't know what the portability requirements are on garage, but https://docs.rs/nix/latest/nix/sys/statvfs/index.html looks appropriate in principle