restic-alarm: send alerts for inactive restic backups stored in S3 buckets

Design

restic-alarm watches over S3(-compatible) buckets containing restic backups, and sends alert emails for buckets that have been inactive (= without new snapshots) for too long. Alerts and inactivity duration settings as well as contact email are configured separately for each bucket.

The buckets to watch and their respective configuration are stored in a separate bucket restic-alarm-state accessed by restic-alarm. Each bucket to watch corresponds to a file restic-alarm-state/watch/<bucket uid> containing the bucket's alert settings.

The S3 key id, secret, region and endpoint URL needed by restic-alarm to access restic-alarm-state and the buckets to watch are passed to the process through (standard) environment variables.

Installation (on NixOS)

1. Create a new S3 key for restic-alarm.
2. Create a new S3 bucket named restic-alarm-state and owned by this new key.
3. Install restic-alarm: import the nixos module provided by the flake as nixosModules.default, then add:

custom.restic-alarm = {
  enable = true;
  env_file = "/path/to/secret/env";
};

where /path/to/secret/env points to a file containing the environment variables for S3 access (region, endpoint, access key id, secret key). The env file should only be readable by root.

A template for the env file would be:

AWS_DEFAULT_REGION=xxxx
AWS_ENDPOINT_URL="https://xxxxx"
AWS_ACCESS_KEY_ID=xxxxx
AWS_SECRET_ACCESS_KEY=xxxxx

Adding a bucket to watched by restic-alarm

1. Give read-only access to restic-alarm's S3 key for your bucket.
2. Add a new file in the restic-alarm-state bucket at path watch/<bucket uid>, containing the following information:

# Email where to send inactivity alerts
email = "your@email"
# Inactivity after which to send an alert (in days).
# Here, send an alert if there has not been a new snapshot in more than 3 days. 
inactivity = 3
# Optional: short name for your bucket (can be anything)
name = "my_backups"
# Optional: interval between successive alerts (in days; default = 1 day)
alert_interval = 1
# Optional: duration after which we stop sending alerts (in days; default = 7 days)
alert_duration = 7

(the file is parsed as a toml file)

Since some of the fields are optional, a minimal configuration file can be as short as:

email = "your@email"
inactivity = 3