restic-alarm: send alerts for inactive restic backups stored in S3 buckets

Design

restic-alarm watches over S3(-compatible) buckets containing restic backups, and sends alert emails for buckets that have been inactive (= without new snapshots) for too long. Alerts and inactivity duration settings as well as contact email are configured separately for each bucket.

The buckets to watch and their respective configuration are stored in a separate bucket restic-alarm-state accessed by restic-alarm. Each bucket to watch corresponds to a file restic-alarm-state/watch/<bucket uid> containing the bucket's alert settings.

The S3 access key and secret, S3 region and endpoint URL needed by restic-alarm to access its bucket and the buckets to watch are passed to the process through (standard) environment variables.

Installation (on NixOS)

1. Create an S3 access key for restic-alarm.
2. Create a S3 bucket owned by this key and named restic-alarm-state.
3. Install restic-alarm: import the nixos module provided by the flake as nixosModules.default, then add:

custom.restic-alarm = {
  enable = true;
  env_file = "/path/to/secret/env";
};

where /path/to/secret/env points to a file containing the environment variables for S3 access (region, endpoint, access key id, secret key). The env file should only be readable by root.

A template for the env file to start from would be:

AWS_DEFAULT_REGION=xxxx
AWS_ENDPOINT_URL="https://xxxxx"
AWS_ACCESS_KEY_ID=xxxxx
AWS_SECRET_ACCESS_KEY=xxxxx

Adding a bucket to be monitored by restic-alarm

1. Give read-only access to your bucket for restic-alarm's S3 key.
2. Add a new file in the restic-alarm-state bucket at path watch/<bucket uid>, containing the following information:

# Email where to send inactivity alerts
email = "your@email"
# Optional: short name for your bucket (can be anything)
name = "my_backups"
# Inactivity after which to send an alert (in days).
# Here, send an alert if there has not been a new snapshot in more than 3 days. 
inactivity = 3
# Optional: interval between successive alerts (in days; default = 1 day)
alert_interval = 1
# Optional: duration after which we stop sending alerts (in days; default = 7 days)
alert_duration = 7

(the file is parsed as a toml file)

Since some of the fields are optional, a minimal configuration file would be e.g.:

email = "your@email"
inactivity = 3