ANSIBLE

How to proceed

For each machine, one by one do:

Check that cluster is healthy
- Check garage
  - check that all nodes are online docker exec -ti xxx /garage status
  - check that tables are in sync docker exec -ti 63a4d7ecd795 /garage repair --yes tables
  - check garage logs
    - no unknown errors or resync should be in progress
    - the following line must appear INFO garage_util::background > Worker exited: Repair worker
- Check that Nomad is healthy
  - nomad server members
  - nomad node status
- Check that Consul is healthy
  - consul members
- Check that Postgres is healthy
Run ansible-playbook -i production.yml --limit <machine> -u <username> site.yml
Run nomad node drain -enable -force -self
Reboot
Run nomad node drain -self -disable
Check that cluster is healthy (basically the whole first point)