nixcfg/README.md

Deuxfleurs on NixOS!

This repository contains code to run Deuxfleur's infrastructure on NixOS.

It sets up the following:

• A Wireguard mesh between all nodes
• Consul, with TLS
• Nomad, with TLS

How to use this?

See the following documentation topics:

• Quick start for adding new nodes after NixOS install
• Architecture of this repo, how the scripts work
• List of TCP and UDP ports used by services

Additionnal documentation topics:

• Succint guide for NixOS installation with LUKX full disk encryption (we don't do that in practice on our servers)
• Example hardware-config.nix for a full disk encryption scenario

Why not Ansible?

I often get asked why not use Ansible to deploy to remote machines, as this would look like a typical use case. There are many reasons, which basically boil down to "I really don't like Ansible":

• Ansible tries to do declarative system configuration, but doesn't do it correctly at all, like Nix does. Example: in NixOS, to undo something you've done, just comment the corresponding lines and redeploy.

• Ansible is massive overkill for what we're trying to do here, we're just copying a few small files and running some basic commands, leaving the rest to NixOS.

• YAML is a pain to manipulate as soon as you have more than two or three indentation levels. Also, why in hell would you want to write loops and conditions in YAML when you could use a proper expression language?

• Ansible's vocabulary is not ours, and it imposes a rigid hierarchy of directories and files which I don't want.

• Ansible is probably not flexible enough to do what we want, at least not without getting a migraine when trying. For example, it's inventory management is too simple to account for the heterogeneity of our cluster nodes while still retaining a level of organization (some configuration options are defined cluster-wide, some are defined for each site - physical location - we deploy on, and some are specific to each node).

• I never remember Ansible's command line flags.

• My distribution's package for Ansible takes almost 400MB once installed, WTF??? By not depending on it, we're reducing the set of tools we need to deploy to a bare minimum: Git, OpenSSH, OpenSSL, socat, pass (and the Consul and Nomad binaries which are, I'll admit, not small).

More

Please read README.more.md for more detailed information