nixcfg/doc/nixos-install.md

Preparation

Download NixOS 21.11 ISO. Burn to USB.

Booting into install environment

Boot the ISO on PC to install.

Become root with sudo su

loadkeys fr
setfont sun12x22

Do network config if necessary, see install guide

Make partitions

cgdisk /dev/sda

Recommended layout:

/dev/sda1 	512M 	ef00 	EFI System partition
/dev/sda2 	100% 	8309 	Linux LUKS 

Setup cryptography

cryptsetup luksFormat /dev/sda2
cryptsetup open /dev/sda2 cryptlvm

Create PV, VG and LVs

pvcreate /dev/mapper/cryptlvm
vgcreate NixosVG /dev/mapper/cryptlvm
lvcreate -L 8G NixosVG -n swap
lvcreate -l 100%FREE NixosVG -n root

Format partitions

mkfs.fat -F 32 -n boot /dev/sda1
mkswap /dev/NixosVG/swap
mkfs.ext4 /dev/NixosVG/root

Mount partitions

swapon /dev/NixosVG/swap
mount /dev/NixosVG/root /mnt
mkdir /mnt/boot
mount /dev/sda1 /mnt/boot

Generate base NixOS configuration

nixos-generate-config --root /mnt

Update hardware-configuration.nix

This section is needed:

  boot.initrd.luks.devices."cryptlvm" = {
    device = "/dev/disk/by-uuid/<uuid of sda2>";
    allowDiscards = true;
  };

And for the root filesystem, remember to add the relatime and discard options so that it looks like this:

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/<...>";
      fsType = "ext4";
      options = [ "relatime" "discard" ];
    };

Update configuration.nix

Just enough so that basic tasks can be done from keyboard and remotely:

• timezone
• keyboard layout
• font sun12x22
• vim
• user
• ssh
• ssh port in firewall

Do the installation

nixos-install

First boot

Reboot machine. Login as root

passwd <user>

If necessary, assign static IP: ip addr add 192.168.1.40/24 dev eno1 or sth (replace ip and device appropriately)

Remotely: ssh-copy-id <user>@<ip>. Check SSH access is good.

Deploy from this repo

It's time!

Changes in this repo:

• create node .nix file, site .nix file if neccessary, and symlink for node .site.nix (create site and cluster files if necessary; use existing files of e.g. the staging cluster as examples/templates)
• make sure values are filled in correctly
• add node to ssh_config with it's LAN IP, we don't have VPN at this stage

Configuration steps on the node:

# On node being installed
mkdir -p /var/lib/deuxfleurs/remote-unlock
cd /var/lib/deuxfleurs/remote-unlock
ssh-keygen -t ed25519 -N "" -f ./ssh_host_ed25519_key

Try to deploy:

# In nixcfg repository from your PC
./deploy.sh <cluster> <nodename>

Reboot.

Check remote unlocking works: ssh -p 222 root@<ip>

Configure wireguard

Create wireguard keys:

On the node:

# On node being installed
mkdir -p /var/lib/deuxfleurs/wireguard-keys
cd /var/lib/deuxfleurs/wireguard-keys
wg genkey | tee private | wg pubkey > public

Get the public key, make sure it is in cluster.nix so that nodes know one another. Also put it anywhere else like in your local wireguard config for instance so that you can access the node from your PC by its wireguard address and not only its LAN address.

Redo a deploy (./deploy.sh <cluster> <nodename>)

Configure Nomad and Consul TLS

If you are bootstraping a new cluster, you need to ./genpki.sh <cluster> to make a TLS PKI for the Nomad+Consul cluster to work. Then redo a deploy.