From f5f0927b9ecb781aca649b0ae32e665b0e06a95f Mon Sep 17 00:00:00 2001
From: Alex Auvolat <alex@adnab.me>
Date: Fri, 25 Feb 2022 21:54:53 +0100
Subject: [PATCH] write some minimal documentation to get nodes up and running

---
 doc/nixos-install.md | 176 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 176 insertions(+)
 create mode 100644 doc/nixos-install.md

diff --git a/doc/nixos-install.md b/doc/nixos-install.md
new file mode 100644
index 0000000..f2fcd2d
--- /dev/null
+++ b/doc/nixos-install.md
@@ -0,0 +1,176 @@
+## Preparation
+
+Download NixOS 21.11 ISO. Burn to USB.
+
+## Booting into install environment
+
+Boot the ISO on PC to install.
+
+Become root with `sudo su`
+
+```bash
+loadkeys fr
+setfont sun12x22
+```
+
+Do network config if necessary, see [install guide](https://nixos.org/manual/nixos/stable/index.html#sec-installation-booting-networking)
+
+## Make partitions
+
+```bash
+cgdisk /dev/sda
+```
+
+Recommended layout:
+
+```
+/dev/sda1 	512M 	ef00 	EFI System partition
+/dev/sda2 	100% 	8309 	Linux LUKS 
+```
+
+## Setup cryptography 
+
+```bash
+cryptsetup luksFormat /dev/sda2
+cryptsetup open /dev/sda2 cryptlvm
+```
+
+## Create PV, VG and LVs
+
+```bash
+pvcreate /dev/mapper/cryptlvm
+vgcreate NixosVG /dev/mapper/cryptlvm
+lvcreate -L 8G NixosVG -n swap
+lvcreate -l 100%FREE NixosVG -n root
+```
+
+## Format partitions
+
+```bash
+mkfs.fat -F 32 -n boot /dev/sda1
+mkswap /dev/NixosVG/swap
+mkfs.ext4 /dev/NixosVG/root
+```
+
+## Mount partitions
+
+```bash
+swapon /dev/NixosVG/swap
+mount /dev/NixosVG/root /mnt
+mkdir /mnt/boot
+mount /dev/sda1 /mnt/boot
+```
+
+## Generate base NixOS configuration
+
+```bash
+nixos-generate-config --root /mnt
+```
+
+## Update `hardware-configuration.nix`
+
+This section is needed:
+
+```nix
+  boot.initrd.luks.devices."cryptlvm" = {
+    device = "/dev/disk/by-uuid/<uuid of sda2>";
+    allowDiscards = true;
+  };
+```
+
+And for the root filesystem, remember to add the `relatime` and `discard` options so that it looks like this:
+
+```nix
+  fileSystems."/" =
+    { device = "/dev/disk/by-uuid/<...>";
+      fsType = "ext4";
+      options = [ "relatime" "discard" ];
+    };
+```
+
+## Update `configuration.nix`
+
+Just enough so that basic tasks can be done from keyboard and remotely:
+
+- timezone
+- keyboard layout
+- font `sun12x22`
+- vim
+- user
+- ssh
+- ssh port in firewall
+
+## Do the installation
+
+```bash
+nixos-install
+```
+
+## First boot
+
+Reboot machine. Login as `root`
+
+```bash
+passwd <user>
+```
+
+If necessary, assign static IP: `ip addr add 192.168.1.40/24 dev eno1` or sth (replace ip and device appropriately)
+
+Remotely: `ssh-copy-id <user>@<ip>`. Check SSH access is good.
+
+## Deploy from this repo
+
+It's time!
+
+**Changes in this repo:**
+
+- create node `.nix` file, site `.nix` file if neccessary, and symlink for node `.site.nix`
+  (create site and cluster files if necessary; use existing files of e.g.
+  the staging cluster as examples/templates)
+- make sure values are filled in correctly
+- add node to `ssh_config` with it's LAN IP, we don't have VPN at this stage
+
+**Configuration steps on the node:**
+
+```bash
+# On node being installed
+mkdir -p /var/lib/deuxfleurs/remote-unlock
+cd /var/lib/deuxfleurs/remote-unlock
+ssh-keygen -t ed25519 -N "" -f ./ssh_host_ed25519_key
+```
+
+**Try to deploy:**
+
+```bash
+# In nixcfg repository from your PC
+./deploy.sh <cluster> <nodename>
+```
+
+Reboot.
+
+Check remote unlocking works: `ssh -p 222 root@<ip>`
+
+## Configure wireguard
+
+**Create wireguard keys:**
+
+On the node:
+
+```bash
+# On node being installed
+mkdir -p /var/lib/deuxfleurs/wireguard-keys
+cd /var/lib/deuxfleurs/wireguard-keys
+wg genkey | tee private | wg pubkey > public
+```
+
+Get the public key, make sure it is in `cluster.nix` so that nodes know one
+another.  Also put it anywhere else like in your local wireguard config for
+instance so that you can access the node from your PC by its wireguard address
+and not only its LAN address.
+
+Redo a deploy (`./deploy.sh <cluster> <nodename>`)
+
+## Configure Nomad and Consul TLS
+
+If you are bootstraping a new cluster, you need to `./genpki.sh <cluster>` to
+make a TLS PKI for the Nomad+Consul cluster to work. Then redo a deploy.