From dccb0c0eef9e04ed3e3f3dd3914027840ab61797 Mon Sep 17 00:00:00 2001 From: LUXEY Adrien Date: Thu, 4 Mar 2021 12:01:33 +0100 Subject: [PATCH] guide about installing a Debian remotely with full disk encryption --- README.md | 10 +- bootstrapper/README.md | 19 ++ bootstrapper/os_install/README.md | 405 ++++++++++++++++++++++++++++++ 3 files changed, 430 insertions(+), 4 deletions(-) create mode 100644 bootstrapper/README.md create mode 100644 bootstrapper/os_install/README.md diff --git a/README.md b/README.md index ca5c790..5b3af52 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,14 @@ Go to [deployer](deployer/), where the fun is at. +## Structure + +* [`/deployer`](deployer/) contains deployment Ansible code for Serenity. **Being outdated** by migration to new server (HammerHead). +* [`/bootstrapper`](bootstrapper/) contains Ansible configuration code for new servers (notably HammerHead). Will only take interest in the bootstrapping phase of the host. Service deployment will be handled by Nomad et al. + ## TODO -* dockerise nextcloud, synapse, gitea -* backup db -* send backups to a remote - +* Host bootstrap Ansible code to [`/bootstrapper`](bootstrapper/) # Random notes diff --git a/bootstrapper/README.md b/bootstrapper/README.md new file mode 100644 index 0000000..c0a2499 --- /dev/null +++ b/bootstrapper/README.md @@ -0,0 +1,19 @@ +# Bootstrap cluster hosts + +* `os_install/` contains documentation about OS installation for remote nodes +* The rest is Ansible configuration + +## Configuration goals + +* Configure access rights + + User creation, upload their ssh keys, set passwordless sudo... + +* Installation of base toolset through package manager + +* Installation of complex tooling (e.g. Docker, Nomad, Vault, Consul) through other means (custom APT repo?) + +* Basic configuration of the server + + * Docker log rotation + diff --git a/bootstrapper/os_install/README.md b/bootstrapper/os_install/README.md new file mode 100644 index 0000000..7809cdb --- /dev/null +++ b/bootstrapper/os_install/README.md @@ -0,0 +1,405 @@ +# Installing a cloud server with full disk encryption + +For hosting purposes, I am installing a new server on [Kimsufi](kimsufi.com/). This is a French public cloud provider (property of [OVH](ovh.com/)). + +It is well known that, at [Deuxfleurs](https://deuxfleurs.fr), we do not fancy hosting in datacenters: because of the lack on control on our machines, and potential privacy threats. In essence, people at Kimsufi could easily -- and without me knowing -- read my hard drive, and thus any sensitive user data stored inside. + +For that reason, I want to make it harder for their staff to access my data. A good first line of defense for that purpose is to encrypt the whole disk (*aka* Full Disk Encryption). Note that motivated attackers with physical access to my server would easily overcome the encryption. **If you really care about who accesses your data, keep your hard drives closest to you, e.g. at your home.** + +--- + +Long story short: this guide presents a step-by-step installation of Debian buster on Kimsufi, using the [`dm-crypt`](https://wiki.archlinux.org/index.php/Dm-crypt) Linux kernel subsystem to encrypt the `/` and the `swap` partitions. +Because I don't have physical access to the server, I need a way to *remotely enter the disk decryption password*, that is `dropbear` (a lightweight SSH server) inside `initramfs` (a minimal operating system that runs from RAM, before Linux boots). + +If you plan on doing the same without Kimsufi, know that it makes no difference: I am basically using a "rescue" console on another computer to configure `/dev/sda` (my server's system drive) for booting Linux with encrypted partitions. + +I must credit several useful guides that helped me succeed: + +* [OpsBlog guide on installing Linux with full-disk encryption and DropBear SSH on Kimsufi](https://opsblog.net/posts/full-disk-encrypted-ubuntu-kimsufi-sever/) + + DropBear allows SSH connection before boot, to remotely decrypt the main partition. + +* [Debian guide on installing Debian from a Linux system](https://www.debian.org/releases/wheezy/mipsel/apds03.html.en) + + + +## Overview of steps + +* Boot the Kimsufi server in rescue mode (gives you a remote shell with access to your system's hard drive). + +* Wipe filesystem, format partitions. I basically want: + + * A "cleartext" `/boot` partition + * An encrypted `/` partition + * An encrypted swap partition + + Docs on the encrypted swap: + * [ArchLinux wiki: dm-crypt - Swap encryption](https://wiki.archlinux.org/index.php/Dm-crypt/Swap_encryption) + * [cryptsetup FAQ](https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions#2-setup): see "2.3 How do I set up encrypted swap?" + +* Install and configure Debian (including disk cryptography and decryption through SSH). + +* Boot and log into the server. + +* Setup automated decryption from a remote (in case of reboot). + +## Detailed process + +### Boot the server in rescue mode + +* Go to the Kimsufi admin panel, click "NetBoot", select "Rescue", pick "rescue64-pro", clock "Next", "Confirm", and then click the "Reboot" button on the admin panel. + +* Kimsufi will mail you the root password for you to log into the server in rescue mode. + +* Will likely cause an error from your SSH client, since the fingerprint of the server differs from the usual one. + +You should end up with a shell that has access to `/dev/sda`, your server's drive. + +### Wipe filesystem & format partitions + +Target partition table: + +``` + +-----------+--------------------+-----------+--------------------+ +Name: | /dev/sda1 | /dev/sda2 | /dev/sda3 | /dev/sda4 | + | | | | | + | | | | | + | | +------------------+ | +------------------+ + | | | /dev/mapper/swap | | | /dev/mapper/main | +Format: | none | | swap | ext4 | | ext4 | +Size: | 1MiB | | 8GiB | 512MiB | | remaining space | +Flags: | bios_grub | | | | | | +Mount: | | | swap | /boot | | / | + +-----------+-+------------------+-----------+-+------------------+ +``` + +We use GPT partition table layout (without UEFI), which demands a small `bios_grub` partition at the beginning of the drive (stores GRUB's `core.img`). Hence `/dev/sda1`. See: + +* [Wikipedia: BIOS boot partition](https://en.wikipedia.org/wiki/BIOS_boot_partition) +* [GNU: BIOS installation](https://www.gnu.org/software/grub/manual/grub/html_node/BIOS-installation.html#BIOS-installation) +* [ArchLinux wiki: Partitioning](https://wiki.archlinux.org/index.php/partitioning#Partition_table) +* [Gentoo wiki: Installation - Disks](https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Disks#Partition_tables) + + +* Wipe all information about previous filesystem: + + DANGER!!! DANGER!!! WIPES EVERYTHING FROM YOUR DISK!!! + + ```bash + wipefs -a /dev/sda + ``` + +* Create GPT disk layout + + ```bash + # Create GPT layout + parted -a optimal /dev/sda mklabel gpt + # Create first 1MiB bios_grub partition + parted /dev/sda -a optimal mkpart bios_grub 0% 1MiB + parted /dev/sda set 1 bios_grub on + # Create third 8GiB swap partition + parted /dev/sda -a optimal mkpart swap 1MiB 8001MiB + # Create second 512MiB /boot partition + parted /dev/sda -a optimal mkpart boot 8001MiB 8513MiB + # Create last / partition using all remaining space + parted /dev/sda -a optimal mkpart main 8513MiB 100% + # Set first partition as bootable + ``` + +* Format partitions + + * The ext4 boot partition + + Format it like so: + + ```bash + mkfs.ext4 /dev/sda3 + ``` + + * The encrypted swap partition + + All configuration of the encrypted swap partition comes after the OS installation. + + * The encrypted main partition + + ```bash + # Set it up using cryptsetup (I searched for good parameters): + cryptsetup --type luks2 --cipher aes-xts-plain64 --hash sha256 --iter-time 2000 --key-size 512 luksFormat /dev/sda4 + + # Get the UUID, save it for later: + cryptsetup luksDump /dev/sda4 | grep UUID | awk '{print $2}' + # Example output: 09762476-ba7c-4732-8856-44a716c23339 + + # Decrypt the partition: + cryptsetup open /dev/sda4 main + + # Format it to ext4: + mkfs.ext4 /dev/mapper/main + ``` + +### Install and configure Debian + +* Mount the partitions + + ```bash + # Decrypt the main partition (if not already done): + cryptsetup open /dev/sda4 main + # mount it: + mount /dev/mapper/main /mnt + # Mount the boot partition: + mkdir /mnt/boot + mount /dev/sda3 /mnt/boot + ``` + +* Bootstrap latest stable Debian into mounted partitions + + ```bash + debootstrap --arch amd64 stable /mnt http://ftp.fr.debian.org/debian/ + ``` + +* Mount system partitions + + ```bash + mount -o bind /dev /mnt/dev + mount -t proc proc /mnt/proc + mount -t sysfs sys /mnt/sys + mount -t devpts devpts /mnt/dev/pts + ``` + +* Chroot + + chroot /mnt /bin/bash + +* Fill out `crypttab` and `fstab` (`UUID` is the output of above `cryptsetup luksDump`) + + `/etc/crypttab` + + ```bash + cat << EOF > /etc/crypttab + # + main UUID=09762476-ba7c-4732-8856-44a716c23339 none luks + swap /dev/sda2 /dev/urandom swap,noearly,cipher=aes-xts-plain64,size=512 + EOF + ``` + + `/etc/fstab` + + ```bash + cat << EOF > /etc/fstab + # + /dev/mapper/main / ext4 errors=remount-ro 0 1 + /dev/sda3 /boot ext4 defaults 0 2 + /dev/mapper/swap none swap sw 0 0 + EOF + ``` + +* Do the `/proc/mounts` -> `/etc/mtab` symlink + + ```bash + ln -s /proc/mounts /etc/mtab + ``` + + Why? [Linux From Stratch](http://archive.linuxfromscratch.org/lfs-museum/3.3/LFS-BOOK-3.3-HTML/chapter06/mtablink.html) and [our guide on OpsBlog](https://opsblog.net/posts/full-disk-encrypted-ubuntu-kimsufi-sever/) both say it's needed. + +* Choose a hostname & DNS domain + + hammerhead & hammerhead.luxeylab.net + +* Fill in network-related files + + Beware of your ethernet interface name! It's only after failing with `eth0` that the boot logs informed me that the iface name was `eno1`. + + `/etc/network/interfaces` + + ```bash + cat << EOF > /etc/network/interfaces + ###################################################################### + # /etc/network/interfaces -- configuration file for ifup(8), ifdown(8) + # See the interfaces(5) manpage for information on what options are + # available. + ###################################################################### + + # loopback interface + auto lo + iface lo inet loopback + + # eno1 through DHCP + auto eno1 + iface eno1 inet dhcp + EOF + ``` + + `/etc/resolv.conf` + + ```bash + cat << EOF > /etc/resolv.conf + # OVH public DNS + nameserver 213.186.33.99 + # https://servers.opennicproject.org/edit.php?srv=ns2.he.de.dns.opennic.glue + nameserver 172.104.136.243 + EOF + ``` + + `/etc/hosts` + + ```bash + cat << EOF > /etc/hosts + 127.0.0.1 localhost + 127.0.1.1 hammerhead hammerhead.luxeylab.net + + # The following lines are desirable for IPv6 capable hosts + ::1 localhost ip6-localhost ip6-loopback + fe00::0 ip6-localnet + ff00::0 ip6-mcastprefix + ff02::1 ip6-allnodes + ff02::2 ip6-allrouters + ff02::3 ip6-allhosts + + 213.186.33.116 ws.ovh.com + 198.245.48.4 ws.ovh.ca + 5.135.179.11 hammerhead hammerhead.luxeylab.net + EOF + ``` + + `/etc/hostname` + + ```bash + echo "hammerhead" > /etc/hostname + ``` + + `/etc/timezone` + + ```bash + dpkg-reconfigure tzdata + ``` + +* Update APT repositories in `/etc/apt/sources.list` + + ```bash + cat << EOF > /etc/apt/sources.list + deb http://ftp.fr.debian.org/debian buster main + deb-src http://ftp.fr.debian.org/debian buster main + + deb http://ftp.fr.debian.org/debian-security/ buster/updates main + deb-src http://ftp.fr.debian.org/debian-security/ buster/updates main + + deb http://ftp.fr.debian.org/debian buster-updates main + deb-src http://ftp.fr.debian.org/debian buster-updates main + EOF + ``` + +* Configure locales & keyboard + + ```bash + # Refresh packages list + apt update + # Install & configure locales + apt install locales && dpkg-reconfigure locales + # Install & configure keyboard + apt install console-setup console-data && dpkg-reconfigure keyboard-configuration + ``` + +* Pick up a kernel & install it + + ```bash + apt-cache search linux-image + # I selected the meta-package `linux-image-amd64` which is the latest + apt install linux-image-amd64 linux-headers-amd64 + ``` + +* Install additional necessary software + + ```bash + apt install cryptsetup dropbear grub-pc man-db ssh + ``` + + Because we use a GPT table layout but no UEFI, we install `grub-pc` and not `grub-efi-amd64`. + + Do think about installing `bash-completion` and `ufw` someday. + + +* Set up SSH keys and the rest (where `` is your client SSH public key) + + ```bash + mkdir /root/.ssh && chmod 600 /root/.ssh + echo "" > /root/.ssh/authorized_keys + echo "" > /etc/dropbear-initramfs/authorized_keys + ``` + +* Configure dropbear-initramfs (SSH on boot) + + ```bash + sed -i 's/#DROPBEAR_OPTIONS=/DROPBEAR_OPTIONS=\"-p 2222 -c \/bin\/cryptroot-unlock\"/' /etc/dropbear-initramfs/config + ``` + + + This changes the SSH listen post to 2222, and enforces that the only command run from `dropbear-initramfs` is `cryptroot-unlock`. + +* Configure sshd custom listen port (SSH after boot) + + ```bash + sed -i 's/#Port 22/Port 2223/' /etc/ssh/sshd_config + ``` + +* [Optional] Authorize root password login + + ```bash + # Remove -s (disable password login) from dropbear-initramfs + sed -i 's/local flags=\"Fs\"/local flags=\"F\"/' /usr/share/initramfs-tools/scripts/init-premount/dropbear + ``` + +* Update GRUB and initramfs + + ```bash + update-grub && update-initramfs -u + ``` + +* Clean up before leaving + + ```bash + exit + umount /mnt/{boot,dev/pts,dev,proc,sys} + umount /mnt + cryptsetup luksClose main + ``` + +### Boot and log into the server + +* I configured my laptop's `~/.ssh/config` like so: + + ``` + Host hammerhead-decrypt + User root + Hostname + Port 2222 + IdentityFile + # Prevents signatures mismatch + UserKnownHostsFile ~/.ssh/known_hosts_hammerhead-decrypt + Host hammerhead + User root + Hostname + Port 2223 + IdentityFile + ``` + +* And here is how I connect to the server: + + ```bash + laptop$ ssh hammerhead-decrypt + Please unlock disk main: + # [...] + Connection to closed. + laptop$ ssh hammerhead + # [...] + root@hamerhead:~# echo A winner is you! + A winner is you! + ``` + + +### Setup automated decryption from a remote + +It is desirable to have a daemon running on a remote server, to automatically decrypt the drive when the encrypted server reboots without warning. + +The remote server is called a *key escrow*. One must be particularly careful about the escrow's security, since it holds the decryption keys for our server. + +Trinity recommends [Tang](https://github.com/latchset/tang) and [Clevis](https://github.com/latchset/clevis). \ No newline at end of file