automation/bootstrapper/os_install/README.md

# Installing a cloud server with full disk encryption

For hosting purposes, I am installing a new server on [Kimsufi](kimsufi.com/). This is a French public cloud provider (property of [OVH](ovh.com/)). 

It is well known that, at [Deuxfleurs](https://deuxfleurs.fr), we do not fancy hosting in datacenters: because of the lack on control on our machines, and potential privacy threats. In essence, people at Kimsufi could easily -- and without me knowing -- read my hard drive, and thus any sensitive user data stored inside. 

For that reason, I want to make it harder for their staff to access my data. A good first line of defense for that purpose is to encrypt the whole disk (*aka* Full Disk Encryption). Note that motivated attackers with physical access to my server would easily overcome the encryption. **If you really care about who accesses your data, keep your hard drives closest to you, e.g. at your home.**

--- 

Long story short: this guide presents a step-by-step installation of Debian buster on Kimsufi, using the [`dm-crypt`](https://wiki.archlinux.org/index.php/Dm-crypt) Linux kernel subsystem to encrypt the `/` and the `swap` partitions.
Because I don't have physical access to the server, I need a way to *remotely enter the disk decryption password*, that is `dropbear` (a lightweight SSH server) inside `initramfs` (a minimal operating system that runs from RAM, before Linux boots).

If you plan on doing the same without Kimsufi, know that it makes no difference: I am basically using a "rescue" console on another computer to configure `/dev/sda` (my server's system drive) for booting Linux with encrypted partitions.

I must credit several useful guides that helped me succeed:

* [OpsBlog guide on installing Linux with full-disk encryption and DropBear SSH on Kimsufi](https://opsblog.net/posts/full-disk-encrypted-ubuntu-kimsufi-sever/)

    DropBear allows SSH connection before boot, to remotely decrypt the main partition.

* [Debian guide on installing Debian from a Linux system](https://www.debian.org/releases/wheezy/mipsel/apds03.html.en)



## Overview of steps 

* Boot the Kimsufi server in rescue mode (gives you a remote shell with access to your system's hard drive).

* Wipe filesystem, format partitions. I basically want:

    * A "cleartext" `/boot` partition
    * An encrypted `/` partition
    * An encrypted swap partition 

        Docs on the encrypted swap: 
        * [ArchLinux wiki: dm-crypt - Swap encryption](https://wiki.archlinux.org/index.php/Dm-crypt/Swap_encryption)
        * [cryptsetup FAQ](https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions#2-setup): see "2.3 How do I set up encrypted swap?"

* Install and configure Debian (including disk cryptography and decryption through SSH).

* Boot and log into the server.

* Setup automated decryption from a remote (in case of reboot).

## Detailed process 

### Boot the server in rescue mode 

* Go to the Kimsufi admin panel, click "NetBoot", select "Rescue", pick "rescue64-pro", clock "Next", "Confirm", and then click the "Reboot" button on the admin panel.

* Kimsufi will mail you the root password for you to log into the server in rescue mode.

* Will likely cause an error from your SSH client, since the fingerprint of the server differs from the usual one. 

You should end up with a shell that has access to `/dev/sda`, your server's drive.

### Wipe filesystem & format partitions 

Target partition table:

```
        +-----------+--------------------+-----------+--------------------+
Name:   | /dev/sda1 | /dev/sda2          | /dev/sda3 | /dev/sda4          |
        |           |                    |           |                    |
        |           |                    |           |                    |
        |           | +------------------+           | +------------------+
        |           | | /dev/mapper/swap |           | | /dev/mapper/main |
Format: | none      | | swap             | ext4      | | ext4             |
Size:   | 1MiB      | | 8GiB             | 512MiB    | | remaining space  |
Flags:  | bios_grub | |                  |           | |                  |
Mount:  |           | | swap             | /boot     | | /                |
        +-----------+-+------------------+-----------+-+------------------+
```

We use GPT partition table layout (without UEFI), which demands a small `bios_grub` partition at the beginning of the drive (stores GRUB's `core.img`). Hence `/dev/sda1`. See: 

* [Wikipedia: BIOS boot partition](https://en.wikipedia.org/wiki/BIOS_boot_partition)
* [GNU: BIOS installation](https://www.gnu.org/software/grub/manual/grub/html_node/BIOS-installation.html#BIOS-installation)
* [ArchLinux wiki: Partitioning](https://wiki.archlinux.org/index.php/partitioning#Partition_table)
* [Gentoo wiki: Installation - Disks](https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Disks#Partition_tables)


* Wipe all information about previous filesystem:

    DANGER!!! DANGER!!! WIPES EVERYTHING FROM YOUR DISK!!!

    ```bash
    wipefs -a /dev/sda
    ```

* Create GPT disk layout

    ```bash
    # Create GPT layout
    parted -a optimal /dev/sda mklabel gpt
    # Create first 1MiB bios_grub partition 
    parted /dev/sda -a optimal mkpart bios_grub 0% 1MiB
    parted /dev/sda set 1 bios_grub on
    # Create third 8GiB swap partition
    parted /dev/sda -a optimal mkpart swap 1MiB 8001MiB
    # Create second 512MiB /boot partition
    parted /dev/sda -a optimal mkpart boot 8001MiB 8513MiB
    # Create last / partition using all remaining space 
    parted /dev/sda -a optimal mkpart main 8513MiB 100%
    # Set first partition as bootable
    ```

* Format partitions

    * The ext4 boot partition

        Format it like so:

        ```bash
        mkfs.ext4 /dev/sda3
        ```

    * The encrypted swap partition

        All configuration of the encrypted swap partition comes after the OS installation. 

    * The encrypted main partition

        ```bash
        # Set it up using cryptsetup (I searched for good parameters):
        cryptsetup --type luks2 --cipher aes-xts-plain64 --hash sha256 --iter-time 2000 --key-size 512 luksFormat /dev/sda4

        # Get the UUID, save it for later:
        cryptsetup luksDump /dev/sda4 | grep UUID | awk '{print $2}'
        # Example output: 09762476-ba7c-4732-8856-44a716c23339

        # Decrypt the partition:
        cryptsetup open /dev/sda4 main

        # Format it to ext4:
        mkfs.ext4 /dev/mapper/main
            ```

### Install and configure Debian 

* Mount the partitions 

    ```bash
    # Decrypt the main partition (if not already done):
    cryptsetup open /dev/sda4 main
    # mount it:
    mount /dev/mapper/main /mnt 
    # Mount the boot partition:
    mkdir /mnt/boot
    mount /dev/sda3 /mnt/boot
    ```

* Bootstrap latest stable Debian into mounted partitions

    ```bash
    debootstrap --arch amd64 stable /mnt http://ftp.fr.debian.org/debian/
    ```

* Mount system partitions

    ```bash
    mount -o bind /dev /mnt/dev
    mount -t proc proc /mnt/proc
    mount -t sysfs sys /mnt/sys
    mount -t devpts devpts /mnt/dev/pts
    ```

* Chroot
        
        chroot /mnt /bin/bash

* Fill out `crypttab` and `fstab` (`UUID` is the output of above `cryptsetup luksDump`) 

    `/etc/crypttab`

    ```bash
    cat << EOF > /etc/crypttab
    # <name> <device>                                  <password>   <options>
    main     UUID=09762476-ba7c-4732-8856-44a716c23339 none         luks
    swap     /dev/sda2                                 /dev/urandom swap,noearly,cipher=aes-xts-plain64,size=512
    EOF
    ```

    `/etc/fstab`

    ```bash
    cat << EOF > /etc/fstab
    # <filesystem>   <dir> <type> <options>          <dump> <pass>
    /dev/mapper/main /     ext4   errors=remount-ro  0      1
    /dev/sda3        /boot ext4   defaults           0      2
    /dev/mapper/swap none  swap   sw                 0      0
    EOF
    ```

* Do the `/proc/mounts` -> `/etc/mtab` symlink

    ```bash
    ln -s /proc/mounts /etc/mtab
    ```

    Why? [Linux From Stratch](http://archive.linuxfromscratch.org/lfs-museum/3.3/LFS-BOOK-3.3-HTML/chapter06/mtablink.html) and [our guide on OpsBlog](https://opsblog.net/posts/full-disk-encrypted-ubuntu-kimsufi-sever/) both say it's needed.

* Choose a hostname & DNS domain

    hammerhead & hammerhead.luxeylab.net 

* Fill in network-related files

    `/etc/sysctl.d/10-ipv6`

    Disable IPv6 auto-configuration

    ```bash
    cat << EOF > /etc/sysctl.d/10-ipv6
    net.ipv6.conf.all.autoconf=0
    net.ipv6.conf.all.accept_ra=0
    EOF
    ```


    `/etc/network/interfaces`

    Beware of your ethernet interface name! It's only after failing with `eth0` that the boot logs informed me that the iface name was `eno1`.

    ```bash
    cat << EOF > /etc/network/interfaces
    ######################################################################
    # /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)
    # See the interfaces(5) manpage for information on what options are
    # available.
    ######################################################################

    # loopback interface
    auto lo
    iface lo inet loopback

    # eno1 
    auto eno1
    allow-hotplug eno1

    iface eno1 inet static 
        address 5.135.179.11
        netmask 255.255.255.0
        gateway 5.135.179.254

    iface eno1 inet6 static 
        address 2001:41d0:8:ba0b::1
        netmask 128

    post-up /sbin/ip -f inet6 route add 2001:41d0:8:baff:ff:ff:ff:ff dev eno1 
    post-up /sbin/ip -f inet6 route add default via 2001:41d0:8:baff:ff:ff:ff:ff 
    pre-down /sbin/ip -f inet6 route del 2001:41d0:8:baff:ff:ff:ff:ff dev eno1
    pre-down /sbin/ip -f inet6 route del default via 2001:41d0:8:baff:ff:ff:ff:ff
    EOF
    ```

    [This OVH IPv6 guide](https://docs.ovh.com/gb/en/dedicated/network-ipv6/) might help you out for the IPv6 part.

    `/etc/resolv.conf`

    ```bash
    cat << EOF > /etc/resolv.conf 
    # OVH public DNS
    nameserver 213.186.33.99
    # https://servers.opennicproject.org/edit.php?srv=ns2.he.de.dns.opennic.glue 
    nameserver 172.104.136.243
    EOF
    ```

    `/etc/hosts`

    ```bash
    cat << EOF > /etc/hosts
    127.0.0.1 localhost
    127.0.1.1 hammerhead hammerhead.luxeylab.net 

    # The following lines are desirable for IPv6 capable hosts
    ::1     localhost ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    ff02::3 ip6-allhosts

    213.186.33.116 ws.ovh.com
    198.245.48.4 ws.ovh.ca
    5.135.179.11 hammerhead hammerhead.luxeylab.net 
    EOF
    ```

    `/etc/hostname`

    ```bash
    echo "hammerhead" > /etc/hostname
    ```

    `/etc/timezone`

    ```bash
    dpkg-reconfigure tzdata
    ```

* Update APT repositories in `/etc/apt/sources.list`

    ```bash
    cat << EOF > /etc/apt/sources.list
    deb http://ftp.fr.debian.org/debian buster main
    deb-src http://ftp.fr.debian.org/debian buster main

    deb http://ftp.fr.debian.org/debian-security/ buster/updates main
    deb-src http://ftp.fr.debian.org/debian-security/ buster/updates main

    deb http://ftp.fr.debian.org/debian buster-updates main
    deb-src http://ftp.fr.debian.org/debian buster-updates main
    EOF
    ```

* Configure locales & keyboard

    ```bash
    # Refresh packages list
    apt update
    # Install & configure locales 
    apt install locales && dpkg-reconfigure locales
    # Install & configure keyboard 
    apt install console-setup console-data && dpkg-reconfigure keyboard-configuration
    ```

* Pick up a kernel & install it

    ```bash
    apt-cache search linux-image
    # I selected the meta-package `linux-image-amd64` which is the latest 
    apt install linux-image-amd64 linux-headers-amd64
    ```

* Install additional necessary software

    ```bash
    apt install cryptsetup dropbear grub-pc man-db ssh 
    ```

    Because we use a GPT table layout but no UEFI, we install `grub-pc` and not `grub-efi-amd64`.

    Do think about installing `bash-completion` and `ufw` someday.


* Set up SSH keys and the rest (where `<PUBLIC_SSH_KEY>` is your client SSH public key)

    ```bash
    mkdir /root/.ssh && chmod 600 /root/.ssh
    echo "<PUBLIC_SSH_KEY>" > /root/.ssh/authorized_keys
    echo "<PUBLIC_SSH_KEY>" > /etc/dropbear-initramfs/authorized_keys
    ```

* Configure dropbear-initramfs (SSH on boot)

    ```bash
    sed -i 's/#DROPBEAR_OPTIONS=/DROPBEAR_OPTIONS=\"-p 2222 -c \/bin\/cryptroot-unlock\"/' /etc/dropbear-initramfs/config
    ```


    This changes the SSH listen post to 2222, and enforces that the only command run from `dropbear-initramfs` is `cryptroot-unlock`.

* Configure sshd custom listen port (SSH after boot)

    ```bash
    sed -i 's/#Port 22/Port 2223/' /etc/ssh/sshd_config
    ```

* [Optional] Authorize root password login 

    ```bash
    # Remove -s (disable password login) from dropbear-initramfs
    sed -i 's/local flags=\"Fs\"/local flags=\"F\"/' /usr/share/initramfs-tools/scripts/init-premount/dropbear
    ```

* Update GRUB and initramfs

    ```bash
    update-grub && update-initramfs -u
    ```

* Clean up before leaving 

    ```bash
    exit
    umount /mnt/{boot,dev/pts,dev,proc,sys}
    umount /mnt
    cryptsetup luksClose main
    ```

### Boot and log into the server

* I configured my laptop's `~/.ssh/config` like so:

    ```
    Host hammerhead-decrypt
        User root
        Hostname <SERVER_URL_OR_IP>
        Port 2222
        IdentityFile <PRIVATE_SSH_KEY_PATH>
        # Prevents signatures mismatch
        UserKnownHostsFile ~/.ssh/known_hosts_hammerhead-decrypt
    Host hammerhead
        User root
        Hostname <SERVER_URL_OR_IP>
        Port 2223
        IdentityFile <PRIVATE_SSH_KEY_PATH>
    ```

* And here is how I connect to the server:

    ```bash
    laptop$ ssh hammerhead-decrypt 
    Please unlock disk main: <PASSWORD>
    # [...]
    Connection to <SERVER_URL_OR_IP> closed.
    laptop$ ssh hammerhead 
    # [...]
    root@hamerhead:~# echo A winner is you!
    A winner is you!
    ```


### Setup automated decryption from a remote

It is desirable to have a daemon running on a remote server, to automatically decrypt the drive when the encrypted server reboots without warning.

The remote server is called a *key escrow*. One must be particularly careful about the escrow's security, since it holds the decryption keys for our server.

Trinity recommends [Tang](https://github.com/latchset/tang) and [Clevis](https://github.com/latchset/clevis).