automation/bootstrapper/os_install/README.md
2021-04-11 17:05:18 +02:00

14 KiB

Installing a cloud server with full disk encryption

For hosting purposes, I am installing a new server on Kimsufi. This is a French public cloud provider (property of OVH).

It is well known that, at Deuxfleurs, we do not fancy hosting in datacenters: because of the lack on control on our machines, and potential privacy threats. In essence, people at Kimsufi could easily -- and without me knowing -- read my hard drive, and thus any sensitive user data stored inside.

For that reason, I want to make it harder for their staff to access my data. A good first line of defense for that purpose is to encrypt the whole disk (aka Full Disk Encryption). Note that motivated attackers with physical access to my server would easily overcome the encryption. If you really care about who accesses your data, keep your hard drives closest to you, e.g. at your home.


Long story short: this guide presents a step-by-step installation of Debian buster on Kimsufi, using the dm-crypt Linux kernel subsystem to encrypt the / and the swap partitions. Because I don't have physical access to the server, I need a way to remotely enter the disk decryption password, that is dropbear (a lightweight SSH server) inside initramfs (a minimal operating system that runs from RAM, before Linux boots).

If you plan on doing the same without Kimsufi, know that it makes no difference: I am basically using a "rescue" console on another computer to configure /dev/sda (my server's system drive) for booting Linux with encrypted partitions.

I must credit several useful guides that helped me succeed:

Overview of steps

  • Boot the Kimsufi server in rescue mode (gives you a remote shell with access to your system's hard drive).

  • Wipe filesystem, format partitions. I basically want:

  • Install and configure Debian (including disk cryptography and decryption through SSH).

  • Boot and log into the server.

  • Setup automated decryption from a remote (in case of reboot).

Detailed process

Boot the server in rescue mode

  • Go to the Kimsufi admin panel, click "NetBoot", select "Rescue", pick "rescue64-pro", clock "Next", "Confirm", and then click the "Reboot" button on the admin panel.

  • Kimsufi will mail you the root password for you to log into the server in rescue mode.

  • Will likely cause an error from your SSH client, since the fingerprint of the server differs from the usual one.

You should end up with a shell that has access to /dev/sda, your server's drive.

Wipe filesystem & format partitions

Target partition table:

        +-----------+--------------------+-----------+--------------------+
Name:   | /dev/sda1 | /dev/sda2          | /dev/sda3 | /dev/sda4          |
        |           |                    |           |                    |
        |           |                    |           |                    |
        |           | +------------------+           | +------------------+
        |           | | /dev/mapper/swap |           | | /dev/mapper/main |
Format: | none      | | swap             | ext4      | | ext4             |
Size:   | 1MiB      | | 8GiB             | 512MiB    | | remaining space  |
Flags:  | bios_grub | |                  |           | |                  |
Mount:  |           | | swap             | /boot     | | /                |
        +-----------+-+------------------+-----------+-+------------------+

We use GPT partition table layout (without UEFI), which demands a small bios_grub partition at the beginning of the drive (stores GRUB's core.img). Hence /dev/sda1. See:

  • Wikipedia: BIOS boot partition

  • GNU: BIOS installation

  • ArchLinux wiki: Partitioning

  • Gentoo wiki: Installation - Disks

  • Wipe all information about previous filesystem:

    DANGER!!! DANGER!!! WIPES EVERYTHING FROM YOUR DISK!!!

    wipefs -a /dev/sda
    
  • Create GPT disk layout

    # Create GPT layout
    parted -a optimal /dev/sda mklabel gpt
    # Create first 1MiB bios_grub partition 
    parted /dev/sda -a optimal mkpart bios_grub 0% 1MiB
    parted /dev/sda set 1 bios_grub on
    # Create third 8GiB swap partition
    parted /dev/sda -a optimal mkpart swap 1MiB 8001MiB
    # Create second 512MiB /boot partition
    parted /dev/sda -a optimal mkpart boot 8001MiB 8513MiB
    # Create last / partition using all remaining space 
    parted /dev/sda -a optimal mkpart main 8513MiB 100%
    # Set first partition as bootable
    
  • Format partitions

    • The ext4 boot partition

      Format it like so:

      mkfs.ext4 /dev/sda3
      
    • The encrypted swap partition

      All configuration of the encrypted swap partition comes after the OS installation.

    • The encrypted main partition

      # Set it up using cryptsetup (I searched for good parameters):
      cryptsetup --type luks2 --cipher aes-xts-plain64 --hash sha256 --iter-time 2000 --key-size 512 luksFormat /dev/sda4
      
      # Get the UUID, save it for later:
      cryptsetup luksDump /dev/sda4 | grep UUID | awk '{print $2}'
      # Example output: 09762476-ba7c-4732-8856-44a716c23339
      
      # Decrypt the partition:
      cryptsetup open /dev/sda4 main
      
      # Format it to ext4:
      mkfs.ext4 /dev/mapper/main
          ```
      
      

Install and configure Debian

  • Mount the partitions

    # Decrypt the main partition (if not already done):
    cryptsetup open /dev/sda4 main
    # mount it:
    mount /dev/mapper/main /mnt 
    # Mount the boot partition:
    mkdir /mnt/boot
    mount /dev/sda3 /mnt/boot
    
  • Bootstrap latest stable Debian into mounted partitions

    debootstrap --arch amd64 stable /mnt http://ftp.fr.debian.org/debian/
    
  • Mount system partitions

    mount -o bind /dev /mnt/dev
    mount -t proc proc /mnt/proc
    mount -t sysfs sys /mnt/sys
    mount -t devpts devpts /mnt/dev/pts
    
  • Chroot

      chroot /mnt /bin/bash
    
  • Fill out crypttab and fstab (UUID is the output of above cryptsetup luksDump)

    /etc/crypttab

    cat << EOF > /etc/crypttab
    # <name> <device>                                  <password>   <options>
    main     UUID=09762476-ba7c-4732-8856-44a716c23339 none         luks
    swap     /dev/sda2                                 /dev/urandom swap,noearly,cipher=aes-xts-plain64,size=512
    EOF
    

    /etc/fstab

    cat << EOF > /etc/fstab
    # <filesystem>   <dir> <type> <options>          <dump> <pass>
    /dev/mapper/main /     ext4   errors=remount-ro  0      1
    /dev/sda3        /boot ext4   defaults           0      2
    /dev/mapper/swap none  swap   sw                 0      0
    EOF
    
  • Do the /proc/mounts -> /etc/mtab symlink

    ln -s /proc/mounts /etc/mtab
    

    Why? Linux From Stratch and our guide on OpsBlog both say it's needed.

  • Choose a hostname & DNS domain

    hammerhead & hammerhead.luxeylab.net

  • Fill in network-related files

    /etc/sysctl.d/10-ipv6

    Disable IPv6 auto-configuration

    cat << EOF > /etc/sysctl.d/10-ipv6
    net.ipv6.conf.all.autoconf=0
    net.ipv6.conf.all.accept_ra=0
    EOF
    

    /etc/network/interfaces

    Beware of your ethernet interface name! It's only after failing with eth0 that the boot logs informed me that the iface name was eno1.

    cat << EOF > /etc/network/interfaces
    ######################################################################
    # /etc/network/interfaces -- configuration file for ifup(8), ifdown(8)
    # See the interfaces(5) manpage for information on what options are
    # available.
    ######################################################################
    
    # loopback interface
    auto lo
    iface lo inet loopback
    
    # eno1 
    auto eno1
    allow-hotplug eno1
    
    iface eno1 inet static 
        address 5.135.179.11
        netmask 255.255.255.0
        gateway 5.135.179.254
    
    iface eno1 inet6 static 
        address 2001:41d0:8:ba0b::1
        netmask 128
    
    post-up /sbin/ip -f inet6 route add 2001:41d0:8:baff:ff:ff:ff:ff dev eno1 
    post-up /sbin/ip -f inet6 route add default via 2001:41d0:8:baff:ff:ff:ff:ff 
    pre-down /sbin/ip -f inet6 route del 2001:41d0:8:baff:ff:ff:ff:ff dev eno1
    pre-down /sbin/ip -f inet6 route del default via 2001:41d0:8:baff:ff:ff:ff:ff
    EOF
    

    This OVH IPv6 guide might help you out for the IPv6 part.

    /etc/resolv.conf

    cat << EOF > /etc/resolv.conf 
    # OVH public DNS
    nameserver 213.186.33.99
    # https://servers.opennicproject.org/edit.php?srv=ns2.he.de.dns.opennic.glue 
    nameserver 172.104.136.243
    EOF
    

    /etc/hosts

    cat << EOF > /etc/hosts
    127.0.0.1 localhost
    127.0.1.1 hammerhead hammerhead.luxeylab.net 
    
    # The following lines are desirable for IPv6 capable hosts
    ::1     localhost ip6-localhost ip6-loopback
    fe00::0 ip6-localnet
    ff00::0 ip6-mcastprefix
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    ff02::3 ip6-allhosts
    
    213.186.33.116 ws.ovh.com
    198.245.48.4 ws.ovh.ca
    5.135.179.11 hammerhead hammerhead.luxeylab.net 
    EOF
    

    /etc/hostname

    echo "hammerhead" > /etc/hostname
    

    /etc/timezone

    dpkg-reconfigure tzdata
    
  • Update APT repositories in /etc/apt/sources.list

    cat << EOF > /etc/apt/sources.list
    deb http://ftp.fr.debian.org/debian buster main
    deb-src http://ftp.fr.debian.org/debian buster main
    
    deb http://ftp.fr.debian.org/debian-security/ buster/updates main
    deb-src http://ftp.fr.debian.org/debian-security/ buster/updates main
    
    deb http://ftp.fr.debian.org/debian buster-updates main
    deb-src http://ftp.fr.debian.org/debian buster-updates main
    EOF
    
  • Configure locales & keyboard

    # Refresh packages list
    apt update
    # Install & configure locales 
    apt install locales && dpkg-reconfigure locales
    # Install & configure keyboard 
    apt install console-setup console-data && dpkg-reconfigure keyboard-configuration
    
  • Pick up a kernel & install it

    apt-cache search linux-image
    # I selected the meta-package `linux-image-amd64` which is the latest 
    apt install linux-image-amd64 linux-headers-amd64
    
  • Install additional necessary software

    apt install cryptsetup dropbear grub-pc man-db ssh 
    

    Because we use a GPT table layout but no UEFI, we install grub-pc and not grub-efi-amd64.

    Do think about installing bash-completion and ufw someday.

  • Set up SSH keys and the rest (where <PUBLIC_SSH_KEY> is your client SSH public key)

    mkdir /root/.ssh && chmod 600 /root/.ssh
    echo "<PUBLIC_SSH_KEY>" > /root/.ssh/authorized_keys
    echo "<PUBLIC_SSH_KEY>" > /etc/dropbear-initramfs/authorized_keys
    
  • Configure dropbear-initramfs (SSH on boot)

    sed -i 's/#DROPBEAR_OPTIONS=/DROPBEAR_OPTIONS=\"-p 2222 -c \/bin\/cryptroot-unlock\"/' /etc/dropbear-initramfs/config
    

    This changes the SSH listen post to 2222, and enforces that the only command run from dropbear-initramfs is cryptroot-unlock.

  • Configure sshd custom listen port (SSH after boot)

    sed -i 's/#Port 22/Port 2223/' /etc/ssh/sshd_config
    
  • [Optional] Authorize root password login

    # Remove -s (disable password login) from dropbear-initramfs
    sed -i 's/local flags=\"Fs\"/local flags=\"F\"/' /usr/share/initramfs-tools/scripts/init-premount/dropbear
    
  • Update GRUB and initramfs

    update-grub && update-initramfs -u
    
  • Clean up before leaving

    exit
    umount /mnt/{boot,dev/pts,dev,proc,sys}
    umount /mnt
    cryptsetup luksClose main
    

Boot and log into the server

  • I configured my laptop's ~/.ssh/config like so:

    Host hammerhead-decrypt
        User root
        Hostname <SERVER_URL_OR_IP>
        Port 2222
        IdentityFile <PRIVATE_SSH_KEY_PATH>
        # Prevents signatures mismatch
        UserKnownHostsFile ~/.ssh/known_hosts_hammerhead-decrypt
    Host hammerhead
        User root
        Hostname <SERVER_URL_OR_IP>
        Port 2223
        IdentityFile <PRIVATE_SSH_KEY_PATH>
    
  • And here is how I connect to the server:

    laptop$ ssh hammerhead-decrypt 
    Please unlock disk main: <PASSWORD>
    # [...]
    Connection to <SERVER_URL_OR_IP> closed.
    laptop$ ssh hammerhead 
    # [...]
    root@hamerhead:~# echo A winner is you!
    A winner is you!
    

Setup automated decryption from a remote

It is desirable to have a daemon running on a remote server, to automatically decrypt the drive when the encrypted server reboots without warning.

The remote server is called a key escrow. One must be particularly careful about the escrow's security, since it holds the decryption keys for our server.

Trinity recommends Tang and Clevis.