From 8d17a07c9be5cd9d400644c34ea50177535d15f6 Mon Sep 17 00:00:00 2001 From: Alex Auvolat Date: Sat, 24 Dec 2022 22:59:37 +0100 Subject: [PATCH] reorganize some things --- README.md | 38 ++++++++++++++----- cluster/prod/app/secretmgr | 1 + cluster/prod/app/secretmgr.py | 1 - cluster/prod/app/shell.nix | 1 - cluster/staging/app/secretmgr | 1 + cluster/staging/app/secretmgr.py | 1 - cluster/staging/app/shell.nix | 1 - .../csi-s3}/deploy/csi-s3.hcl | 0 .../csi-s3}/deploy/dummy-volume.hcl | 0 .../nextcloud}/config/litestream.yml | 0 .../nextcloud}/deploy/nextcloud.hcl | 0 .../nextcloud}/secrets/nextcloud/admin_pass | 0 .../nextcloud}/secrets/nextcloud/admin_user | 0 .../secrets/nextcloud/s3_access_key | 0 .../secrets/nextcloud/s3_secret_key | 0 .../ssb}/deploy/go-ssb-room.hcl | 0 .../{bad.ssb => app/ssb}/deploy/ssb-room.hcl | 0 .../telemetry-elastic}/config/apm-config.yaml | 0 .../telemetry-elastic}/config/filebeat.yml | 0 .../config/grafana-litestream.yml | 0 .../provisioning/datasources/elastic.yaml | 0 .../config/otel-config.yaml | 0 .../deploy/telemetry-system.hcl | 0 .../telemetry-elastic}/deploy/telemetry.hcl | 0 .../yugabyte}/deploy/yugabyte.hcl | 0 .../example-hardware-configuration.nix | 0 .../luks-fde}/nixos-install-luks.md | 10 ++--- secretmgr/{secretmgr.py => secretmgr} | 3 +- secretmgr/shell.nix | 15 -------- sshtool | 5 +++ 30 files changed, 43 insertions(+), 34 deletions(-) create mode 120000 cluster/prod/app/secretmgr delete mode 120000 cluster/prod/app/secretmgr.py delete mode 120000 cluster/prod/app/shell.nix create mode 120000 cluster/staging/app/secretmgr delete mode 120000 cluster/staging/app/secretmgr.py delete mode 120000 cluster/staging/app/shell.nix rename experimental/{bad.csi-s3 => app/csi-s3}/deploy/csi-s3.hcl (100%) rename experimental/{bad.csi-s3 => app/csi-s3}/deploy/dummy-volume.hcl (100%) rename experimental/{bad.nextcloud => app/nextcloud}/config/litestream.yml (100%) rename experimental/{bad.nextcloud => app/nextcloud}/deploy/nextcloud.hcl (100%) rename experimental/{bad.nextcloud => app/nextcloud}/secrets/nextcloud/admin_pass (100%) rename experimental/{bad.nextcloud => app/nextcloud}/secrets/nextcloud/admin_user (100%) rename experimental/{bad.nextcloud => app/nextcloud}/secrets/nextcloud/s3_access_key (100%) rename experimental/{bad.nextcloud => app/nextcloud}/secrets/nextcloud/s3_secret_key (100%) rename experimental/{bad.ssb => app/ssb}/deploy/go-ssb-room.hcl (100%) rename experimental/{bad.ssb => app/ssb}/deploy/ssb-room.hcl (100%) rename experimental/{bad.telemetry-elastic => app/telemetry-elastic}/config/apm-config.yaml (100%) rename experimental/{bad.telemetry-elastic => app/telemetry-elastic}/config/filebeat.yml (100%) rename experimental/{bad.telemetry-elastic => app/telemetry-elastic}/config/grafana-litestream.yml (100%) rename experimental/{bad.telemetry-elastic => app/telemetry-elastic}/config/grafana/provisioning/datasources/elastic.yaml (100%) rename experimental/{bad.telemetry-elastic => app/telemetry-elastic}/config/otel-config.yaml (100%) rename experimental/{bad.telemetry-elastic => app/telemetry-elastic}/deploy/telemetry-system.hcl (100%) rename experimental/{bad.telemetry-elastic => app/telemetry-elastic}/deploy/telemetry.hcl (100%) rename experimental/{bad.yugabyte => app/yugabyte}/deploy/yugabyte.hcl (100%) rename {doc => experimental/luks-fde}/example-hardware-configuration.nix (100%) rename {doc => experimental/luks-fde}/nixos-install-luks.md (94%) rename secretmgr/{secretmgr.py => secretmgr} (98%) delete mode 100644 secretmgr/shell.nix diff --git a/README.md b/README.md index c86a067..3527dbb 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,35 @@ This repository contains code to run Deuxfleur's infrastructure on NixOS. -It sets up the following: +## Our abstraction stack -- A Wireguard mesh between all nodes -- Consul, with TLS -- Nomad, with TLS +We try to build a generic abstraction stack between our different resources (CPU, RAM, disk, etc.) and our services (Chat, Storage, etc.), we develop our own tools when needed. + +Our first abstraction level is the NixOS level, which installs a bunch of standard components: + + * **Wireguard:** provides encrypted communication between remote nodes + * **Nomad:** schedule containers and handle their lifecycle + * **Consul:** distributed key value store + lock + service discovery + * **Docker:** package, distribute and isolate applications + +Then, inside our Nomad+Consul orchestrator, we deploy a number of base services: + + * **[Garage](https://git.deuxfleurs.fr/Deuxfleurs/garage/):** S3-compatible lightweight object store for self-hosted geo-distributed deployments (we also have a legacy glusterfs cluster) + * **[DiploNAT](https://git.deuxfleurs.fr/Deuxfleurs/diplonat):** network automation (firewalling, upnp igd) + * **[Bottin](https://git.deuxfleurs.fr/Deuxfleurs/bottin):** authentication and authorization (LDAP protocol, consul backend) + * **[Guichet](https://git.deuxfleurs.fr/Deuxfleurs/guichet):** a dashboard for our users and administrators + * **Stolon + PostgreSQL:** distributed relational database + * **Prometheus + Grafana:** monitoring + +Some services we provide based on this abstraction: + + * **Websites:** Garage (static) + fediverse blog (Plume) + * **Chat:** Synapse + Element Web (Matrix protocol) + * **Email:** Postfix SMTP + Dovecot IMAP + opendkim DKIM + Sogo webmail | Alps webmail (experimental) + * **Visioconference:** Jitsi + * **Collaboration:** CryptPad + +As a generic abstraction is provided, deploying new services should be easy. ## How to use this? @@ -16,11 +40,7 @@ See the following documentation topics: - [How to add new nodes to a cluster (rapid overview)](doc/adding-nodes.md) - [Architecture of this repo, how the scripts work](doc/architecture.md) - [List of TCP and UDP ports used by services](doc/ports) - -Additionnal documentation topics: - -- [Succint guide for NixOS installation with LUKX full disk encryption](doc/nixos-install-luks.md) (we don't do that in practice on our servers) -- [Example `hardware-config.nix` for a full disk encryption scenario](doc/example-hardware-configuration.nix) - [Why not Ansible?](doc/why-not-ansible.md) + diff --git a/cluster/prod/app/secretmgr b/cluster/prod/app/secretmgr new file mode 120000 index 0000000..6aff4ad --- /dev/null +++ b/cluster/prod/app/secretmgr @@ -0,0 +1 @@ +../../../secretmgr/secretmgr \ No newline at end of file diff --git a/cluster/prod/app/secretmgr.py b/cluster/prod/app/secretmgr.py deleted file mode 120000 index 107653c..0000000 --- a/cluster/prod/app/secretmgr.py +++ /dev/null @@ -1 +0,0 @@ -../../../secretmgr/secretmgr.py \ No newline at end of file diff --git a/cluster/prod/app/shell.nix b/cluster/prod/app/shell.nix deleted file mode 120000 index b10effc..0000000 --- a/cluster/prod/app/shell.nix +++ /dev/null @@ -1 +0,0 @@ -../../../secretmgr/shell.nix \ No newline at end of file diff --git a/cluster/staging/app/secretmgr b/cluster/staging/app/secretmgr new file mode 120000 index 0000000..6aff4ad --- /dev/null +++ b/cluster/staging/app/secretmgr @@ -0,0 +1 @@ +../../../secretmgr/secretmgr \ No newline at end of file diff --git a/cluster/staging/app/secretmgr.py b/cluster/staging/app/secretmgr.py deleted file mode 120000 index 107653c..0000000 --- a/cluster/staging/app/secretmgr.py +++ /dev/null @@ -1 +0,0 @@ -../../../secretmgr/secretmgr.py \ No newline at end of file diff --git a/cluster/staging/app/shell.nix b/cluster/staging/app/shell.nix deleted file mode 120000 index b10effc..0000000 --- a/cluster/staging/app/shell.nix +++ /dev/null @@ -1 +0,0 @@ -../../../secretmgr/shell.nix \ No newline at end of file diff --git a/experimental/bad.csi-s3/deploy/csi-s3.hcl b/experimental/app/csi-s3/deploy/csi-s3.hcl similarity index 100% rename from experimental/bad.csi-s3/deploy/csi-s3.hcl rename to experimental/app/csi-s3/deploy/csi-s3.hcl diff --git a/experimental/bad.csi-s3/deploy/dummy-volume.hcl b/experimental/app/csi-s3/deploy/dummy-volume.hcl similarity index 100% rename from experimental/bad.csi-s3/deploy/dummy-volume.hcl rename to experimental/app/csi-s3/deploy/dummy-volume.hcl diff --git a/experimental/bad.nextcloud/config/litestream.yml b/experimental/app/nextcloud/config/litestream.yml similarity index 100% rename from experimental/bad.nextcloud/config/litestream.yml rename to experimental/app/nextcloud/config/litestream.yml diff --git a/experimental/bad.nextcloud/deploy/nextcloud.hcl b/experimental/app/nextcloud/deploy/nextcloud.hcl similarity index 100% rename from experimental/bad.nextcloud/deploy/nextcloud.hcl rename to experimental/app/nextcloud/deploy/nextcloud.hcl diff --git a/experimental/bad.nextcloud/secrets/nextcloud/admin_pass b/experimental/app/nextcloud/secrets/nextcloud/admin_pass similarity index 100% rename from experimental/bad.nextcloud/secrets/nextcloud/admin_pass rename to experimental/app/nextcloud/secrets/nextcloud/admin_pass diff --git a/experimental/bad.nextcloud/secrets/nextcloud/admin_user b/experimental/app/nextcloud/secrets/nextcloud/admin_user similarity index 100% rename from experimental/bad.nextcloud/secrets/nextcloud/admin_user rename to experimental/app/nextcloud/secrets/nextcloud/admin_user diff --git a/experimental/bad.nextcloud/secrets/nextcloud/s3_access_key b/experimental/app/nextcloud/secrets/nextcloud/s3_access_key similarity index 100% rename from experimental/bad.nextcloud/secrets/nextcloud/s3_access_key rename to experimental/app/nextcloud/secrets/nextcloud/s3_access_key diff --git a/experimental/bad.nextcloud/secrets/nextcloud/s3_secret_key b/experimental/app/nextcloud/secrets/nextcloud/s3_secret_key similarity index 100% rename from experimental/bad.nextcloud/secrets/nextcloud/s3_secret_key rename to experimental/app/nextcloud/secrets/nextcloud/s3_secret_key diff --git a/experimental/bad.ssb/deploy/go-ssb-room.hcl b/experimental/app/ssb/deploy/go-ssb-room.hcl similarity index 100% rename from experimental/bad.ssb/deploy/go-ssb-room.hcl rename to experimental/app/ssb/deploy/go-ssb-room.hcl diff --git a/experimental/bad.ssb/deploy/ssb-room.hcl b/experimental/app/ssb/deploy/ssb-room.hcl similarity index 100% rename from experimental/bad.ssb/deploy/ssb-room.hcl rename to experimental/app/ssb/deploy/ssb-room.hcl diff --git a/experimental/bad.telemetry-elastic/config/apm-config.yaml b/experimental/app/telemetry-elastic/config/apm-config.yaml similarity index 100% rename from experimental/bad.telemetry-elastic/config/apm-config.yaml rename to experimental/app/telemetry-elastic/config/apm-config.yaml diff --git a/experimental/bad.telemetry-elastic/config/filebeat.yml b/experimental/app/telemetry-elastic/config/filebeat.yml similarity index 100% rename from experimental/bad.telemetry-elastic/config/filebeat.yml rename to experimental/app/telemetry-elastic/config/filebeat.yml diff --git a/experimental/bad.telemetry-elastic/config/grafana-litestream.yml b/experimental/app/telemetry-elastic/config/grafana-litestream.yml similarity index 100% rename from experimental/bad.telemetry-elastic/config/grafana-litestream.yml rename to experimental/app/telemetry-elastic/config/grafana-litestream.yml diff --git a/experimental/bad.telemetry-elastic/config/grafana/provisioning/datasources/elastic.yaml b/experimental/app/telemetry-elastic/config/grafana/provisioning/datasources/elastic.yaml similarity index 100% rename from experimental/bad.telemetry-elastic/config/grafana/provisioning/datasources/elastic.yaml rename to experimental/app/telemetry-elastic/config/grafana/provisioning/datasources/elastic.yaml diff --git a/experimental/bad.telemetry-elastic/config/otel-config.yaml b/experimental/app/telemetry-elastic/config/otel-config.yaml similarity index 100% rename from experimental/bad.telemetry-elastic/config/otel-config.yaml rename to experimental/app/telemetry-elastic/config/otel-config.yaml diff --git a/experimental/bad.telemetry-elastic/deploy/telemetry-system.hcl b/experimental/app/telemetry-elastic/deploy/telemetry-system.hcl similarity index 100% rename from experimental/bad.telemetry-elastic/deploy/telemetry-system.hcl rename to experimental/app/telemetry-elastic/deploy/telemetry-system.hcl diff --git a/experimental/bad.telemetry-elastic/deploy/telemetry.hcl b/experimental/app/telemetry-elastic/deploy/telemetry.hcl similarity index 100% rename from experimental/bad.telemetry-elastic/deploy/telemetry.hcl rename to experimental/app/telemetry-elastic/deploy/telemetry.hcl diff --git a/experimental/bad.yugabyte/deploy/yugabyte.hcl b/experimental/app/yugabyte/deploy/yugabyte.hcl similarity index 100% rename from experimental/bad.yugabyte/deploy/yugabyte.hcl rename to experimental/app/yugabyte/deploy/yugabyte.hcl diff --git a/doc/example-hardware-configuration.nix b/experimental/luks-fde/example-hardware-configuration.nix similarity index 100% rename from doc/example-hardware-configuration.nix rename to experimental/luks-fde/example-hardware-configuration.nix diff --git a/doc/nixos-install-luks.md b/experimental/luks-fde/nixos-install-luks.md similarity index 94% rename from doc/nixos-install-luks.md rename to experimental/luks-fde/nixos-install-luks.md index 3f0feca..9e173f7 100644 --- a/doc/nixos-install-luks.md +++ b/experimental/luks-fde/nixos-install-luks.md @@ -1,6 +1,6 @@ ## Preparation -Download NixOS 21.11 ISO. Burn to USB. +Download NixOS. Burn to USB. ## Booting into install environment @@ -120,7 +120,7 @@ Remotely: `ssh-copy-id @`. Check SSH access is good. ## Deploy from this repo -See [this documentation](quick-start.md). +See the documentation in `/doc` in this repo. The old procedure described here is partly obsolete. ## Old guide @@ -154,7 +154,7 @@ Reboot. Check remote unlocking works: `ssh -p 222 root@` -## Configure wireguard +### Configure wireguard ```bash # On node being installed @@ -172,11 +172,11 @@ Redo a deploy (`./deploy.sh `) Check VPN works. Change IP in `ssh_config` to use VPN IP instead of LAN IP (required for deploy when away from home). -## Commit changes to `nixcfg` repo +### Commit changes to `nixcfg` repo This is a good point to commit your new/modified `.nix` files. -## Configure Nomad and Consul TLS +### Configure Nomad and Consul TLS If you are bootstraping a new cluster, you need to `./genpki.sh ` to make a TLS PKI for the Nomad+Consul cluster to work. Then redo a deploy. diff --git a/secretmgr/secretmgr.py b/secretmgr/secretmgr similarity index 98% rename from secretmgr/secretmgr.py rename to secretmgr/secretmgr index 8b17f61..3c0ec08 100755 --- a/secretmgr/secretmgr.py +++ b/secretmgr/secretmgr @@ -1,4 +1,5 @@ -#!/usr/bin/env python3 +#!/usr/bin/env nix-shell +#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.pip ps.consul ps.ldap ps.passlib ps.requests ps.six ])" # DEPENDENCY: python-consul import consul diff --git a/secretmgr/shell.nix b/secretmgr/shell.nix deleted file mode 100644 index c9b8053..0000000 --- a/secretmgr/shell.nix +++ /dev/null @@ -1,15 +0,0 @@ -{ - pkgs ? import {} -}: - -with pkgs; mkShell { - nativeBuildInputs = [ - nomad - docker-compose - python39Packages.pip - python39Packages.ldap - python39Packages.consul - python39Packages.passlib - ]; -} - diff --git a/sshtool b/sshtool index 262f0e3..24c19af 100755 --- a/sshtool +++ b/sshtool @@ -1,6 +1,11 @@ #!/usr/bin/env bash CMDFILE="$1" +if [ -z "$CMDFILE" ] || [ ! -f "$CMDFILE" ]; then + echo "sshtool is not meant to be called on its own." + echo "See scripts that use it (e.g. deploy_nixos) for usage examples." + exit 1 +fi shift 1 cd $(dirname $CMDFILE)