reorganize some things

README.md

@@ -2,11 +2,35 @@
 
 This repository contains code to run Deuxfleur's infrastructure on NixOS.
 
-It sets up the following:
+## Our abstraction stack
 
-- A Wireguard mesh between all nodes
-- Consul, with TLS
-- Nomad, with TLS
+We try to build a generic abstraction stack between our different resources (CPU, RAM, disk, etc.) and our services (Chat, Storage, etc.), we develop our own tools when needed.
+
+Our first abstraction level is the NixOS level, which installs a bunch of standard components:
+
+  * **Wireguard:** provides encrypted communication between remote nodes
+  * **Nomad:** schedule containers and handle their lifecycle
+  * **Consul:** distributed key value store + lock + service discovery
+  * **Docker:** package, distribute and isolate applications
+  
+Then, inside our Nomad+Consul orchestrator, we deploy a number of base services:
+
+  * **[Garage](https://git.deuxfleurs.fr/Deuxfleurs/garage/):** S3-compatible lightweight object store for self-hosted geo-distributed deployments (we also have a legacy glusterfs cluster)
+  * **[DiploNAT](https://git.deuxfleurs.fr/Deuxfleurs/diplonat):** network automation (firewalling, upnp igd)
+  * **[Bottin](https://git.deuxfleurs.fr/Deuxfleurs/bottin):** authentication and authorization (LDAP protocol, consul backend)
+  * **[Guichet](https://git.deuxfleurs.fr/Deuxfleurs/guichet):** a dashboard for our users and administrators
+  * **Stolon + PostgreSQL:** distributed relational database
+  * **Prometheus + Grafana:** monitoring
+
+Some services we provide based on this abstraction:
+
+  * **Websites:** Garage (static) + fediverse blog (Plume)
+  * **Chat:** Synapse + Element Web (Matrix protocol)
+  * **Email:** Postfix SMTP + Dovecot IMAP + opendkim DKIM + Sogo webmail  | Alps webmail (experimental)
+  * **Visioconference:** Jitsi
+  * **Collaboration:** CryptPad
+
+As a generic abstraction is provided, deploying new services should be easy.
 
 ## How to use this?
 
@@ -16,11 +40,7 @@ See the following documentation topics:
 - [How to add new nodes to a cluster (rapid overview)](doc/adding-nodes.md)
 - [Architecture of this repo, how the scripts work](doc/architecture.md)
 - [List of TCP and UDP ports used by services](doc/ports)
-
-Additionnal documentation topics:
-
-- [Succint guide for NixOS installation with LUKX full disk encryption](doc/nixos-install-luks.md) (we don't do that in practice on our servers)
-- [Example `hardware-config.nix` for a full disk encryption scenario](doc/example-hardware-configuration.nix)
 - [Why not Ansible?](doc/why-not-ansible.md)
 
 
+

cluster/prod/app/secretmgr

@@ -0,0 +1 @@
+../../../secretmgr/secretmgr

cluster/prod/app/secretmgr.py

@@ -1 +0,0 @@
-../../../secretmgr/secretmgr.py

cluster/prod/app/shell.nix

@@ -1 +0,0 @@
-../../../secretmgr/shell.nix

cluster/staging/app/secretmgr

@@ -0,0 +1 @@
+../../../secretmgr/secretmgr

cluster/staging/app/secretmgr.py

@@ -1 +0,0 @@
-../../../secretmgr/secretmgr.py

cluster/staging/app/shell.nix

@@ -1 +0,0 @@
-../../../secretmgr/shell.nix

experimental/luks-fde/nixos-install-luks.md

@@ -1,6 +1,6 @@
 ## Preparation
 
-Download NixOS 21.11 ISO. Burn to USB.
+Download NixOS. Burn to USB.
 
 ## Booting into install environment
 
@@ -120,7 +120,7 @@ Remotely: `ssh-copy-id <user>@<ip>`. Check SSH access is good.
 
 ## Deploy from this repo
 
-See [this documentation](quick-start.md).
+See the documentation in `/doc` in this repo. The old procedure described here is partly obsolete.
 
 ## Old guide
 
@@ -154,7 +154,7 @@ Reboot.
 
 Check remote unlocking works: `ssh -p 222 root@<ip>`
 
-## Configure wireguard
+### Configure wireguard
 
 ```bash
 # On node being installed
@@ -172,11 +172,11 @@ Redo a deploy (`./deploy.sh <cluster> <nodename>`)
 
 Check VPN works. Change IP in `ssh_config` to use VPN IP instead of LAN IP (required for deploy when away from home).
 
-## Commit changes to `nixcfg` repo
+### Commit changes to `nixcfg` repo
 
 This is a good point to commit your new/modified `.nix` files.
 
-## Configure Nomad and Consul TLS
+### Configure Nomad and Consul TLS
 
 If you are bootstraping a new cluster, you need to `./genpki.sh <cluster>` to
 make a TLS PKI for the Nomad+Consul cluster to work. Then redo a deploy.

secretmgr/secretmgr

@@ -1,4 +1,5 @@
-#!/usr/bin/env python3
+#!/usr/bin/env nix-shell
+#!nix-shell -i python3 -p "python3.withPackages(ps: [ ps.pip ps.consul ps.ldap ps.passlib ps.requests ps.six ])"
 
 # DEPENDENCY: python-consul
 import consul

secretmgr/shell.nix

@@ -1,15 +0,0 @@
-{
-  pkgs ? import <nixpkgs> {}
-}:
-
-with pkgs; mkShell {
-  nativeBuildInputs = [
-    nomad 
-    docker-compose
-    python39Packages.pip 
-    python39Packages.ldap 
-    python39Packages.consul 
-    python39Packages.passlib
-  ];
-}
-

sshtool

@@ -1,6 +1,11 @@
 #!/usr/bin/env bash
 
 CMDFILE="$1"
+if [ -z "$CMDFILE" ] || [ ! -f "$CMDFILE" ]; then
+	echo "sshtool is not meant to be called on its own."
+	echo "See scripts that use it (e.g. deploy_nixos) for usage examples."
+	exit 1
+fi
 shift 1
 
 cd $(dirname $CMDFILE)