Merge branch 'master' of git.deuxfleurs.fr:Deuxfleurs/infrastructure

README.md

@@ -46,7 +46,7 @@ To ease the development, we make the choice of a fully integrated environment
 
 ### Deploying/Updating new services is done from your machine
 
-*The following instructions are provided for ops that already have access to the servers.*
+*The following instructions are provided for ops that already have access to the servers (meaning: their SSH public key is known by the cluster).*
 
 Deploy Nomad on your machine:
 
@@ -74,16 +74,37 @@ Create an alias (and put it in your `.bashrc`) to bind APIs on your machine:
 alias bind_df="ssh \
   -p110 \
   -N \
-  -L 4646:127.0.0.1:4646 \
-  -L 8500:127.0.0.1:8500 \
-  -L 8082:traefik-admin.service.2.cluster.deuxfleurs.fr:8082 \
-  -L 5432:psql-proxy.service.2.cluster.deuxfleurs.fr:5432 \
   -L 1389:bottin2.service.2.cluster.deuxfleurs.fr:389 \
+  -L 4646:127.0.0.1:4646 \
+  -L 5432:psql-proxy.service.2.cluster.deuxfleurs.fr:5432 \
+  -L 8082:traefik-admin.service.2.cluster.deuxfleurs.fr:8082 \
+  -L 8500:127.0.0.1:8500 \
   <a server from the cluster>"
 ```
 
 and run: 
 
-```
     bind_df
+
+Adrien uses `.ssh/config` configuration instead. I works basically the same. Here it goes:
+
 ```
+# in ~/.ssh/config 
+
+Host deuxfleurs
+    User adrien
+    Hostname deuxfleurs.fr
+    # If you don't use the default ~/.ssh/id_rsa to connect to Deuxfleurs
+    IdentityFile <some_key_path>
+    PubKeyAuthentication yes
+    ForwardAgent No
+    LocalForward 1389 bottin2.service.2.cluster.deuxfleurs.fr:389
+    LocalForward 4646 127.0.0.1:4646
+    LocalForward 5432 psql-proxy.service.2.cluster.deuxfleurs.fr:5432
+    LocalForward 8082 traefik-admin.service.2.cluster.deuxfleurs.fr:8082
+    LocalForward 8500 127.0.0.1:8500
+```
+
+Now, to connect, do the following:
+
+    ssh deuxfleurs -N 

app/.gitignore

@@ -1 +1,2 @@
+env/
 __pycache__

app/README.md

@@ -1,6 +1,4 @@
-## Understand this folder hierarchy
-
-This folder contains the following hierarchy:
+# Folder hierarchy
 
 - `<module>/build/<image_name>/`: folders with dockerfiles and other necessary resources for building container images
 - `<module>/config/`: folder containing configuration files, referenced by deployment file
@@ -8,18 +6,34 @@ This folder contains the following hierarchy:
 - `<module>/deploy/`: folder containing the HCL file(s) necessary for deploying the module
 - `<module>/integration/`: folder containing files for integration testing using docker-compose
 
+# Secret Manager `secretmgr.py`
+
+The Secret Manager ensures that all secrets are present where they should in the cluster.
+
+**You need access to the cluster** (SSH port forwarding) for it to find any secret on the cluster. Refer to the previous directory's [README](../README.md), at the bottom of the file.
+
 ## How to install `secretmgr.py` dependencies
 
-How to install its dependencies:
-
 ```bash
-# on fedora:
-dnf install -y openldap-devel
-# on ubuntu:
-apt-get install -y libldap2-dev
+### Install system dependencies first:
+## On fedora
 
-# for eveyrone:
-pip3 install --user --requirement requirements.txt
+dnf install -y openldap-devel cyrus-sasl-devel
+## On ubuntu
+apt-get install -y libldap2-dev libsasl2-dev
+
+### Now install the Python dependencies from requirements.txt:
+
+## Either using a virtual environment
+# (requires virtualenv python module)
+python3 -m virtualenv env 
+# Must be done everytime you create a new terminal window in this folder:
+. env/bin/activate 
+# Install the deps
+pip install -r requirements.txt
+
+## Either by installing the dependencies for your system user:
+pip3 install --user -r requirements.txt
 ```
 
 ## How to use `secretmgr.py`
@@ -42,7 +56,7 @@ Rotate secrets for app `dummy`, overwriting existing ones (be careful, this is d
 ./secretmgr.py regen dummy
 ```
 
-## How to upgrade our packaged apps to a new version?
+# Upgrading one of our packaged apps to a new version
 
  1. Edit `docker-compose.yml`
  2. Change the `VERSION` variable to the desired version

app/backup/secrets/backup/id_ed25519

@@ -0,0 +1 @@
+USER_LONG Private ed25519 key of the container doing the backup

app/backup/secrets/backup/id_ed25519.pub

@@ -0,0 +1 @@
+USER Public ed25519 key of the container doing the backup (this key must be in authorized_keys on the backup target host)

app/backup/secrets/backup/target_ssh_dir

@@ -0,0 +1 @@
+USER Directory where to store backups on target host

app/backup/secrets/backup/target_ssh_fingerprint

@@ -0,0 +1 @@
+USER SSH fingerprint of the target machine (format: copy here the corresponding line from your known_hosts file)

app/backup/secrets/backup/target_ssh_host

@@ -0,0 +1 @@
+USER Hostname of the backup target host

app/backup/secrets/backup/target_ssh_port

@@ -0,0 +1 @@
+USER SSH port number to connect to the target host

app/backup/secrets/backup/target_ssh_user

@@ -0,0 +1 @@
+USER SSH username to log in as on the target host

app/garage/secrets/garage/garage-ca.crt

@@ -0,0 +1 @@
+USER_LONG garage-ca.crt (generated with Garage's genkeys.sh script)

app/garage/secrets/garage/garage-ca.key

@@ -0,0 +1 @@
+USER_LONG garage-ca.key (generated with Garage's genkeys.sh script)

app/garage/secrets/garage/garage.crt

@@ -0,0 +1 @@
+USER_LONG garage.crt (generated with Garage's genkeys.sh script)

app/garage/secrets/garage/garage.key

@@ -0,0 +1 @@
+USER_LONG garage.key (generated with Garage's genkeys.sh script)

app/im/secrets/chat/coturn/static-auth

@@ -1 +1 @@
-USER cotorn static-auth (what is this?)
+USER coturn static-auth (what is this?)

app/im/secrets/chat/easybridge/as_token

@@ -0,0 +1 @@
+CMD openssl rand -hex 32

app/im/secrets/chat/easybridge/db_pass

@@ -0,0 +1 @@
+SERVICE_PASSWORD easybridge

app/im/secrets/chat/easybridge/db_user

@@ -0,0 +1 @@
+CONST easybridge

app/im/secrets/chat/easybridge/hs_token

@@ -0,0 +1 @@
+CMD openssl rand -hex 32

app/im/secrets/chat/easybridge/web_session_key

@@ -0,0 +1,2 @@
+CMD openssl rand -hex 32
+

app/im/secrets/chat/fb2mx/as_token

@@ -1 +1 @@
-USER fb2mx API server token
+CMD openssl rand -hex 32

app/im/secrets/chat/fb2mx/hs_token

@@ -1 +1 @@
-USER fb2mx homeserver token
+CMD openssl rand -hex 32

app/im/secrets/chat/synapse/homeserver.signing.key

@@ -0,0 +1 @@
+USER Synapse homeserver ed25519 signing key

app/im/secrets/chat/synapse/registration_shared_secret

@@ -1 +1 @@
-USER Shared secret for homeserver registrations (?)
+CMD head -c 32 /dev/urandom | base64

app/plume/secrets/plume/pgsql_pw

@@ -1 +1 @@
-CMD openssl rand -base64 32
+SERVICE_PASSWORD plume

app/secretmgr.py

@@ -43,6 +43,9 @@ USER_LONG <description>
 CMD <command>
 (a secret that is generated by running this command)
 
+CMD_ONCE <command>
+(same, but value is not changed when doing a regen)
+
 CONST <constant value>
 (the secret has a constant value set here)
 
@@ -81,6 +84,7 @@ consul_server = consul.Consul()
 USER             = "USER"
 USER_LONG        = "USER_LONG"
 CMD              = "CMD"
+CMD_ONCE         = "CMD_ONCE"
 CONST            = "CONST"
 CONST_LONG       = "CONST_LONG"
 SERVICE_DN       = "SERVICE_DN"
@@ -103,12 +107,15 @@ class bcolors:
 
 def read_secret(key, file_path):
     lines = [l.strip() for l in open(file_path, "r")]
+    if len(lines) == 0:
+        print(bcolors.FAIL, "ERROR:", bcolors.ENDC, "Empty file in", file_path)
+        sys.exit(-1)
     l0 = lines[0].split(" ")
     stype = l0[0]
     secret = {"type": stype, "key": key}
     if stype in [USER, USER_LONG]:
         secret["desc"] = " ".join(l0[1:])
-    elif stype == CMD:
+    elif stype in [CMD, CMD_ONCE]:
         secret["cmd"] = " ".join(l0[1:])
     elif stype == CONST:
         secret["value"] = " ".join(l0[1:])
@@ -151,6 +158,7 @@ def get_secrets_services(secrets):
         if svc not in services:
             services[svc] = {
                 "dn": "cn=%s,%s"%(svc, SERVICE_DN_SUFFIX),
+                "desc": "(not provided)",
                 "pass": None,
                 "dn_at": [],
                 "pass_at": [],
@@ -275,7 +283,7 @@ def gen_secrets_base(secrets, regen):
                     line = input().strip()
                     if line == ".":
                         break
-                    vals.append(line)
+                    lines.append(line)
                 val = "\n".join(lines)
                 consul_server.kv.put(key, val)
                 print(bcolors.OKCYAN, "Value set.", bcolors.ENDC)
@@ -289,7 +297,7 @@ def gen_secrets_base(secrets, regen):
             consul_server.kv.put(key, secret["value"])
             print(bcolors.OKCYAN, "Value set.", bcolors.ENDC)
 
-        if secret["type"] == CMD:
+        if secret["type"] == CMD or (secret["type"] == CMD_ONCE and data is None):
             print("----")
             print(key)
             print("Executing command:", secret["cmd"])

op_guide/create_database/README.md

@@ -8,6 +8,8 @@ Go to guichet.deuxfleurs.fr
   4. Hash it with `slappasswd`
   5. Add a `userpassword` entry with the hash
 
+This step can also be done using the automated tool `secretmgr.py` in the app folder.
+
 ## 2. Connect to postgres with the admin users
 
 ```bash

op_guide/plume/README.md

@@ -1,3 +1,5 @@
+## Creating a new Plume user
+
   1. Bind nomad on your machine with SSH (check the README file at the root of this repo)
   2. Go to http://127.0.0.1:4646
   3. Select `plume` -> click `exec` button (top right)