Merge branch 'master' of git.deuxfleurs.fr:Deuxfleurs/infrastructure

This commit is contained in:
Quentin 2021-01-20 10:21:42 +01:00
commit 8eaa7914d0
29 changed files with 92 additions and 26 deletions

View file

@ -46,7 +46,7 @@ To ease the development, we make the choice of a fully integrated environment
### Deploying/Updating new services is done from your machine ### Deploying/Updating new services is done from your machine
*The following instructions are provided for ops that already have access to the servers.* *The following instructions are provided for ops that already have access to the servers (meaning: their SSH public key is known by the cluster).*
Deploy Nomad on your machine: Deploy Nomad on your machine:
@ -74,16 +74,37 @@ Create an alias (and put it in your `.bashrc`) to bind APIs on your machine:
alias bind_df="ssh \ alias bind_df="ssh \
-p110 \ -p110 \
-N \ -N \
-L 4646:127.0.0.1:4646 \
-L 8500:127.0.0.1:8500 \
-L 8082:traefik-admin.service.2.cluster.deuxfleurs.fr:8082 \
-L 5432:psql-proxy.service.2.cluster.deuxfleurs.fr:5432 \
-L 1389:bottin2.service.2.cluster.deuxfleurs.fr:389 \ -L 1389:bottin2.service.2.cluster.deuxfleurs.fr:389 \
-L 4646:127.0.0.1:4646 \
-L 5432:psql-proxy.service.2.cluster.deuxfleurs.fr:5432 \
-L 8082:traefik-admin.service.2.cluster.deuxfleurs.fr:8082 \
-L 8500:127.0.0.1:8500 \
<a server from the cluster>" <a server from the cluster>"
``` ```
and run: and run:
```
bind_df bind_df
Adrien uses `.ssh/config` configuration instead. I works basically the same. Here it goes:
``` ```
# in ~/.ssh/config
Host deuxfleurs
User adrien
Hostname deuxfleurs.fr
# If you don't use the default ~/.ssh/id_rsa to connect to Deuxfleurs
IdentityFile <some_key_path>
PubKeyAuthentication yes
ForwardAgent No
LocalForward 1389 bottin2.service.2.cluster.deuxfleurs.fr:389
LocalForward 4646 127.0.0.1:4646
LocalForward 5432 psql-proxy.service.2.cluster.deuxfleurs.fr:5432
LocalForward 8082 traefik-admin.service.2.cluster.deuxfleurs.fr:8082
LocalForward 8500 127.0.0.1:8500
```
Now, to connect, do the following:
ssh deuxfleurs -N

1
app/.gitignore vendored
View file

@ -1 +1,2 @@
env/
__pycache__ __pycache__

View file

@ -1,6 +1,4 @@
## Understand this folder hierarchy # Folder hierarchy
This folder contains the following hierarchy:
- `<module>/build/<image_name>/`: folders with dockerfiles and other necessary resources for building container images - `<module>/build/<image_name>/`: folders with dockerfiles and other necessary resources for building container images
- `<module>/config/`: folder containing configuration files, referenced by deployment file - `<module>/config/`: folder containing configuration files, referenced by deployment file
@ -8,18 +6,34 @@ This folder contains the following hierarchy:
- `<module>/deploy/`: folder containing the HCL file(s) necessary for deploying the module - `<module>/deploy/`: folder containing the HCL file(s) necessary for deploying the module
- `<module>/integration/`: folder containing files for integration testing using docker-compose - `<module>/integration/`: folder containing files for integration testing using docker-compose
# Secret Manager `secretmgr.py`
The Secret Manager ensures that all secrets are present where they should in the cluster.
**You need access to the cluster** (SSH port forwarding) for it to find any secret on the cluster. Refer to the previous directory's [README](../README.md), at the bottom of the file.
## How to install `secretmgr.py` dependencies ## How to install `secretmgr.py` dependencies
How to install its dependencies:
```bash ```bash
# on fedora: ### Install system dependencies first:
dnf install -y openldap-devel ## On fedora
# on ubuntu:
apt-get install -y libldap2-dev
# for eveyrone: dnf install -y openldap-devel cyrus-sasl-devel
pip3 install --user --requirement requirements.txt ## On ubuntu
apt-get install -y libldap2-dev libsasl2-dev
### Now install the Python dependencies from requirements.txt:
## Either using a virtual environment
# (requires virtualenv python module)
python3 -m virtualenv env
# Must be done everytime you create a new terminal window in this folder:
. env/bin/activate
# Install the deps
pip install -r requirements.txt
## Either by installing the dependencies for your system user:
pip3 install --user -r requirements.txt
``` ```
## How to use `secretmgr.py` ## How to use `secretmgr.py`
@ -42,7 +56,7 @@ Rotate secrets for app `dummy`, overwriting existing ones (be careful, this is d
./secretmgr.py regen dummy ./secretmgr.py regen dummy
``` ```
## How to upgrade our packaged apps to a new version? # Upgrading one of our packaged apps to a new version
1. Edit `docker-compose.yml` 1. Edit `docker-compose.yml`
2. Change the `VERSION` variable to the desired version 2. Change the `VERSION` variable to the desired version

View file

@ -0,0 +1 @@
USER_LONG Private ed25519 key of the container doing the backup

View file

@ -0,0 +1 @@
USER Public ed25519 key of the container doing the backup (this key must be in authorized_keys on the backup target host)

View file

@ -0,0 +1 @@
USER Directory where to store backups on target host

View file

@ -0,0 +1 @@
USER SSH fingerprint of the target machine (format: copy here the corresponding line from your known_hosts file)

View file

@ -0,0 +1 @@
USER Hostname of the backup target host

View file

@ -0,0 +1 @@
USER SSH port number to connect to the target host

View file

@ -0,0 +1 @@
USER SSH username to log in as on the target host

View file

@ -0,0 +1 @@
USER_LONG garage-ca.crt (generated with Garage's genkeys.sh script)

View file

@ -0,0 +1 @@
USER_LONG garage-ca.key (generated with Garage's genkeys.sh script)

View file

@ -0,0 +1 @@
USER_LONG garage.crt (generated with Garage's genkeys.sh script)

View file

@ -0,0 +1 @@
USER_LONG garage.key (generated with Garage's genkeys.sh script)

View file

@ -1 +1 @@
USER cotorn static-auth (what is this?) USER coturn static-auth (what is this?)

View file

@ -0,0 +1 @@
CMD openssl rand -hex 32

View file

@ -0,0 +1 @@
SERVICE_PASSWORD easybridge

View file

@ -0,0 +1 @@
CONST easybridge

View file

@ -0,0 +1 @@
CMD openssl rand -hex 32

View file

@ -0,0 +1,2 @@
CMD openssl rand -hex 32

View file

@ -1 +1 @@
USER fb2mx API server token CMD openssl rand -hex 32

View file

@ -1 +1 @@
USER fb2mx homeserver token CMD openssl rand -hex 32

View file

@ -0,0 +1 @@
USER Synapse homeserver ed25519 signing key

View file

@ -1 +1 @@
USER Shared secret for homeserver registrations (?) CMD head -c 32 /dev/urandom | base64

View file

@ -1 +1 @@
CMD openssl rand -base64 32 SERVICE_PASSWORD plume

View file

@ -43,6 +43,9 @@ USER_LONG <description>
CMD <command> CMD <command>
(a secret that is generated by running this command) (a secret that is generated by running this command)
CMD_ONCE <command>
(same, but value is not changed when doing a regen)
CONST <constant value> CONST <constant value>
(the secret has a constant value set here) (the secret has a constant value set here)
@ -81,6 +84,7 @@ consul_server = consul.Consul()
USER = "USER" USER = "USER"
USER_LONG = "USER_LONG" USER_LONG = "USER_LONG"
CMD = "CMD" CMD = "CMD"
CMD_ONCE = "CMD_ONCE"
CONST = "CONST" CONST = "CONST"
CONST_LONG = "CONST_LONG" CONST_LONG = "CONST_LONG"
SERVICE_DN = "SERVICE_DN" SERVICE_DN = "SERVICE_DN"
@ -103,12 +107,15 @@ class bcolors:
def read_secret(key, file_path): def read_secret(key, file_path):
lines = [l.strip() for l in open(file_path, "r")] lines = [l.strip() for l in open(file_path, "r")]
if len(lines) == 0:
print(bcolors.FAIL, "ERROR:", bcolors.ENDC, "Empty file in", file_path)
sys.exit(-1)
l0 = lines[0].split(" ") l0 = lines[0].split(" ")
stype = l0[0] stype = l0[0]
secret = {"type": stype, "key": key} secret = {"type": stype, "key": key}
if stype in [USER, USER_LONG]: if stype in [USER, USER_LONG]:
secret["desc"] = " ".join(l0[1:]) secret["desc"] = " ".join(l0[1:])
elif stype == CMD: elif stype in [CMD, CMD_ONCE]:
secret["cmd"] = " ".join(l0[1:]) secret["cmd"] = " ".join(l0[1:])
elif stype == CONST: elif stype == CONST:
secret["value"] = " ".join(l0[1:]) secret["value"] = " ".join(l0[1:])
@ -151,6 +158,7 @@ def get_secrets_services(secrets):
if svc not in services: if svc not in services:
services[svc] = { services[svc] = {
"dn": "cn=%s,%s"%(svc, SERVICE_DN_SUFFIX), "dn": "cn=%s,%s"%(svc, SERVICE_DN_SUFFIX),
"desc": "(not provided)",
"pass": None, "pass": None,
"dn_at": [], "dn_at": [],
"pass_at": [], "pass_at": [],
@ -275,7 +283,7 @@ def gen_secrets_base(secrets, regen):
line = input().strip() line = input().strip()
if line == ".": if line == ".":
break break
vals.append(line) lines.append(line)
val = "\n".join(lines) val = "\n".join(lines)
consul_server.kv.put(key, val) consul_server.kv.put(key, val)
print(bcolors.OKCYAN, "Value set.", bcolors.ENDC) print(bcolors.OKCYAN, "Value set.", bcolors.ENDC)
@ -289,7 +297,7 @@ def gen_secrets_base(secrets, regen):
consul_server.kv.put(key, secret["value"]) consul_server.kv.put(key, secret["value"])
print(bcolors.OKCYAN, "Value set.", bcolors.ENDC) print(bcolors.OKCYAN, "Value set.", bcolors.ENDC)
if secret["type"] == CMD: if secret["type"] == CMD or (secret["type"] == CMD_ONCE and data is None):
print("----") print("----")
print(key) print(key)
print("Executing command:", secret["cmd"]) print("Executing command:", secret["cmd"])

View file

@ -8,6 +8,8 @@ Go to guichet.deuxfleurs.fr
4. Hash it with `slappasswd` 4. Hash it with `slappasswd`
5. Add a `userpassword` entry with the hash 5. Add a `userpassword` entry with the hash
This step can also be done using the automated tool `secretmgr.py` in the app folder.
## 2. Connect to postgres with the admin users ## 2. Connect to postgres with the admin users
```bash ```bash

View file

@ -1,3 +1,5 @@
## Creating a new Plume user
1. Bind nomad on your machine with SSH (check the README file at the root of this repo) 1. Bind nomad on your machine with SSH (check the README file at the root of this repo)
2. Go to http://127.0.0.1:4646 2. Go to http://127.0.0.1:4646
3. Select `plume` -> click `exec` button (top right) 3. Select `plume` -> click `exec` button (top right)