merge from upstream

README.md

@@ -5,21 +5,25 @@ deuxfleurs.fr
 
 ## Our abstraction stack
 
-We try to build a generic abstraction stack between our different resources (CPU, RAM, disk, etc.) and our services (Chat, Storage, etc.):
+We try to build a generic abstraction stack between our different resources (CPU, RAM, disk, etc.) and our services (Chat, Storage, etc.), we develop our own tools when needed:
 
-  * ansible (physical node conf)
-  * nomad (schedule containers)
-  * consul (distributed key value store / lock / service discovery)
-  * garage/glusterfs (file storage)
-  * stolon + postgresql (distributed relational database)
-  * docker (container tool)
-  * bottin (LDAP server, auth)
+  * **[garage](https://git.deuxfleurs.fr/Deuxfleurs/garage/):** S3-compatible lightweight object store for self-hosted geo-distributed deployments (we also have a legacy glusterfs cluster)
+  * **[diplonat](https://git.deuxfleurs.fr/Deuxfleurs/diplonat):** network automation (firewalling, upnp igd)
+  * **[bottin](https://git.deuxfleurs.fr/Deuxfleurs/bottin):** authentication and authorization (LDAP protocol, consul backend)
+  * **[guichet](https://git.deuxfleurs.fr/Deuxfleurs/guichet):** a dashboard for our users and administrators
+  * **ansible:** physical node configuration
+  * **nomad:** schedule containers and handle their lifecycle
+  * **consul:** distributed key value store + lock + service discovery
+  * **stolon + postgresql:** distributed relational database
+  * **docker:** package, distribute and isolate applications
 
 Some services we provide:
 
-  * Chat (Matrix/Riot)
-  * Email (Postfix/Dovecot/Sogo)
-  * Storage (Seafile)
+  * **Websites:** garage (static) + fediverse blog (plume)
+  * **Chat:** Synapse + Element Web (Matrix protocol)
+  * **Email:** Postfix SMTP + Dovecot IMAP + opendkim DKIM + Sogo webmail (legacy) | Alps webmail (experimental)
+  * **Storage:** Seafile (legacy) | Nextcloud (experimental)
+  * **Visio:** Jitsi
 
 As a generic abstraction is provided, deploying new services should be easy.
 
@@ -40,14 +44,6 @@ To ease the development, we make the choice of a fully integrated environment
 
 ## Start hacking
 
-### Clone the repository
-
-```
-git clone https://gitlab.com/superboum/deuxfleurs.fr.git
-git submodule init
-git submodule update
-```
-
 ### Deploying/Updating new services is done from your machine
 
 *The following instructions are provided for ops that already have access to the servers.*
@@ -82,6 +78,7 @@ alias bind_df="ssh \
   -L 8500:127.0.0.1:8500 \
   -L 8082:traefik-admin.service.2.cluster.deuxfleurs.fr:8082 \
   -L 5432:psql-proxy.service.2.cluster.deuxfleurs.fr:5432 \
+  -L 1389:bottin2.service.2.cluster.deuxfleurs.fr:389 \
   <a server from the cluster>"
 ```
 

app/.gitignore

@@ -0,0 +1 @@
+__pycache__

app/README.md

@@ -0,0 +1,52 @@
+## Understand this folder hierarchy
+
+This folder contains the following hierarchy:
+
+- `<module>/build/<image_name>/`: folders with dockerfiles and other necessary resources for building container images
+- `<module>/config/`: folder containing configuration files, referenced by deployment file
+- `<module>/secrets/`: folder containing secrets, which can be synchronized with Consul using `secretmgr.py`
+- `<module>/deploy/`: folder containing the HCL file(s) necessary for deploying the module
+- `<module>/integration/`: folder containing files for integration testing using docker-compose
+
+## How to install `secretmgr.py` dependencies
+
+How to install its dependencies:
+
+```bash
+# on fedora:
+dnf install -y openldap-devel
+# on ubuntu:
+apt-get install -y libldap2-dev
+
+# for eveyrone:
+pip3 install --user --requirement requirements.txt
+```
+
+## How to use `secretmgr.py`
+
+Check that all secrets are correctly deployed for app `dummy`:
+
+```bash
+./secretmgr.py check dummy
+```
+
+Generate secrets for app `dummy` if they don't already exist:
+
+```bash
+./secretmgr.py gen dummy
+```
+
+Rotate secrets for app `dummy`, overwriting existing ones (be careful, this is dangerous!):
+
+```bash
+./secretmgr.py regen dummy
+```
+
+## How to upgrade our packaged apps to a new version?
+
+ 1. Edit `docker-compose.yml`
+ 2. Change the `VERSION` variable to the desired version
+ 3. Increment the docker image tag by 1 (eg: superboum/riot:v13 -> superboum/riot:v14)
+ 4. Run `docker-compose build`
+ 5. Run `docker-compose push`
+ 6. Done

app/build/README.md

@@ -1,8 +0,0 @@
-## How to upgrade our packaged apps to a new version?
-
- 1. Edit `docker-compose.yml`
- 2. Change the `VERSION` variable to the desired version
- 3. Increment the docker image tag by 1 (eg: superboum/riot:v13 -> superboum/riot:v14)
- 4. Run `docker-compose build`
- 5. Run `docker-compose push`
- 6. Done

app/build/blog-quentin/Dockerfile

@@ -1,16 +0,0 @@
-FROM amd64/debian:stretch as builder
-
-COPY ./quentin.dufour.io/Gemfile /root/quentin.dufour.io/Gemfile
-
-WORKDIR /root/quentin.dufour.io
-
-RUN apt-get update && \
-    apt-get install -y ruby-dev gem build-essential bundler zlib1g-dev libxml2-dev && \
-    bundle install
-
-COPY ./quentin.dufour.io/ /root/quentin.dufour.io/
-RUN bundle exec jekyll build
-
-FROM superboum/amd64_webserver:v2
-COPY --from=builder /root/quentin.dufour.io/_site /srv/http
-

app/build/blog-quentin/README.md

@@ -1 +0,0 @@
-sudo docker build -t superboum/amd64_blog:v19 .

app/build/coturn/Dockerfile

@@ -1,8 +0,0 @@
-FROM amd64/debian:buster
-
-RUN apt-get update && \
-    apt-get dist-upgrade -y && \
-    apt-get install -y \
-      coturn
-
-CMD ["/usr/bin/turnserver"]

app/build/coturn/README.md

@@ -1,17 +0,0 @@
-
-## Génère l'image
-```
-sudo docker build -t registry.gitlab.com/superboum/ankh-morpork/amd64_coturn:v1 .
-```
-
-## Run bash dans le container
-```
-sudo docker run --rm -t -i registry.gitlab.com/superboum/ankh-morpork/amd64_coturn:v1 bash
-sudo docker run --rm -t -i -p 3478:3478/udp -p 3479:3479/udp -p 3478:3478/tcp -p 3479:3479/tcp registry.gitlab.com/superboum/ankh-morpork/amd64_coturn:v1 
-```
-
-## Used ports
-- udp/tcp 3478 3479 
-
-## Publish
-sudo docker push registry.gitlab.com/superboum/ankh-morpork/amd64_coturn:v1

app/build/landing/README.md

@@ -1,3 +0,0 @@
-```
-docker build -t superboum/amd64_landing:v8 .
-```

app/build/static/Dockerfile

@@ -1,9 +0,0 @@
-FROM golang:1.11.1-stretch as builder
-
-COPY ./goStatic /goStatic
-WORKDIR /goStatic
-RUN CGO_ENABLED=0 go build -a -o web-server .
-
-FROM scratch
-COPY --from=builder /goStatic/web-server /
-ENTRYPOINT ["/web-server"]

app/build/static/README.md

@@ -1,5 +0,0 @@
-
-```
-sudo docker build -t superboum/amd64_webserver:v3 .
-sudo docker push superboum/amd64_webserver:v3
-``` 

app/build/static/goStatic

@@ -1 +0,0 @@
-Subproject commit 3f97f57aaee09a142afe3ca0f1a5d51acd856436

app/build/webpull/.gitignore

@@ -1 +0,0 @@
-main

app/build/webpull/Dockerfile.nodejs

@@ -1,9 +0,0 @@
-FROM node:13.8-buster
-
-RUN apt-get update && \
-    apt-get install -y git
-
-COPY ./main /srv/httpd
-WORKDIR /srv
-CMD ["/srv/httpd"]
-    

app/build/webpull/Dockerfile.ruby

@@ -1,12 +0,0 @@
-FROM fedora:32
-
-ENV LC_ALL=C.UTF-8
-ENV LANG=C.UTF-8
-ENV LANGUAGE=en_US.UTF-8
-ENV RUBYOPT --disable-did_you_mean
-
-RUN dnf install -y git ruby ruby-devel rubygems rubygem-bundler @development-tools redhat-rpm-config gcc-c++ zlib-devel
-
-COPY ./main /srv/httpd
-WORKDIR /srv
-CMD ["/srv/httpd"]

app/build/webpull/README.md

@@ -1,23 +0,0 @@
-# webpull
-
-Webpull allows you to update your live website without deploying a new docker container but by simply calling an URL
-
-You need to specify a secret token at boot:
-
-```
-WEBPULL_TOKEN=s3cr3et ./webpull
-```
-
-## Node.js version
-
-```
-go build ./main.go
-sudo docker build -f ./Dockerfile.nodejs -t superboum/amd64_webpull_pug:v1  .
-```
-
-## Ruby version
-
-```
-go build ./main.go
-sudo docker build -f ./Dockerfile.ruby -t superboum/amd64_webpull_ruby:v1 .
-```

app/build/webpull/main.go

@@ -1,100 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"errors"
-	"io"
-	"os/exec"
-	"os"
-	"log"
-	"net/http"
-	"strings"
-)
-
-func myexec(w io.Writer, main string, params ...string) error {
-	cmd := exec.Command(main, params...)
-	cmd.Stdout = w
-	cmd.Stderr = w
-	err := cmd.Run()
-	if err != nil {
-		fmt.Fprintf(w, "Failed to run: %s %s\n", main, strings.Join(params, " "))
-	}
-	return err
-}
-
-func update(w io.Writer) error {
-	fmt.Fprintf(w, "Start update...\n")
-	_, err := os.Stat("./.git")
-	if err != nil {
-		fmt.Fprintf(w, ".git folder does not exist, creating it...\n")
-		err := myexec(w, "git", "init")
-		if err != nil {
-			return err
-		}
-	}
-
-	err = myexec(w, "git", "remote", "get-url", "origin")
-	if err != nil {
-		repo, exists := os.LookupEnv("WEBPULL_REPO")
-		if !exists {
-			fmt.Fprintf(w, "You must define WEBPULL_REPO env variable...\n")
-			return errors.New("Missing environment variable WEBPULL_REPO")
-		}
-		fmt.Fprintf(w, "git remote is not yet set...\n")
-		err := myexec(w, "git", "remote", "add", "origin", repo)
-		if err != nil {
-			return err
-		}
-	}
-
-	err = myexec(w, "git", "pull", "origin", "master")
-	if err != nil {
-		fmt.Fprintf(w, "Failed to pull...\n")
-		return err
-	}
-
-	_, err = os.Stat("./.webpull")
-	if err != nil {
-		fmt.Fprintf(w, "You must create an executable file named '.webpull' at the root of your repository.\nIf you have nothing to run, just create an empty bash script...\n")
-		return err
-	}
-
-	err = myexec(w, "./.webpull")
-	if err != nil {
-		fmt.Fprintf(w, "An error occured during script execution\n")
-		return err
-	}
-
-	fmt.Fprintf(w, "Success.\n")
-	return nil
-}
-
-func main() {
-	token, exists := os.LookupEnv("WEBPULL_TOKEN")
-	if !exists {
-		log.Fatal("Environment variable 'WEBPULL_TOKEN' must be defined")
-	}
-
-	if update(os.Stdout) != nil {
-		log.Fatal("Initial 'update' failed")
-	}
-
-	fs := http.FileServer(http.Dir("./static"))
-	http.HandleFunc("/update", func(w http.ResponseWriter, r *http.Request) {
-		keys, ok := r.URL.Query()["token"]
-		if !ok || len(keys[0]) < 1 {
-			http.Error(w, "Missing 'token' query parameter", 401)
-			return
-		}
-
-		if keys[0] != token {
-			http.Error(w, "Wrong token", 401)
-			return
-		}
-
-		update(w)
-	})
-	http.Handle("/", fs)
-
-	log.Fatal(http.ListenAndServe(":8080", nil))
-}

app/config/secrets/.gitignore

@@ -1,11 +0,0 @@
-# Blacklist everything cleverly
-*
-!*/
-
-# Whitelist some patterns
-!*.sample
-!*.gen
-!*.sh
-!.gitignore
-
-# Whitelist specific files

app/config/secrets/chat/fb2mx/db_url.sample

@@ -1 +0,0 @@
-postgres://username:password@hostname/dbname

app/config/secrets/plume/pgsql_pw.sh

@@ -1,2 +0,0 @@
-#!/bin/bash
-openssl rand -base64 32 > pgsql_pw

app/config/secrets/plume/secret_key.sh

@@ -1,2 +0,0 @@
-#!/bin/bash
-openssl rand -base64 32 > secret_key

app/deployment/web_static.hcl

@@ -1,117 +0,0 @@
-job "web_static" {
-  datacenters = ["dc1"]
-  type = "service"
-
-  constraint {
-    attribute = "${attr.cpu.arch}"
-    value     = "amd64"
-  }
-
-  group "landing" {
-    network {
-      port "deuxfleurs_port" { to = 8080 }
-    }
-
-    task "server" {
-      driver = "docker"
-      config {
-        image = "superboum/amd64_webpull_pug:v4"
-        ports = [ "deuxfleurs_port" ]
-      }
-
-      template {
-        data = <<EOH
-WEBPULL_REPO="https://git.deuxfleurs.fr/Deuxfleurs/site.git"
-WEBPULL_TOKEN="{{ key "secrets/web/home_token" | trimSpace }}"
-EOH
-        destination = "secrets/env"
-        env = true
-      }
-
-      resources {
-        memory = 200
-      }
-
-      service {
-        tags = [
-          "webstatic",
-          "traefik.enable=true",
-          "traefik.frontend.entryPoints=https,http",
-          "traefik.frontend.rule=Host:deuxfleurs.fr,www.deuxfleurs.fr,deuxfleurs.org,www.deuxfleurs.org;PathPrefix:/",
-          "traefik.frontend.priority=10"
-          # Ideally we would have a rewrite regex like this https://regex101.com/r/WHrABU/1
-          # See how it does not handle "http://deuxfleurs.fr/"? Not a big deal since HTTPS is redirected somewhere else
-          #"traefik.frontend.redirect.regex=^https?://(www\.deuxfleurs\.fr|deuxfleurs\.org|www\.deuxfleurs\.org)(.*)$",
-          #"traefik.frontend.redirect.replacement=https://deuxfleurs.fr/$${2}",
-          # Only set permanent redirect when it is guaranteed to work
-          #"traefik.frontend.redirect.permanent=true",
-        ]
-        port = "deuxfleurs_port"
-        address_mode = "host"
-        name = "landing"
-        check {
-          type = "tcp"
-          port = "deuxfleurs_port"
-          interval = "60s"
-          timeout = "5s"
-          check_restart {
-            limit = 3
-            grace = "90s"
-            ignore_warnings = false
-          }
-        }
-      }
-    }
-  }
-
-  group "quentin" {
-    network {
-      port "quentin_port" { to = 8080 }
-    }
-
-    task "server" {
-      driver = "docker"
-      config {
-        image = "superboum/amd64_webpull_ruby:v1"
-        ports = [ "quentin_port" ]
-      }
-
-      template {
-        data = <<EOH
-WEBPULL_REPO="https://git.deuxfleurs.fr/quentin/quentin.dufour.io.git"
-WEBPULL_TOKEN="{{ key "secrets/web/quentin.dufour.io_token" | trimSpace }}"
-EOH
-        destination = "secrets/env"
-        env = true
-      }
-
-      resources {
-        memory = 500
-      }
-
-      service {
-        tags = [
-          "webstatic",
-          "traefik.enable=true",
-          "traefik.frontend.entryPoints=https",
-          "traefik.frontend.rule=Host:quentin.dufour.io,www.quentin.dufour.io;PathPrefix:/"
-        ]
-        port = "quentin_port"
-        address_mode = "host"
-        name = "blog-quentin"
-        check {
-          type = "tcp"
-          port = "quentin_port"
-          interval = "60s"
-          timeout = "5s"
-          check_restart {
-            limit = 3
-            grace = "90s"
-            ignore_warnings = false
-          }
-        }
-      }
-    }
-  }
-}
-

app/directory/deploy/directory.hcl

@@ -35,7 +35,7 @@ job "directory" {
       }
 
       template {
-        data = file("../config/configuration/directory/bottin/config.json")
+        data = file("../config/bottin/config.json")
         destination = "secrets/config.json"
       }
 
@@ -78,7 +78,7 @@ job "directory" {
       }
 
       template {
-        data = file("../config/configuration/directory/guichet/config.json.tpl")
+        data = file("../config/guichet/config.json.tpl")
         destination = "secrets/config.json"
       }
 

app/docker-compose.yml

@@ -3,7 +3,7 @@ services:
 
   mariadb:
     build:
-      context: ./mariadb
+      context: ./seafile/build/mariadb
       args:
         VERSION: 4 # fake for now
     image: superboum/amd64_mariadb:v4 
@@ -11,24 +11,24 @@ services:
   # Instant Messaging
   riot:
     build:
-      context: ./riotweb
+      context: ./im/build/riotweb
       args:
         # https://github.com/vector-im/riot-web/releases
-        VERSION: 1.7.14
-    image: particallydone/amd64_riotweb:v18
+        VERSION: 1.7.16
+    image: superboum/amd64_riotweb:v19
 
   synapse:
     build: 
-      context: ./matrix-synapse
+      context: ./im/build/matrix-synapse
       args:
         # https://github.com/matrix-org/synapse/releases
-        VERSION: 1.24.0
-    image: particallydone/amd64_synapse:v39
+        VERSION: 1.25.0
+    image: superboum/amd64_synapse:v40
 
   # Email
   sogo:
     build:
-      context: ./sogo
+      context: ./email/build/sogo
       args:
         # fake for now
         VERSION: 5.0.0
@@ -36,7 +36,7 @@ services:
 
   alps:
     build:
-      context: ./alps
+      context: ./email/build/alps
       args:
         VERSION: 5cef0aaff2b8b6ee3e00b566123517e241d8cfb8
     image: superboum/amd64_alps:v1
@@ -44,48 +44,48 @@ services:
   # VoIP
   jitsi-meet:
     build:
-      context: ./jitsi-meet
+      context: ./jitsi/build/jitsi-meet
       args:
         # https://github.com/jitsi/jitsi-meet
         PREFIXV: stable/jitsi-meet_
-        VERSION: 4966
-    image: superboum/amd64_jitsi_meet:v2
+        VERSION: 5390
+    image: superboum/amd64_jitsi_meet:v3
 
   jitsi-conference-focus:
     build:
-      context: ./jitsi-conference-focus
+      context: ./jitsi/build/jitsi-conference-focus
       args:
         # https://github.com/jitsi/jicofo
         PREFIXV: stable/jitsi-meet_
-        VERSION: 4966
-    image: superboum/amd64_jitsi_conference_focus:v5
+        VERSION: 5390
+    image: superboum/amd64_jitsi_conference_focus:v6
 
   jitsi-videobridge:
     build:
-      context: ./jitsi-videobridge
+      context: ./jitsi/build/jitsi-videobridge
       args:
         # https://github.com/jitsi/jitsi-videobridge
         PREFIXV: stable/jitsi-meet_
-        VERSION: 4966
-    image: superboum/amd64_jitsi_videobridge:v15
+        VERSION: 5390
+    image: superboum/amd64_jitsi_videobridge:v16
 
   jitsi-xmpp:
     build:
-      context: ./jitsi-xmpp
+      context: ./jitsi/build/jitsi-xmpp
       args:
         VERSION: 0.11.2-1
     image: superboum/amd64_jitsi_xmpp:v8
 
   plume:
     build:
-      context: ./plume
+      context: ./plume/build/plume
       args:
-        VERSION: 0cd26dfbf4ab7be467325ed77230cf371147a98e
-    image: superboum/plume:v1
+        VERSION: 0.6.0
+    image: superboum/plume:v2
 
   postfix:
     build:
-      context: ./postfix
+      context: ./email/build/postfix
       args:
         # https://packages.debian.org/fr/buster/postfix
         VERSION: 3.4.14-0+deb10u1

app/dummy/secrets/dummy/test_cmd

@@ -0,0 +1 @@
+CMD head -c 10 /dev/urandom | base64

app/dummy/secrets/dummy/test_const

@@ -0,0 +1 @@
+CONST this is a constant

app/dummy/secrets/dummy/test_const_long

@@ -0,0 +1,5 @@
+CONST_LONG
+this is a 
+constant
+on several
+lines

app/dummy/secrets/dummy/test_service_dn

@@ -0,0 +1 @@
+SERVICE_DN dummy Dummy service for testing secretmgr.py

app/dummy/secrets/dummy/test_service_password

@@ -0,0 +1 @@
+SERVICE_PASSWORD dummy

app/dummy/secrets/dummy/test_user

@@ -0,0 +1 @@
+USER Test user value