merge from upstream

This commit is contained in:
LUXEY Adrien 2021-01-19 14:33:44 +01:00
commit 65421d947e
238 changed files with 652 additions and 443 deletions

View file

@ -5,21 +5,25 @@ deuxfleurs.fr
## Our abstraction stack
We try to build a generic abstraction stack between our different resources (CPU, RAM, disk, etc.) and our services (Chat, Storage, etc.):
We try to build a generic abstraction stack between our different resources (CPU, RAM, disk, etc.) and our services (Chat, Storage, etc.), we develop our own tools when needed:
* ansible (physical node conf)
* nomad (schedule containers)
* consul (distributed key value store / lock / service discovery)
* garage/glusterfs (file storage)
* stolon + postgresql (distributed relational database)
* docker (container tool)
* bottin (LDAP server, auth)
* **[garage](https://git.deuxfleurs.fr/Deuxfleurs/garage/):** S3-compatible lightweight object store for self-hosted geo-distributed deployments (we also have a legacy glusterfs cluster)
* **[diplonat](https://git.deuxfleurs.fr/Deuxfleurs/diplonat):** network automation (firewalling, upnp igd)
* **[bottin](https://git.deuxfleurs.fr/Deuxfleurs/bottin):** authentication and authorization (LDAP protocol, consul backend)
* **[guichet](https://git.deuxfleurs.fr/Deuxfleurs/guichet):** a dashboard for our users and administrators
* **ansible:** physical node configuration
* **nomad:** schedule containers and handle their lifecycle
* **consul:** distributed key value store + lock + service discovery
* **stolon + postgresql:** distributed relational database
* **docker:** package, distribute and isolate applications
Some services we provide:
* Chat (Matrix/Riot)
* Email (Postfix/Dovecot/Sogo)
* Storage (Seafile)
* **Websites:** garage (static) + fediverse blog (plume)
* **Chat:** Synapse + Element Web (Matrix protocol)
* **Email:** Postfix SMTP + Dovecot IMAP + opendkim DKIM + Sogo webmail (legacy) | Alps webmail (experimental)
* **Storage:** Seafile (legacy) | Nextcloud (experimental)
* **Visio:** Jitsi
As a generic abstraction is provided, deploying new services should be easy.
@ -40,14 +44,6 @@ To ease the development, we make the choice of a fully integrated environment
## Start hacking
### Clone the repository
```
git clone https://gitlab.com/superboum/deuxfleurs.fr.git
git submodule init
git submodule update
```
### Deploying/Updating new services is done from your machine
*The following instructions are provided for ops that already have access to the servers.*
@ -82,6 +78,7 @@ alias bind_df="ssh \
-L 8500:127.0.0.1:8500 \
-L 8082:traefik-admin.service.2.cluster.deuxfleurs.fr:8082 \
-L 5432:psql-proxy.service.2.cluster.deuxfleurs.fr:5432 \
-L 1389:bottin2.service.2.cluster.deuxfleurs.fr:389 \
<a server from the cluster>"
```

1
app/.gitignore vendored Normal file
View file

@ -0,0 +1 @@
__pycache__

52
app/README.md Normal file
View file

@ -0,0 +1,52 @@
## Understand this folder hierarchy
This folder contains the following hierarchy:
- `<module>/build/<image_name>/`: folders with dockerfiles and other necessary resources for building container images
- `<module>/config/`: folder containing configuration files, referenced by deployment file
- `<module>/secrets/`: folder containing secrets, which can be synchronized with Consul using `secretmgr.py`
- `<module>/deploy/`: folder containing the HCL file(s) necessary for deploying the module
- `<module>/integration/`: folder containing files for integration testing using docker-compose
## How to install `secretmgr.py` dependencies
How to install its dependencies:
```bash
# on fedora:
dnf install -y openldap-devel
# on ubuntu:
apt-get install -y libldap2-dev
# for eveyrone:
pip3 install --user --requirement requirements.txt
```
## How to use `secretmgr.py`
Check that all secrets are correctly deployed for app `dummy`:
```bash
./secretmgr.py check dummy
```
Generate secrets for app `dummy` if they don't already exist:
```bash
./secretmgr.py gen dummy
```
Rotate secrets for app `dummy`, overwriting existing ones (be careful, this is dangerous!):
```bash
./secretmgr.py regen dummy
```
## How to upgrade our packaged apps to a new version?
1. Edit `docker-compose.yml`
2. Change the `VERSION` variable to the desired version
3. Increment the docker image tag by 1 (eg: superboum/riot:v13 -> superboum/riot:v14)
4. Run `docker-compose build`
5. Run `docker-compose push`
6. Done

View file

@ -1,8 +0,0 @@
## How to upgrade our packaged apps to a new version?
1. Edit `docker-compose.yml`
2. Change the `VERSION` variable to the desired version
3. Increment the docker image tag by 1 (eg: superboum/riot:v13 -> superboum/riot:v14)
4. Run `docker-compose build`
5. Run `docker-compose push`
6. Done

View file

@ -1,16 +0,0 @@
FROM amd64/debian:stretch as builder
COPY ./quentin.dufour.io/Gemfile /root/quentin.dufour.io/Gemfile
WORKDIR /root/quentin.dufour.io
RUN apt-get update && \
apt-get install -y ruby-dev gem build-essential bundler zlib1g-dev libxml2-dev && \
bundle install
COPY ./quentin.dufour.io/ /root/quentin.dufour.io/
RUN bundle exec jekyll build
FROM superboum/amd64_webserver:v2
COPY --from=builder /root/quentin.dufour.io/_site /srv/http

View file

@ -1 +0,0 @@
sudo docker build -t superboum/amd64_blog:v19 .

View file

@ -1,8 +0,0 @@
FROM amd64/debian:buster
RUN apt-get update && \
apt-get dist-upgrade -y && \
apt-get install -y \
coturn
CMD ["/usr/bin/turnserver"]

View file

@ -1,17 +0,0 @@
## Génère l'image
```
sudo docker build -t registry.gitlab.com/superboum/ankh-morpork/amd64_coturn:v1 .
```
## Run bash dans le container
```
sudo docker run --rm -t -i registry.gitlab.com/superboum/ankh-morpork/amd64_coturn:v1 bash
sudo docker run --rm -t -i -p 3478:3478/udp -p 3479:3479/udp -p 3478:3478/tcp -p 3479:3479/tcp registry.gitlab.com/superboum/ankh-morpork/amd64_coturn:v1
```
## Used ports
- udp/tcp 3478 3479
## Publish
sudo docker push registry.gitlab.com/superboum/ankh-morpork/amd64_coturn:v1

View file

@ -1,3 +0,0 @@
```
docker build -t superboum/amd64_landing:v8 .
```

View file

@ -1,9 +0,0 @@
FROM golang:1.11.1-stretch as builder
COPY ./goStatic /goStatic
WORKDIR /goStatic
RUN CGO_ENABLED=0 go build -a -o web-server .
FROM scratch
COPY --from=builder /goStatic/web-server /
ENTRYPOINT ["/web-server"]

View file

@ -1,5 +0,0 @@
```
sudo docker build -t superboum/amd64_webserver:v3 .
sudo docker push superboum/amd64_webserver:v3
```

@ -1 +0,0 @@
Subproject commit 3f97f57aaee09a142afe3ca0f1a5d51acd856436

View file

@ -1 +0,0 @@
main

View file

@ -1,9 +0,0 @@
FROM node:13.8-buster
RUN apt-get update && \
apt-get install -y git
COPY ./main /srv/httpd
WORKDIR /srv
CMD ["/srv/httpd"]

View file

@ -1,12 +0,0 @@
FROM fedora:32
ENV LC_ALL=C.UTF-8
ENV LANG=C.UTF-8
ENV LANGUAGE=en_US.UTF-8
ENV RUBYOPT --disable-did_you_mean
RUN dnf install -y git ruby ruby-devel rubygems rubygem-bundler @development-tools redhat-rpm-config gcc-c++ zlib-devel
COPY ./main /srv/httpd
WORKDIR /srv
CMD ["/srv/httpd"]

View file

@ -1,23 +0,0 @@
# webpull
Webpull allows you to update your live website without deploying a new docker container but by simply calling an URL
You need to specify a secret token at boot:
```
WEBPULL_TOKEN=s3cr3et ./webpull
```
## Node.js version
```
go build ./main.go
sudo docker build -f ./Dockerfile.nodejs -t superboum/amd64_webpull_pug:v1 .
```
## Ruby version
```
go build ./main.go
sudo docker build -f ./Dockerfile.ruby -t superboum/amd64_webpull_ruby:v1 .
```

View file

@ -1,100 +0,0 @@
package main
import (
"fmt"
"errors"
"io"
"os/exec"
"os"
"log"
"net/http"
"strings"
)
func myexec(w io.Writer, main string, params ...string) error {
cmd := exec.Command(main, params...)
cmd.Stdout = w
cmd.Stderr = w
err := cmd.Run()
if err != nil {
fmt.Fprintf(w, "Failed to run: %s %s\n", main, strings.Join(params, " "))
}
return err
}
func update(w io.Writer) error {
fmt.Fprintf(w, "Start update...\n")
_, err := os.Stat("./.git")
if err != nil {
fmt.Fprintf(w, ".git folder does not exist, creating it...\n")
err := myexec(w, "git", "init")
if err != nil {
return err
}
}
err = myexec(w, "git", "remote", "get-url", "origin")
if err != nil {
repo, exists := os.LookupEnv("WEBPULL_REPO")
if !exists {
fmt.Fprintf(w, "You must define WEBPULL_REPO env variable...\n")
return errors.New("Missing environment variable WEBPULL_REPO")
}
fmt.Fprintf(w, "git remote is not yet set...\n")
err := myexec(w, "git", "remote", "add", "origin", repo)
if err != nil {
return err
}
}
err = myexec(w, "git", "pull", "origin", "master")
if err != nil {
fmt.Fprintf(w, "Failed to pull...\n")
return err
}
_, err = os.Stat("./.webpull")
if err != nil {
fmt.Fprintf(w, "You must create an executable file named '.webpull' at the root of your repository.\nIf you have nothing to run, just create an empty bash script...\n")
return err
}
err = myexec(w, "./.webpull")
if err != nil {
fmt.Fprintf(w, "An error occured during script execution\n")
return err
}
fmt.Fprintf(w, "Success.\n")
return nil
}
func main() {
token, exists := os.LookupEnv("WEBPULL_TOKEN")
if !exists {
log.Fatal("Environment variable 'WEBPULL_TOKEN' must be defined")
}
if update(os.Stdout) != nil {
log.Fatal("Initial 'update' failed")
}
fs := http.FileServer(http.Dir("./static"))
http.HandleFunc("/update", func(w http.ResponseWriter, r *http.Request) {
keys, ok := r.URL.Query()["token"]
if !ok || len(keys[0]) < 1 {
http.Error(w, "Missing 'token' query parameter", 401)
return
}
if keys[0] != token {
http.Error(w, "Wrong token", 401)
return
}
update(w)
})
http.Handle("/", fs)
log.Fatal(http.ListenAndServe(":8080", nil))
}

View file

@ -1,11 +0,0 @@
# Blacklist everything cleverly
*
!*/
# Whitelist some patterns
!*.sample
!*.gen
!*.sh
!.gitignore
# Whitelist specific files

View file

@ -1 +0,0 @@
postgres://username:password@hostname/dbname

View file

@ -1,2 +0,0 @@
#!/bin/bash
openssl rand -base64 32 > pgsql_pw

View file

@ -1,2 +0,0 @@
#!/bin/bash
openssl rand -base64 32 > secret_key

View file

@ -1,117 +0,0 @@
job "web_static" {
datacenters = ["dc1"]
type = "service"
constraint {
attribute = "${attr.cpu.arch}"
value = "amd64"
}
group "landing" {
network {
port "deuxfleurs_port" { to = 8080 }
}
task "server" {
driver = "docker"
config {
image = "superboum/amd64_webpull_pug:v4"
ports = [ "deuxfleurs_port" ]
}
template {
data = <<EOH
WEBPULL_REPO="https://git.deuxfleurs.fr/Deuxfleurs/site.git"
WEBPULL_TOKEN="{{ key "secrets/web/home_token" | trimSpace }}"
EOH
destination = "secrets/env"
env = true
}
resources {
memory = 200
}
service {
tags = [
"webstatic",
"traefik.enable=true",
"traefik.frontend.entryPoints=https,http",
"traefik.frontend.rule=Host:deuxfleurs.fr,www.deuxfleurs.fr,deuxfleurs.org,www.deuxfleurs.org;PathPrefix:/",
"traefik.frontend.priority=10"
# Ideally we would have a rewrite regex like this https://regex101.com/r/WHrABU/1
# See how it does not handle "http://deuxfleurs.fr/"? Not a big deal since HTTPS is redirected somewhere else
#"traefik.frontend.redirect.regex=^https?://(www\.deuxfleurs\.fr|deuxfleurs\.org|www\.deuxfleurs\.org)(.*)$",
#"traefik.frontend.redirect.replacement=https://deuxfleurs.fr/$${2}",
# Only set permanent redirect when it is guaranteed to work
#"traefik.frontend.redirect.permanent=true",
]
port = "deuxfleurs_port"
address_mode = "host"
name = "landing"
check {
type = "tcp"
port = "deuxfleurs_port"
interval = "60s"
timeout = "5s"
check_restart {
limit = 3
grace = "90s"
ignore_warnings = false
}
}
}
}
}
group "quentin" {
network {
port "quentin_port" { to = 8080 }
}
task "server" {
driver = "docker"
config {
image = "superboum/amd64_webpull_ruby:v1"
ports = [ "quentin_port" ]
}
template {
data = <<EOH
WEBPULL_REPO="https://git.deuxfleurs.fr/quentin/quentin.dufour.io.git"
WEBPULL_TOKEN="{{ key "secrets/web/quentin.dufour.io_token" | trimSpace }}"
EOH
destination = "secrets/env"
env = true
}
resources {
memory = 500
}
service {
tags = [
"webstatic",
"traefik.enable=true",
"traefik.frontend.entryPoints=https",
"traefik.frontend.rule=Host:quentin.dufour.io,www.quentin.dufour.io;PathPrefix:/"
]
port = "quentin_port"
address_mode = "host"
name = "blog-quentin"
check {
type = "tcp"
port = "quentin_port"
interval = "60s"
timeout = "5s"
check_restart {
limit = 3
grace = "90s"
ignore_warnings = false
}
}
}
}
}
}

View file

@ -35,7 +35,7 @@ job "directory" {
}
template {
data = file("../config/configuration/directory/bottin/config.json")
data = file("../config/bottin/config.json")
destination = "secrets/config.json"
}
@ -78,7 +78,7 @@ job "directory" {
}
template {
data = file("../config/configuration/directory/guichet/config.json.tpl")
data = file("../config/guichet/config.json.tpl")
destination = "secrets/config.json"
}

View file

@ -3,7 +3,7 @@ services:
mariadb:
build:
context: ./mariadb
context: ./seafile/build/mariadb
args:
VERSION: 4 # fake for now
image: superboum/amd64_mariadb:v4
@ -11,24 +11,24 @@ services:
# Instant Messaging
riot:
build:
context: ./riotweb
context: ./im/build/riotweb
args:
# https://github.com/vector-im/riot-web/releases
VERSION: 1.7.14
image: particallydone/amd64_riotweb:v18
VERSION: 1.7.16
image: superboum/amd64_riotweb:v19
synapse:
build:
context: ./matrix-synapse
context: ./im/build/matrix-synapse
args:
# https://github.com/matrix-org/synapse/releases
VERSION: 1.24.0
image: particallydone/amd64_synapse:v39
VERSION: 1.25.0
image: superboum/amd64_synapse:v40
# Email
sogo:
build:
context: ./sogo
context: ./email/build/sogo
args:
# fake for now
VERSION: 5.0.0
@ -36,7 +36,7 @@ services:
alps:
build:
context: ./alps
context: ./email/build/alps
args:
VERSION: 5cef0aaff2b8b6ee3e00b566123517e241d8cfb8
image: superboum/amd64_alps:v1
@ -44,48 +44,48 @@ services:
# VoIP
jitsi-meet:
build:
context: ./jitsi-meet
context: ./jitsi/build/jitsi-meet
args:
# https://github.com/jitsi/jitsi-meet
PREFIXV: stable/jitsi-meet_
VERSION: 4966
image: superboum/amd64_jitsi_meet:v2
VERSION: 5390
image: superboum/amd64_jitsi_meet:v3
jitsi-conference-focus:
build:
context: ./jitsi-conference-focus
context: ./jitsi/build/jitsi-conference-focus
args:
# https://github.com/jitsi/jicofo
PREFIXV: stable/jitsi-meet_
VERSION: 4966
image: superboum/amd64_jitsi_conference_focus:v5
VERSION: 5390
image: superboum/amd64_jitsi_conference_focus:v6
jitsi-videobridge:
build:
context: ./jitsi-videobridge
context: ./jitsi/build/jitsi-videobridge
args:
# https://github.com/jitsi/jitsi-videobridge
PREFIXV: stable/jitsi-meet_
VERSION: 4966
image: superboum/amd64_jitsi_videobridge:v15
VERSION: 5390
image: superboum/amd64_jitsi_videobridge:v16
jitsi-xmpp:
build:
context: ./jitsi-xmpp
context: ./jitsi/build/jitsi-xmpp
args:
VERSION: 0.11.2-1
image: superboum/amd64_jitsi_xmpp:v8
plume:
build:
context: ./plume
context: ./plume/build/plume
args:
VERSION: 0cd26dfbf4ab7be467325ed77230cf371147a98e
image: superboum/plume:v1
VERSION: 0.6.0
image: superboum/plume:v2
postfix:
build:
context: ./postfix
context: ./email/build/postfix
args:
# https://packages.debian.org/fr/buster/postfix
VERSION: 3.4.14-0+deb10u1

View file

@ -0,0 +1 @@
CMD head -c 10 /dev/urandom | base64

View file

@ -0,0 +1 @@
CONST this is a constant

View file

@ -0,0 +1,5 @@
CONST_LONG
this is a
constant
on several
lines

View file

@ -0,0 +1 @@
SERVICE_DN dummy Dummy service for testing secretmgr.py

View file

@ -0,0 +1 @@
SERVICE_PASSWORD dummy

View file

@ -0,0 +1 @@
USER Test user value

Some files were not shown because too many files have changed in this diff Show more