Deuxfleurs
/
infrastructure
Archived
24
2
Fork 2
Code Issues 22 Pull Requests Projects 1 Releases Wiki Activity

Update guide

This commit is contained in:
Quentin 2022-01-28 17:00:50 +01:00
parent 2775eeb0fe
commit 89937f2107
1 changed files with 231 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

234
op_guide/stolon/manual_backup.md
View File

@ -27,25 +27,253 @@ First setup a SSH tunnel on your machine that bind postgresql, eg:
ssh -L 5432:psql-proxy.service.2.cluster.deuxfleurs.fr:5432 ...
```
*Later, we will use `/tmp/sql` as our working directory. Depending on your distribution, this
folder may be a `tmpfs` and thus mounted on RAM. If it is the case, choose another folder, that is not a `tmpfs`, otherwise you will fill your RAM
and fail your backup. I am using NixOS and the `/tmp` folder is a regular folder, persisted on disk, which explain why I am using it.*
Then export your password in `PGPASSWORD` and launch the backup:
```bash
export PGPASSWORD=xxx
mkdir -p /tmp/sql
cd /tmp/sql
pg_basebackup \
--host=127.0.0.1 \
--username=replicator \
--pgdata=/tmp/sql \
--format=tar \
--wal-method=none \
--wal-method=stream \
--gzip \
--compress=6 \
--progress \
--max-rate=2M
--max-rate=5M
```
*Something you should now: while it seems optional, fetching the WAL is mandatory. At first, I thought it was a way to have a "more recent backup".
But after some reading, it appears that the base backup is corrupted because it is not a snapshot at all, but a copy of the postgres folder with no specific state.
The whole point of the WAL is, in fact, to fix this corrupted archive...*
*Take a cup of coffe, it will take some times...*
The result I get (the important file is `base.tar.gz`, `41921.tar.gz` will probably be missing as it is a secondary tablespace I will deactivate soon):
```
[nix-shell:/tmp/sql]$ ls
41921.tar.gz backup_manifest base.tar.gz pg_wal.tar.gz
```
From now, disconnect from the production to continue your work.
You don't need it anymore and it will prevent some disaster if you fail a command.
## Importing the backup
## Dump SQL
> The backup taken with `pg_basebckup` is an exact copy of your data directory so, all you need to do to restore from that backup is to point postgres at that directory and start it up.
```bash
mkdir -p /tmp/sql/pg_data && cd /tmp/sql/pg_data
tar xzfv ../base.tar.gz
```
Now you should have something like that:
```
[nix-shell:/tmp/sql/pg_data]$ ls
backup_label base pg_commit_ts pg_hba.conf pg_logical pg_notify pg_serial pg_stat pg_subtrans pg_twophase pg_wal postgresql.conf tablespace_map
backup_label.old global pg_dynshmem pg_ident.conf pg_multixact pg_replslot pg_snapshots pg_stat_tmp pg_tblspc PG_VERSION pg_xact stolon-temp-postgresql.conf
```
Now we will extract the WAL:
```bash
mkdir -p /tmp/sql/wal && cd /tmp/sql/wal
tar xzfv ../pg_wal.tar.gz
```
You should have something like that:
```
[nix-shell:/tmp/sql/wal]$ ls
00000003000014AF000000C9 00000003000014AF000000CA 00000003.history archive_status
```
Now, We must edit/read some files before launching our ephemeral server:
- Set `listen_addresses = '0.0.0.0'` in `postgresql.conf`
- Add `restore_command = 'cp /mnt/wal/%f %p' ` in `postgresql.conf`
- Check `port` in `postgresql.conf`, in our case it is `5433`.
- Create an empty file named `recovery.signal`
*Do not create a `recovery.conf` file, it might be written on the internet but this is a deprecated method and your postgres daemon will refuse to boot if it finds one.*
*Currently, we use port 5433 in oour postgresql configuration despite 5432 being the default port. Indeed, in production, clients access the cluster transparently through the Stolon Proxy that listens on port 5432 and redirect the requests to the correct PostgreSQL instance, listening secretly on port 5433! To export our binary backup in text, we will directly query our postgres instance without passing through the proxy, which is why you must note this port.*
Now we will start our postgres container on our machine.
At the time of writing the live version is `superboum/amd64_postgres:v9`.
We must start by getting `postgres` user id. Our container are run by default with this user, so you only need to run:
```bash
docker run --rm -it superboum/amd64_postgres:v9 id
```
And we get:
```
uid=999(postgres) gid=999(postgres) groups=999(postgres),101(ssl-cert)
```
Now `chown` your `pg_data`:
```bash
chown 999:999 -R /tmp/sql/{pg_data,wal}
chmod 700 -R /tmp/sql/{pg_data,wal}
```
And finally:
```
docker run \
--rm \
-it \
-p 5433:5433 \
-v /tmp/sql/:/mnt/ \
superboum/amd64_postgres:v9 \
postgres -D /mnt/pg_data
```
I have the following output:
```
2022-01-28 14:46:39.750 GMT [1] LOG: skipping missing configuration file "/mnt/pg_data/postgresql.auto.conf"
2022-01-28 14:46:39.763 UTC [1] LOG: starting PostgreSQL 13.3 (Debian 13.3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2022-01-28 14:46:39.764 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5433
2022-01-28 14:46:39.767 UTC [1] LOG: listening on Unix socket "/tmp/.s.PGSQL.5433"
2022-01-28 14:46:39.773 UTC [7] LOG: database system was interrupted; last known up at 2022-01-28 14:33:13 UTC
cp: cannot stat '/mnt/wal/00000004.history': No such file or directory
2022-01-28 14:46:40.318 UTC [7] LOG: starting archive recovery
2022-01-28 14:46:40.321 UTC [7] LOG: restored log file "00000003.history" from archive
2022-01-28 14:46:40.336 UTC [7] LOG: restored log file "00000003000014AF000000C9" from archive
2022-01-28 14:46:41.426 UTC [7] LOG: could not open directory "pg_tblspc/41921/PG_13_202007201": No such file or directory
2022-01-28 14:46:41.445 UTC [7] LOG: could not open directory "pg_tblspc/41921/PG_13_202007201": No such file or directory
2022-01-28 14:46:41.457 UTC [7] LOG: redo starts at 14AF/C9000028
2022-01-28 14:46:41.500 UTC [7] LOG: restored log file "00000003000014AF000000CA" from archive
2022-01-28 14:46:42.461 UTC [7] LOG: consistent recovery state reached at 14AF/CA369AB0
2022-01-28 14:46:42.461 UTC [1] LOG: database system is ready to accept read only connections
cp: cannot stat '/mnt/wal/00000003000014AF000000CB': No such file or directory
2022-01-28 14:46:42.463 UTC [7] LOG: redo done at 14AF/CA369AB0
2022-01-28 14:46:42.463 UTC [7] LOG: last completed transaction was at log time 2022-01-28 14:35:04.698438+00
2022-01-28 14:46:42.480 UTC [7] LOG: could not open directory "pg_tblspc/41921/PG_13_202007201": No such file or directory
2022-01-28 14:46:42.493 UTC [7] LOG: restored log file "00000003000014AF000000CA" from archive
cp: cannot stat '/mnt/wal/00000004.history': No such file or directory
2022-01-28 14:46:43.462 UTC [7] LOG: selected new timeline ID: 4
2022-01-28 14:46:44.441 UTC [7] LOG: archive recovery complete
2022-01-28 14:46:44.444 UTC [7] LOG: restored log file "00000003.history" from archive
2022-01-28 14:46:45.614 UTC [1] LOG: database system is ready to accept connections
```
*Notes: the missing tablespace is a legacy tablesplace used in the past to debug Matrix. It will be removed soon, we can safely ignore it. Other errors on cp seems to be intended as postgres might want to know how far it can rewind with the WAL but I a not 100% sure.*
Your ephemeral instance should work:
```bash
export PGPASSWORD=xxx # your postgres (admin) account password
psql -h 127.0.0.1 -p 5433 -U postgres postgres
```
And your databases should appear:
```
[nix-shell:~/Documents/dev/infrastructure]$ psql -h 127.0.0.1 -p 5433 -U postgres postgres
psql (13.5, server 13.3 (Debian 13.3-1.pgdg100+1))
Type "help" for help.
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+------------+------------+-----------------------
xxxxx | xxxxx | UTF8 | en_US.utf8 | en_US.utf8 |
xxxxx | xxxxx | UTF8 | en_US.utf8 | en_US.utf8 |
xxxxx | xxxxx | UTF8 | en_US.utf8 | en_US.utf8 |
postgres | postgres | UTF8 | en_US.utf8 | en_US.utf8 |
xxxx | xxxxx | UTF8 | en_US.utf8 | en_US.utf8 |
xxxx | xxxxx | UTF8 | en_US.utf8 | en_US.utf8 |
template0 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | en_US.utf8 | en_US.utf8 | =c/postgres +
| | | | | postgres=CTc/postgres
(8 rows)
```
## Dump your ephemeral database as SQL
Now we can do a SQL export of our ephemeral database.
We use zstd to automatically compress the outputed file.
We use multiple parameters:
- `-vv` gives use some idea on the progress
- `-9` is a quite high value and should compress efficiently. Decrease it if your machine is low powered
- `-T0` asks zstd to use all your cores. By default, zstd uses only one core.
```bash
pg_dumpall -h 127.0.0.1 -p 5433 -U postgres \
| zstd -vv -9 -T0 --format=zstd > dump-`date --rfc-3339=seconds | sed 's/ /T/'`.sql.zstd
```
I get the following result:
```
[nix-shell:/tmp/sql]$ ls -lah dump*
-rw-r--r-- 1 quentin users 749M janv. 28 16:07 dump-2022-01-28T16:06:29+01:00.sql.zstd
```
Now you can stop your ephemeral server.
## Restore your SQL file
First, start a blank server:
```bash
docker run \
--rm -it \
--name postgres \
-p 5433:5432 \
superboum/amd64_postgres:v9 \
bash -c '
set -ex
mkdir /tmp/psql
initdb -D /tmp/psql --no-locale --encoding=UTF8
echo "host all postgres 0.0.0.0/0 md5" >> /tmp/psql/pg_hba.conf
postgres -D /tmp/psql
'
```
Then set the same password as your prod for the `posgtgres` user (it will be required as part of the restore):
```bash
docker exec -ti postgres bash -c "echo \"ALTER USER postgres WITH PASSWORD '$PGPASSWORD';\" | psql"
echo '\l' | psql -h 127.0.0.1 -p 5433 -U postgres postgres
# the database should have no entry (except `posgtres`, `template0` and `template1`) otherwise ABORT EVERYTHING, YOU ARE ON THE WRONG DB
```
And finally, restore your SQL backup:
```bash
zstdcat -vv dump-* | \
grep -P -v '^(CREATE|DROP) ROLE postgres;' | \
psql -h 127.0.0.1 -p 5433 -U postgres --set ON_ERROR_STOP=on postgres
```
*Note: we must skip CREATE/DROP ROLE postgres during the restore as it aready exists and would generate an error.
Because we want to be extra careful, we specifically asked to crash on every error and do not want to change this behavior.
So, instead, we simply remove any entry that contains the specific regex stated in the previous command.*
## Ref
https://philipmcclarence.com/backing-up-and-restoring-postgres-using-pg_basebackup/
https://www.cybertec-postgresql.com/en/pg_basebackup-creating-self-sufficient-backups/
https://www.postgresql.org/docs/14/continuous-archiving.html
https://www.postgresql.org/docs/14/backup-dump.html#BACKUP-DUMP-RESTORE
https://dba.stackexchange.com/questions/75033/how-to-restore-everything-including-postgres-role-from-pg-dumpall-backup