nixcfg/doc/telemetry.md

# create elasticsearch folders on all nodes

```bash
mkdir -p /mnt/ssd/telemetry/es_data/nodes
chown 1000 /mnt/ssd/telemetry/es_data/nodes
```

# generate ca and tls certs for elasticsearch cluster

start a `bash` in an elasticsearch image, such as `docker.elastic.co/elasticsearch/elasticsearch:7.17.0`: `docker run -ti docker.elastic.co/elasticsearch/elasticsearch:7.17.0 bash`

generate a ca and node certs:

```bash
./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
```

copy `elastic-certificates.p12` to `/mnt/ssd/telemetry/es_data` in all nodes, and chown it:

```bash
chown 1000 /mnt/ssd/telemetry/es_data/elastic-certificates.p12
```

# create elasticsearch passwords

in elasticsearch container

```bash
./bin/elasticsearch-setup-passwords auto
```

save passwords in consul, at:

- `secrets/telemetry/elastic_passwords/apm_system` for user `apm_system`
- `secrets/telemetry/elastic_passwords/kibana_system` for user `kibana_system`
- `secrets/telemetry/elastic_passwords/elastic` for user `elastic`

check kibana works, login to kibana with user `elastic`

# create role and user for apm

create role `apm_writer`, give privileges:

- cluster privileges `manage_ilm`, `read_ilm`, `manage_ingest_pipelines`, `manage_index_templates`
- on index `apm-*` privileges `create_doc`, `create_index`, `view_index_metadata`, `manage`
- on index `apm-*sourcemap` privilege `read_cross_cluster`

create user `apm` with roles `apm_writer` and `apm_system`. give it a randomly generated password that you save in `secrets/telemetry/elastic_passwords/apm`

check apm data is ingested correctly (visible in kibana)

# create role and user for grafana

create role `grafana`, give privileges:

- on index `apm-*` privileges `read` and `view_index_metadata`

create user `grafana` with role `grafana`. give it a randomly generated password that you save in `secrets/telemetry/elastic_passwords/grafana`

check grafana works