nixcfg/doc/telemetry.md

# generate ca and tls certs for elasticsearch cluster

start a `bash` in an elasticsearch image, such as `docker.elastic.co/elasticsearch/elasticsearch:8.2.0`: `docker run -ti docker.elastic.co/elasticsearch/elasticsearch:8.2.0 bash`

generate a ca and node certs:

```bash
./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
```

write these files in Consul at `secrets/telemetry/elasticsearch/elastic-certificates.p12` and `secrets/telemetry/elasticsearch/elastic-stack-ca.p12`

# start nomad services

```bash
nomad run telemetry-system.hcl
nomad run telemetry.hcl
```

# create elasticsearch passwords

in an elasticsearch container that was launched by Nomad:

```bash
./bin/elasticsearch-reset-password -u elastic
./bin/elasticsearch-reset-password -u kibana
```

save passwords in consul, at:

- `secrets/telemetry/elastic_passwords/kibana_system` for user `kibana_system`
- `secrets/telemetry/elastic_passwords/elastic` for user `elastic`

check kibana works, login to kibana with user `elastic`

grafana and apm-server will use the elastic user (the admin) to write data to elasticsearch