nixcfg/doc/telemetry.md

create elasticsearch folders on all nodes

mkdir -p /mnt/ssd/telemetry/es_data/nodes
chown 1000 /mnt/ssd/telemetry/es_data/nodes

generate ca and tls certs for elasticsearch cluster

start a bash in an elasticsearch image, such as docker.elastic.co/elasticsearch/elasticsearch:7.17.0: docker run -ti docker.elastic.co/elasticsearch/elasticsearch:7.17.0 bash

generate a ca and node certs:

./bin/elasticsearch-certutil ca
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

copy elastic-certificates.p12 to /mnt/ssd/telemetry/es_data in all nodes, and chown it:

chown 1000 /mnt/ssd/telemetry/es_data/elastic-certificates.p12

create elasticsearch passwords

in elasticsearch container

./bin/elasticsearch-setup-passwords auto

save passwords in consul, at:

• secrets/telemetry/elastic_passwords/apm_system for user apm_system
• secrets/telemetry/elastic_passwords/kibana_system for user kibana_system
• secrets/telemetry/elastic_passwords/elastic for user elastic

check kibana works, login to kibana with user elastic

create role and user for apm

create role apm_writer, give privileges:

• cluster privileges manage_ilm, read_ilm, manage_ingest_pipelines, manage_index_templates
• on index apm-* privileges create_doc, create_index, view_index_metadata, manage
• on index apm-*sourcemap privilege read_cross_cluster

create user apm with roles apm_writer and apm_system. give it a randomly generated password that you save in secrets/telemetry/elastic_passwords/apm

check apm data is ingested correctly (visible in kibana)

create role and user for grafana

create role grafana, give privileges:

• on index apm-* privileges read and view_index_metadata

create user grafana with role grafana. give it a randomly generated password that you save in secrets/telemetry/elastic_passwords/grafana

check grafana works