Prometheus
As explained by TechWorld with Nana Prometheus Architecture explained and https://medium.com/techno101/how-to-send-a-mail-using-prometheus-alertmanager-7e880a3676db
Prometheus Server
Time Series Database (TSDB)
Data Retrieval Worker
Web Server/API/UI
Metrics
HELP {Describing text }
TYPE {Counter, gauge, histogram}
The data retrieval worker pulls /metrics
e.g. http://host.name:9090/metrics
.
Exporter
Exports metrics from target to own /metrics
endpoint.
E.g. exporters for
databases (mysql, postgresql)
linux server
build tools
List of prometheus official exporters
The
node_exporter
(e.g.apt install prometheus-node-exporter
) exports all linux system related information.The
blackbox_exporter
can probe remote locations for reachability.Custom exporters can be written with client libs .
Configuration
Rules and endpoints can be configured in
prometheus.yaml
scrape_interval
,evaluation_interval
can be configure globally and per jobRules are in
$filename.rules
files and can be included
Alertmanager
Is queried using PromQL
We can use this to setup alerts
Configuration
alert rules are configured in alert.rules
./promtool check rules alert.rules
⚠️ Prometheus itself does not send the actual alert messages to users, this is the responsibility of the Alertmanager (deployed independently). When an alert reaches the Firing state, Prometheus notifies the Alertmanager, which in turn will route the alert to the right channel (e.g. Slack, Pagerduty…) and the right people.
https://devconnected.com/monitoring-disk-i-o-on-linux-with-the-node-exporter/