Master Prometheus Monitoring System: A Crash Course

Table of Contents

1. Introduction
2. Prometheus Architecture
3. Metric Types in Prometheus
   • Counter
   • Gauge
   • Histogram
   • Summary
4. PromQL
5. Service Discovery in Prometheus
   • Static Configs
   • Dynamic Discovery
   • Kubernetes Service Discovery
6. Exporters in Prometheus
7. Client Libraries in Prometheus
8. Visualization of Metrics in Prometheus
9. Alerting in Prometheus
   • Alertmanager
   • DeadMansSnitch
10. Limitations of Prometheus
11. Thanos
12. Conclusion

Prometheus: An Open-Source Monitoring System

Prometheus is a popular open-source monitoring system that collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts when specified conditions are observed. Unlike traditional push-Based approaches such as Graphite, Prometheus uses a pull-based model. It scrapes HTTP endpoints that are exposed by the application.

Prometheus Architecture

Prometheus offers four Core metric types: Counter, Gauge, Histogram, and Summary. A Counter's value can only be increased or be reset to zero on restart. You can use a counter to represent the number of requests served, tasks completed, or errors. A Gauge represents a single numerical value that can arbitrarily go up and down. Gauges are typically used for measured values like temperatures or Current memory usage. A Histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. For example, you can measure the latency of HTTP requests. Finally, a Summary samples observations (usually things like request durations and response sizes). In addition, Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets you select and aggregate time-series data in real-time. For example, this query http_requests_total{job="service-a", handler="/api/login"} returns all-time series with the metric http_requests_total and the given job and handler labels.

Metric Types in Prometheus

Prometheus offers four core metric types: Counter, Gauge, Histogram, and Summary. A Counter's value can only be increased or be reset to zero on restart. You can use a counter to represent the number of requests served, tasks completed, or errors. A Gauge represents a single numerical value that can arbitrarily go up and down. Gauges are typically used for measured values like temperatures or current memory usage. A Histogram samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. For example, you can measure the latency of HTTP requests. Finally, a Summary samples observations (usually things like request durations and response sizes).

PromQL

Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets you select and aggregate time-series data in real-time. For example, this query http_requests_total{job="service-a", handler="/api/login"} returns all-time series with the metric http_requests_total and the given job and handler labels.

Service Discovery in Prometheus

To configure targets, there are multiple service discovery mechanisms built into Prometheus. The simplest one is static_configs. You can add hostnames or IP addresses to instruct Prometheus to scrape them. Also, you have an option for dynamic discovery. Suppose you host your infrastructure in one of the public clouds, such as AWS or GCP. Prometheus can dynamically discover targets based on the tags and labels you add to your virtual machines. In AWS, you can use ec2_sd_configs. This job will select all the EC2 instances with the tag prometheus-node-exporter equal to true. It is way more convenient and flexible than specifying targets manually. And, of course, you have Kubernetes service discovery that is based on labels. If you are using the Prometheus operator, you can Create a service monitor object such as this.

Exporters in Prometheus

Prometheus community built a lot of exporters that you can use to convert application-specific metrics to the Prometheus format. For example, you can use Postgress exporter or node-exporter to extract CPU memory and network metrics from the server and expose it to Prometheus.

Client Libraries in Prometheus

Prometheus also provides client libraries that you can use in your application to expose metrics in Prometheus format. You have a wide variety of libraries for many languages, including golang, python, java, and others.

Visualization of Metrics in Prometheus

Prometheus can Visualize metrics using the native interface; however, it's very common to use it with Grafana. You would need to add Prometheus as a data source, and then you can build dashboards using Prometheus metrics.

Alerting in Prometheus

To send alerts, you can use Alertmanager. First, you create a rule that will be evaluated by the Prometheus. If the condition is met, Prometheus will notify Alertmanager to send the alert. For example, this rule will be triggered when the CPU usage of the nginx server exceeds 80 percent for 15 minutes. Alertmanager has a variety of integrations. It can send alerts as slack messages. You can send emails, or you can integrate with Pagerduty.

DeadMansSnitch

To keep your Prometheus healthy, you need a way to monitor your monitoring system. One of the ways is to use DeadMansSnitch. The way it works, you create an alert that is always active. It sends alerts to the DeadMansSnitch service every 15 minutes. In case your Alertmanager misses one of the alerts, DeadMansSnitch will notify you that something is broken in your monitoring system. It's a paid but very cheap service that can also monitor your cron jobs.

Limitations of Prometheus

While Prometheus is a solid monitoring solution, it has a few limitations. For example, if you want to deploy a highly available Prometheus setup, you would use Thanos. Also, it allows you to integrate multiple environments in a single place. Instead of going to different Grafana dashboards, you get access to all your services from a single Grafana.

Thanos

Thanos is built on top of the Prometheus, so it's relatively easy to upgrade your existing Prometheus deployments to Thanos. If you want to learn more about how to deploy and manage Prometheus in Kubernetes or VMs, I have other tutorials that can help you to get started.

Conclusion

Prometheus is a powerful open-source monitoring system that offers a wide range of features and integrations. With its pull-based model, functional query language, and support for multiple metric types, it's a popular choice for monitoring modern applications. Whether you're using it with Grafana or Alertmanager, or integrating it with other tools like Thanos, Prometheus is a versatile and reliable solution for monitoring your infrastructure.