Prometheus Essentials

Prometheus is an open-source monitoring system built around a time-series database and a pull-based collection model. Instead of your services pushing metrics somewhere, Prometheus periodically scrapes an HTTP endpoint each target exposes (usually /metrics), stores the samples with timestamps, and gives you PromQL to query and alert on them. It's the de facto standard for Kubernetes and cloud-native observability, and pairs almost universally with Grafana for dashboards and Alertmanager for routing alerts.

The Data Model

Every metric is a time series identified by a metric name plus a set of key-value labels. There are four core metric types, and picking the right one matters — Prometheus doesn't enforce it, but PromQL functions assume it.

# Counter — monotonically increasing, resets only on restart (request counts, errors)
http_requests_total{method="GET",status="200",handler="/api/users"} 84213

# Gauge — goes up and down (memory usage, queue depth, active connections)
process_resident_memory_bytes 104857600
queue_depth{queue="emails"} 12

# Histogram — buckets observations into configurable ranges (request latency)
http_request_duration_seconds_bucket{le="0.1"} 24054
http_request_duration_seconds_bucket{le="0.5"} 33444
http_request_duration_seconds_bucket{le="+Inf"} 34305
http_request_duration_seconds_sum 1856.3
http_request_duration_seconds_count 34305

# Summary — like a histogram but computes quantiles client-side
http_request_duration_seconds{quantile="0.99"} 0.42

Histograms are almost always the better choice over summaries: summaries compute fixed quantiles on the client and can't be aggregated across instances, while histogram buckets can be summed across every pod and then have quantiles computed server-side with histogram_quantile().

Scraping & Exporters

Prometheus needs targets to scrape, configured in prometheus.yml. For apps that don't natively expose Prometheus metrics (databases, hardware, third-party services), you run an exporter — a small process that translates that system's native metrics into the Prometheus text format.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: "api-service"
    metrics_path: /metrics
    static_configs:
      - targets: ["api-1:8080", "api-2:8080"]

  - job_name: "node-exporter"
    static_configs:
      - targets: ["node1:9100", "node2:9100"]

  - job_name: "postgres-exporter"
    static_configs:
      - targets: ["postgres-exporter:9187"]

  # Kubernetes service discovery — no static target list to maintain
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: "${1}"

In Kubernetes, kubernetes_sd_configs is what makes Prometheus scale — it queries the Kubernetes API to discover pods/services/endpoints dynamically, and relabel_configs rewrites label metadata (like using an annotation to opt a pod into scraping) before the scrape even happens.

PromQL

PromQL is Prometheus's query language. The single most important habit: never graph a raw counter directly — always wrap it in rate() or irate() over a range first, since a counter's raw value is just an ever-growing number that resets on restart.

# Per-second rate of requests over the last 5 minutes
rate(http_requests_total[5m])

# Error rate as a percentage, aggregated across instances
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m])) * 100

# p95 latency from histogram buckets
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# Group by label and sort
topk(5, sum(rate(http_requests_total[5m])) by (handler))

# Compare current value to 1 hour ago
process_resident_memory_bytes - process_resident_memory_bytes offset 1h

# Predict when a disk will fill up based on recent trend
predict_linear(node_filesystem_free_bytes[1h], 4 * 3600) < 0

`sum by (le)` before `histogram_quantile` is the pattern to memorize — you must aggregate across instances before computing the quantile, not after, or the math is wrong.

Alerting Rules

Alerting rules live in rule files and are evaluated on evaluation_interval. When an expression is true, the alert enters pending state; if it stays true for for: duration, it becomes firing and gets pushed to Alertmanager. Prometheus itself never sends notifications — that's Alertmanager's job.

groups:
  - name: api-alerts
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
          sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.job }}"
          description: "Error rate is {{ $value | humanizePercentage }} for the last 5 minutes."

      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} is down"
          description: "Prometheus has failed to scrape {{ $labels.instance }} for 2 minutes."

`for:` exists to absorb noise — without it, a single blip that self-resolves in a few seconds would fire and resolve an alert, paging someone for nothing. The built-in `up` metric (1 if the last scrape succeeded, 0 otherwise) is the simplest and most reliable way to detect a dead target.

Practical Tips & Gotchas

• Never put unbounded values (user IDs, request IDs, raw URLs) in a label — it causes cardinality explosions that can crash Prometheus's memory usage.

• Local on-disk storage is not built for long-term retention or HA — for that, pair Prometheus with remote-write to Thanos, Cortex, or Mimir.

• `rate()` needs at least two samples in the window — if your `scrape_interval` is 15s, don't compute `rate(x[10s])`; the range must cover several scrape intervals.

• Counters can reset (pod restart, process crash) — `rate()` and `increase()` handle resets correctly by detecting the drop; raw subtraction between two points does not.

• The default retention is 15 days — plan capacity and remote storage before you need six months of history for a postmortem.

• Use recording rules to precompute expensive, frequently-dashboarded queries — it keeps Grafana dashboards fast and avoids re-running heavy aggregations on every page load.