Grafana: Alerts, Provisioning & Cloud

Grafana Alerting

Grafana Unified Alerting (v8+) centralizes alert rules, contact points, and notification policies. Rules evaluate periodically and fire when conditions are met.

• Alert rule: condition on a query (e.g., error rate > 5% for 5 minutes)

• Contact point: where to send alerts — Slack, PagerDuty, email, webhook, OpsGenie

• Notification policy: routing tree — which alerts go to which contact point based on labels

• Silences: suppress alerts for a time range (maintenance windows)

• Alert states: Normal → Pending (condition met but not for long enough) → Firing → Resolved

# Alert rule (Alerting → Alert rules → New alert rule)
# Or define via Terraform / provisioning YAML

# Example conditions:
# A: query — rate(http_requests_total{status=~"5.."}[5m])
# B: query — rate(http_requests_total[5m])
# C: expression — A / B > 0.05  (error rate > 5%)
# Condition: C is above 0.05 for 5m

# Labels on the rule:
severity: critical
team: backend
env: production

Provisioning (Infrastructure as Code)

Grafana supports provisioning dashboards, data sources, and alert rules via YAML files — dashboards are read from disk on startup.

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 15s

# /etc/grafana/provisioning/dashboards/default.yaml
apiVersion: 1
providers:
  - name: Default
    type: file
    options:
      path: /var/lib/grafana/dashboards
      # Grafana reads all .json files from this directory

Docker Compose Setup

# docker-compose.yml — Grafana + Prometheus + Node Exporter
services:
  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - grafana-data:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning

  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  node-exporter:
    image: prom/node-exporter:latest
    ports: ["9100:9100"]

volumes:
  grafana-data:

Grafana Cloud

• Hosted Grafana: free tier — 10k series Prometheus, 50GB Loki logs, 50GB Tempo traces, 14-day retention

• Grafana Agent: lightweight collector that scrapes metrics and ships to Grafana Cloud

• Prometheus remote_write: push metrics from self-hosted Prometheus to Grafana Cloud

• Grafana OnCall: on-call scheduling and escalation policies (included in Cloud)

• k6: load testing tool — results visualized natively in Grafana

• Grafana Mimir: horizontally scalable Prometheus-compatible backend (open-source)

• Grafana Alloy: next-gen agent, replaces Grafana Agent — supports OTEL natively