Grafana: Data Sources & PromQL

Connecting Data Sources

• Configuration → Data Sources → Add data source

• Prometheus: URL (e.g., http://prometheus:9090), no auth for internal; enable exemplars for trace linking

• Loki: URL (e.g., http://loki:3100) — log aggregation, pairs with Prometheus

• InfluxDB: Flux or InfluxQL query language — for time-series databases

• PostgreSQL / MySQL: direct SQL queries on relational databases

• CloudWatch: AWS metrics and logs — requires IAM role or access key

• Elasticsearch / OpenSearch: logs, traces, full-text search

• Jaeger / Tempo / Zipkin: distributed tracing backends

• TestData: built-in fake data source — great for building dashboards without real data

PromQL Essentials

PromQL (Prometheus Query Language) is used in Grafana panels when the data source is Prometheus. Understanding it is essential for building useful dashboards.

# Instant vector — current value of a metric
http_requests_total

# Filter by label
http_requests_total{job="api", status="200"}

# Range vector — values over a time window
http_requests_total[5m]

# rate() — per-second rate from a counter (use with range vector)
rate(http_requests_total{status!="200"}[5m])

# irate() — instantaneous rate (last two samples) — more responsive but noisy
irate(http_requests_total[5m])

# increase() — total increase over window (rate * duration)
increase(http_requests_total[1h])

# sum() — aggregate across labels
sum(rate(http_requests_total[5m])) by (status)

# avg, min, max, count
avg(node_cpu_seconds_total{mode="idle"}) by (instance)

# Histogram quantiles (p50, p95, p99)
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

# Arithmetic
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) * 100)

# Comparison — only return when condition is true
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes < 0.1

LogQL — Querying Loki

# Stream selector (required)
{app="nginx", env="production"}

# Filter by log content
{app="api"} |= "ERROR"
{app="api"} != "health"
{app="api"} |~ "status=5[0-9][0-9]"   # regex match

# Parse JSON logs
{app="api"} | json | level="error"

# Extract fields and aggregate
sum(rate({app="api"} | json | status="500" [5m])) by (endpoint)

# Log volume over time (for bar chart or time series)
sum(count_over_time({app="api"}[1m])) by (level)

Grafana Query Inspector

• Panel menu → Inspect → Query: see the exact query sent to the data source

• Panel menu → Inspect → Data: see raw response data in table format

• Panel menu → Inspect → Stats: query execution time, number of data points

• Use Query Inspector to debug why a panel shows "No data" or unexpected values