Reliability, Performance & Multi-Pipeline

Persistent Queue & Dead Letter Queue

# logstash.yml
queue.type: persisted           # buffers to disk between input and filter/output —
queue.max_bytes: 4gb            # without it, a crash loses events still in memory

dead_letter_queue.enable: true  # events that fail processing (e.g. an Elasticsearch
                                 # mapping conflict) land here instead of vanishing

Multiple Pipelines

# pipelines.yml — independently tuned, scaled, and restartable
- pipeline.id: apache-logs
  path.config: "/etc/logstash/conf.d/apache.conf"
  pipeline.workers: 4

- pipeline.id: app-metrics
  path.config: "/etc/logstash/conf.d/metrics.conf"
  pipeline.workers: 2
  pipeline.batch.size: 500

# A spike or failure processing app-metrics doesn't require restarting
# (or even touching) the apache-logs pipeline.

Performance Tuning

pipeline.workers controls parallel filter/output threads (usually defaults to CPU core count); pipeline.batch.size controls how many events each worker groups together before processing — a larger batch improves throughput at the cost of slightly higher per-event latency. Watch for catastrophic-backtracking-prone grok patterns (ambiguous, overlapping alternatives) under high throughput — test patterns against real sample lines with the Grok Debugger before deploying.

Type Consistency

Elasticsearch locks a field's type from the first document establishing the index mapping. Grok always extracts strings — sending a raw grok-extracted "status" field straight to the output can later conflict if another source sends the same field name as an actual number. Normalize types with mutate { convert => {...} } in the filter stage before output, rather than relying on the Dead Letter Queue to catch it after the fact.