Pipeline Basics: Input, Filter, Output

A Complete Pipeline Config

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "apache" {
    grok {
      match => { "message" => "%{IP:client_ip} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:status}" }
    }

    date {
      match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      target => "@timestamp"  // use the LOG's own time, not ingestion time
    }

    mutate {
      convert => { "status" => "integer" }  // grok output is always strings
    }
  }
}

output {
  elasticsearch {
    hosts => ["es1:9200", "es2:9200"]  // failover across multiple nodes
    index => "logs-apache-%{+YYYY.MM.dd}"
  }
}

Codecs

input {
  tcp {
    port => 5000
    codec => json  // parses each line as JSON automatically —
  }                // no separate grok/parsing filter needed
}

Beats vs. Logstash

Beats (Filebeat, Metricbeat) are lightweight, single-purpose shippers meant to run on many edge hosts with minimal overhead. Logstash does the heavier parsing/enrichment, typically run centrally, often receiving data forwarded from Beats agents. A typical architecture: many Filebeats → fewer centralized Logstash instances → Elasticsearch.