Hadoop: YARN, Ecosystem & Modern Alternatives

YARN (Resource Manager)

YARN (Yet Another Resource Negotiator) decouples resource management from application logic. It allows running diverse workloads (MapReduce, Spark, Flink, Tez) on the same cluster.

YARN components:

ResourceManager (master)
  - Scheduler: allocates cluster resources to applications
  - ApplicationsManager: manages application lifecycle

NodeManager (on each worker node)
  - Launches and manages containers
  - Reports resource usage to ResourceManager
  - Monitors container health

ApplicationMaster (per application)
  - Runs inside a container
  - Negotiates resources from ResourceManager
  - Coordinates task execution

Container: unit of resources (CPU cores + memory)

Scheduling queues:
  FIFO Scheduler    — simple, single queue, not suitable for shared clusters
  Capacity Scheduler — multiple queues with guaranteed capacity (default)
  Fair Scheduler    — shares resources equally among all apps

# YARN commands
yarn node -list                          # list NodeManagers
yarn application -list                   # running applications
yarn application -status <app_id>
yarn application -kill <app_id>
yarn logs -applicationId <app_id>

# Queue status
yarn queue -status default

# ResourceManager web UI: http://resourcemanager:8088

Hadoop Ecosystem

Data Storage:
  HDFS          — distributed file system (core)
  HBase         — NoSQL on top of HDFS (random read/write, wide-column)
  Kudu          — columnar storage with fast analytics + random access

Data Processing:
  MapReduce     — batch processing (low-level, rarely used directly now)
  Spark         — in-memory batch/streaming (replaces MapReduce for most use cases)
  Tez           — DAG execution engine (used by Hive, Pig)
  Flink         — streaming-first, low-latency processing

SQL on Hadoop:
  Hive          — SQL interface over HDFS/ORC (batch queries)
  Impala        — fast SQL on HDFS (interactive, no Tez/MapReduce)
  Presto/Trino  — federated SQL (HDFS, S3, databases)
  Spark SQL     — SQL within Spark

Data Ingestion:
  Sqoop         — import/export between RDBMS and HDFS
  Kafka         — high-throughput event streaming
  Flume         — log/event collection
  NiFi          — data flow automation (drag-and-drop)

Orchestration:
  Oozie         — workflow scheduler for Hadoop jobs
  Airflow       — Python-based DAG scheduler (now preferred)

Serialization:
  Avro          — row-based, schema evolution, Kafka-native
  Parquet       — columnar, great for analytics (Spark/Hive preferred)
  ORC           — columnar, optimized for Hive

Hive Basics

-- Create external table (data stays in HDFS)
CREATE EXTERNAL TABLE logs (
    timestamp STRING,
    level     STRING,
    message   STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '	'
STORED AS TEXTFILE
LOCATION '/data/logs/';

-- Create ORC table (optimized format)
CREATE TABLE sales_orc (
    sale_id   INT,
    amount    DECIMAL(10,2),
    sale_date DATE
)
STORED AS ORC
TBLPROPERTIES ("orc.compress"="SNAPPY");

-- Partitioned table (critical for performance)
CREATE TABLE events (
    user_id INT,
    action  STRING
)
PARTITIONED BY (year INT, month INT, day INT)
STORED AS PARQUET;

-- Load data with partition
INSERT INTO events PARTITION (year=2024, month=3, day=15)
SELECT user_id, action FROM raw_events WHERE dt = '2024-03-15';

-- Query (partition pruning is automatic)
SELECT COUNT(*) FROM events WHERE year=2024 AND month=3;

When to Use Hadoop vs Alternatives

• Hadoop strengths: petabyte-scale data already on HDFS, existing cluster investment, tight HBase integration.

• Use Spark instead of MapReduce: 10-100x faster, same HDFS/YARN, Python/Scala/SQL APIs.

• Use cloud object storage (S3, GCS) instead of HDFS: cheaper, no NameNode bottleneck, managed service.

• Use Trino/Athena for interactive SQL: no ETL needed, query S3 directly, pay-per-query.

• Use Kafka + Flink for streaming: lower latency than Spark Streaming, exactly-once semantics.

• Managed services: AWS EMR, Google Dataproc, Azure HDInsight — skip Hadoop cluster management.

• Hadoop is mature but declining: new projects typically choose Spark + cloud storage + Airflow.