RDDs, SparkSession & Basic Operations

Getting Started

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName('MyApp').getOrCreate()
sc = spark.sparkContext  # underlying SparkContext, used for RDDs/broadcast/accumulator

# RDD — Spark's original low-level abstraction, still available for
# fine-grained control; DataFrames (built on top) are recommended for most work
rdd = sc.parallelize([1, 2, 3, 4, 5])
squared = rdd.map(lambda x: x ** 2)   # lazy — no computation happens yet
result = squared.collect()             # ACTION — pulls ALL results to the driver

# take(n) instead of collect() on large results — avoids pulling the
# entire distributed dataset into the driver's own (limited) memory
top_5 = squared.take(5)

Key-Value Operations

text = sc.textFile('data.txt')
words = text.flatMap(lambda line: line.split(' '))

# reduceByKey combines locally on each partition BEFORE shuffling —
# much less data moves over the network than groupByKey would require
counts = words.map(lambda w: (w, 1)).reduceByKey(lambda a, b: a + b)
print(counts.collect())

pandas Interop

import pandas as pd

spark_df = spark.createDataFrame(pandas_df)

# Same caution as collect() — pulls the FULL distributed result into
# the driver's single-machine memory as a pandas DataFrame
back_to_pandas = spark_df.filter(spark_df.status == 'active').toPandas()

Running Jobs

pyspark                       # interactive shell — spark/sc already in scope
spark-submit my_job.py        # submit to a cluster (YARN, Kubernetes, standalone)
spark-submit --master yarn --num-executors 10 my_job.py