Dask: Schedulers, Distributed Execution & Tuning

Local vs Distributed Scheduler

# Default local schedulers -- threads/processes on ONE machine,
# good for larger-than-memory-but-fits-on-disk or multi-core CPU work
result = ddf.compute()  # uses local threaded scheduler by default

# dask.distributed -- Client/scheduler/workers architecture, can span
# an actual multi-machine cluster; commonly used even on a single
# machine for its richer diagnostics dashboard
from dask.distributed import Client
client = Client()  # starts a local cluster + web dashboard

# Dashboard shows: live task progress, per-worker CPU/memory,
# the task graph itself -- diagnosing bottlenecks, not just waiting blind

compute() vs persist()

# compute() -- brings the full result back as ONE local, concrete object
local_df = ddf.compute()  # a real pandas DataFrame, must fit in memory

# persist() -- triggers computation but keeps the result DISTRIBUTED
# in cluster memory, still lazily-referenceable for further chained ops.
# Useful for an expensive intermediate result reused across several
# subsequent computations -- computed once, not recomputed each time.
expensive = ddf[ddf.amount > 100].persist()
result_a = expensive.groupby('category').sum().compute()
result_b = expensive.groupby('region').mean().compute()

Shuffles: A Real Performance Cliff

• Per-partition operations (filter, per-partition column math) parallelize cheaply -- each partition's own data, no coordination needed.

• Operations requiring data reorganization ACROSS partitions (a full sort, some groupby-aggregate patterns) require actual data movement between workers -- a shuffle.

• Explains why some seemingly simple operations are much slower than others that look similar in code.

Dask-ML & Choosing Dask vs Spark

• Dask-ML scales specific bottleneck operations (parallel hyperparameter search, out-of-core fitting for partial_fit-compatible algorithms) -- complements scikit-learn, doesn't reimplement it.

• Dask: Python-native, closely mirrors pandas/NumPy -- gentler learning curve for a PyData-fluent team.

• Spark: JVM-based (PySpark for Python), larger/more mature ecosystem -- often the default at very large enterprise scale, especially with existing Hadoop/Spark infrastructure.