Dask Parallel and Distributed Computing
Scale pandas/NumPy workflows beyond memory and across clusters.
When to Use
- Datasets exceed available RAM
- Need to parallelize pandas or NumPy operations
- Processing multiple files efficiently (CSVs, Parquet)
- Building custom parallel workflows
- Distributing workloads across multiple cores/machines
Dask Collections
| Collection |
Like |
Use Case |
| DataFrame |
pandas |
Tabular data, CSV/Parquet |
| Array |
NumPy |
Numerical arrays, matrices |
| Bag |
list |
Unstructured data, JSON logs |
| Delayed |
Custom |
Arbitrary Python functions |
Key concept: All collections are lazy—computation happens only when you call .compute().
Lazy Evaluation
| Function |
Behavior |
Use |
dd.read_csv() |
Lazy load |
Large CSVs |
dd.read_parquet() |
Lazy load |
Large Parquet |
| Operations |
Build graph |
Chain transforms |
.compute() |
Execute |
Get final result |
Key concept: Dask builds a task graph of operations, optimizes it, then executes in parallel. Call .compute() once at the end, not after every operation.
Schedulers
| Scheduler |
Best For |
Start |
| threaded |
NumPy/Pandas (releases GIL) |
Default |
| processes |
Pure Python (GIL bound) |
scheduler='processes' |
| synchronous |
Debugging |
scheduler='synchronous' |
| distributed |
Monitoring, scaling, clusters |
Client() |
Distributed Scheduler
| Feature |
Benefit |
| Dashboard |
Real-time progress monitoring |
| Cluster scaling |
Add/remove workers |
| Fault tolerance |
Retry failed tasks |
| Worker resources |
Memory management |
Chunking Concepts
DataFrame Partitions
| Concept |
Description |
| Partition |
Subset of rows (like a mini DataFrame) |
| npartitions |
Number of partitions |
| divisions |
Index boundaries between partitions |
Array Chunks
| Concept |
Description |
| Chunk |
Subset of array (n-dimensional block) |
| chunks |
Tuple of chunk sizes per dimension |
| Optimal size |
~100 MB per chunk |
Key concept: Chunk size is critical. Too small = scheduling overhead. Too large = memory issues. Target ~100 MB.
DataFrame Operations
Supported (parallel)
| Category |
Operations |
| Selection |
filter, loc, column selection |
| Aggregation |
groupby, sum, mean, count |
| Transforms |
apply (row-wise), map_partitions |
| Joins |
merge, join (shuffles data) |
| I/O |
read_csv, read_parquet, to_parquet |
Avoid or Use Carefully
| Operation |
Issue |
Alternative |
iterrows |
Kills parallelism |
map_partitions |
apply(axis=1) |
Slow |
map_partitions |
Repeated compute() |
Inefficient |
Single compute() at end |
sort_values |
Expensive shuffle |
Avoid if possible |
Common Patterns
ETL Pipeline
scan_* or read_* (lazy load)- Chain filters and transforms
- Single
.compute() or .to_parquet()
Multi-File Processing
| Pattern |
Description |
| Glob patterns |
dd.read_csv('data/*.csv') |
| Partition per file |
Natural parallelism |
| Output partitioned |
to_parquet('output/') |
Custom Operations
| Method |
Use Case |
map_partitions |
Apply function to each partition |
map_blocks |
Apply function to each array block |
delayed |
Wrap arbitrary Python functions |
Best Practices
| Practice |
Why |
| Don't load locally first |
Let Dask handle loading |
| Single compute() at end |
Avoid redundant computation |
| Use Parquet |
Faster than CSV, columnar |
| Match partition to files |
One partition per file |
| Check task graph size |
len(ddf.__dask_graph__()) < 100k |
| Use distributed for debugging |
Dashboard shows progress |
Common Pitfalls
| Pitfall |
Solution |
| Loading with pandas first |
Use dd.read_* directly |
| compute() in loops |
Collect all, single compute() |
| Too many partitions |
Repartition to ~100 MB each |
| Memory errors |
Reduce chunk size, add workers |
| Slow shuffles |
Avoid sorts/joins when possible |
vs Alternatives
| Tool |
Best For |
Trade-off |
| Dask |
Scale pandas/NumPy, clusters |
Setup complexity |
| Polars |
Fast in-memory |
Must fit in RAM |
| Vaex |
Out-of-core single machine |
Limited operations |
| Spark |
Enterprise, SQL-heavy |
Infrastructure |
Resources