CSE57 · DataOps & MLOps

Spark Batch Processing

Aggregate a large synthetic sales file with Spark: partitions, shuffles and the cost of a bad join.

📶 L300⏱ 40 min📘 Guided

🎯 You will learn to

Know partitions
Avoid big shuffles
Read a job plan

📘 Guided lab

This mission is a guided walkthrough: follow the steps on your own computer or practice account. The in-browser simulator for it is planned for a later release.

✅ Steps & checkpoints

Load the file
Group by state
Join with a small table
Broadcast the small table

🗺️ Spark job stages

The driver splits work into tasks over partitions; a group-by shuffles data between stages, and a skewed key slows one task.

The 2D diagram shows the exact flow, ports and trust boundaries. The 3D scene is a concept view of the same parts; packets travel along the same arrows.

🤔 Think about it

Why is a broadcast join faster for a small table?

Your answer (kept only on this device)

🛡️ Safety & cost

Synthetic data only. No personal data is used.

🧹 Cleanup

Nothing to clean up.

📚 Reference documents

Official sources, checked on 5 Nov 2026. Cloud services change often; check the official page before relying on a feature. All references