🏭 DataOps & MLOps
Pipelines with quality contracts, medallion layers, late and duplicate events, lineage and cost.
6 missions🧪 3 live
CSE56ETL Data Quality ChallengeRaw exam results arrive with duplicates, blanks and bad marks. Clean them layer by layer.
CSE57Spark Batch ProcessingAggregate a large synthetic sales file with Spark: partitions, shuffles and the cost of a bad join.
DO01ETL with Schema & Quality ContractsA data contract says which columns, types and ranges are allowed. Enforce it from bronze to gold.
DO02Spark Partitioning & CostChoose partition columns, fix data skew and compare job time and cost.
DO03dbt Models, Tests & LineageBuild staging and mart models with dbt, add tests and read the lineage graph.
DO04CDC, Late Events, Dedupe & IdempotencyChange events arrive twice and out of order. Make the pipeline give the same answer every time.
🗺️ DataOps medallion pipeline
Raw data lands in bronze, quality rules clean it into silver (bad rows quarantined) and business tables are built in gold; lineage tracks every step.
Drag to turn, scroll to zoom, tap a block to see what it is.
The 2D diagram shows the exact flow, ports and trust boundaries. The 3D scene is a concept view of the same parts; packets travel along the same arrows.
📚 Reference documents
- Apache Spark documentation · Apache
- dbt documentation · dbt Labs
- OpenLineage documentation · OpenLineage