Part 6

Part 6 turns the transformations learned in part 5 into persistent file-backed pipelines. The repeated stages are ingest, validate, transform, summarize and present: read typed records, join and check them, preview results, and only then write output.

Part 7 will keep the same stages while replacing selected hand-written operations with supported dataframe and analytical-library calls.