AI curriculum

Course outline for Data Engineering Foundations for AI Systems

A retrieval or evaluation dataset can look valid while omitting records, exposing personal data, or leaking evaluation examples into training data. Build a small source-qualified dataset pipeline with reconciliation, deduplication, and versioned transformations. Test incremental updates and split isolation before trusting downstream AI results.

Technology: Data Engineering and Databases
Duration: 6 hours
Modules: 3 · 2 hours each
Level: Level 1

About Data Engineering Foundations for AI Systems

Data Engineering Foundations for AI Systems Course Objectives

Compare ETL and ELT boundaries for preparing AI datasets.
Implement a repeatable batch transform with source keys and sensitive-field exclusions.
Reconcile completeness and detect duplicates independently of schema checks.
Record source lineage, transformation versions, and dataset splits.
Detect overlap between supplied training and evaluation fixtures.
Test reruns, late arrivals, and a downstream schema-change impact.

Pre-requisites

Basic scripting and generic SQL skills.
Familiarity with tabular data, files, and simple record transformations; a focused bridge can refresh these skills.

Before this course

Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.

No prerequisite course is required. The entry skills above still apply.

Recommended preparation

Readiness checks

Generic SQL and tabular-data handling
Basic scripting with files and records

Can query and transform tabular data; supply a focused SQL/scripting bridge if needed. PostgreSQL optimization, functions, and outbox mechanics are not gates.

Find your place in the learning map

Lab Setup

Scripting runtime, local SQL database, pipeline/checkpoint starter, and supplied split/deduplication helpers.
Synthetic support records, dummy sensitive fields, source control totals, and duplicate, split-overlap, late-arrival, and schema-change fixtures.
Use local scripts and simulated streams; no live model, model training, embedding generation, hosted scheduler, or real personal data is needed.

Detailed Course Outline

Module 1: Define AI dataset preparation boundaries

Data pipelines: Define source keys and retrieval/evaluation output contracts; track accepted and rejected records before downstream AI use rather than trusting schema validity alone.
ETL: Place extraction, validation, and transformation before target loading, identifying the boundary where malformed source rows are rejected.
ELT: Place transformation after source loading and compare responsibility boundaries with ETL for the same tabular workload.

Practical outcome: Specify source keys and retrieval/evaluation dataset contracts; implement one supplied transform with sensitive-field exclusions and compare ETL with ELT placement.

Module 2: Process and validate dataset records

Batch processing: Prepare a bounded AI dataset repeatably; reconcile counts and control totals and apply supplied deduplication and sensitive-field exclusion rules.
Streaming: Process simulated support-record updates and late arrivals; distinguish fresh source data from a downstream retrieval dataset awaiting refresh.
Data quality: Separate schema validity, completeness, duplicates, sensitive fields, and training/evaluation overlap; use supplied checks without claiming to detect all contamination.

Practical outcome: Run a batch and simulated updates, quarantine malformed rows, reconcile source counts and control totals, deduplicate records, and inspect training/evaluation overlap.

Module 3: Make dataset versions and updates traceable

Orchestration: Record transformation dependencies, versions, and checkpoints so controlled reruns follow the pipeline's declared processing boundaries.
Data lineage: Link retrieval and evaluation records to source keys, split membership, and transformation versions; identify downstream AI evidence affected by a correction.
Incremental processing contracts: Specify rerun, late-arrival, backfill, and schema-change behavior for AI datasets; preserve unique records and split membership during bounded replay.

Practical outcome: Record transformation versions, source-to-output lineage, split membership, and checkpoints; define late-arrival, backfill, and schema-change behavior for one AI dataset consumer.

Practical exercise

Build a synthetic support-record pipeline with accepted/rejected outputs, retrieval records, a held-out evaluation split, and a lineage manifest using supplied split and deduplication helpers.
Quarantine malformed rows, exclude a dummy personal-data field, detect an omitted row through reconciliation, and flag supplied duplicate and training/evaluation-overlap fixtures.
Rerun the input and backfill a late record; verify stable source keys, no duplicate accepted rows, and preserved split membership.
Change one source field and record the consumer failure, corrected transformation version, and bounded replay from the documented checkpoint.

What you will produce

Prepare a repeatable AI dataset with completeness checks, sensitive-field exclusions, duplicate and split-overlap detection, and traceable incremental updates.

How the work is checked

Accepted and rejected rows reconcile with source counts and control totals.
Incomplete, duplicate, and training/evaluation-overlap fixtures are detected; the dummy sensitive field is absent from accepted outputs.
Rerun and late-record backfill preserve unique source keys and documented split membership.
Lineage and transformation versions explain the schema-change failure and replay boundary.

Six-hour local AI dataset lab with supplied split/deduplication helpers and simulated streams. Detect the provided overlap cases without claiming to detect every semantic duplicate or contamination source. Excluding named fields is not complete anonymization. No model training, RAG implementation, or production streaming platform is included.

When another approach is better

A simple full batch can be safer than incremental state when the dataset is small.
Do not choose streaming when delayed batch delivery meets the decision's timing requirements.

How we train

Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.

Discuss your learning goals