AI curriculum

Course outline for Data Engineering Foundations for AI Systems

Move ordinary changing records from sources to trustworthy outputs. Implement extraction, transformation, loading, quality checks, orchestration, and lineage. Test reruns, late arrivals, corrections, and schema changes. Then apply the same contracts to an AI dataset with explicit split isolation and sensitive-field rules.

Data Engineering Foundations for AI Systems Course Objectives

  • Define source and consumer contracts for a small data pipeline.
  • Implement ETL and compare an executed ELT path over the same records.
  • Run batch processing and an actual local bounded event consumer.
  • Reconcile completeness, duplicates, validity, and sensitive-field exclusions.
  • Execute dependency-ordered steps with recoverable checkpoints and lineage.
  • Test incremental changes, deletions, late arrivals, backfill, and schema evolution.
  • Apply a bounded training/evaluation split check without overstating contamination detection.

Pre-requisites

  • Basic Python, files, tables, and SQL queries.
  • Database Systems for AI Applications or equivalent keys, updates, and persistence concepts.
  • Run a small script and interpret counts and timestamps.
  • No scheduler, broker, Spark, or model-training experience is required.

Before this course

Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.

Required preparation

Recommended preparation

Readiness checks

  • Basic Python, files, tables, and SQL queries.
  • Database Systems for AI Applications or equivalent keys, updates, and persistence concepts.
  • Run a small script and interpret counts and timestamps.
  • No scheduler, broker, Spark, or model-training experience is required.

Basic Python, files, tables, and SQL queries.

Find your place in the learning map

Lab Setup

Delivery requirements: the environment, access, and exercise materials must be prepared and verified before delivery. This outline does not certify that a lab package is ready.

Compatible stable Python, SQLite, tests, dependency-runner/checkpoint starter, bounded local event producer/consumer, synthetic orders/support records, and split helpers. The transport delivers actual local messages but is not Kafka. First-use adapter and contract instruction stays inside guided hours. Delivery requires checked restart boundaries, reference outputs, stable dependencies, and cleanup. No cloud scheduler, model, or personal data is required.

Detailed Course Outline

Module 1: Define what moves and where it changes (2 hours)

  • Pipeline contracts: Sources, consumers, pipeline stages, and output contracts.
  • Keys types and timestamps: Keys, types, units, timestamps, and schema expectations.
  • Source extraction boundaries: Extraction boundaries, snapshots, and changing source records.
  • ETL and ELT: ETL transformations before load and ELT transformations after load.
  • Python and SQL paths: Python and SQL starter paths with visible intermediate results.
  • Ownership and retention: Ownership, sensitivity, retention, and rejected-record handling.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Implement an ETL path and run a small ELT variant with explicit responsibility boundaries.

Module 2: Process records and prove what happened to them (2 hours)

  • Repeatable batches: Repeatable batch transforms and deterministic source-key handling.
  • Schema and business validity: Schema validity versus business validity and missing data.
  • Source reconciliation: Source reconciliation, control totals, rejects, and duplicate accounting.
  • Quarantine and repair: Quarantine records with reasons and repair/reprocessing paths.
  • Sensitive-field exclusions: Named sensitive-field exclusions and their privacy limits.
  • Omitted-row tests: Independent tests that detect a silently omitted valid row.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Produce reconciled accepted/rejected outputs and detect an omission despite valid output schemas.

Module 3: Handle new, changed, and late records (2 hours)

  • Full and incremental refresh: Full refresh versus incremental extraction and update contracts.
  • Event and processing time: Event time, processing time, and high-water-mark limitations.
  • Local event operation: Actual local producer/consumer updates through a prepared transport.
  • Idempotent updates and tombstones: Idempotent upserts, duplicate deliveries, tombstones, and source corrections.
  • Late arrivals and backfill: Late-arrival policy, backfill windows, and durable progress.
  • Restart boundaries: Restart between output publication and checkpoint advancement.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Restart a bounded local consumer and apply late, duplicate, corrected, and deleted records correctly.

Module 4: Coordinate stages and trace affected outputs (2 hours)

  • Stage dependency and retries: Dependency order, stage inputs, failures, retries, and safe reruns.
  • Executable orchestration: A small executable orchestrator over prepared local storage.
  • Checkpoints and safe reuse: Checkpoints, atomic output/progress boundaries, and invalid reuse.
  • Record lineage: Record-level provenance and transformation/input versions.
  • Schema evolution and replay: Schema change, consumer failure, migration, and bounded replay.
  • Correction impact: Lineage queries for correction impact and recovery evidence.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Stop and resume the pipeline, then identify outputs affected by a source-schema change.

Module 5: Transfer sound contracts to AI datasets (2 hours)

  • Dataset consumer contracts: Retrieval, training, and evaluation outputs as distinct consumers.
  • Stable split assignment: Stable grouping keys, split assignment, and correction history.
  • Duplicate and group overlap: Exact duplicate and declared group-overlap checks.
  • Downstream sensitive copies: Sensitive-field exclusions and downstream copy responsibilities.
  • Backfill and manifests: Late records, backfills, split preservation, and versioned manifests.
  • Simpler full-batch choices: Choosing full batch when incremental complexity has no benefit.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Prepare a versioned AI dataset with tested group isolation and explicit remaining contamination risks.

Practical exercise

  • Build an order-record pipeline using Python and SQLite. Specify source keys, currencies/units, event time, schema, accepted outputs, quarantine outputs, and consumer requirements. Execute ETL and a small ELT variant, comparing the same transformation and where raw data persists. Run a full batch with invalid, duplicate, and omitted-row tests. Reconcile source identifiers, outcome categories, and control totals rather than trusting counts alone.
  • Process actual locally delivered updates with a prepared producer/consumer transport. Handle a duplicate, late arrival, changed value, and tombstone. Stop after an output write but before progress acknowledgement, restart, and verify the stated idempotency/transaction policy. A timestamp watermark alone can miss late or equal-timestamp records; define the bounded extraction/backfill policy.
  • Execute extraction, validation, transform, and publication through a small dependency runner. Inject a stage failure, resume safely, and inspect lineage. Change a source field, observe a consumer failure, version the transformation, and perform a bounded replay with independent output checks. Prepared storage and transport adapters prevent this from becoming another scheduler course.
  • Finally adapt synthetic support records into retrieval and held-out evaluation outputs. Apply stable group assignment and supplied overlap checks, then correct and backfill records without silently changing their split. Exclude a named dummy sensitive field. Add an independent negative test before feedback. No model training, embeddings, complete anonymization, or universal contamination detection is claimed by these checks.

Delivery and timing

  • Guided hours include teaching, demonstrations, learner practice, feedback, questions, and contingency as allocated above. Breaks, bulk downloads, and prerequisite setup are additional. Allocations need a timed learner pilot before delivery; they are not measured completion times.

What you will produce

Implement and recover a bounded pipeline with quality checks, incremental updates, and record-level lineage.

How the work is checked

  • Both ETL and ELT paths execute with explicit raw-data boundaries.
  • Source keys and control totals reconcile accepted, duplicate, and rejected records.
  • Actual local updates follow correction, deletion, and late-arrival policies.
  • Restart does not silently lose records or create duplicate accepted effects.
  • Executed orchestration and lineage explain failure, recovery, and changed outputs.
  • Schema evolution is observed, repaired, versioned, and replayed.
  • AI transfer preserves declared split groups and states the overlap check's limits.

10 guided hours. General pipeline mechanics precede AI dataset transfer. Learners implement ordinary changes and recovery through prepared local storage and transport adapters. Actual local messages are not a production broker or high-volume stream system. No embeddings, training, cloud scheduler, or universal contamination/anonymization claim is made. Spark and Kafka operation belongs in Data Platforms for AI Workloads. A scheduled full refresh can be the simpler sufficient design.

When another approach is better

  • A simple full batch can be safer than incremental state when the dataset is small.
  • Do not choose streaming when delayed batch delivery meets the decision's timing requirements.

How we train

Contact us for full course details, including duration, delivery options and lab requirements.

Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.

Discuss your learning goals