AI curriculum
Course outline for Data Platforms for AI Workloads
Operate a small batch and stream platform before choosing a larger architecture. Transform records with Spark, produce and consume events with Kafka, and test restart, replay, lag, and retention assumptions. Connect storage, table formats, and analytical queries to clear responsibilities. Compare the platform with a simpler batch and apply the result to AI dataset refresh.
Data Platforms for AI Workloads Course Objectives
- Separate object storage, lake tables, compute, messaging, and analytical serving.
- Execute Spark DataFrame transformations and SQL with checked outputs.
- Operate Kafka topics, producers, consumers, offsets, and consumer groups.
- Test consumer restart, replay, duplicate effects, and lag under continuing arrivals.
- Distinguish lakehouse table semantics from file/object storage semantics.
- Compare Hadoop ecosystem and managed-service roles without assuming equivalence.
- Justify recovery, retention, capacity, and cost choices or recommend simpler batch.
Pre-requisites
- Data Engineering Foundations for AI Systems or equivalent executable pipeline, quality, lineage, and incremental concepts.
- Basic SQL, Python, terminal use, and interpreting rates and timestamps.
- Basic relational transactions and file formats.
- No Spark, Kafka, Hadoop, cloud-account, or cluster-administration experience is required.
Before this course
Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.
Required preparation
Recommended preparation
Readiness checks
- Data Engineering Foundations for AI Systems or equivalent executable pipeline, quality, lineage, and incremental concepts.
- Basic SQL, Python, terminal use, and interpreting rates and timestamps.
- Basic relational transactions and file formats.
- No Spark, Kafka, Hadoop, cloud-account, or cluster-administration experience is required.
Data Engineering Foundations for AI Systems or equivalent executable pipeline, quality, lineage, and incremental concepts.
Find your place in the learning mapLab Setup
Delivery requirements: the environment, access, and exercise materials must be prepared and verified before delivery. This outline does not certify that a lab package is ready.
Preconfigured disposable Spark/PySpark, Kafka KRaft, compatible Python clients, DuckDB, and a small Spark/Iceberg sandbox. Materials include synthetic events, bounded datasets, producer/consumer/sink starters, failure hooks, manifest checks, and resource limits. Delivery requires compatible stable releases, JVM/Python/client and Spark/Iceberg support, licenses, platform resources, and cleanup. Incompatibility blocks delivery; screenshots do not replace operation. Local storage and dated managed-storage profiles avoid real cloud credentials or purchases.
Detailed Course Outline
Module 1: Give each platform component a clear job (2 hours)
- Data lakes and curated datasets: Data lakes, raw files, curated datasets, and table-level metadata.
- Object storage responsibilities: Object keys, prefixes, partitions, formats, versions, and deletion responsibility.
- Lakehouse tables and snapshots: Lakehouse tables, snapshots, catalogs, and atomic table operations.
- Iceberg table inspection: Iceberg table inspection through a prepared compatible sandbox.
- DuckDB analytical query: Analytical serving with a small DuckDB query over Parquet.
- Hadoop roles and alternatives: Hadoop roles: HDFS, YARN, MapReduce, and modern alternatives.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Inspect stored files, a table snapshot, and an analytical query without confusing their responsibilities.
Module 2: Transform a dataset with Spark (2 hours)
- Spark foundations: Spark basics, session startup, DataFrames, schemas, and lazy execution.
- PySpark transformations: PySpark reads, column expressions, filtering, and typed transformations.
- Joins and data quality: Joins, grouping, nulls, duplicates, and data-quality checks.
- Actions and Parquet output: Actions, output partitions, Parquet writes, and visible job execution.
- Spark SQL comparison: Spark SQL over the same data and independent result comparisons.
- Distributed execution concepts: Local execution versus distributed scheduling and shuffle concepts.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Run a learner-extended transformation and verify its records, schema, and output files.
Module 3: Make Spark output reproducible and inspect its cost (2 hours)
- Reproducible dataset output: Source keys, rerun policy, versioned output paths, and manifests.
- Plans partitions and shuffles: Execution plans, partitions, shuffles, and bounded skew observations.
- Repartitioning experiment: A controlled repartitioning experiment with actual stage observations.
- Schema failure and correction: Schema change and a failed transformation followed by correction.
- Snapshot comparison: Table snapshot inspection and a prepared version comparison.
- Simpler local processing: Local notebook/script alternatives when distributed work is unnecessary.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Compare two Spark runs, explain changed work, and preserve identifiable dataset versions.
Module 4: Move actual events through Kafka (2 hours)
- Kafka foundations: Kafka basics: broker, topic, partition, record key, and offset.
- KRaft startup and inspection: KRaft local startup, topic creation, and CLI inspection.
- Producer operation: Producer acknowledgements and an actual Python producer.
- Consumer groups and offsets: Consumer groups, partition assignment, polling, and offset commits.
- Partition ordering: Keyed partition ordering versus global order.
- Actual event consumption: Actual event delivery into a harmless local consumer sink.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Produce and consume keyed updates and inspect partition/offset evidence independently.
Module 5: Restart, replay, and catch up honestly (2 hours)
- Restart and retained events: Consumer restart, retained records, committed offsets, and reset policies.
- Sink and commit boundaries: Sink writes versus offset commits and duplicate-effect exposure.
- Transactional deduplication: Stable event identity and transactional deduplication in the lab sink.
- Observed consumer lag: Throttled consumers, observed lag, and continuing producer arrivals.
- Catch-up and retention: Replay/catch-up calculations, retention windows, and capacity margins.
- Failure and guarantee limits: Failure tests and limits of Kafka's exactly-once processing claims.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Reproduce restart/replay behavior and measure lag recovery without claiming arbitrary exactly-once effects.
Module 6: Choose a platform from operated evidence (2 hours)
- Bounded platform integration: Kafka-to-storage responsibility and a prepared bounded integration path.
- Batch output and snapshots: Spark batch output, dataset snapshots, and analytical inspection.
- Dataset and index freshness: AI dataset preparation versus downstream retrieval-index freshness.
- Managed-platform profiles: Managed object-storage and compute profiles with dated cost assumptions.
- Operating cost and recovery: Outage, catch-up, retained bytes, processing, and operating costs.
- Scheduled-batch comparison: Architecture selection versus a working scheduled-batch baseline.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Defend a bounded architecture or choose the simpler batch using lab evidence and declared workload assumptions.
Practical exercise
- Prepare synthetic order updates and a later AI dataset-refresh variant. Inspect Parquet files and an Iceberg table using a prepared compatible environment. Query Parquet with DuckDB. These are short operational introductions, not complete table-format or analytical-database development tracks. Hadoop and cloud vendor services remain sourced comparisons, not unannounced required deployments.
- Extend and run PySpark transformations: schema checks, filter, join, deduplication, aggregate, and versioned Parquet output. Run equivalent SQL for one result and compare actual records. Observe lazy evaluation, a shuffle, a partition change, and a schema failure. Save actual run plans/observations and manifests. Local mode teaches operation but does not establish multi-node scaling or fault tolerance.
- Start actual Kafka, create a topic, and use producer/consumer clients. Capture keys, partitions, offsets, and consumed results. Stop and restart the consumer, replay retained events, and test a sink-write/offset-commit interruption. Compare a non-idempotent sink with a stable-key transactional deduplication variant. Do not claim that committing an offset atomically commits an arbitrary external side effect. Partition order is not global business-event order.
- Throttle the consumer while the producer continues. Measure lag growth and recovery after capacity changes. Calculate catch-up time only when service capacity exceeds arrivals, and reject retention that fails to cover the outage plus catch-up and margin. Explain which parts are observed local behavior and which are estimates at larger workload assumptions.
- Use a prepared integration adapter to land a bounded consumed batch for Spark processing and analytical inspection. Track source events, dataset version, and downstream freshness separately. Compare a working scheduled-file batch with the platform route. Make a new failure/recovery prediction independently before opening feedback. No real cloud purchase or production index is needed.
Delivery and timing
- Guided hours include teaching, demonstrations, learner practice, feedback, questions, and contingency as allocated above. Breaks, bulk downloads, and prerequisite setup are additional. Allocations need a timed learner pilot before delivery; they are not measured completion times.
What you will produce
Operate Spark and Kafka, inspect lake storage, and justify a bounded data platform against simpler batch processing.
How the work is checked
- Actual file, table-snapshot, and analytical-query observations have separate meanings.
- Learner-extended Spark transformations execute and match checked result sets.
- Actual Kafka producer/consumer records preserve key/partition/offset evidence.
- Restart/replay tests explain duplicate sink effects and deduplication boundaries.
- Measured lag and estimated catch-up distinguish observations from assumptions.
- Retention covers the declared recovery interval or the design is rejected.
- The architecture distinguishes dataset readiness from retrieval freshness.
- The decision evaluates operational cost and a simpler working batch alternative.
12 guided hours. Spark receives four dedicated hours, Kafka four, and storage/integration four. Iceberg and DuckDB are bounded operational introductions. Stable Spark/Iceberg, JVM, Python, and client compatibility is a delivery prerequisite, not implied by independently latest packages. No full Hadoop installation, cloud administration, production cluster tuning, end-to-end streaming lakehouse, or distributed fault-tolerance certification is claimed. Spark Structured Streaming is comparison-only; Kafka clients provide required streaming practice.
When another approach is better
- Prefer a scheduled batch and simple analytical store when latency and scale do not require a distributed stream platform.
- Do not add a lakehouse or Hadoop ecosystem solely because other organizations use one.
How we train
Contact us for full course details, including duration, delivery options and lab requirements.
Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.
Discuss your learning goals