AI curriculum

Course outline for Data Platforms for AI Workloads

An AI dataset or retrieval corpus needs trustworthy versions and recoverable updates, not every data-platform component. Compare lake, lakehouse, batch, and stream responsibilities for a supplied AI data workload. Use curated traces to decide whether distributed processing and rapid refresh justify their operating and recovery costs.

Technology: Data Engineering and Databases
Duration: 4 hours
Modules: 2 · 2 hours each
Level: Level 2

About Data Platforms for AI Workloads

Data Platforms for AI Workloads Course Objectives

Separate source storage, dataset versions, retrieval refresh, and analytical queries.
Map Spark, Kafka, and Hadoop ecosystem roles to an AI data workload.
Diagnose a retrieval-refresh consumer's capacity and backpressure from traces.
Estimate replay, retention, storage, and processing costs under stated assumptions.
Compare vendor-neutral roles with supplied managed-cloud profiles without assuming equivalent semantics.
Justify a distributed architecture or a simpler scheduled batch.

Pre-requisites

Familiarity with SQL, data pipelines, and data-quality checks.
Basic command-line or programming experience and ability to interpret counts, rates, and timestamps.

Before this course

Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.

Required preparation

Data Engineering Foundations for AI Systems

Recommended preparation

Can describe data pipelines and quality before selecting batch/streaming storage and processing components.

Find your place in the learning map

Lab Setup

Spreadsheet or local notebook and diagram editor.
Synthetic AI dataset versions, Spark/Kafka/Hadoop processing examples, object-storage manifests, retrieval-refresh traces, and analytical query fixtures.
Dated managed-cloud profiles and local calculations; no model training, full cluster, cloud subscription, or complete platform implementation is required.

Detailed Course Outline

Module 1: Assign AI data storage and processing roles

Data lakes: Store versioned AI source datasets separately from retrieval indexes and processing jobs; assess whether scheduled preparation meets freshness and recovery needs.
Lakehouses: Compare table versions and governed dataset access with underlying object storage; identify responsibilities for repeatable AI data preparation without assuming snapshots prevent leakage.
Spark: Inspect distributed AI dataset preparation traces; compare deduplication and transformation workloads with a simpler batch before accepting cluster operating costs.
Kafka: Analyze document-update streams and retrieval-refresh consumers; calculate lag and replay needs without confusing delivered events with a fresh index.

Practical outcome: Map versioned source datasets, table management, batch preparation, and retrieval-refresh streams; diagnose a lagging index-refresh consumer from curated traces.

Module 2: Select a recoverable AI data architecture

Hadoop ecosystem: Map ecosystem storage and processing roles to workload needs rather than selecting components solely because they appear together.
Object storage: Compare source/artifact storage profiles such as S3, Azure Blob Storage, and Google Cloud Storage; estimate version retention and replay costs under stated assumptions.
Analytical databases: Use supplied query profiles for dataset inspection and AI usage analysis; distinguish analytical serving from online retrieval and authoritative agent-state storage.

Practical outcome: Compare ecosystem and query-store roles, calculate replay/retention needs, and distinguish dataset/analytics storage from derived retrieval indexes using supplied managed-cloud profiles.

Practical exercise

Produce an architecture decision and responsibility map for a synthetic AI dataset and retrieval-refresh workload.
Inspect a valid dataset batch and overloaded refresh consumer; identify whether stale retrieval reflects source changes, processing lag, or missing lineage.
Calculate backlog growth, replay time, retained bytes, and a cost range from supplied rates; map managed offerings to roles rather than claiming feature parity.
Reject retention that cannot cover the stated outage and replay window; compare with a simpler scheduled refresh and preserve identifiable dataset versions.

What you will produce

Choose AI data-platform components using dataset versioning, retrieval freshness, consumer capacity, replay, retention, and cost requirements.

How the work is checked

The architecture separates source datasets, processing, messaging, analytical queries, and derived retrieval indexes.
The overloaded retrieval-refresh consumer is diagnosed from rates, lag, and dataset-version evidence.
Replay, retention, and cost calculations state their units and assumptions.
The selection rejects insufficient recovery coverage and evaluates a batch alternative.

Four-hour curated-fixture decision lab for AI datasets and retrieval refresh, not model training or a complete data platform. Replay estimates include continuing arrivals; catch-up requires capacity above arrival rate. Retention covers outage plus catch-up with a margin. Managed-service profiles need current verification and do not establish equivalent recovery semantics.

When another approach is better

Prefer a scheduled batch and simple analytical store when latency and scale do not require a distributed stream platform.
Do not add a lakehouse or Hadoop ecosystem solely because other organizations use one.

How we train

Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.

Discuss your learning goals