AI curriculum

Course outline for Observability Foundations for AI Systems

An AI request may complete successfully yet use stale retrieval evidence or fail during a tool call. Instrument a supplied mock retrieval-model-tool path and correlate its logs, metrics, and traces. Diagnose execution failures and retest a bounded correction while keeping service reliability separate from answer quality.

Technology: AI Operations, Evaluation and Observability
Duration: 6 hours
Modules: 3 · 2 hours each
Level: Level 1

About Observability Foundations for AI Systems

Observability Foundations for AI Systems Course Objectives

Define logs, metrics, and traces for a retrieval-model-tool request path.
Instrument a supplied local service with correlation and sensitive-input redaction.
Query latency, errors, and mock token/cost measurements with bounded metric labels.
Distinguish infrastructure failures from recorded answer-quality failures.
Calculate a reliability SLO with an explicit numerator, denominator, and window.
Retest a correction and document incomplete-trace and causal-inference limits.

Pre-requisites

Basic application development or operations experience.
Familiarity with HTTP services and command-line tools.
Ability to run a supplied service and containers; cloud deployment experience is helpful but not required.

Before this course

Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.

No prerequisite course is required. The entry skills above still apply.

Recommended preparation

Cloud Foundations for AI Engineers

Has application or operations experience with HTTP and shell tools; cloud terminology helps but is not required to instrument a local service.

Find your place in the learning map

Lab Setup

Local application runtime and container support; verify the supplied OpenTelemetry, Prometheus, and Grafana stack before teaching time.
Supplied mock retrieval-model-tool service, synthetic traffic, token/cost counters, answer-quality labels, and redaction/missing-span fixtures.
Stage roles are explained in the scaffold; no AI prerequisite, live model, hosted monitoring, or production telemetry is required.

Detailed Course Outline

Module 1: Evidence for an AI request

Logs: Record structured retrieval, model, and tool events with correlation identifiers; exclude named sensitive prompts, source fields, and dummy credentials from exported evidence.
Metrics: Define bounded-label traffic, stage latency, errors, and mock token/cost counters; distinguish aggregate service measurements from individual answer-quality judgments.
Traces: Represent retrieval, model, and tool steps as related spans; locate execution failures and recognize gaps without assuming successful spans imply correct answers.

Practical outcome: Define a telemetry contract for mock retrieval, model, and tool stages with correlation identifiers, bounded metric labels, and prompt/source-data exclusions.

Module 2: Instrumentation and inspection

OpenTelemetry: Instrument the supplied service with consistent span context and telemetry fields; connect application events while preserving configured sensitive-field exclusions.
Prometheus: Query supplied traffic, error, and latency series; check measurement definitions and label choices before interpreting changes in service behavior.
Grafana: Build a focused dashboard linking request volume, failures, and latency; connect aggregate signals to the specific failing request evidence.

Practical outcome: Instrument the supplied local request path and build queries/dashboard panels linking traffic, stage latency, errors, and supplied token/cost counters.

Module 3: Reliability versus answer quality

Distributed tracing: Follow an AI request across mock service boundaries to locate a tool failure; distinguish recorded stage attribution, answer-quality labels, and proof of causation.
SLOs: Define a reliability objective with explicit numerator, denominator, and measurement window; calculate fixture results without equating reliability with answer correctness.
Correction and retest evidence: Compare equal-duration windows using the same workload and metric definition; record observed errors, remaining failures, and limits of causal inference.

Practical outcome: Diagnose an injected tool-service failure, inspect a successful-but-incorrect answer fixture, calculate a reliability SLO, and retest a bounded correction.

Practical exercise

Submit instrumentation changes, dashboard queries, and an incident packet for a supplied mock retrieval-model-tool service.
Run healthy traffic, a downstream tool failure, and a missing-span case; compare execution status with a separately supplied answer-quality label.
Correlate complete traces, flag incomplete evidence, and verify that dummy credentials and named sensitive input fields do not appear in exported telemetry.
Compare latency and fixture token/cost counts; calculate the SLO before/after the fix with equal-duration windows and the same workload, while recording answer-quality findings separately.

What you will produce

Diagnose and retest a mock AI request path using correlated telemetry while separating execution reliability, cost, and answer-quality evidence.

How the work is checked

Complete traces correlate retrieval, model, and tool stages without exposed dummy credentials or named sensitive inputs; incomplete traces are flagged.
The incident packet distinguishes the injected tool failure from an incorrect-answer fixture and relates stage latency to supplied token/cost counters.
The SLO calculation and retest use equal-duration windows, the same workload and metric definition, and separately recorded observed errors.

Six-hour local instrumentation lab with mock AI stages and supplied quality labels. Cost counters use fixture assumptions, not provider billing evidence. Redaction covers named fixture fields, not all possible sensitive data. Teach observability fundamentals here; detailed prompt/agent tracing and quality evaluation follow in specialist courses.

When another approach is better

A single diagnostic script may be sufficient for a small one-off problem without a full monitoring stack.
Sampled or incomplete traces cannot justify a complete account of every request.
Telemetry alone cannot demonstrate business impact or answer correctness.

How we train

Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.

Discuss your learning goals