AI curriculum

Course outline for AI Observability

A plausible AI response can hide missing evidence, an invalid tool call, or an action that never committed. Build a redacted execution ledger that connects model stages, retrieved sources, policy versions, approvals, and actual outcomes. Use it to investigate failures and total operating effort without claiming access to hidden reasoning.

Technology: AI Operations, Evaluation and Observability
Duration: 6 hours
Modules: 3 · 2 hours each
Level: Level 4

About AI Observability

AI Observability Course Objectives

Define redacted trace fields and retention rules for model interactions.
Separate input, output, and cache usage using supplied provider accounting.
Calculate per-stage latency and cost and record human wait and rework.
Reconstruct proposed, approved, and committed actions with context and policy versions.
Identify conclusions that missing trace evidence cannot support.

Pre-requisites

Ability to inspect a validated LLM application, equivalent to Building Applications with LLMs.
Logging, metrics, and trace competence equivalent to Observability Foundations for AI Systems.
Familiarity with agent tool calls or a short supplied tool-trace bridge; retrieval experience is helpful.

Before this course

Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.

Required preparation

Recommended preparation

Readiness checks

Can read model/tool-call traces and relate requests to results

Can inspect application behavior and telemetry before tracing model/tool stages; supply an agent/tool-trace bridge or take Agentic AI Foundations for deeper preparation.

Find your place in the learning map

Lab Setup

Computer with a trace viewer, spreadsheet, or analysis notebook.
Use synthetic prompts, supplied pricing/accounting fixtures, and local mock tool traces; no live users, provider account, or production logs are required.
Supply policy/context versions, redaction rules, and incomplete traces for investigation.

Detailed Course Outline

Module 1: Safe model interaction records

Prompt traces: Record redacted input references, context versions, and compaction events; choose necessary evidence fields and retention rules instead of indiscriminate prompt storage.
Model traces: Capture model identity, request metadata, output status, and stage relationships; distinguish observable interaction records from inaccessible internal model reasoning.
Token usage: Separate input, output, and cache accounting using supplied provider definitions; reconcile usage categories and flag missing or incompatible records.

Practical outcome: Define a redacted trace contract containing model identity, input/output/cache categories, context version, and compaction events without storing unnecessary private content.

Module 2: Resource and failure diagnosis

Latency: Measure model service time, tool time, queue wait, and human review separately; locate bottlenecks across the full execution path.
Cost: Apply supplied pricing to recorded usage and state exclusions; include review and rework effort rather than treating token charges as total cost.
Failure analysis: Classify missing retrieval, malformed output, denied execution, and missing result evidence; state which conclusions incomplete traces cannot support.

Practical outcome: Produce a per-stage resource report distinguishing service time, waiting, review, and rework; classify missing evidence, invalid output, and execution failures.

Module 3: Observable action history

Agent traces: Link workflow stages, context versions, worker activity, and observable outcomes; distinguish progress narration from evidence that a task actually completed.
Tool calls: Inspect arguments, access decisions, tool results, and final state; separate an attempted operation from a denied or committed effect.
Decision provenance: Connect retrieved sources, policy versions, proposals, approvals, and committed state; preserve unknowns without claiming hidden reasoning or causal proof.

Practical outcome: Build an execution ledger linking retrieved sources, policy versions, tool results, approvals, and committed state; list unknowns rather than infer hidden reasoning.

Practical exercise

Submit a redacted execution ledger and stage-level cost/failure report for a supplied failed retrieval-and-action run.
Compare a valid committed action with missing retrieval, malformed arguments, a denied write, and a missing action-result event.
Classify every labeled failure fixture and distinguish proposal, approval, and commit in each trace.
Reconcile recorded token categories with the supplied cost table, remove all dummy secrets, and explicitly mark conclusions blocked by missing evidence.

What you will produce

Reconstruct an AI run from a redacted ledger of evidence, approvals, and actual effects.

How the work is checked

The ledger separates proposed, approved, and committed states and records their versions.
Cost totals reconcile to supplied accounting and include declared exclusions.
The failure report classifies all supplied cases and lists unsupported conclusions.
Exported records contain no seeded dummy secrets.

Six-hour synthetic trace investigation; production instrumentation, retention policy, and causal effectiveness studies need separate work.

When another approach is better

Raw prompt logging is inappropriate when required evidence can be captured with safer metadata.
Token totals alone are insufficient when review, waiting, or rework dominates operating cost.
Provenance cannot reveal hidden model reasoning or prove a change caused improvement.

How we train

Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.

Discuss your learning goals