AI curriculum

Course outline for AI Observability

Instrument a support assistant that retrieves facts, calls a real model, and proposes a bounded local action. Generate observable records rather than starting with completed ledgers. Diagnose quality and execution failures, reconcile usage and costs, and connect a proposed action to independently checked application state.

AI Observability Course Objectives

  • Instrument actual model, retrieval, and tool activity with the OpenTelemetry Python SDK.
  • Test sensitive-field exclusions and bounded retention across generated records.
  • Reconcile real input, output, cache, retry, and missing-usage categories.
  • Measure stage latency and calculate model plus broader operating costs.
  • Link proposals, approvals, attempted tools, and independently observed commits.
  • Diagnose an unseen quality or execution failure without inventing missing evidence.

Pre-requisites

  • Observability Foundations for AI Systems or equivalent running telemetry, logs, metrics, traces, and query competence.
  • Building Applications with LLMs or equivalent model-request handling, validation, and application tests.
  • Basic retrieval and agent/tool concepts, with a short orientation if needed.
  • Python code editing and interpretation of durations, token usage, and prices.

Before this course

Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.

Required preparation

Recommended preparation

Readiness checks

  • Observability Foundations for AI Systems or equivalent running telemetry, logs, metrics, traces, and query competence.
  • Building Applications with LLMs or equivalent model-request handling, validation, and application tests.
  • Basic retrieval and agent/tool concepts, with a short orientation if needed.
  • Python code editing and interpretation of durations, token usage, and prices.

Observability Foundations for AI Systems or equivalent running telemetry, logs, metrics, traces, and query competence.

Find your place in the learning map

Lab Setup

Delivery requirements: the environment, access, and exercise materials must be prepared and verified before delivery. This outline does not certify that a lab package is ready.

Python support assistant, stable OpenTelemetry trace SDK/OTLP exporter, familiar Collector/Tempo stack, synthetic facts, local action store, independent approval adapter, redaction/expiry hooks, and an authorized supported model endpoint. Credentials, bounded budget, usage/cache definitions, dated pricing, field exclusions, retention behavior, stack compatibility, and cleanup require verification. No production logs or consequential actions are used.

Detailed Course Outline

Module 1: Record a real model interaction safely (2 hours)

  • Application-owned AI spans: OpenTelemetry Python refresher and explicit application-owned AI span schema.
  • Actual model call identities: Actual model call with template, model, parameter, and context identities.
  • Redacted evidence references: Redacted input references, output status, and relevant evidence links.
  • Sensitive export paths: Sensitive content in attributes, errors, tool arguments, and exports.
  • Retention and expiry checks: Allowlisted fields, short retention, and controlled expiry tests.
  • Hidden-reasoning limits: Observable interactions versus inaccessible hidden model reasoning.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Instrument a real model request and inspect exported records with tested dummy-secret exclusion and expiry.

Module 2: Link evidence to decisions and effects (2 hours)

  • Retrieval evidence linkage: Retrieval identifiers, ranked sources, context versions, and missing evidence.
  • Workers and context history: Parent spans, workers, checkpoints, and context-compaction references.
  • Proposal approval and commit: Model proposal, independent approval, and executor commit as distinct states.
  • Tool attempts and state: Tool attempt, denied access, returned result, and observed application state.
  • Action and policy identities: Stable action identity, policy version, and application result linkage.
  • Independent effect readback: Independent readback and explicit gaps after missing result events.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Generate a linked model/tool ledger and distinguish approved intent from an independently confirmed local effect.

Module 3: Reconcile time and money from actual records (2 hours)

  • Provider usage definitions: Provider usage definitions and overlapping versus exclusive token categories.
  • Token and cache categories: Input, output, cached input, cache writes, and missing usage handling.
  • Retries and missing usage: Retries, canceled requests, and observed versus estimated billable work.
  • Stage and human time: Queue wait, model service, tool time, and human waiting.
  • Dated cost reconciliation: Dated provider prices, units, region/tier assumptions, and cost reconciliation.
  • Cost per useful result: Infrastructure, review, rework, and cost per accepted useful result.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Reconcile actual request usage and stage timings with a reproducible cost calculation and explicit exclusions.

Module 4: Diagnose failures without trusting the narration (2 hours)

  • Retrieval and answer failures: Missing retrieval versus unsupported generated claims and invalid output.
  • Tool and commit failures: Malformed arguments, denied writes, failed tools, and missing commit evidence.
  • Real and injected evidence: Controlled adverse hooks alongside actual successful model/tool operation.
  • Correctness versus authorization: Source support and correctness checks separate from execution acceptance.
  • Incomplete diagnostic evidence: Unseen diagnostic incident with incomplete records and competing explanations.
  • Correction and independent retest: Learner correction and independent success/failure/recovery retest.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Investigate an unseen incident, state evidence gaps, and verify a correction against answer quality and final state.

Module 5: Export an investigation others can check (2 hours)

  • Investigation reconstruction: Reconstructing sources, policy, context, proposal, approval, and commit history.
  • Redaction regression: Redaction regression tests over backend, file export, and exception records.
  • Expiry and copy limits: Expiry checks in the declared lab store and downstream-copy limitations.
  • Quality and effort summary: Quality, latency, cost, failure, and reviewer-effort summary with uncertainty.
  • Independent diagnostic test: New learner-authored diagnostic test before feedback or answer disclosure.
  • Operational handoff: Operational handoff, unknowns, retention ownership, and corrective follow-up.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Submit a generated redacted dossier with reproducible calculations, independently checked effects, and unsupported conclusions explicitly withheld.

Practical exercise

  • The proposed starter provides retrieval, model-client, local state-store, protected executor, approval input, and exporter adapters. The learner authors SDK spans, safe fields, usage normalization, cost calculation, and an independent negative test. The learner changes and runs real instrumentation rather than just filling in a ledger. A real authorized model request must generate part of the records. Local tools perform harmless real state changes; supplied adverse records only supplement deliberately induced failures and an unseen diagnostic incident.
  • Trace a source-backed answer and a proposed action. Approval is supplied through an independent reviewer input, never inferred from model text. Check state with a separate readback query. Compare success, unsupported answer, malformed arguments, denied action, and a missing result event. Do not equate a successful tool response with durable state or infer failure solely from a missing event. Diagnose what the available evidence supports and what further observation is needed.
  • Inspect actual provider usage and responses, including any cache subset fields. Reconcile categories without counting cached input twice. Zero measured cache use is valid; a labeled synthetic cache fixture tests calculation, not an observed cache saving. Missing usage is unknown, not free. Use dated official prices for the selected endpoint. Measure model, tool, and queue durations, record simulated human wait explicitly, and estimate review/rework with declared rates. Score a bounded set of known answers for correctness and source support separately from action approval. Test redaction before export and expiry in the declared local retention store. Complete the new diagnostic and final test independently before opening feedback.

Delivery and timing

  • Guided hours include teaching, demonstrations, learner practice, feedback, questions, and contingency as allocated above. Breaks, bulk downloads, account approval, and prerequisite setup are additional. Allocations need a timed learner pilot before delivery; they are not measured completion times.

What you will produce

Instrument real model and tool operations, reconcile usage and cost, and diagnose failures with redacted inspectable evidence.

How the work is checked

  • Generated exports link actual model/retrieval/tool operations with safe identifiers.
  • Dummy secrets remain absent from declared exports, errors, and backend views.
  • Retention expiry is tested with its copy/storage limitations stated.
  • Usage and dated costs reconcile with no hidden double counting or invented cache saving.
  • Stage timing and human wait remain distinct and missing fields remain unknown.
  • Proposal, approval, attempted action, and confirmed commit have independent evidence.
  • Unseen diagnosis and retest check answer quality, permitted effects, and recovery.

10 guided hours. OpenTelemetry Python is the required instrumentation library, using a versioned application schema rather than assumed stable GenAI conventions. A real authorized model call and harmless real tool effects are required. Controlled faults and synthetic cache fixtures supplement observations, not replace them. Records do not expose hidden reasoning, prove explanations, establish causality, or certify universal privacy or provider erasure. Deeper datasets and judges belong to the evaluation course.

When another approach is better

  • Raw prompt logging is inappropriate when required evidence can be captured with safer metadata.
  • Token totals alone are insufficient when review, waiting, or rework dominates operating cost.
  • Provenance cannot reveal hidden model reasoning or prove a change caused improvement.

How we train

Contact us for full course details, including duration, delivery options and lab requirements.

Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.

Discuss your learning goals