AI curriculum
Course outline for Observability Foundations for AI Systems
Find why an ordinary order-status service is slow or failing. Collect structured logs, metrics, and distributed traces from running services. Query real data and build a dashboard. Define a reliability objective, diagnose an unseen fault, make one correction, and retest before transferring the method to AI stages.
Observability Foundations for AI Systems Course Objectives
- Emit useful structured logs with request identity and sensitive-field exclusions.
- Choose bounded counters, gauges, distributions, labels, and units.
- Instrument an actual service using the OpenTelemetry Python SDK.
- Export through a Collector and inspect related spans in Tempo.
- Execute Prometheus queries and build a Grafana incident dashboard.
- Calculate an explicit SLO and compare repeatable before/after observations.
- Diagnose and correct an independent injected fault before AI-stage transfer.
Pre-requisites
- Run a small Python HTTP service and read its code and errors.
- Use functions, HTTP requests, test scripts, and local processes or containers.
- Interpret counts, durations, fractions, and basic charts.
- No model, agent, telemetry SDK, dashboard, or monitoring experience is assumed.
Before this course
Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.
No prerequisite course is required. The entry skills above still apply.
Recommended preparation
Readiness checks
- Run a small Python HTTP service and read its code and errors.
- Use functions, HTTP requests, test scripts, and local processes or containers.
- Interpret counts, durations, fractions, and basic charts.
- No model, agent, telemetry SDK, dashboard, or monitoring experience is assumed.
Run a small Python HTTP service and read its code and errors.
Find your place in the learning mapLab Setup
Delivery requirements: the environment, access, and exercise materials must be prepared and verified before delivery. This outline does not certify that a lab package is ready.
Two disposable Python HTTP services, request generator, stable OpenTelemetry trace/metric SDK and OTLP exporter, compatible Collector/Tempo/Prometheus/Grafana stack, synthetic data, and fault hooks. Compatible releases, exporter components, ports, resource limits, trace storage, queries, dashboard connections, and cleanup require verification. Endpoints remain restricted; real backend queries and learner instrumentation cannot be replaced by supplied screenshots.
Detailed Course Outline
Module 1: Ask what evidence a request needs (2 hours)
- Telemetry questions: Service behavior, user-visible success, and questions telemetry can answer.
- Structured logs: Structured logs, timestamps, severity, request identifiers, and useful fields.
- Metric types and cardinality: Counters, gauges, histograms, units, and bounded label cardinality.
- Trace relationships: Trace spans, duration, parent relationships, and incomplete requests.
- Sensitive-field exclusion: Sensitive-field exclusion, safe exceptions, and short retention ownership.
- Independent service results: Actual ordinary-service requests and independently checked results.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Add structured events and define metric/span contracts for a running order-status request.
Module 2: Follow a request across services (2 hours)
- OpenTelemetry API and SDK: OpenTelemetry API versus SDK and first manual span creation.
- Tracer provider and spans: Tracer provider, resource identity, span status, and exception records.
- Cross-service context propagation: Context injection/extraction across two actual HTTP services.
- Collector pipeline: OTLP exporter and Collector receiver, processor, exporter, and pipeline.
- Tempo trace lookup: Tempo backend basics, stored trace lookup, and Grafana trace view.
- Missing telemetry diagnosis: Missing context, dropped spans, sampling, and telemetry pipeline failure.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Export actual related spans through a Collector into Tempo and repair one broken context link.
Module 3: Query rates without confusing the units (2 hours)
- Metric export path: OpenTelemetry metric instruments and the prepared Prometheus export path.
- Prometheus series: Prometheus scrape targets, time series, labels, and sample intervals.
- PromQL rates and resets: Actual PromQL selectors, aggregation, rates, and counter resets.
- Matched error populations: Error ratio with matched populations and empty-window handling.
- Latency histograms: Histogram buckets, latency distributions, and percentile limitations.
- Independent traffic reconciliation: Bounded request generation and independent count/duration reconciliation.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Run learner-written Prometheus queries against generated traffic and explain each result's units and denominator.
Module 4: Make a dashboard answer an incident question (2 hours)
- Grafana foundations: Grafana basics, data sources, panels, queries, and time ranges.
- Incident dashboard panels: Request rate, errors, latency, and dependency panels over real series.
- Units and misleading averages: Correct units, legends, intervals, and misleading averages.
- Trace and log correlation: Tempo data source, trace lookup, and correlated log identifiers.
- Evidence and competing hypotheses: Aggregate symptoms versus request-specific evidence and alternative hypotheses.
- Saved dashboard interpretation: Saved dashboard export and an independent changed-traffic interpretation.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Build a working service dashboard and follow an observed symptom into a stored distributed trace.
Module 5: Correct a fault and earn the retest (2 hours)
- SLI and SLO: SLI measurement versus SLO target and explicit observation window.
- Eligible and good events: Eligible requests, good events, latency threshold, and excluded events.
- Error-budget arithmetic: Error budget arithmetic without extrapolating a short laboratory window.
- Independent fault diagnosis: Unseen injected dependency fault and evidence-based competing hypotheses.
- Correction and comparable retest: Learner correction, comparable traffic, and before/after measurements.
- Rollback and functional recovery: Failed correction, rollback, and independent functional recovery check.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Diagnose an unsolved service failure, correct it, and calculate comparable SLO results before and after.
Module 6: Keep the method when a stage uses AI (2 hours)
- AI service-path transfer: Queue, retrieval, model, and tool stages as service-path transfer.
- Operational and answer quality: Operational success versus answer correctness and evidence quality.
- Supplied AI-stage adapter: Supplied AI-stage adapter without requiring model-provider operation.
- Bounded redacted metadata: Bounded labels and redacted metadata instead of raw prompt metrics.
- Independent recovery test: New independent latency/failure test and comparable recovery observations.
- Operational handoff limits: Evidence gaps, sampling limits, and a concise operational handoff.
- Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.
Practical outcome: Extend the same telemetry design to an AI-stage adapter without replacing ordinary observability or claiming model-quality measurement.
Practical exercise
- Proposed instructor materials provide two ordinary HTTP services, request generator, disposable Collector/Tempo/Prometheus/Grafana deployment, test data, and fault hooks. Start the services, connect the exporter, run a query, and create a panel with guidance. The learner adds SDK spans, context propagation, bounded metric instruments, structured logs, PromQL expressions, dashboard panels, and an independently authored retest. Prepared topology does not replace instrumentation or actual backend operation.
- Generate traffic and inspect real telemetry. Check the export path, then query Prometheus and look up a stored trace in Tempo through Grafana. A console-only export or screenshot of a supplied dashboard does not complete the exercise. Use request-result counts as an independent check on metric populations. Do not put request IDs or raw user content into metric labels.
- Complete a new fault investigation before opening its separate feedback. The learner identifies the responsible stage, states alternative explanations, modifies the service, and reruns the same workload and observation window. Calculate good/eligible events and latency compliance, including missing samples and zero denominators. Confirm functionality separately from dashboard recovery. Add a bounded AI-stage adapter only after the general service path works. Model answer quality remains a separate evaluation concern.
Delivery and timing
- Guided hours include teaching, demonstrations, learner practice, feedback, questions, and contingency as allocated above. Breaks, bulk downloads, account approval, and prerequisite setup are additional. Allocations need a timed learner pilot before delivery; they are not measured completion times.
What you will produce
Instrument ordinary services, operate a bounded telemetry stack, and diagnose and retest a fault with independent evidence.
How the work is checked
- Structured logs, units, identifiers, exclusions, and bounded labels match actual records.
- SDK instrumentation exports related spans through the configured Collector to Tempo.
- Actual PromQL queries reconcile request populations and explain rate/window behavior.
- Grafana panels use real service series with correct units and useful incident views.
- An unseen fault receives an evidence-grounded diagnosis, correction, and independent retest.
- SLO calculations state numerator, denominator, target, window, and limits.
- AI transfer preserves the distinction between service operation and answer quality.
12 guided hours. General observability is the primary subject. OpenTelemetry Python, Prometheus, Grafana, Collector, and Tempo have bounded actual operation. Ordinary structured logs avoid requiring an unstable logs SDK. AI stages are final transfer without a required model account. Telemetry does not prove answer quality or causality. No backend clustering, production monitoring rollout, long-term reliability certification, or full on-call program is included.
When another approach is better
- A single diagnostic script may be sufficient for a small one-off problem without a full monitoring stack.
- Sampled or incomplete traces cannot justify a complete account of every request.
- Telemetry alone cannot demonstrate business impact or answer correctness.
How we train
Contact us for full course details, including duration, delivery options and lab requirements.
Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.
Discuss your learning goalsRelated Courses
These courses list this course as preparation.