AI curriculum

Course outline for AI Application Evaluation

A better-looking answer or lower latency does not show that an AI application has improved. Define a task contract and compare two versions on fixed cases with independent quality checks. Include retrieval failures, reviewer disagreement, review time, and rework so that the report does not confuse faster generation with better outcomes.

Technology: AI Application Engineering
Duration: 4 hours
Modules: 2 · 2 hours each
Level: Level 4

About AI Application Evaluation

AI Application Evaluation Course Objectives

Define a task contract, scoring rubric, and representative held-out cases.
Detect structurally valid outputs that fail the intended contract.
Compare deterministic, human, and model-based judgments with documented limits.
Test retrieval and answer quality as separate dimensions.
Build a repeatable regression comparison including fresh-session cases.
Report latency, cost assumptions, human touch time, and rework without unsupported causal claims.

Pre-requisites

Building Applications with LLMs or equivalent ability to specify and inspect an AI application contract.
Familiarity with an AI application or prototype.
Basic scripting or test automation skills.
Retrieval-Augmented Generation is helpful; supplied retrieval traces support the focused evaluation exercise.

Before this course

Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.

Required preparation

Building Applications with LLMs

Recommended preparation

Retrieval-Augmented Generation

Has an application contract or prototype to test; RAG familiarity helps with the retrieval-specific part of evaluation.

Find your place in the learning map

Lab Setup

Computer with a scripting runtime, prepared test runner, and spreadsheet or notebook.
Supplied versioned outputs, retrieval traces, separate development and held-out cases, independently labeled judge fixtures including reversed answer order, timing records, and reviewer-effort records.
A local model judge is optional; no paid judge, live user experiment, or production telemetry is required.

Detailed Course Outline

Module 1: Define credible quality evidence (2 hours)

Evaluation design: Task contracts, scoring rubrics, failure categories, and comparison limits; define completion before optimizing prompts or interpreting observed improvements.
Golden datasets: Representative labeled cases, source support, and held-out splits; keep development examples separate from the fixed scored comparison.
LLM-as-judge: Rubric-based model judgments, order sensitivity, and independent labels; examine disagreement without treating judge agreement as evidence of truth.
Deterministic evaluation: Schema checks, required behaviors, and source-backed assertions; expose outputs that satisfy weak types but violate the intended contract.

Practical outcome: Create a held-out dataset and definition of done, expose a weak schema-only check, and compare a deterministic rubric with supplied judge outputs while recording sensitivity and disagreement.

Module 2: Compare versions and total effort (2 hours)

Human evaluation: Independent scoring, explicit rubrics, and reviewer disagreement; distinguish credible correctness evidence from persuasive wording or shared assumptions.
Regression testing: Fixed cases, versioned outputs, and fresh-session runs; detect regressions without hidden conversation dependencies or tuning on scored cases.
RAG evaluation: Expected evidence, retrieved context, and claim support; separate missing retrieval from unsupported generation and incorrectly attached source references.
Cost and latency evaluation: Token assumptions, response time, waiting, human touch time, and rework; report separate measures and alternative explanations rather than causal claims.

Practical outcome: Run a two-version regression report with fresh-session fixtures, separate retrieval from generation failures, and include latency, cost assumptions, waiting, human touch time, rework, and downstream defects with stated comparison limits.

Practical exercise

Produce a reusable evaluation dataset, rubric, test script, and two-version comparison report.
Accept source-supported contract-compliant answers and flag malformed, unsupported, or superficially typed outputs.
Score at least twelve fixed cases, report per-category results and reviewer disagreements, and rerun the suite without hidden conversation state.
Compare quality, latency, reviewer effort, and rework separately; state at least two alternative explanations for an observed improvement rather than asserting causation.

What you will produce

Produce a reproducible application-version comparison that separates answer quality, retrieval quality, and total operating effort.

How the work is checked

The frozen rubric detects semantic failure that a schema-only check misses.
The dataset and versioned inputs permit an independent rerun without tuning on held-out cases.
Results separate retrieval, generation, resource use, and reviewer disagreement; supplied judge decisions are checked against independent labels and reversed answer order.
The conclusion records comparison limits and avoids causal claims from a before/after table.

Four hours using prepared versioned outputs and an evaluation runner. Keep development cases separate from the fixed held-out comparison; do not tune a prompt or rubric on the scored cases. Calibrate supplied judge decisions against independent labels, and inspect order sensitivity and reviewer disagreement without treating agreement as truth. Exercises fit within module time; online experiments and causal inference require additional methods.

When another approach is better

Do not use a model judge as the sole acceptance authority for outcomes that have independent factual or deterministic checks.
Do not select a version solely on response latency when errors, review effort, or rework outweigh that gain.

How we train

Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.

Discuss your learning goals