AI curriculum

Course outline for AI Application Evaluation

Determine whether an AI application meets its task requirements and whether a change genuinely improves its measured behavior. Create datasets and rubrics, implement evaluators, compare human and model judgments, and run a regression suite. Consider quality, latency, cost, and user effort together rather than accepting a fluent answer or a single average score as sufficient evidence. Distinguish exact-result correctness, uncertain predictive performance, and the quality and reliability of outputs that can legitimately vary.

AI Application Evaluation Course Objectives

  • Define task-specific criteria, representative cases, and a held-out evaluation set.
  • Choose exact, semantic, statistical, or combined checks appropriate to the task.
  • Compare human and LLM-based judging against independently reviewed examples.
  • Separate retrieval, answer, and application-effect failures.
  • Run a reproducible version comparison with quality, latency, cost, and review effort.

Pre-requisites

  • Write simple scripts and run tests over structured inputs and outputs.
  • Understand the behavior and purpose of an AI application under test.
  • Artificial Intelligence Foundations and Applied Generative AI or equivalent foundations and model-interaction competence.
  • Building Applications with LLMs is recommended for builders; an existing application can supply equivalent experience.
  • Retrieval-Augmented Generation helps with the RAG segment; its essential request/context structure is introduced here.

Before this course

Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.

Required preparation

Recommended preparation

Readiness checks

  • Write simple scripts and run tests over structured inputs and outputs.
  • Understand the behavior and purpose of an AI application under test.
  • Artificial Intelligence Foundations and Applied Generative AI or equivalent foundations and model-interaction competence.
  • Building Applications with LLMs is recommended for builders; an existing application can supply equivalent experience.
  • Retrieval-Augmented Generation helps with the RAG segment; its essential request/context structure is introduced here.

Can script and test structured inputs and outputs and understands an AI application under test; RAG structure is introduced for the retrieval segment, and a full application-building course is not mandatory.

Find your place in the learning map

Lab Setup

Delivery requirements: the following environment, materials, access, and exercise timing must be prepared and verified before delivery. This outline does not certify that a lab package is ready.

A selected scripting runtime, an application endpoint or runnable local model, an evaluator scaffold, and a model capable of the judge exercise. The lab provider must arrange access and quotas; learners need no paid accounts. Synthetic fixtures inject repeatable defects, while actual inference produces comparison and judge outputs. Human review can use reference judgments plus a learner attempt; no separate live reviewer panel is required. The lab must include a tiny fixed predictor and labeled cases for contrast with generated outputs. Scaffold the repeated-run harness so the learner spends time choosing checks and interpreting results rather than building a benchmark platform.

Detailed Course Outline

Module 1: Define success and build evaluation data (2 hours)

  • Task types and uncertainty: Distinguish exact-result tasks, uncertain predictions, and sampled generation before choosing checks.
  • Success criteria and risks: User tasks, acceptance criteria, risks, and failure categories.
  • Evaluation datasets and rubrics: Golden datasets, reference labels, rubrics, and ambiguous cases.
  • Held-out cases and slices: Development versus held-out sets, leakage, and representative slices.
  • Exact and effect checks: Exact values, numeric tolerances, invariants, and expected application effects.
  • Predictive performance metrics: Classification and regression metrics across representative unseen examples.
  • Valid generated-answer variation: Several valid generated answers: factual support, completeness, and task constraints.

Practical outcome: Build checks for a fixed calculation, a predictor, and variable generated answers.

Module 2: Evaluate outputs, retrieval, and judges (2 hours)

  • Human reference judgments: Compare human scoring with independent review and resolve disagreements against explicit criteria.
  • Semantic rubrics: Semantic rubrics for correctness, relevance, completeness, and usable style.
  • Executed model judges: Execute LLM-as-judge prompts and inspect variability in the resulting judgments.
  • Judge bias and disagreement: Position/verbosity bias, order changes, and disagreement with reference judgments.
  • Retrieval and answer support: RAG retrieval quality versus answer support and citation correctness.
  • Hard constraints: Unacceptable facts or actions remain failures even when wording variation is allowed.
  • Small-sample limits: Explain small-sample uncertainty and the limits of aggregate evaluation scores.

Practical outcome: Compare judging methods and diagnose a retrieval or answer-quality failure.

Module 3: Regression testing and release decisions (2 hours)

  • Versioned evaluation inputs: Versioned inputs, prompts, models, settings, and evaluator code.
  • Two-configuration runs: Run two actual configurations and inspect regressions at the individual-case level.
  • Repeated-run evidence: Repeated runs across cases: quality spread, failure frequency, and repeatability.
  • Controlling variability: Control model/settings/context; distinguish output variability from evaluator variability.
  • Reproducibility controls and limits: Seeds and low temperature as controls, not universal reproducibility guarantees.
  • Clean-session checks: Use clean-session checks where retained state can change the observed result.
  • Latency, cost, and human effort: End-to-end latency, token usage, inference costs, review, and rework.
  • Quality and contract thresholds: Fixed safety/contract checks alongside quality thresholds and measured trade-offs.
  • Offline and online evidence: Offline evaluation versus production monitoring and controlled online experiments.

Practical outcome: Rerun a comparison and explain whether the evidence supports adopting the change.

Practical exercise

  • Use three small related tasks: an exact calculation, a prediction from a fixed model, and a generated summary. Write exact/tolerance checks for the calculation, use held-out labels for the predictor, and author a rubric for the summary. Do not reject a correct paraphrase merely because it differs from a reference answer; still reject invented facts and missing requirements.
  • Freeze the criteria and check cases before comparing two configurations of the small application. Generate repeated outputs for each selected check input and report per-case quality variation and failure frequency, not only a best answer or one overall average. Run a model judge on a subset, compare it with independently reviewed judgments, and inspect reversed answer order. Use a small provided retrieval case to distinguish missing evidence from unsupported generation. Investigate a planted regression and report latency, cost assumptions, and observed human review effort separately.

What you will produce

Author task-appropriate evaluators and run repeated configuration comparisons with per-case quality, failure, latency, cost, and review-effort evidence.

How the work is checked

  • The dataset includes normal, boundary, and adverse cases without development leakage.
  • Exact/tolerance checks, predictive metrics, and generation rubrics match their tasks.
  • A valid paraphrase passes while a fluent but factually wrong answer fails.
  • Real repeated runs produce a report with identifiable inputs, versions, and failure frequencies.
  • Model-output variation and evaluator disagreement are recorded separately.
  • Retrieval failure and unsupported generation are correctly distinguished.
  • The recommendation preserves hard constraints and does not turn a small sample into a guarantee.

Six guided hours include demonstrations, practice, feedback, and questions. Breaks and pre-course setup are outside the total. Tiny contrast tasks and a scaffolded runner support exact calculations, fixed-model predictions, and variable generated summaries; these are not three new application builds. Classical metrics are applied, not re-derived. This is application-level evaluation, not a full statistics, model-training, production-observability, or autonomous-agent evaluation course. Online experiments receive orientation, not a claimed live A/B test. Detailed confidence intervals and large-scale reliability studies are outside scope. Repeatability and correctness are different: a fixed model can repeat an incorrect prediction, while sampled generation can yield different acceptable answers. Verify inference, judge access, quotas, fixtures, and pacing before delivery.

When another approach is better

  • Use exact or tolerance checks when the task has a specified answer; semantic similarity alone is insufficient.
  • Use representative held-out metrics for prediction rather than judge one plausible example.
  • Do not treat model-judge scores or a small offline sample as a production reliability guarantee.

How we train

Contact us for full course details, including duration, delivery options and lab requirements.

Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.

Discuss your learning goals