AI curriculum
Course outline for AI Application Evaluation
Anecdotal model outputs do not show whether an application is improving. Define representative tests and compare quality, retrieval, cost, and latency with explicit criteria.
About AI Application Evaluation
Anecdotal model outputs do not show whether an application is improving. Define representative tests and compare quality, retrieval, cost, and latency with explicit criteria.
AI Application Evaluation Course Objectives
- Create an evaluation dataset and scoring rubric.
- Compare deterministic, human, and model-based judgments.
- Build a regression report with quality and resource measures.
Pre-requisites
- Familiarity with an AI application or prototype.
- Basic scripting or test automation skills.
Lab Setup
- Computer with a scripting runtime and test runner.
- Use supplied application outputs and timing records; a local model-based judge is optional.
Detailed Course Outline
Proposed modulesModule 1: Designing tests
- Evaluation design
- Golden datasets
- LLM-as-judge
- Deterministic evaluation
Practical outcome: Create a test set with explicit scoring criteria.
Module 2: Comparing application versions
- Human evaluation
- Regression testing
- RAG evaluation
- Cost and latency evaluation
Practical outcome: Produce a version comparison with error categories.
Practical exercise
- Evaluate two application configurations and explain conflicting quality and latency results.
How we train
Contact us for full course details, including duration, delivery options and lab requirements.
Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.
Discuss your learning goals