AI curriculum

Course outline for LLM and Agent Evaluation

Build an evaluator for a small support assistant with answer-only and permitted action tasks. Construct held-out data, generate real outputs from two configurations, and compare quality with independently checked outcomes. Contrast human ratings with actual automated judge calls. Separate small-sample evidence from reliability claims and simulated online monitoring from real user experiments. Use Jev in two roles: a bounded rubric evaluator and an evaluated assistant decision component. Keep retrieval, end-state, tool-trace, and human checks separate.

LLM and Agent Evaluation Course Objectives

  • Design quality metrics and online signals with explicit units and limitations.
  • Construct a versioned dataset with independent labels and leakage-controlled held-out inputs.
  • Implement retrieval, answer-support, correctness, state, authority, and stop checks.
  • Run two actual configurations repeatedly in fresh isolated sessions.
  • Collect independent human ratings and execute automated judge calls.
  • Adjudicate disagreements and report regressions, review effort, and uncertain thresholds.
  • Design safe online monitoring without claiming a live causal experiment.
  • Operate Jev as a rubric evaluator and evaluate its bounded assistant decisions.
  • Compare decision errors, abstention, review cost, and frozen-threshold held-out results.

Pre-requisites

  • AI Application Evaluation or equivalent basic application evaluation and test design.
  • Agentic AI Foundations or equivalent agent loops, tool effects, permissions, and stopping concepts.
  • Basic retrieval, Python tests, data tables, and rate/cost calculations.
  • No judge framework or statistical inference expertise is assumed.
  • No prior Jev or decision-model experience is assumed.

Before this course

Complete the required courses or bring equivalent experience. Recommended courses add useful depth; they are not enrolment gates.

Required preparation

Recommended preparation

Readiness checks

  • AI Application Evaluation or equivalent basic application evaluation and test design.
  • Agentic AI Foundations or equivalent agent loops, tool effects, permissions, and stopping concepts.
  • Basic retrieval, Python tests, data tables, and rate/cost calculations.
  • No judge framework or statistical inference expertise is assumed.
  • No prior Jev or decision-model experience is assumed.

AI Application Evaluation or equivalent basic application evaluation and test design.

Find your place in the learning map

Lab Setup

Delivery requirements: the environment, access, and exercise materials must be prepared and verified before delivery. This outline does not certify that a lab package is ready.

Python evaluation runner, synthetic sources, local retrieval, protected harmless action store, resettable sessions, two controlled real configurations, supported generative-judge endpoint, official TypeSafe Jev access, independent human raters, development/calibration data, sealed final cases, and separate feedback. Stable support, explicit model/client pins, classroom rights, individual credentials, permitted probes, data/retention terms, quotas, capped budgets, complete materials, and cleanup require verification. A stable model identifier does not remove early-access/account eligibility limits. Missing Jev access blocks that objective; another classifier or replay cannot silently replace it. No real users, high-impact decisions, production credentials, or local Jev weights are required.

Detailed Course Outline

Module 1: Define success before collecting scores (2 hours)

  • Task contracts and consequences: Task contracts, user goals, error consequences, and observable expected outcomes.
  • Answer-quality dimensions: Answer correctness, support, completeness, relevance, and appropriate abstention.
  • Quality and authority: Quality metrics versus authority invariants and application task completion.
  • Dataset construction: Dataset construction, labeling, source versions, and representative case categories.
  • Development and sealed test groups: Disjoint development, calibration, and sealed final-test groups with leakage checks.
  • Frozen manifests and independent cases: Instructor custody, versioned manifests, independent labels, and controlled final release.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Author a task/rubric contract and three disjoint dataset manifests with independently specified sealed final expectations.

Module 2: Separate retrieved evidence from generated answers (2 hours)

  • Retrieval relevance: Retrieval relevance, recall at a stated cutoff, and source-label scope.
  • Retrieval and generation failure: Correct retrieval with poor generation versus missing evidence with fluent output.
  • Claim-level answer checks: Claim-level support, answer correctness, completeness, and refusal appropriateness.
  • Learner-authored scorers: Learner-implemented scorers and deterministic scorer unit tests.
  • Actual development outputs: Real model answers over the development set and actual retrieval results.
  • Fixture limits and blind spots: Adverse fixture supplements and metric blind spots.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Run retrieval and answer scorers on actual generated answers and explain one disagreement between the metrics.

Module 3: Check the effect instead of the completion message (2 hours)

  • Expected state and effects: Expected state, allowed effects, tool arguments, and independent readback.
  • Actual harmless tools: Actual harmless tool execution with protected local test state.
  • No-tool success: Legitimate no-tools success and unnecessary tool-use penalties.
  • Unauthorized-effect checks: Denied writes, malformed arguments, and unauthorized-effect rejection.
  • Cancellation and commit boundaries: Stopped stream versus canceled work versus already committed action.
  • Evaluator recovery regression: Controlled interruption/recovery and regression tests for the evaluator itself.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Implement end-state and authority checks and run actual success, no-tools, failure, and interruption tests.

Module 4: Compare real configurations without leaking answers (2 hours)

  • Configuration identities: Frozen configuration identities, model settings, prompts, and tool policies.
  • Same sealed final inputs: Development comparisons and separately sealed final inputs with independent expected results.
  • Fresh-session isolation: Fresh sessions, cleared memory/state, and no answer-key context.
  • Repeated actual generation: Repeated actual model/agent generation with final runs held in sealed storage.
  • Sealed scorer outputs: Development-only scorer results and complete sealed final-run failure accounting.
  • Sealed comparison records: Development regressions and sealed final outputs, labels, usage, and feedback.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Run two configurations repeatedly, inspect development results, and collect final runs without exposing any final feedback.

Module 5: Ask people and judges independently (2 hours)

  • Development rubric calibration: Human-rubric anchors and practice ratings on development outputs only.
  • Independent human ratings: Independent development ratings before discussion or automated-score disclosure.
  • Actual generative judges: Real development model-judge calls with structured rubric results and validation.
  • Judge bias and injection: Development-only order swaps, verbosity, authorized injection probes, and self-preference.
  • Disagreement measures: Human/human and human/judge disagreement with explicit denominators.
  • Adjudication and unsafe acceptance: Review effort, adjudication, missing ratings, and unsafe judge acceptance.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Develop a human rubric and generative judge using development outputs, then compare independent development ratings.

Module 6: Use Jev as a bounded answer evaluator (3 hours)

  • Bounded decision questions: Bounded questions, dynamic options, typed answers, and rubric levels.
  • Jev System One operation: Jev System One request state, question identifiers, and response inspection.
  • Rubric-based answer scoring: Separate relevance, grounding, and completeness questions over generated answers.
  • Question types and abstention: Choice, Noul predicates, Score distributions, and appropriate abstention.
  • Rules judges and decision models: Deterministic rules, generative judges, and decision models against human judgments.
  • Typed output and evaluator bias: Type validity versus correctness and evaluator bias on new cases.
  • Guided-time allocation: 1 hour for explanation and a worked example, 1 hour for learner practice, 0.5 hours for feedback, and 0.5 hours for questions and contingency.

Practical outcome: Implement and run a real Jev rubric evaluator on development outputs and prepare separate rubric acceptance rules for calibration.

Module 7: Evaluate the decision model inside the assistant (3 hours)

  • Routing and review destination: Ticket routing and triage with an explicit review destination.
  • Tool-call checks and permissions: Proposed tool-call checks versus application-enforced permissions and input validation.
  • Document and next-step decisions: Document-support checks and next-step selection within permitted actions.
  • Missing options and drift: Missing correct options, abstention, ambiguous inputs, and distribution change.
  • Class-specific error measures: Per-class confusion, precision, recall, and critical false accepts.
  • Frozen thresholds and final tests: Calibration-selected thresholds, complete evaluator freeze, and final-test release controls.
  • Guided-time allocation: 1 hour for explanation and a worked example, 1 hour for learner practice, 0.5 hours for feedback, and 0.5 hours for questions and contingency.

Practical outcome: Test actual development Jev decisions, select thresholds on separate calibration data, and freeze the complete evaluation package.

Module 8: Decide what the comparison actually supports (2 hours)

  • Per-category comparison: Per-category quality, grounding, task success, and invariant failure rates.
  • Paired regression and variability: Paired regressions and repeat variation without invented statistical significance.
  • Judge acceptance errors: Judge over-acceptance, under-acceptance, abstention, and sensitivity checks.
  • Threshold and reviewer consequences: Threshold consequences, reviewer workload, and cost per accepted result.
  • Independent new-case evaluation: Independent new-case evaluation before feedback and no held-out retuning.
  • Versioned regression evidence: Versioned regression report, rejection reasons, and next-data requirements.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Defend or reject a configuration using computed evidence, adjudication, review cost, and explicit sample limitations.

Module 9: Plan online monitoring without risking users (2 hours)

  • Offline and online boundaries: Offline evaluation versus deployed observation and controlled online experiments.
  • Experiment design: Eligible population, units, assignment, outcomes, guardrails, and review signals.
  • Safe local monitoring simulation: Safe local monitoring simulation using generated test-run events.
  • Drift bias and privacy: Selection bias, delayed labels, drift, privacy, and missing feedback.
  • Stop and rollback ownership: Stop conditions, rollback ownership, and consequential-action restrictions.
  • Further evidence and approvals: Further sample needs and approval gates before a real experiment.
  • Guided-time allocation: 0.75 hours for explanation and a worked example, 0.75 hours for learner practice, 0.25 hours for feedback, and 0.25 hours for questions and contingency.

Practical outcome: Simulate a monitoring/stop decision and write an online design that explicitly does not claim real-user or causal evidence.

Practical exercise

  • The proposed starter supplies synthetic support documents, a retrieval adapter, two configurable actual model/agent runners, harmless protected tool/state adapters, reset hooks, an empty result schema, and separate development output materials. The learner constructs labeled development cases and implements retrieval/quality scoring, state/authority/stop checks, a runner extension, and an independent test. Expected results come from source facts and policy, not evaluated models or judges. The instructor supplies independent calibration and hidden final material, including complete unsolved final cases. These materials are proposed, not certified supplied.
  • Create three disjoint versioned splits for answer and decision evaluation in Module 1. Group related families and near duplicates together; check cross-split leakage. Modules 2–3 use development cases for scorers, real harmless tool effects, permitted no-tools success, denied writes, malformed arguments, and interruption around a local commit. Seeded unauthorized-state mutations must fail regardless of judge praise. Distinguish a stopped stream, canceled pending work, and already committed state.
  • In Module 4 visibly generate and score development outputs from both configurations. Freeze generation settings before the instructor-controlled runner optionally collects at least ten hidden final inputs for each configuration twice in fresh reset sessions: forty actual final generations. Seal raw outputs, expected labels, traces, retrieved IDs, stop/state records, failures, per-case scores, aggregate feedback, and usage. Show only a content-free collection receipt. Hidden input execution is instructor- controlled; learner-authored visible cases alone are not unseen assessment. Delaying final collection until after the complete freeze is also valid. Never silently drop failed or timed-out runs. No sealed final data is teaching material in Modules 5–7.
  • Module 5 develops human rubric interpretation and an actual generative judge using development outputs only. Module 6 implements and changes actual Jev questions, options, rubrics, and client code on separate development outputs from both configurations. Teach Noul as the vendor's predicate answer type. Use development probe cases for order, verbosity, omission, and authorized inert injection checks. Module 7 develops the Jev tool-check component on development data, then uses the separate calibration material to select action-specific and rubric-specific thresholds and estimate review demand. Calibration examples never count as final cases. If calibration feedback changes a model, scorer, judge, question, or rubric, version the change and obtain fresh calibration material before threshold selection again.
  • At the end of Module 7, hash and freeze generation configurations, deterministic scorers, human rubric/instructions, generative judge prompt/model, Jev model/questions/ options/rubrics, thresholds, abstention/fallback rules, all manifests, subset-selection rule, and denominators. Instructor verifies the actual package before any final-output release. Block release if anything is unfrozen or exposed; replace leaked final cases.
  • In Module 8 release only the preselected blinded subset of at least twelve final generated outputs to two independent human raters, with task/source evidence but no reference answer, configuration identity, automated results, or other human scores. Lock both original submissions before automated-score disclosure. Then run the frozen deterministic checks, forty actual generative-judge calls, and forty actual Jev rubric requests on all final generated outputs. Multi-question batching is allowed where supported; budget more requests if it is not. Neither cached scores nor a different classifier labeled Jev completes the required named operation.
  • Also test at least twelve independently labeled final decision cases after the freeze. They cover clear allowed/denied calls, missing evidence/options, abstention, no authorized tool, ambiguity, invalid inputs, changed data, and permitted inert injection cases. Categories can overlap. Run actual Jev calls for eligible cases; log invalid-input blocks as local checks, not Jev responses. Unresolved hosted probe permission leaves that probe blocked/local-only and the hosted conclusion unproven. Model approval never grants executor permission.
  • Unblind final per-case/aggregate results and independent labels once in Module 8. Adjudicate all disagreements in the co-rated subset and every critical judge/invariant conflict across both sets. Preserve original ratings and scores, missing data, usage/retries, model identities, questions/options/rubrics/thresholds, and release hashes. Report separate relevance/grounding/completeness wrong accepts/rejects and review rates. Report decision-class precision/recall and critical false accepts with denominators. Measure actual review/adjudication effort and latency/cost, including failures. Small samples and twelve co-rated outputs demonstrate a method, not population reliability or calibrated probabilities. No average hides unsafe grounding acceptance.
  • After final feedback, never retune or reuse these cases as unseen final evidence. A changed evaluator or system requires a new regression version, development work, fresh calibration if affected, and new disjoint final cases. Keep the old set as disclosed regression history only. Independent tasks and their expected answers remain hidden until submission. Feedback explains wrong criteria, unsafe passes, authority denial, missing evidence, and uncertainty without rewriting original scores.
  • Module 9 simulates monitoring from generated test events and writes a safe future online design. No real users, consequential actions, or causal improvement claim. Only future authorized model operation satisfies these exercises; no calls ran here. Application injection probes use inert synthetic content and authorized boundaries. Obtain provider clarification/permission before hosted adversarial probes. Do not probe provider infrastructure, bypass safeguards, or export sensitive customer data.

Delivery and timing

  • Guided hours include teaching, demonstrations, learner practice, feedback, questions, and contingency as allocated above. Breaks, bulk downloads, account approval, and prerequisite setup are additional. Allocations need a timed learner pilot before delivery; they are not measured completion times.

What you will produce

Evaluate real model and agent behavior, compare human and automated judges, and test Jev as evaluator and decision component.

How the work is checked

  • Dataset, split, labels, source versions, and configuration identities are preserved.
  • Implemented metrics distinguish retrieval, answer quality, effects, authority, and stopping.
  • At least forty actual held-out runs cover both configurations and two fresh-session repeats.
  • Success, legitimate no-tools work, failures, and recovery receive independent checks.
  • Real judge calls and independent human ratings have retained evidence and adjudication.
  • Forbidden effects fail regardless of answer quality; disagreement and missing data remain visible.
  • Regression/threshold decisions include quality, review cost, variation, and sample limitations.
  • Online simulation is labeled and does not masquerade as a user experiment.
  • Actual Jev rubric calls retain relevance, grounding, completeness, and human-disagreement evidence.
  • Independent unseen decision tests include missing options, abstention, and no authorized tool.
  • Per-class precision/recall and critical false accepts accompany explicit denominators and uncertainty.
  • Question/option/rubric/model/threshold versions freeze before held-out scoring and never borrow provider thresholds.
  • Application permissions override model decisions; retries, latency, cost, and fallback remain visible.
  • Three disjoint splits and sealed final artifacts prevent final-data-driven judge design.
  • Verified complete freeze precedes final raw-output release, human rating, and automated scoring.
  • Independent final human submissions precede automated-score disclosure and final adjudication.
  • Rubric-specific wrong accepts/rejects and review rates accompany decision-class precision/recall.
  • Final feedback never supports retuning on the same set as fresh held-out evidence.

20 guided hours. Real generation, harmless tool effects, automated judging, and Jev System One operation are required future practice. Jev is both a rubric evaluator and an evaluated bounded-decision component. Valid typed output does not establish correctness. Confidence concentration is not calibrated correctness, and thresholds do not transfer automatically. Deterministic code retains arithmetic and permission authority. Final data and outputs stay sealed until scorers, rubrics, questions, judges, and thresholds freeze on development data. Online work is design and local simulation, not an authorized live user experiment. Small samples cannot certify safety, statistical significance, calibration, or universal injection resistance.

When another approach is better

  • A judge score alone is unsuitable when correctness depends on externally observable state or permission.
  • A small fixture suite cannot establish population-level reliability or calibrated confidence.
  • Do not run live online evaluation on consequential users without a separate safety and experiment plan.

How we train

Contact us for full course details, including duration, delivery options and lab requirements.

Ask about exercises, instructor feedback and the prior knowledge you need. Tool-specific courses marked provisional may become modules in a broader course.

Discuss your learning goals