Course outline for LLM and Agent Evaluation

Module 1: Evaluation contract and settings

• Offline evaluation: Define task contracts and compare configurations on held-out fixtures; separate development examples from final acceptance cases to limit evaluation leakage.
• Online evaluation: Design a simulated monitoring plan for deployed behavior, review effort, and adverse outcomes; distinguish observed signals from a live causal experiment.
• Quality metrics: Choose answer-support and correctness criteria based on error consequences; keep quality scores separate from authority invariants and task completion.

Practical outcome: Define a bounded task contract, error-cost table, held-out split, and simulated online monitoring plan without claiming a live experiment.

Module 2: Answers versus actual effects

• Agent success metrics: Check durable state, permitted effects, and stopping behavior independently; distinguish stopped streams, canceled pending work, and already committed actions.
• RAG metrics: Evaluate retrieved evidence and answer support separately; detect missing sources and unsupported claims even when an answer appears fluent.
• Regression suites: Version valid and adverse cases covering malformed arguments, denied writes, and interruptions; fail forbidden effects regardless of narrative quality.

Practical outcome: Build independent checks for answer support, tool arguments, permitted effects, durable state, and stopping; fail fluent answers when state or authority is wrong.

Module 3: Judgment and comparison

• Evaluation datasets: Construct representative synthetic cases with explicit expected outcomes and held-out splits; repeat fresh-session comparisons without tuning on final results.
• Human evaluation: Use explicit rubrics, adjudicated examples, and disagreement records; include reviewer effort and error consequences when choosing acceptance thresholds.
• Automated judges: Compare judge outputs with human adjudication and independent state checks; document disagreements and reject scores that overlook unauthorized effects.

Practical outcome: Compare two configurations over repeated fresh-session fixtures, adjudicate judge disagreements, and report quality, review effort, and threshold limitations.

Practical exercise

• Submit a versioned suite, held-out results table, and threshold/disagreement dossier for two configurations or supplied runs.
• Include valid tasks, absent evidence, denied writes, malformed arguments, and interrupted execution; inspect actual final state.
• Run at least ten held-out cases per configuration and require every authority invariant to pass independently of answer scores.
• Reject all seeded unauthorized-effect cases, adjudicate every human/judge disagreement, and report task acceptance and total review effort separately.