PythonIntermediateApplied AI EngineerForward Deployed EngineerApplied AI Architect
LLM eval harness with judge calibration and a CI gate
Evaluate a triage prompt with code graders and a calibrated LLM judge, report pass rates with Wilson intervals, cost and latency per slice, and block regressions in CI.
Locked
The code is part of the full course
The file map and notes are open to everyone. Get lifetime access to every template's source, the .zip download and all four tracks for a one-time $249.
Run it
shell
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env # add ANTHROPIC_API_KEY
python -m app.run_evals run --sut keyword --no-judge --out results/keyword # offline
python -m app.run_evals calibrate
python -m app.run_evals run --sut claude --out results/candidate
python -m app.compare results/keyword/results.json results/candidate/results.jsonTest it
shell
pytest -qOffline: no API key or network needed.
Taught in
- Module 8 Why Evals Are the FDE's Superpower
- Module 8 Exercise: Build an Eval Harness
- Module 8 Graders: Code, Humans and LLM-as-Judge
- Module 8 Exercise: Calibrate an LLM-as-Judge
- Module 8 Release Gates, Noise and Nondeterminism
- Module 8 Exercise: A Release Gate for Prompt Changes
- E4 Evals Are the Job
- E4 Exercise: A Golden-Set Runner That Tells the Truth
- E4 LLM-as-Judge Done Right
- E4 Exercise: A CI Regression Gate
- E1 Exercise: A Field-Level Prompt Regression Suite
- E6 Design It: An Eval Platform for Weekly Changes