← All templates
PythonIntermediateApplied AI EngineerForward Deployed EngineerApplied AI Architect

LLM eval harness with judge calibration and a CI gate

Evaluate a triage prompt with code graders and a calibrated LLM judge, report pass rates with Wilson intervals, cost and latency per slice, and block regressions in CI.

Locked

The code is part of the full course

The file map and notes are open to everyone. Get lifetime access to every template's source, the .zip download and all four tracks for a one-time $249.

Run it

shell
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env  # add ANTHROPIC_API_KEY
python -m app.run_evals run --sut keyword --no-judge --out results/keyword  # offline
python -m app.run_evals calibrate
python -m app.run_evals run --sut claude --out results/candidate
python -m app.compare results/keyword/results.json results/candidate/results.json

Test it

shell
pytest -q

Offline: no API key or network needed.

Taught in