Applied AI Engineer Playbook. Build production LLM systems: prompting, agents, retrieval, evals, latency and cost, then the system design round. For: backend and ML engineers.
Applied AI Engineer map
Build it for production.Defend it in the loop.
- 0%
- complete
- 0/61
- graded tasks done
- 56h
- to go
Start here
E1 · Prompting
Anatomy of a Production Prompt
Ready for the full loop?
Loop mode is closed book: no tutor, hints or reveals, and your first graded attempt on each round is the score.
Modules
0 of 114 lessons- 0/6E1·PromptingFull course
Production Prompting & Structured Outputs
System prompts, JSON schemas, extraction reliability and prompt regression tests.
Leo Martins, Staff engineer6 lessons · 2.7h- ReadAnatomy of a Production PromptFree preview18m
- TaskA Warranty-Claim Extractor That Checks Its Own Output40mLocked: part of the full course
- ReadStructured Outputs and Validation in Depth18mLocked: part of the full course
- TaskA Field-Level Prompt Regression Suite35mLocked: part of the full course
- WriteDefend Your Prompt: Maintenance Requests at Larkspur40mLocked: part of the full course
- QuizModule E1 Quiz10mLocked: part of the full course
- 0/8E2·AgentsFull course
Tool Use & Agents
Tool loops, MCP servers, sub-agents, failure handling and traces.
Leo Martins, Staff engineer8 lessons · 3.6h- ReadThe Tool Loop in Depth18mLocked: part of the full course
- TaskHarden a Trip-Support Agent Loop40mLocked: part of the full course
- ReadDesigning Tools Models Use Well18mLocked: part of the full course
- TaskPause for Approval, Resume Later45mLocked: part of the full course
- ReadDebugging Agents from Traces16mLocked: part of the full course
- ReadSub-agents and Orchestrator-Workers16mLocked: part of the full course
- TaskAn Orchestrator With Cheap, Isolated Workers45mLocked: part of the full course
- QuizModule E2 Quiz15mLocked: part of the full course
- 0/6E3·RetrievalFull course
Retrieval at Scale
Chunking, hybrid search, reranking, retrieval evals and citations.
Leo Martins, Staff engineer6 lessons · 2.7h- ReadRetrieval Architecture Choices20mLocked: part of the full course
- TaskA Chunker and BM25 From Scratch40mLocked: part of the full course
- TaskHybrid Retrieval With Rank Fusion and Filters35mLocked: part of the full course
- ReadMeasuring Retrieval16mLocked: part of the full course
- TaskGrounded Answers With Verified Citations40mLocked: part of the full course
- QuizModule Quiz: Retrieval at Scale10mLocked: part of the full course
- 0/6E4·EvalsFull course
Evals as Engineering
Golden sets, LLM-as-judge calibration and regression gates in CI.
Leo Martins, Staff engineer6 lessons · 2.8h- ReadEvals Are the Job18mLocked: part of the full course
- TaskA Golden-Set Runner That Tells the Truth40mLocked: part of the full course
- ReadLLM-as-Judge Done Right18mLocked: part of the full course
- TaskA Pairwise Judge You Can Trust40mLocked: part of the full course
- TaskA CI Regression Gate35mLocked: part of the full course
- QuizModule E4 Quiz14mLocked: part of the full course
- 0/6E5·ProductionFull course
Production Concerns
Streaming, latency budgets, prompt caching, rate limits, fallbacks, observability and cost.
Leo Martins, Staff engineer6 lessons · 2.4h- ReadLatency Budgets: Where the Seconds Go16mLocked: part of the full course
- TaskA Retry Policy You Can Defend40mLocked: part of the full course
- ReadCost at Scale: Caching, Routing, Batch and the Number That Matters18mLocked: part of the full course
- TaskA Model Router With a Daily Budget40mLocked: part of the full course
- ReadObservability and Graceful Degradation18mLocked: part of the full course
- QuizModule E5 Quiz14mLocked: part of the full course
- 0/9E6·System designFull course
LLM System Design Interview
Design rounds for LLM systems: requirements, architecture, evals, failure modes and cost, graded against an interviewer's rubric.
Leo Martins, Staff engineer9 lessons · 5.2h- ReadHow the LLM Design Round Works16mLocked: part of the full course
- ReadWorked Example: An Internal Knowledge Assistant20mLocked: part of the full course
- WriteDesign It: A Support Agent at 10,000 Tickets a Day45mLocked: part of the full course
- WriteDesign It: Document Q&A at p95 Under 2 Seconds40mLocked: part of the full course
- WriteDesign It: An Eval Platform for Weekly Changes40mLocked: part of the full course
- TalkDesign Interview: A Multi-Tenant LLM Gateway45mLocked: part of the full course
- TalkDesign Interview: An Autonomous Coding Agent45mLocked: part of the full course
- WriteDesign It: Multimodal Document Intake for a Customs Broker45mLocked: part of the full course
- QuizModule E6 Quiz14mLocked: part of the full course
- 0/11E7·Mock loopFull course
Mock Interview Loop
A full practice loop: timed progressive coding, system design, project deep dive and values.
Leo Martins, Staff engineer11 lessons · 6.7h- ReadHow to Run Your Mock Loop12mLocked: part of the full course
- DrillRound 1: A Prompt-Template Registry90mLocked: part of the full course
- WriteRound 2 Prep: Design a Pull-Request Review Assistant (Written)45mLocked: part of the full course
- TalkRound 2, Live: The PR Review Assistant Design Interview45mLocked: part of the full course
- WriteBefore Round 3: Your Six-Line Project Summary15mLocked: part of the full course
- TalkRound 3: Technical Deep Dive30mLocked: part of the full course
- TalkRound 4: Experiences and Goals30mLocked: part of the full course
- TalkRound 5: The Customer-Engineer Session30mLocked: part of the full course
- WriteTake-Home: Build It, Measure It, Write the README60mLocked: part of the full course
- TalkRound 6: The Take-Home Panel Walk-Through30mLocked: part of the full course
- ReadYour Loop Scorecard and Next Steps15mLocked: part of the full course
- 0/15E8·Gannet Home EnergyFull course
Agent Evals and Reliability
Evaluate agents the way they fail in production: end-state and trajectory grading in a sandbox, pass^k with clustered intervals, simulated users, error analysis into a ranked taxonomy, overfitting-safe hill-climbing, synthetic and adversarial suites with attack success rate as a gate, rerun policies for flaky cases, and a broken harness to debug. Builds on E4.
Teodora Kessler, Head of AI Quality15 lessons · 7.9h- ReadAgent Evals: End State, Trajectory and pass^k20mLocked: part of the full course
- TaskGrade 30 Agent Runs and Report pass^340mLocked: part of the full course
- ReadSimulated Users and Multi-Turn Evals16mLocked: part of the full course
- TaskA Simulated-User Harness for a Support Agent40mLocked: part of the full course
- TaskFrom 60 Failures to a Ranked Taxonomy40mLocked: part of the full course
- ReadHill-Climbing Without Overfitting16mLocked: part of the full course
- DrillDrill: The Frozen-Test Protocol45mLocked: part of the full course
- ReadSynthetic Data and Adversarial Suites18mLocked: part of the full course
- TaskSynthetic Cases from Real Seeds40mLocked: part of the full course
- TaskScore an Injection Suite Against Two Defences40mLocked: part of the full course
- TaskFlaky or Regression? A Rerun Policy35mLocked: part of the full course
- TaskDebug a Broken Eval Harness40mLocked: part of the full course
- WriteWritten: An Eval Plan for a New Agent Product45mLocked: part of the full course
- TalkRole-Play: The VP Who Wants to Ship on 20 Examples25mLocked: part of the full course
- QuizModule E8 Quiz12mLocked: part of the full course
- 0/12E9·Tallowmere Industrial Supply (fictional)Full course
Multimodal Input, Streaming and Throughput at Scale
Route PDFs and phone photos into Claude with page-verified extraction and an abstain path, ground answers on the native Citations API, parse the raw streaming protocol with cancellation and TTFT, run async pools and Message Batches jobs that respect rate limits, and find the exact byte that breaks a prompt cache.
Imani Ferrowell, VP of Order Operations12 lessons · 6h- ReadDocuments and Images as Input18mLocked: part of the full course
- TaskPurchase-Order Intake with an Abstain Path45mLocked: part of the full course
- ReadNative Citations: What the API Guarantees15mLocked: part of the full course
- TaskRebuild a Grounded Answer on Native Citations40mLocked: part of the full course
- ReadThe Streaming Protocol, Event by Event16mLocked: part of the full course
- TaskAn SSE Parser with Cancellation and TTFT45mLocked: part of the full course
- ReadThroughput at Scale: Async Clients, Rate Limits and Batches16mLocked: part of the full course
- TaskAn Async Backfill That Respects Rate Limits40mLocked: part of the full course
- TaskA Message Batches Job, End to End40mLocked: part of the full course
- TaskFind the Byte That Breaks the Cache35mLocked: part of the full course
- WriteWritten Design: Streaming UX for the Order Desk35mLocked: part of the full course
- QuizModule Quiz: Multimodal, Streaming and Scale12mLocked: part of the full course
- 0/12E10·Lumenfold Media (fictional)Full course
Inference, Embeddings and Model Choices
What happens below the API: prefill and decode, the KV cache, batching and rate-limit pools; embeddings and approximate search; when fine-tuning or distillation is justified; OpenTelemetry for LLM calls; and choosing and migrating between Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1.
Noor Haddad, Head of ML Platform12 lessons · 5.5h- ReadInference and Serving: What Happens Between the Request and the First Token16mLocked: part of the full course
- TaskToken Buckets, Priorities and Cost per Tenant45mLocked: part of the full course
- TalkDesign Interview: Serve an Open-Weight Model Next to Claude45mLocked: part of the full course
- ReadEmbeddings Internals: Why Dense Retrieval Misses, and What to Do About It16mLocked: part of the full course
- TaskBrute Force Versus IVF, Measured40mLocked: part of the full course
- ReadFine-Tuning and Distillation: When, With What Data, and How You'd Know15mLocked: part of the full course
- WriteWrite It: Should Lumenfold Fine-Tune for Headlines?30mLocked: part of the full course
- ReadObservability: Mapping Your LLM Logs onto OpenTelemetry15mLocked: part of the full course
- TaskTurn Call Logs into OpenTelemetry Spans35mLocked: part of the full course
- ReadChoosing Models and Migrating Safely: Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.118mLocked: part of the full course
- TaskA Linter for the Opus 5.5 Migration40mLocked: part of the full course
- QuizModule Quiz: Inference, Embeddings and Model Choices12mLocked: part of the full course
- 0/9E11·Fernhollow Travel (fictional)Full course
TypeScript for Claude
The TypeScript side of the job, run locally with Node (the browser course runs Python only): typed Messages API calls, streaming to a Next.js route handler, tool loops, structured outputs and prompt caching with @anthropic-ai/sdk, and an MCP server with @modelcontextprotocol/sdk. Practised through code-reading checks, a code review, a Python-to-TypeScript port and a local lab.
Ada Wexley, Staff Engineer9 lessons · 3.7h- ReadTypeScript With Claude: Setup, Types and Your First Call15mLocked: part of the full course
- ReadStreaming Claude to a Next.js Route Handler15mLocked: part of the full course
- ReadTool Loops, Structured Outputs and Prompt Caching in TypeScript18mLocked: part of the full course
- ReadAn MCP Server in TypeScript15mLocked: part of the full course
- QuizCode-Reading Check: What Does This Do?15mLocked: part of the full course
- WriteWrite It: Review Fernhollow's Assistant Route35mLocked: part of the full course
- WriteWrite It: Port a Python Tool Loop to TypeScript40mLocked: part of the full course
- WriteLab: Run the TypeScript Kit on Your Machine60mLocked: part of the full course
- QuizModule Quiz: TypeScript for Claude10mLocked: part of the full course
- 0/14E12·Kestrel Code (fictional)Full course
LLM Inference & Serving in depth
One layer below E10: the inference pipeline and the roofline, KV-cache size math, GQA, paged KV and prefix caching, continuous batching and chunked prefill, speculative decoding and quantization, tensor, pipeline and expert parallelism, the levers you control on a managed API like Claude's, and the latency and cost playbook that decides between self-hosting and the API.
Ines Moreau, VP of Infrastructure14 lessons · 6.6h- ReadThe Inference Pipeline: Prefill, Decode and the Roofline20mLocked: part of the full course
- ReadThe KV Cache: Size Math, GQA, Paging and Prefix Caching20mLocked: part of the full course
- TaskKV-Cache Memory and How Many Conversations Fit35mLocked: part of the full course
- ReadBatching and Scheduling: Continuous Batching, Chunked Prefill and Disaggregation20mLocked: part of the full course
- TaskSimulate Static vs Continuous Batching45mLocked: part of the full course
- ReadMaking Decode Faster: Speculative Decoding, Quantization and Fused Kernels20mLocked: part of the full course
- TaskIs Speculative Decoding Worth It?35mLocked: part of the full course
- ReadParallelism and Hardware: Splitting a Model, MoE and the GPU Memory Budget20mLocked: part of the full course
- ReadServing Stacks and the API View: What You Control When Someone Else Runs the GPUs20mLocked: part of the full course
- ReadThe Latency and Cost Playbook: SLOs, Load Tests, Capacity Math and Self-Host vs API20mLocked: part of the full course
- TaskCapacity Planner, GPUs vs the API45mLocked: part of the full course
- TalkDesign Interview: An Inference Tier for a Coding Assistant45mLocked: part of the full course
- WriteWrite It: Self-Host or API? A Memo to the CTO35mLocked: part of the full course
- QuizModule Quiz: LLM Inference and Serving15mLocked: part of the full course
Unlock every track.The first lessons of every track are free. Get all four tracks, every graded task and three mock loops with a one-time payment.
Get full access · $249