Evaltudepreview

Learn the method

A 100-lesson curriculum on evaluating LLM systems.

How to frame the decision, define what correct means, design evaluation data, diagnose failures, and understand what serving choices cost — the reusable engineering that the worked evaluations apply. 10 core parts plus an optional foundation, 100 lessons.

Start the curriculum Start with evaluation foundations

Recommended paths

Evaluation engineer

Build the system → define and evaluate quality → design data → diagnose failures → measure and operate in production.

Part 1 → Part 2 → Part 3 → Part 4 → Part 5 → Part 7 → Part 8 → Part 9
Inference / platform engineer

Build → understand serving → performance & economics → observe → the complete system → improve continuously → (optional) how the weights were trained.

Part 1 → Part 6 → Part 7 → Part 8 → Part 2 → Part 9 → Optional

The parts

  1. Part 1 — LLM Application FoundationsFrom software to LLM engineering · Prompts, schemas and application patterns5 lessons
  2. Part 2 — Complete LLM SystemsThe system, not the model · Extending the model · Reliability and configuration7 lessons
  3. Part 3 — Evaluation FoundationsWhat evaluation is · Defining correctness · Scoring methods8 lessons
  4. Part 4 — Evaluation Data and Dataset DesignSplits and contamination · Representative and adversarial data · Labels and ambiguity8 lessons
  5. Part 5 — Failure Analysis and System ImprovementReading failures · Classifying failures · Where failures live8 lessons
  6. Part 6 — Inference FundamentalsHow a model runs · The KV cache and attention variants · Batching and latency15 lessons
  7. Part 7 — Inference Performance and EconomicsLatency metrics · Throughput and load · The economics11 lessons
  8. Part 8 — Production ObservabilityObservability foundations · Tracing LLM systems · Measuring production quality10 lessons
  9. Part 9 — Continuous EvaluationThe safety net · Comparing and evolving · Scaling judgment10 lessons
  10. Part 10 — Integrated Case StudiesSetting up the case study · Diagnosis and correction · Building, comparing, interpreting12 lessons
  11. Optional Foundation — Build a Small Language ModelData and parameters · Training and generation6 lessons