About this course
A model that hits 95% accuracy in a notebook is still worlds away from shippable. This course does not
teach you to train a bigger model — it teaches you to put a model inside a real system: one that receives
dirty data, meets distributions it never saw, gets depended on by downstream services, and quietly rots
over time. In every session an AI mentor walks you through building a small offline tool by hand — a
system-cost ledger, an acceptance gate, a behavioral test bench, a feature validator, contract tests, a
distribution-drift detector, a monitoring aggregator — all pure Python + numpy, fixed seeds, results in
seconds, verifiable by print. No cloud, no external services, yet each toy is the minimal reproduction of
a load-bearing wall you will actually rely on in production. By the end, the way you look at a model shifts
from "how accurate is it" to "how will it break in production, and how do I catch it first".
What you'll learn
- Decompose an AI system and compute the system-level cost of one model under different decision thresholds
- Hand-compute precision/recall from a confusion matrix, sweep thresholds by cost, and catch data leakage
- Write behavioral tests for a model like software tests (MFT / invariance / directional / slice)
- Wrap a model as a schema'd service, and build a feature validator and contract tests to block bad input and breaking changes
- Build data-quality gates and pipeline expectation tests, and detect training/serving skew
- Detect distribution drift by hand with PSI / KS, aggregate production monitoring metrics, and set alarms
Syllabus
1System Design: From Models to AI-Enabled Systems3 sessions
2Model Quality: Offline Evaluation Done Right3 sessions
3Behavioral Testing: Test a Model Like Software3 sessions
4Serving & Contracts: Wrapping a Model and Testing Its Contract3 sessions
5Data Quality & Pipeline Testing0 sessions
Sessions are on the way.
6Monitoring & Distribution Drift0 sessions
Sessions are on the way.
7MLOps & Responsible ML: Versioning, Reproducibility, Fairness0 sessions
Sessions are on the way.
Same series · 世界名校知名实验室系列
Flagship University Lab Series
Modeled on Stanford / MIT / Berkeley syllabi — learn from scratch with a mentor Agent guiding you in real time.
◎ Modeled on Stanford CS336
LLM from ScratchHand-build every piece of an LLM, along the 17-lecture skeleton of Stanford CS336View course →◎ Modeled on Berkeley’s LLM Agents course
LLM Agents: From Reasoning to Agentic SystemsAlong the 12-lecture skeleton of Berkeley's LLM Agents course — reasoning, frameworks, applications and safety, all hands-onView course →◎ Modeled on Stanford CS329A (Self-Improving AI Agents)
Self-Improving AI AgentsHand-build AI agents that make themselves better, along the skeleton of Stanford CS329AView course →◎ Modeled on MIT’s How to AI (Almost) Anything (multimodal)
Multimodal AI: Teaching Models to See, Hear, and ConnectTurn images, sound and text into tensors by hand — then build alignment, fusion, cross-modal retrieval and multimodal models piece by pieceView course →◎ Modeled on Princeton COS 511 (Theoretical ML)
Learning Theory: PAC, Boosting & Online LearningHand-derive PAC bounds, hand-build AdaBoost, hand-run regret curves — turn learning theory from formulas into experiments you runView course →◎ Modeled on Stanford CS234 (Reinforcement Learning)
Deep Reinforcement Learning: MDPs to Policy GradientsProve convergence by hand, backprop a DQN from scratch, give policy gradient a critic — pure numpy, zero gym, zero GPUView course →