Deep Reinforcement Learning: MDPs to Policy Gradients

RLHF, AlphaGo, game-playing agents — they all stand on the same foundation: an
MDP that writes sequential decision-making as math, and one update rule that makes
value converge sweep by sweep. This is a flagship build-it-yourself course: no gym,
no GPU. Every world — a grid, a cliff, a continuous cart — is one you assemble in
pure numpy; every algorithm is one you derive by hand, complete a few critical
lines of, and actually run to convergence in your own sandbox. You'll prove value
iteration is a γ-contraction and watch the error shrink by a factor of γ each
sweep; watch SARSA and Q-learning develop two personalities at the cliff edge;
trigger the deadly triad until the weights blow up to infinity, then tame them;
build a mini-DQN with hand-written backprop in pure numpy and watch the return
curve climb; derive the policy gradient theorem, add a baseline, then bolt on a
critic to make actor-critic; and finally turn exploration from intuition into
theory with regret bounds via UCB and Thompson sampling — and see what offline
RL's distribution shift is afraid of. The course follows the knowledge skeleton of
Stanford CS234, but every session is an original hands-on script rewritten for the
online sandbox: formulas typeset, algorithms drawn, key knobs draggable, code run.