Home / ML Series
Machine learning from first principles
Eighteen notebooks. Every model is written in NumPy, with scikit-learn and Stable-Baselines3 used only as benchmarks.
01 / How each model is verified
Check 1Known-answer tests
Synthetic data where the correct output is known in advance.
Check 2Gradient checks
Finite differences on every hand-derived gradient, down to 1e-10.
Check 3Reference agreement
Same inputs through scikit-learn or Stable-Baselines3.
Check 4Independent oracle
A different algorithm for the same answer, like value iteration.
02 / All eighteen
No.AlgorithmDatasetResult
S·01Linear RegressionNormal equation, SVD least squares and gradient descent, matching scikit-learn exactly.California Housing · 20,640 block groupsTest R² 0.4985
S·02Logistic RegressionWeighted loss against SMOTE, thresholds chosen on validation. Powers the Fraud Decision Lab.ULB credit-card fraud · 0.17% positivesAUC-PR 0.741
S·03Decision TreeCART with Gini impurity, tuned for recall on the class where a miss is costly.Breast Cancer Wisconsin · 569 tumoursRecall 0.969
S·04Random ForestBagging with feature subsampling. Found and fixed two data-leakage bugs on the way.Titanic · 891 passengersOOB 82.7%
S·05Gradient BoostingTrees fitted to the negative gradient of log-loss, with shrinkage.UCI Adult Census · 48,790 rowsAUC 0.912
S·06K-Nearest NeighborsShows how feature scaling defines the distance metric itself.Iris · 150 flowersCV 96.7%
S·07Naive BayesMultinomial likelihood in log space with Laplace smoothing.20 Newsgroups · by-date splitAcc 79.7%
S·08Support Vector MachineHinge-loss primal by subgradient descent, matching LinearSVC to four decimals.Breast Cancer WisconsinROC-AUC 0.995
S·09Linear Discriminant AnalysisShared-covariance Gaussians solved with a Cholesky factorisation.IMDb reviews · 2,000 TF-IDF featuresCV 88.0%
S·10Multilayer PerceptronBackpropagation by hand, verified by a gradient check at 7.4e-9 before training.MNIST · 70,000 digitsTest 96.4%
U·01K-MeansLloyd's algorithm as coordinate descent. Thirty seeds revealed seven local optima.Iris · labels held outARI 0.645
U·02Hierarchical ClusteringFour linkages via Lance–Williams, backed by 19 unit tests.Mall Customers · 200 customersARI 1.000
U·03PCAVariance maximisation, eigendecomposition and low-rank reconstruction.UCI Wine · 13 measurementsΔ 1e-14
U·04DBSCANRFM features from 541k transactions. The noise points turned out to be the biggest spenders.UCI Online Retail · 4,335 customersARI 1.0
U·05Gaussian Mixture (EM)EM with Cholesky log-densities for numerical stability.Olivetti Faces · 60 PCA componentsLL matches
R·01Q-LearningReaches the computed optimum on slippery FrozenLake, with value iteration as an oracle.FrozenLake 4×474.9% / 74.4%
R·02Deep Q-NetworkReplay buffer and target network in NumPy, gradient check 2e-9.CartPole-v1 · 5 seeds397.7 return
R·03PPOClipped objective and GAE derived by hand, gradient check 1e-10.LunarLander-v3 · 300k steps164 vs SB3 137