Cheat sheet

Part 4 · Evaluation Metrics — Cheat Sheet

Six illustrated pages — foundations, classification, regression, probability estimation, clustering, and model validation.

← Read the full post (Part 4)Updated June 2026
Part 4 · Evaluation Metrics — Cheat Sheet — printable cheat sheet
Download PNG

Or read the searchable version below.

This six-page cheat sheet covers all of Session 7-8. Page 6 (Model validation) is the focus of Part 5 — Model Validation & Cross-Validation and is included here as context.

Page 1 · Foundations

Page 1 of 6 of the Evaluation Metrics cheat sheet. Eleven numbered concepts covering: (1) model evaluation — scoring model performance; (2) model validation — checking whether performance generalizes to unseen / production data; (3) metric-business-goal alignment — the metric must match the problem; (4) problem families need different metrics — classification, scoring, probability estimation, clustering; (5) classification problems — predicting labels; (6) scoring / regression problems — predicting numbers; (7) no-target problems — finding structure without labels; (8) probability estimation problems — predicting class probabilities, not only labels; (9) model parameters — learned from data; (10) hyperparameters — chosen / tuned around training, not learned directly; (11) train / validation / test sets — three splits for learn, tune, and final unbiased check.
The setup behind every evaluation decision: what you're measuring (evaluation vs validation), what kind of problem you have, and the data splits you'll need.

Page 2 · Classification metrics

Page 2 of 6. Thirteen concepts: (12) confusion matrix — TP, FP, FN, TN; (13) accuracy = correct predictions / total; (14) why accuracy fails on imbalanced data — 1% positives example, model predicts all negative, accuracy 99% with 0% recall; (15) precision = TP / (TP + FP) — among predicted positives how many are correct; (16) recall / sensitivity = TP / (TP + FN) — among real positives how many are found; (17) specificity = TN / (TN + FP) — among real negatives how many are correctly rejected; (18) precision-recall trade-off — lowering the threshold raises recall and drops precision; (19) F1 = 2PR / (P + R) — harmonic balance of precision and recall; (20) MCC — Matthews Correlation Coefficient, balanced binary-classifier quality, range -1 to +1; (21) accuracy vs recall vs MCC comparison — why MCC is better under imbalance with a four-scenario table; (22) Cohen's kappa = (P(o) - P(e)) / (1 - P(e)) — better than random guessing; (23) what to do with imbalanced data — collect more data, change metric, resample, SMOTE, trees, anomaly detection; (24) choosing the right classification metric by application — accuracy / precision / recall / specificity depending on the business question.
How to judge a classifier, especially when classes are imbalanced. The confusion matrix is the foundation; every metric is just a different ratio of its four cells.

Page 3 · Regression / scoring metrics

Page 3 of 6. Six concepts: (25) residuals e_i = y_i - ŷ_i — prediction error as the basis of regression metrics; (26) RMSE = sqrt((1/n) sum e_i²) — squared-error metric, same units as the target, penalizes large errors heavily; (27) R-squared / coefficient of determination R² = 1 - RSS / TSS — fraction of target variation explained, range (-∞, 1], 1 = perfect, 0 = no better than mean, < 0 = worse than mean; (28) TSS / RSS decomposition — TSS = total sum of squares, RSS = residual sum of squares, explained = TSS - RSS, more explained variation → higher R²; (29) MAE = (1/n) sum |e_i| — absolute error, less punitive to large errors than RMSE, more robust to outliers; (30) why absolute error is often not a good project objective — absolute error loss L(e) = |e| is non-smooth at zero, squared error loss L(e) = e² is smooth — reportable but not always ideal to optimize.
Measuring numeric prediction error. Every regression metric is a different way of squashing the residual vector into one number.

Page 4 · Probability-estimation metrics

Page 4 of 6. Five concepts: (31) probability outputs / confidence scores — models can output probabilities, not only labels (example: four observations with scores 0.05, 0.30, 0.70, 0.90); (32) decision thresholds — convert probabilities into predictions by cutting at a threshold (default 0.50), changing the threshold changes precision and recall; (33) ROC curve — trade-off between TPR and FPR across thresholds, plotted against a random-baseline diagonal; (34) AUC — area under the ROC curve summarises overall discriminative ability, 0.5 = random, 1.0 = perfect, higher is better but harder to explain to business users; (35) double density plots — show score distributions for actual positives and negatives overlaid, the overlap region = mistakes / confusion, easier visual explanation of threshold mistakes.
Evaluating scores and thresholds. When the model outputs a probability instead of a label, where you put the threshold is its own decision.

Page 5 · Clustering evaluation

Page 5 of 6. Four concepts: (36) choosing k in clustering — try different values, plot within-cluster cost (WCSS) and find the elbow (k = 3 or 4 in the example); (37) compactness / separation — intra-cluster distance should be smaller than inter-cluster distance, good clustering = tight groups + clear separation; (38) silhouette — cohesion vs separation, range -1 to 1, -1.0 = poor (wrong cluster), 0 = okay (overlap), 1.0 = good (well separated), higher silhouette = points are closer to their own cluster than to others; (39) trying different clustering algorithms — k-medians (L1, minimizes Manhattan distances, robust to outliers), k-medoids (PAM, uses actual data points as centers, robust to noise), Gaussian mixtures (GMM, soft assignments, captures overlap), density-based (DBSCAN, finds dense regions of arbitrary shape, marks low-density points as noise).
Judging structure without labels. Clustering metrics measure how compact your clusters are and how distinct they are from one another.

Page 6 · Model validation / generalization

Page 6 of 6. Thirteen concepts: (40) model problems — bias (model too simple), variance (model too sensitive), overfit (good on train, poor on new data), nonsignificance (observed improvement likely due to chance); (41) bias-variance trade-off — increase complexity → lower bias, higher variance, find the sweet spot with lowest test error; (42) training error vs test error — training error can dramatically underestimate test error; (43) train-test split is not enough — if you keep choosing models using the test set, it becomes 'used' and unreliable; (44) hold-out validation — 60% train / 20% validation / 20% test, test set never used during model selection; (45) single split vs multiple splits — one split can be lucky or unlucky, multiple splits are more reliable; (46) cross-validation resampling — create many train / validation splits, evaluate on each, then average; (47) k-fold cross-validation — split data into k folds, each fold becomes validation once, average metric over k runs is the CV score; (48) LOOCV / repeated k-fold — leave-one-out (each example out once), or repeated k-fold (R shuffles); (49) retraining with best parameters — after CV, choose best hyperparameters, retrain on full training data, deploy; (50) bootstrapping — original sample → resample with replacement into Sample 1, 2, …, B → useful for small samples and confidence intervals; (51) significance testing / p-values — compare model to a naive / null model, low p-value (e.g. < 0.05) suggests the improvement is significant; (52) distribution drift — training (historical) distribution vs live (current) distribution can change over time, harming performance, monitor and adapt.
Checking whether performance holds up on new data. Validation is the discipline of making sure the score you compute on your dataset survives contact with production.