Core Concepts & Evaluation

Supervised vs. Unsupervised

Supervised learning trains on labeled input-output pairs to predict outputs for new inputs — classification predicts a discrete category, regression predicts a continuous value. Unsupervised learning finds structure in unlabeled data — clustering (k-means) groups similar items; dimensionality reduction (PCA) compresses features while preserving variance.

Overfitting, Underfitting & the Bias-Variance Tradeoff

from sklearn.model_selection import train_test_split

# Split BEFORE any fitting/scaling — evaluating on training data hides overfitting
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Bias-variance tradeoff:
# - High bias (too simple)  -> underfitting -> poor on BOTH train and test
# - High variance (too complex, fits noise) -> overfitting -> great on train, poor on test
# Total error ~= bias^2 + variance + irreducible error

# Signs of overfitting: train accuracy 99%, test accuracy 70%
# Fixes: regularization, more data, simpler model, early stopping

Evaluation Metrics

from sklearn.metrics import classification_report, roc_auc_score

# Confusion matrix underlies precision/recall/F1:
#                  predicted positive   predicted negative
# actual positive        TP                    FN
# actual negative        FP                    TN
#
# precision = TP / (TP + FP)  — of predicted positives, how many were right
# recall    = TP / (TP + FN)  — of actual positives, how many were found

print(classification_report(y_test, model.predict(X_test)))

# Accuracy is misleading on imbalanced data: 99% negative class means
# "always predict negative" scores 99% accuracy while being useless.
# ROC-AUC is threshold-independent — measures ranking quality across ALL
# thresholds, not just one fixed cutoff (usually 0.5).
auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])

Cross-Validation & Hyperparameter Tuning

from sklearn.model_selection import cross_val_score, GridSearchCV

# K-fold — more robust performance estimate than a single train/test split
scores = cross_val_score(model, X_train, y_train, cv=5)

# Hyperparameters (learning rate, tree depth) are set BEFORE training,
# unlike model parameters (weights) which are LEARNED from data
param_grid = {'max_depth': [3, 5, 10], 'n_estimators': [50, 100, 200]}
grid = GridSearchCV(model, param_grid, cv=5)
grid.fit(X_train, y_train)
best_model = grid.best_estimator_

# Validation set (or CV) tunes hyperparameters — the TEST set stays untouched
# until the very end, so it remains an unbiased final performance estimate.