Feature Engineering, Models & Pitfalls

Feature Scaling — the Right Way

from sklearn.preprocessing import StandardScaler

# CORRECT — fit scaler on train only, then transform both
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)   # reuses train's mean/std, does NOT refit

# WRONG — fitting on the whole dataset before splitting leaks test-set
# statistics into training, producing an overly optimistic evaluation
# scaler.fit_transform(X)  # then splitting X_scaled afterward — DON'T

# Matters most for distance/gradient-based models (k-NN, SVM, neural nets);
# tree-based models (decision trees, random forests) are scale-insensitive.

Ensembles: Random Forest vs. Gradient Boosting

from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier

# Bagging — many trees trained independently on random subsets, votes averaged
rf = RandomForestClassifier(n_estimators=100, max_depth=10)

# Boosting — trees built SEQUENTIALLY, each correcting the previous
# ensemble's errors (residuals). Often more accurate, more prone to
# overfitting if learning_rate/depth aren't tuned carefully.
gb = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, max_depth=3)

Regularization

from sklearn.linear_model import Lasso, Ridge

lasso = Lasso(alpha=0.1)  # L1 — can drive some weights to exactly zero
ridge = Ridge(alpha=0.1)  # L2 — shrinks weights smoothly, rarely to exactly zero

# Handling class imbalance
from sklearn.utils.class_weight import compute_class_weight
model = RandomForestClassifier(class_weight='balanced')  # weights minority class errors more

Common Pitfalls

Data leakage: information from outside legitimate training data (often the target itself, or future data) leaks into features — e.g. including "account was closed" to predict churn. The model looks great in evaluation and fails in production. Feature selection has the same trap: selecting features using the full dataset before splitting leaks test-set patterns into the decision of which features to keep — nest feature selection inside each CV fold, or after the split, using training data only.