Evaluating ML Models

Use this skill for rigorously assessing model performance, comparing alternatives, diagnosing issues, and optimizing hyperparameters.

When to use this skill

• Model training complete — need systematic performance assessment
• Comparing multiple models/algorithms — statistical model comparison
• Diagnosing overfitting/underfitting — bias-variance analysis
• Hyperparameter tuning — finding optimal configurations
• Selecting appropriate metrics — matching metrics to business objectives
• Experiment tracking — reproducible experimentation
• Production readiness check — validation before deployment

When NOT to use this skill

• Feature engineering and preprocessing → use engineering-ml-features
• Exploratory data analysis → use analyzing-data
• Building interactive data apps → use @building-data-apps
• Notebook setup and workflows → use @working-in-notebooks

Quick tool selection

Task	Default choice	Notes
Cross-validation	sklearn.model_selection	Standard CV, stratified, time series, grouped
Classification metrics	sklearn.metrics	Accuracy, precision, recall, F1, ROC-AUC, PR-AUC
Regression metrics	sklearn.metrics	MAE, RMSE, R², MAPE
Hyperparameter tuning	Optuna	Bayesian optimization with pruning
Distributed tuning	Ray Tune	Large-scale distributed search
Experiment tracking	MLflow	Open-source, model registry
Cloud experiment tracking	Weights & Biases	Collaboration-focused
Model comparison	scipy.stats	Paired t-test, McNemar's test

Evaluation workflow

1. Choose cross-validation strategy

Match CV to your data characteristics:

Data Type	CV Strategy	Implementation
Standard tabular	K-Fold	KFold(n_splits=5, shuffle=True)
Classification (imbalanced)	Stratified K-Fold	StratifiedKFold(n_splits=5)
Time series	Time Series Split	TimeSeriesSplit(n_splits=5)
Grouped/ clustered	Group K-Fold	GroupKFold(n_splits=5)

2. Select appropriate metrics

Classification:

• Balanced: Accuracy
• Imbalanced: F1, Precision-Recall, ROC-AUC
• Cost-sensitive: Custom business metrics

Regression:

• Scale-independent: MAPE, R²
• Error magnitude: MAE (robust), RMSE (penalizes large errors)

3. Analyze performance

• Cross-validation mean ± std (estimate variance)
• Validation curves (bias-variance tradeoff)
• Learning curves (data sufficiency)
• Error analysis by segment

4. Hyperparameter optimization

Use Bayesian optimization for efficient search:

import optuna

def objective(trial):
    params = {
        'n_estimators': trial.suggest_int('n_estimators', 10, 200),
        'max_depth': trial.suggest_int('max_depth', 2, 32, log=True)
    }
    model = RandomForestClassifier(**params)
    return cross_val_score(model, X, y, cv=5).mean()

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=100)

5. Track and compare experiments

Log everything for reproducibility:

• Hyperparameters
• Metrics (CV and test)
• Model artifacts
• Dataset versions

Core implementation rules

1. Always use proper validation

from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=cv, scoring='roc_auc')
print(f"CV AUC: {scores.mean():.3f} (+/- {scores.std() * 2:.3f})")

2. Match metrics to problem type

# Classification with imbalance
from sklearn.metrics import classification_report, roc_auc_score

print(classification_report(y_true, y_pred))
y_proba = model.predict_proba(X_test)[:, 1]
print(f"ROC-AUC: {roc_auc_score(y_test, y_proba):.3f}")

# Regression
from sklearn.metrics import mean_absolute_error, root_mean_squared_error

print(f"MAE: {mean_absolute_error(y_true, y_pred):.3f}")
print(f"RMSE: {root_mean_squared_error(y_true, y_pred):.3f}")

3. Validate on hold-out test set

# Final evaluation on untouched test set
# Never optimize hyperparameters on test data!
y_pred = best_model.predict(X_test)
test_score = accuracy_score(y_test, y_pred)
print(f"Test accuracy: {test_score:.3f}")

4. Analyze errors systematically

# Error by segment
errors = y_pred != y_true
error_df = X_test[errors].copy()
error_df['true'] = y_true[errors]
error_df['pred'] = y_pred[errors]

# Analyze patterns
print(error_df.groupby('category').size())

Common anti-patterns

Anti-pattern	Solution
❌ Single train/test split without CV	Use stratified k-fold for robust estimates
❌ Optimizing accuracy on imbalanced data	Use F1, PR-AUC, or class-balanced metrics
❌ Data leakage in preprocessing	Fit preprocessors on train only, use pipelines
❌ Not checking calibration for probabilities	Use sklearn.calibration.CalibratedClassifierCV
❌ Ignoring inference speed/memory	Profile prediction time, consider model size
❌ No error analysis	Segment errors by features to find patterns
❌ Overfitting validation set	Keep final test set completely untouched
❌ Not tracking random seeds	Set random_state everywhere for reproducibility

Progressive disclosure

Detailed reference guides for specific topics:

• references/cross-validation.md — CV strategies for different data types (K-Fold, Stratified, Time Series, Group K-Fold)
• references/metrics-guide.md — Choosing and interpreting classification and regression metrics
• references/hyperparameter-tuning.md — Optuna and Ray Tune optimization patterns
• references/experiment-tracking.md — MLflow and Weights & Biases setup

Related skills

• @engineering-ml-features — Upstream feature engineering before evaluation
• @orchestrating-data-pipelines — Production model deployment
• @assuring-data-pipelines — Model monitoring in production

References

• scikit-learn Model Selection
• scikit-learn Metrics
• Optuna Documentation
• Ray Tune Documentation
• MLflow Tracking
• Weights & Biases Documentation