Hyperparameter Tuning

This skill enables an AI agent to systematically search for optimal hyperparameter configurations for machine learning models. It covers defining search spaces, selecting search strategies (grid, random, Bayesian, Hyperband), running trials with cross-validation, applying early stopping to prune poor configurations, and analyzing results to identify the best-performing parameters. The agent balances exploration and exploitation to find strong configurations within a given computational budget.

Workflow

1. Define the search space: Specify each hyperparameter with its type (categorical, integer, float) and range. Use log-uniform distributions for parameters that span orders of magnitude (e.g., learning rate from 1e-5 to 1e-1). Group related parameters and define conditional search spaces where certain parameters only apply when others take specific values.

2. Select the search strategy: Choose the tuning algorithm based on compute budget and search space size. Grid search is exhaustive but only feasible for small spaces. Random search is a strong baseline that scales better. Bayesian optimization (Tree-structured Parzen Estimators or Gaussian Processes) is most sample-efficient for expensive evaluations. Hyperband and ASHA combine early stopping with random search for deep learning workloads.

3. Configure evaluation: Set up k-fold cross-validation (typically 5-fold) for reliable performance estimates on small to medium datasets. For large datasets or expensive models, use a single holdout validation set. Define the objective metric to optimize (e.g., validation F1, AUC-ROC, RMSE) and whether to minimize or maximize it.

4. Run trials with pruning: Execute the search, launching trials in parallel when possible. Enable pruning to terminate underperforming trials early based on intermediate results (e.g., after a few epochs of training), freeing compute for more promising configurations.

5. Analyze and select results: Inspect the optimization history to understand which hyperparameters matter most (importance analysis). Visualize parameter interactions with contour plots or parallel coordinate plots. Select the best configuration and retrain the final model on the full training set with those parameters.

Supported Technologies

• Frameworks: Optuna, Ray Tune, scikit-learn GridSearchCV/RandomizedSearchCV, Hyperopt, Keras Tuner