Hyperparameter Tuning

Understanding Hyperparameter Tuning

Hyperparameter tuning chooses the settings of a learning algorithm by comparing its performance on data not used to fit each candidate. Model parameters, such as regression coefficients or tree split thresholds, are learned during fitting. Hyperparameters, such as regularization strength or a tree’s minimum leaf size, control how fitting works. They are fixed for an individual fit, but a search procedure changes them between fits.

These settings change the algorithm’s available solutions or its optimization behavior. Increasing a tree’s minimum leaf size prevents splits that would create very small leaves, restricting complexity. Stronger regularization penalizes certain coefficient values more heavily. A learning rate changes the size of optimization updates. Tuning therefore means comparing different learning procedures, not simply trying to maximize training accuracy. A flexible model can fit training observations well while predicting new observations poorly.

Cross-validation provides repeated training and validation comparisons. With K folds, each candidate is fitted K times, holding out a different fold each time. The search averages the validation scores and selects a candidate according to the chosen metric. Every candidate should use the same splits so comparisons reflect settings rather than different validation examples. Ordinary folds assume observations can reasonably be exchanged; repeated subjects require group-aware splits, while forecasting usually requires time-ordered splits.

Grid search evaluates every combination in a specified grid. Three depths and four leaf sizes create twelve candidates, not seven. Randomized search evaluates a chosen number of sampled configurations, making its evaluation budget easier to control. It can explore broad spaces efficiently, but neither method guarantees a nearly optimal result. Search ranges, sampling distributions, and available computation matter. Adaptive methods, including Bayesian optimization, use earlier evaluations to guide later proposals; they still require sound validation.

The winning cross-validation score is a selection result, not an unbiased final evaluation: choosing the highest score also selects favorable noise. Reserve an untouched test set, or use nested cross-validation when evaluating the entire tuning procedure. After selection, refit the winning configuration on all development data. A pipeline must fit preprocessing separately inside each training fold, preventing validation information from influencing scaling, imputation, or feature selection.

Practical walkthrough

Start by defining the prediction task and evaluation metric. For classification, accuracy measures the fraction of correct labels, but it can conceal poor performance on a rare class. Choose the metric before inspecting search results. Also decide which observations must remain together and whether chronological ordering matters. Set aside the test set before any learned preprocessing or tuning.

Next, build a pipeline containing preprocessing and the estimator. Pass this whole pipeline to GridSearchCV, rather than preprocessing the development dataset first. Pipeline parameters use the form step_name__parameter_name. In the example, tree__min_samples_leaf changes the tree’s leaf-size constraint. Standardization is included to demonstrate safe pipeline placement; it is unnecessary for this one-feature tree because rescaling preserves the feature ordering.

Specify a small, defensible search space. The example compares leaf sizes one and four using two explicit stratified folds. Stratification approximately preserves class proportions; it does not solve dependence between observations. Both candidates receive identical folds. Two candidates across two folds require four validation fits, followed by one full-development refit because refit=True. Larger grids multiply this cost quickly.

Run the search using development data only. Inspect cv_results_ to see every candidate’s mean validation score, rather than reporting only best_params_. For real datasets, inspect fold-to-fold variation and computational cost too; a tiny mean advantage may not justify a much slower model.

Finally, predict on the untouched test set with the refitted search object. Treat this evaluation as a final check, not another optimization signal. If you revise choices after seeing test results, those observations are no longer an untouched test set.

Worked example

from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

# Fixed illustrative data; no downloads or random data generation.
X_dev = [[0], [1], [2], [3], [10], [11], [12], [13]]
y_dev = [0, 0, 0, 0, 1, 1, 1, 1]
X_test = [[-1], [14]]
y_test = [0, 1]

pipeline = Pipeline([
    ('scale', StandardScaler()),
    ('tree', DecisionTreeClassifier(random_state=0))
])
cv = StratifiedKFold(n_splits=2, shuffle=False)
search = GridSearchCV(
    pipeline,
    {'tree__min_samples_leaf': [1, 4]},
    scoring='accuracy',
    cv=cv,
    refit=True
)
search.fit(X_dev, y_dev)

for params, score in zip(
    search.cv_results_['params'],
    search.cv_results_['mean_test_score']
):
    leaf = params['tree__min_samples_leaf']
    print(f'min_samples_leaf={leaf}: CV accuracy={score:.2f}')

print('Best leaf size:', search.best_params_['tree__min_samples_leaf'])
predictions = search.predict(X_test)
print('Test predictions:', predictions.tolist())
print(f'Test accuracy: {accuracy_score(y_test, predictions):.2f}')

Expected output

min_samples_leaf=1: CV accuracy=1.00
min_samples_leaf=4: CV accuracy=0.50
Best leaf size: 1
Test predictions: [0, 1]
Test accuracy: 1.00

This example requires scikit-learn. Each fold trains on four observations: two from each class. With minimum leaf size one, a threshold between the two feature clusters separates the classes, including the held-out observations. With minimum leaf size four, splitting would require at least eight training observations, so each fold’s tree remains a single leaf and achieves 0.50 accuracy. The search selects size one and refits on all eight development observations. Scaling statistics are learned separately in each fold. These deliberately simple data explain the algorithm’s constraint; two test observations cannot establish reliable performance for a real application.

Common mistakes and how to avoid them

• Fitting a scaler or feature selector before cross-validation.

  Put learned transformations inside the searched pipeline. Otherwise validation observations influence the representation used for training, even when their labels are not used.

• Treating best_score_ as final generalization performance.

  Use an untouched test set or nested cross-validation. The winning score has already influenced configuration selection and can be optimistic.

• Assuming a larger search always produces a better model.

  Set a budget, use plausible ranges, and inspect score stability. More candidates increase computation and opportunities to select validation noise; they do not guarantee better future predictions.

• Minimizing a negative-loss scorer.

  Scikit-learn search maximizes scores. With neg_mean_squared_error, a score of -2 is better than -5. Negate the reported score when presenting the corresponding positive loss.

Review questions

Further reading

• scikit-learn user guide