Ensemble Methods

Understanding Ensemble Methods

An ensemble combines predictions from several fitted models into one prediction. Its purpose is to improve generalization: performance on new observations, not merely on training examples. Combining models does not guarantee beating the strongest member. The benefit depends on member quality and on whether their errors differ. Five models making the same mistake offer little protection against that mistake.

The scikit-learn guide distinguishes voting, bagging, random forests, boosting, and stacking. Voting is the simplest starting point. In hard voting, classifiers choose labels and the ensemble selects the most frequent label. Soft voting averages predicted class probabilities before choosing a class; those probabilities should be reasonably calibrated. For regression, averaging numerical predictions plays a similar role. Weights can give selected members more influence, but choosing weights requires validation rather than guesswork.

Bagging trains separate models on resampled training sets, usually bootstrap samples drawn with replacement. Averaging can reduce variance: sensitivity to the particular observations used for training. Deep decision trees often benefit because small data changes can substantially alter their splits. Random forests add another source of diversity by considering random subsets of features at splits. Their trees can be trained independently, making parallel computation possible. More trees generally increase memory use and prediction cost, with diminishing benefits.

Boosting builds an additive model sequentially. In gradient boosting, each new learner approximates a direction that reduces the current model's loss; for squared-error regression, this corresponds to fitting residuals. Classification uses loss-dependent updates, not simply repeated fitting to incorrect labels. The learning rate controls each learner's contribution, while the number and complexity of learners control capacity. Smaller learning rates often require more stages. Excessive capacity can still overfit.

Stacking learns how to combine base-model predictions using a final estimator, also called a meta-model. Its training inputs should normally be out-of-fold predictions: predictions for observations excluded from the corresponding base-model fit. Otherwise, the final estimator learns from unrealistically optimistic inputs. Stacking offers flexibility but adds fitting cost and evaluation complexity. No ensemble family or boosting library is universally best; compare candidates against a simple baseline using splits and metrics appropriate to the problem.

Practical walkthrough

Use the example to inspect aggregation before attempting a large forest or boosting search. Prerequisites are Python 3.10 or later and an already installed, compatible scikit-learn release, such as 1.4 or later. The program creates all data in memory and performs no downloads. Familiarity with lists, fitting, and prediction is sufficient.

First, inspect the training data. Each observation has one numeric feature, so every row is a one-element list. Labels are binary. The two query observations are separate from training; their labels are deliberately omitted because this demonstration explains predictions rather than estimating accuracy. No scaling or other learned preprocessing is needed for this single-feature illustration.

Next, create three nearest-neighbor classifiers using one, three, and five neighbors. These are different model configurations, not bootstrap samples. Consequently, this is a voting ensemble, not bagging. Each classifier receives the same training observations. Varying the neighborhood size lets the members disagree about the boundary between classes.

Fit the VotingClassifier, then inspect its estimators_ attribute. These are the fitted member estimators created by the ensemble. Print each member's predictions in the original estimator order, followed by the ensemble's predictions. With three members and two classes, equal-weight hard voting cannot produce a tied class count.

For a real comparison, reserve an untouched test set and choose configurations using validation or cross-validation within training data. Fit any scaler or imputer only within each training fold, preferably through a pipeline. Respect time order or entity groups when required. Then compare the selected ensemble with its members on the same evaluation observations.

Worked example

from sklearn.ensemble import VotingClassifier
from sklearn.neighbors import KNeighborsClassifier

X_train = [[0.0], [2.0], [4.0], [6.0], [8.0]]
y_train = [0, 0, 1, 1, 1]
X_query = [[2.9], [5.1]]

ensemble = VotingClassifier(
    estimators=[
        ("near", KNeighborsClassifier(n_neighbors=1)),
        ("middle", KNeighborsClassifier(n_neighbors=3)),
        ("wide", KNeighborsClassifier(n_neighbors=5)),
    ],
    voting="hard",
)
ensemble.fit(X_train, y_train)

member_predictions = [
    model.predict(X_query).tolist()
    for model in ensemble.estimators_
]
print("Members:", member_predictions)
print("Ensemble:", ensemble.predict(X_query).tolist())

Expected output

Members: [[0, 1], [0, 1], [1, 1]]
Ensemble: [0, 1]

For query 2.9, the closest training point is 2.0, labeled 0. The three closest points are 2.0, 4.0, and 0.0, whose labels give class 0 two votes. Using all five training points instead favors class 1 by three labels to two. The ensemble therefore receives member votes 0, 0, and 1 and returns 0. At 5.1, all three members predict 1. Distances and class counts produce no relevant ties, so the printed output is deterministic without a random seed. This verifies the voting mechanism; without query labels or representative evaluation data, it establishes neither accuracy nor superiority over individual models.

Common mistakes and how to avoid them

• Assuming more models automatically mean better predictions.

  Compare against the strongest individual model on validation data. Inspect where members disagree and whether their errors overlap; duplicated predictions add computational cost without useful diversity.

• Treating learning_rate and n_estimators as interchangeable boosting controls.

  Tune them together on training-only validation splits. Reducing the learning rate weakens each stage's contribution, so the same number of stages may underfit; adding stages can eventually overfit.

• Training a stacking meta-model on base models' in-sample predictions.

  Generate out-of-fold predictions for meta-model training. Keep preprocessing inside those folds, and never use final test labels to fit the combiner or choose its settings.

• Calling every combination of models a random forest.

  Identify how members are trained and combined. This example varies neighbor counts and votes; a random forest trains randomized trees and, in scikit-learn classification, averages their predicted class probabilities.

Review questions

Further reading

• scikit-learn user guide