Understanding Random Forest
A random forest combines decision trees rather than relying on one tree’s boundaries. Each tree repeatedly splits observations using questions such as “Is temperature at most 18?” Its leaves store predictions. Deep trees can capture nonlinear patterns and interactions, but small changes to their training data may produce substantially different structures. This sensitivity is called variance. Combining trees can reduce it, especially when their errors are not strongly correlated.
Two mechanisms encourage variety. By default, scikit-learn forests train each tree on a bootstrap sample: rows drawn from the training set with replacement. Some rows repeat; others are absent from that tree’s sample. During splitting, the tree considers a randomly selected set of candidate features. This selection occurs at individual splits, not simply once for the entire tree. The max_features parameter controls how many features are considered.
For regression, the forest averages numerical tree predictions. For classification, scikit-learn averages the trees’ class-probability estimates and selects the class with the largest average. Describing this as “voting” is useful shorthand, but it is not necessarily a majority vote over each tree’s predicted label. A leaf containing mixed classes contributes probabilities rather than an entirely confident decision.
The main controls affect different trade-offs. n_estimators sets the number of trees: more trees generally stabilize predictions while increasing computation and memory use. max_depth limits tree depth, and min_samples_leaf prevents very small leaves. Restricting tree complexity can reduce overfitting but may miss useful patterns. Lower max_features can make trees less similar, although considering fewer features can also weaken individual trees. No setting guarantees improvement over a single tree or another model.
Forests usually need no feature scaling because splits compare values within individual columns. However, scikit-learn does not automatically interpret arbitrary categorical strings as input features; encode them appropriately. Missing-value support depends on the estimator, version, and configuration, so verify it rather than assuming universal support. Forests also sacrifice the compact explanation of one small tree, and standard forest regression does not extrapolate beyond observed training target values. Treat a forest as a flexible baseline whose usefulness must be checked on representative unseen observations, not as an automatic solution.
Practical walkthrough
Use Python 3.10 or later with scikit-learn 1.4 or later already installed. The example creates every value in memory; it requires no downloaded dataset. X_train contains six observations with two numerical features, while y_train supplies their class labels. Class 0 occupies the low-value region and class 1 the high-value region. The two feature columns deliberately duplicate each other.
First, distinguish training observations from prediction inputs. X_new contains two unseen feature vectors, and neither is passed to fit. They have no supplied labels, so this demonstration produces predictions, not a test accuracy estimate. In a real project, reserve labeled evaluation data before learning preprocessing steps or selecting parameters. Choose splits that respect time order or repeated observations from the same subject when necessary.
Next, construct five trees with max_depth=1. Each tree can make only one split, keeping the mechanism inspectable. max_features=1 means that only one of the two candidate columns is considered at a split. Set random_state to make randomized choices reproducible within a fixed software environment.
Here, bootstrap=False deliberately gives every tree all training rows. Together with duplicated columns, this makes the output easy to derive exactly. This is a teaching configuration, not a recommendation: it removes row resampling and makes the trees equivalent despite randomized feature selection.
Finally, call fit, then predict and predict_proba. Read probability columns using classes_, not an assumed label order. Inspect the tree count through estimators_. For substantive evaluation, compare against a simple baseline, tune using training-only cross-validation, and consult a reserved test set only after selection.
Worked example
from sklearn.ensemble import RandomForestClassifier
X_train = [[0, 0], [1, 1], [2, 2], [8, 8], [9, 9], [10, 10]]
y_train = [0, 0, 0, 1, 1, 1]
X_new = [[1.5, 1.5], [8.5, 8.5]]
forest = RandomForestClassifier(
n_estimators=5,
max_depth=1,
max_features=1,
bootstrap=False,
random_state=42,
n_jobs=1,
)
forest.fit(X_train, y_train)
print("Trees:", len(forest.estimators_))
print("Classes:", forest.classes_.tolist())
print("Predictions:", forest.predict(X_new).tolist())
print("Probabilities:", forest.predict_proba(X_new).tolist())
Expected output
Trees: 5
Classes: [0, 1]
Predictions: [0, 1]
Probabilities: [[1.0, 0.0], [0.0, 1.0]]
Each tree receives the same six training rows. Whichever duplicated feature it considers, the best split is at 5, halfway between 2 and 8. The left leaf contains only class 0; the right contains only class 1. Consequently, all five trees assign the first new observation to class 0 and the second to class 1. Averaging their identical probability vectors leaves those vectors unchanged. The exact output illustrates the prediction interface and aggregation rule, not an improvement from ensembling. Probabilities of 1.0 reflect pure training leaves here; they do not establish certainty about future observations or calibrated real-world risks.
Common mistakes and how to avoid them
Choosing parameters repeatedly based on the test score.
Use cross-validation within training data for selection. Fit encoders, imputers, and feature selectors inside each training fold, preferably through a pipeline; preserve the test set for final evaluation.
Treating feature_importances_ as a causal explanation.
These values summarize training split improvements and can favor features with many possible split points. Consider permutation importance on validation data, account for correlated features, and avoid interpreting either measure as causation.
Assuming additional trees repair unsuitable data or unrestricted complexity.
More trees mainly stabilize aggregation. Investigate label quality, distribution changes, and validation performance; adjust min_samples_leaf or max_depth when individual trees learn overly specific patterns.
Review questions
Why is tree diversity useful?
Averaging helps less when every tree makes the same errors. Bootstrap samples and randomized feature candidates encourage different trees, allowing some errors to cancel rather than accumulate.
How do n_estimators and max_depth differ?
n_estimators controls ensemble size and computational cost. max_depth limits each tree’s complexity. Adding shallow trees does not automatically recover patterns that those trees cannot represent.
Why does this example not demonstrate variance reduction?
Every tree sees all rows and equivalent feature columns, producing the same split. It demonstrates aggregation mechanically, but provides neither diverse trees nor repeated evaluations of prediction variability.
What does a classifier’s score method report, and when is it insufficient?
It reports mean accuracy by default. With imbalanced classes or unequal error costs, inspect class-specific precision and recall, a confusion matrix, and metrics aligned with the actual prediction task.
