Understanding Random Forest
A random forest is a supervised learning model that combines many decision trees. Each tree repeatedly divides observations using feature thresholds, such as whether account age exceeds twelve months. A leaf stores a prediction derived from training observations reaching it. Forests support classification, which predicts categories, and regression, which predicts numerical outcomes. Their usefulness comes from combining different, imperfect views of the same prediction problem.
Two sources of randomness encourage diversity. With bootstrap sampling enabled, each tree trains on rows sampled with replacement: some original rows appear repeatedly, while others are omitted. At each split, the algorithm considers a randomly selected set of candidate features. This is not generally one fixed feature subset per tree. Trees can therefore discover different partitions instead of all relying on the same dominant predictor.
For regression, the forest averages tree predictions. Scikit-learn's RandomForestClassifier averages the trees' class probabilities, then predicts the class with the largest average. Averaging helps reduce variance: a single deep tree can change substantially when its training sample changes, whereas a collection of less-correlated trees is usually more stable. This does not remove bias, guarantee superiority over a single tree, or compensate for unrepresentative training data.
For applied data science, forests are useful because they capture nonlinear relationships and interactions without requiring those relationships to be specified in advance. Feature scaling is usually unnecessary because splits compare values within individual columns. However, mixed data types still require deliberate handling. Text categories generally need encoding for scikit-learn forests. Missing-value support depends on the estimator, version, and configuration; an explicit imputation strategy is often useful. Neither encoding nor imputation should learn from held-out data.
Interpret results as predictive associations, not causal effects. Built-in impurity importance summarizes how features improve training splits, but can favor features with many possible split points. Permutation importance measures performance deterioration after shuffling a feature on validation data; correlated predictors complicate its interpretation. Forests also trade interpretability and memory for flexibility. Regression forests generally cannot extrapolate beyond observed target values, and classification probabilities may need calibration before being used as decision risks. Deployment assumes future observations sufficiently resemble the population and measurement process represented during evaluation.
Practical walkthrough
Start by defining one observation, the target, and the prediction time. For customer cancellation, an observation might be an account at the beginning of a month, with cancellation during the following month as the target. Exclude information recorded after that prediction time, even when it is available in your historical table.
Next, reserve test data before exploring model choices. A stratified random split can suit independent classification observations. Repeated customers require grouped splitting; forecasting requires chronological splitting. The evaluation design should resemble the conditions under which predictions will actually be used, rather than merely producing convenient partitions.
Create preprocessing for the training features. For example, combine numeric imputation and categorical one-hot encoding with a ColumnTransformer, then place that transformer and RandomForestClassifier inside a Pipeline. Fit the pipeline only on training data. During cross-validation, fit a fresh pipeline within each training fold so validation rows cannot influence imputation values or learned categories.
Compare against a simple baseline. Select metrics that reflect the task: precision and recall expose different classification errors, while mean absolute error expresses regression error in target units. Accuracy alone can conceal poor detection of a rare class. Choose any decision threshold using validation data, not the final test set.
Tune a small number of meaningful controls. n_estimators changes the number of trees; max_features affects diversity; min_samples_leaf and max_depth constrain individual trees. More trees cost computation and eventually offer diminishing returns. Fix random_state for reproducibility. After selecting the workflow, evaluate once on untouched test data and inspect errors across relevant subgroups.
Worked example
# Standard-library-only demonstration of classification aggregation.
# These illustrative tree outputs are supplied, not learned from data.
from statistics import mean
# Each tuple contains P(no cancellation), P(cancellation).
tree_probabilities = [
(0.40, 0.60),
(0.40, 0.60),
(0.95, 0.05),
]
forest_probabilities = [
mean(tree[c] for tree in tree_probabilities)
for c in range(2)
]
tree_labels = [max(range(2), key=lambda c: p[c])
for p in tree_probabilities]
forest_label = max(range(2), key=lambda c: forest_probabilities[c])
print("Tree labels:", tree_labels)
print("Forest probabilities:",
", ".join(f"{p:.3f}" for p in forest_probabilities))
print("Forest label:", forest_label)
Expected output
Tree labels: [1, 1, 0]
Forest probabilities: 0.583, 0.417
Forest label: 0
This example isolates the aggregation step used by scikit-learn's forest classifier. It does not train trees or estimate predictive performance. Each supplied pair represents one tree's probabilities for the same unseen account, with class 1 meaning cancellation. Two trees individually prefer cancellation, but the forest predicts class 0: the average cancellation probability is (0.60 + 0.60 + 0.05) / 3, or 0.417 after rounding. Thus, averaging probabilities can differ from majority voting over hard labels. In a fitted model, probability columns follow classes_. The resulting probability is a model estimate, not evidence that cancellation occurs at that rate without calibration checks.
Common mistakes and how to avoid them
Treating the forest as a model that accepts any table unchanged.
Inspect feature types, encode categories, and choose a supported missing-value strategy. Put learned transformations inside a pipeline fitted only on training data.
Increasing the number of trees to fix every generalization problem.
More trees mainly stabilize aggregation. Investigate leakage, population shifts, and overly complex individual trees; tune leaf size or depth using appropriate validation splits.
Calling the highest-importance feature the cause of the outcome.
Describe importance as model reliance. Check validation permutation importance and correlated predictors. Establishing causation requires additional assumptions and an appropriate study design.
Repeatedly checking the test score while choosing features or thresholds.
Use training folds and validation data for those decisions. Reserve the test set for a final assessment; repeated selection against it undermines its independence.
Review questions
Why does randomizing trees help an ensemble?
Averaging helps most when trees make different errors. Bootstrap samples and random candidate features reduce similarity between trees, making their combined prediction less sensitive to a particular training sample.
How do max_features and min_samples_leaf address different concerns?
max_features controls candidate features at each split, affecting tree diversity and split quality. min_samples_leaf limits how small leaves become, discouraging predictions based on very few observations.
Why is preprocessing before cross-validation a leakage risk?
Imputation values or category mappings learned from all rows expose validation information to training. A pipeline fitted separately within each fold keeps those learned transformations restricted to that fold's training portion.
What does low permutation importance mean when two features are strongly correlated?
The model may substitute one feature for the other, so shuffling either alone causes little deterioration. Low individual importance therefore does not establish that their shared information is irrelevant.
