Decision Trees

Understanding Decision Trees

A decision tree learns a sequence of questions that maps feature values to a prediction. Imagine classifying a parcel as “standard” or “heavy” using its measured mass. The first question might be “Is mass at most 5 kilograms?” Each answer sends the parcel down a branch. An internal node contains a question; a leaf contains the final prediction. Training chooses the questions from labeled examples rather than requiring you to write the rules yourself.

For numerical features, scikit-learn trees use tests of the form feature <= threshold. During classification training, the algorithm compares candidate splits and favors those that reduce class mixing. The default Gini impurity is 1 minus the sum of squared class proportions. A node containing equal numbers of two classes has impurity 0.5; a node containing only one class has impurity zero. Split quality considers both child impurities, weighted by how many training examples reach each child.

This search is greedy: it chooses a useful split now rather than searching every possible complete tree. A classification leaf predicts its most frequent training class when samples have equal weight. Estimated class probabilities reflect class proportions in that leaf, not guaranteed certainty about future cases. Regression trees follow the same branching idea but predict numerical values; with the usual squared-error criterion, leaves predict their training targets’ mean.

Trees capture nonlinear boundaries and feature interactions without feature scaling. Changing kilograms to grams preserves ordering, so standardization is generally unnecessary. However, numerical input requirements still matter: categorical strings need suitable encoding. A shallow tree can be inspected as a short rule list, while hundreds of leaves quickly undermine that advantage. Small changes in training examples can also change the learned structure.

Depth limits, minimum leaf sizes, and pruning constrain complexity. These controls trade flexibility for stability: too little complexity underfits, while too much can memorize noise. Choose settings using validation data or cross-validation, then evaluate once on untouched test data. The supplied scikit-learn guide organizes these topics under Decision Trees, model selection, and leakage prevention. Random forests and boosting combine trees, often improving predictive performance while sacrificing the simplicity of one readable tree.

Practical walkthrough

Use Python 3.10 or later with a compatible scikit-learn installation; this example uses APIs available in scikit-learn 1.3 and later. Basic familiarity with lists, imports, and method calls is sufficient. All observations are defined in memory, so running the example requires no downloads or external files. Pandas is unnecessary for this one-feature demonstration.

First, define six training parcels. X_train is a list of rows, with one mass measurement per row. Even one feature needs this two-dimensional structure: six samples by one feature. y_train contains six corresponding class labels. These labels are deliberately constructed for teaching, not collected evidence about shipping policy.

Next, keep four separate parcels for testing. Their measurements and labels never enter fit. The fixed split makes the demonstration reproducible, but it is not a representative sampling strategy for a real parcel population. In a real project, choose a split that respects time ordering or repeated entities where relevant.

Construct DecisionTreeClassifier with max_depth=1. This permits one question followed by two leaves, making the entire classifier inspectable. Setting random_state=0 controls internal randomness; it does not make the dataset representative or prevent overfitting by itself. Fit only the training observations. No preprocessing is needed here; if you later add an encoder or imputer, fit it only on training data, preferably within a pipeline.

Finally, print the fitted threshold, predictions, and correct count. Trace each test mass through the question manually. Treat the count as a check of the code’s behavior, not an estimate of deployment accuracy. Do not tune depth against these test results.

Worked example

from sklearn.tree import DecisionTreeClassifier

# One feature: parcel mass in kilograms.
X_train = [[1.0], [2.0], [3.0], [7.0], [8.0], [9.0]]
y_train = ["standard", "standard", "standard", "heavy", "heavy", "heavy"]

# Held out: these observations are never used by fit.
X_test = [[2.5], [5.0], [5.5], [8.5]]
y_test = ["standard", "standard", "heavy", "heavy"]

model = DecisionTreeClassifier(max_depth=1, random_state=0)
model.fit(X_train, y_train)
predictions = model.predict(X_test).tolist()
correct = sum(predicted == actual
              for predicted, actual in zip(predictions, y_test))

print(f"Root threshold: {model.tree_.threshold[0]:.1f} kg")
print(f"Predictions: {predictions}")
print(f"Correct: {correct}/{len(y_test)}")

Expected output

Root threshold: 5.0 kg
Predictions: ['standard', 'standard', 'heavy', 'heavy']
Correct: 4/4

The training classes separate between masses 3 and 7, so the tree chooses their midpoint, 5.0, as its threshold. Before splitting, three examples belong to each class, giving Gini impurity 0.5. Both resulting leaves are pure, so their weighted impurity is zero. Masses 2.5 and 5.0 take the <= branch and receive “standard”; 5.5 and 8.5 receive “heavy.” The four matches are deterministic consequences of this constructed dataset. They do not establish a reliable boundary in the unobserved gap between 3 and 7, nor show that mass causes a real shipping classification. Additional representative observations would be needed to assess generalization.

Common mistakes and how to avoid them

• Passing a flat list of measurements to predict.

  Keep the sample-by-feature shape: use [[2.5], [8.5]] for two parcels with one feature each. A flat list does not specify the required two-dimensional input.

• Increasing depth until training accuracy is perfect.

  Compare depth and min_samples_leaf settings using validation folds. Prefer a simpler tree when validation performance is similar; training perfection can reflect memorized exceptions.

• Using the test set to choose preprocessing or tree settings.

  Reserve the test set for final evaluation. Fit transformations inside each training fold through a pipeline, and choose settings from validation results instead.

• Treating a readable rule as a proven explanation of reality.

  Describe the rule as the model’s prediction procedure. Check data quality and stability across splits; an association learned from labels does not establish causation.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide