Understanding Decision Trees
A decision tree learns a sequence of feature tests that partitions the training examples into groups. The root contains every training row. Each internal node asks a question such as “is temperature less than or equal to 3.5?” Rows satisfying the condition go left; others go right. A terminal node, called a leaf, supplies the prediction. Prediction follows one path, rather than comparing a new row with every training example.
For classification, training seeks splits that make the resulting groups more homogeneous in their labels. The default criterion in scikit-learn’s DecisionTreeClassifier is Gini impurity: G = 1 − sum(p_k²), where p_k is the fraction of examples belonging to class k at that node. A node containing equal numbers of two classes has impurity 0.5; a node containing only one class has impurity zero. These fractions describe training labels, not uncertainty established by an independent experiment.
For each candidate split, the algorithm compares the parent impurity with the weighted average of the two child impurities. Weighting by child sample counts matters: isolating one pure example should not outweigh leaving almost everyone else mixed. With the default best splitter, the tree selects the largest available impurity decrease, then repeats within each child. This is greedy optimization: the best immediate split need not produce the globally best tree. Numerical thresholds typically lie between adjacent distinct training values.
An unweighted classifier predicts the most frequent class in its leaf; predict_proba returns the leaf’s class proportions. A regression tree instead predicts a number, commonly the leaf’s mean target under squared-error loss. Both create piecewise-constant predictions with boundaries aligned to individual features. Trees capture interactions because later tests depend on earlier branches, but regression trees do not extrapolate a continuing trend beyond observed targets.
Unrestricted growth can produce tiny leaves that memorize noise. max_depth limits path length; min_samples_leaf requires support for each resulting leaf. Cost-complexity pruning, controlled by ccp_alpha, trades fit against tree size. Stronger restrictions can underfit; weaker ones can overfit. Small trees are inspectable, but their thresholds may change substantially after small data changes. An understandable predictive rule is still not evidence of causation.
Practical walkthrough
Start by defining one row, the prediction target, and when each feature becomes available. For example, a feature recorded after the outcome would leak information even if the train/test split were otherwise correct. The demonstration below uses six invented numeric observations and binary labels solely to expose the splitting calculation; it is not a performance benchmark.
Keep evaluation observations separate before fitting anything. For ordinary independent observations, a stratified split can preserve class representation. Repeated observations from the same person may require group separation; forecasting usually requires chronological separation. These choices express assumptions about what genuinely new data will look like. A random split is not automatically appropriate.
Prepare features using training data only. Trees generally do not need standardization because their tests depend on ordering, not distances. However, string categories still need suitable numeric encoding for scikit-learn trees. Arbitrary category numbers impose an artificial order; one-hot encoding is often a safer starting point for unordered categories. If imputation or encoding learns information from data, place it in a pipeline and fit it within each training fold.
Fit a deliberately shallow baseline, then inspect its threshold, leaf sizes, and predictions. The example fixes max_depth to one, producing a stump: a root test and two leaves. Its probe inputs are not evaluation data because they have no reference labels.
For a real task, compare depth, minimum leaf size, or pruning strength using validation or cross-validation on training data. Choose a metric appropriate to error costs and class balance. Evaluate the selected pipeline once on untouched test data. Do not interpret training purity as proof of generalization.
Worked example
from sklearn.tree import DecisionTreeClassifier
# Fixed teaching data: one numeric feature per row.
X_train = [[1.0], [2.0], [3.0], [4.0], [5.0], [6.0]]
y_train = [0, 0, 0, 1, 1, 1]
model = DecisionTreeClassifier(max_depth=1, random_state=0)
model.fit(X_train, y_train)
# Unlabeled probes illustrate routing, not test accuracy.
X_probe = [[2.5], [3.5], [4.5]]
tree = model.tree_
left, right = tree.children_left[0], tree.children_right[0]
weighted_gini = (
tree.n_node_samples[left] * tree.impurity[left]
+ tree.n_node_samples[right] * tree.impurity[right]
) / tree.n_node_samples[0]
print(f"threshold: {tree.threshold[0]:.2f}")
print(f"root Gini: {tree.impurity[0]:.2f}")
print(f"weighted child Gini: {weighted_gini:.2f}")
print(f"Gini decrease: {tree.impurity[0] - weighted_gini:.2f}")
print("predictions:", model.predict(X_probe).tolist())
print("class order:", model.classes_.tolist())
for row in model.predict_proba(X_probe):
print("probabilities:", ", ".join(f"{p:.2f}" for p in row))
Expected output
threshold: 3.50
root Gini: 0.50
weighted child Gini: 0.00
Gini decrease: 0.50
predictions: [0, 0, 1]
class order: [0, 1]
probabilities: 1.00, 0.00
probabilities: 1.00, 0.00
probabilities: 0.00, 1.00
Run this example in Python with scikit-learn installed; it requires no downloads or preprocessing. The root has three examples of each class, so its Gini impurity is 1 − 0.5² − 0.5² = 0.5. Splitting at 3.5 creates two pure leaves, each containing three observations. Their weighted impurity is zero, giving a decrease of 0.5. The probe exactly at 3.5 goes left because the comparison includes equality. Probability columns follow classes_, here zero then one. Values of 1.00 reflect pure training leaves, not guaranteed correctness on future cases. No accuracy claim follows from these unlabeled probes.
Common mistakes and how to avoid them
Treating a depth limit as guaranteed protection against overfitting.
Validate the limit alongside min_samples_leaf or pruning strength. Compare training and validation results; even a shallow tree can exploit accidental patterns in a small dataset.
Selecting tree settings by repeatedly checking test accuracy.
Use training-fold cross-validation for selection, including preprocessing. Reserve test labels for final evaluation so model choices do not adapt to the test set.
Reading a pure leaf as certainty or a causal rule.
Inspect its sample count and evaluate held-out behavior. Leaf proportions summarize observed labels; changing the split feature is not proven to change the outcome.
Assuming readable rules must be stable.
Compare fitted rules across validation folds. If thresholds or chosen features change sharply, report that instability rather than presenting one tree as the unique explanation.
Review questions
Why weight child impurities by sample count?
The split should account for every training row. Without weighting, a tiny pure child could appear as influential as a large, highly mixed child.
What makes decision-tree training greedy?
Each node chooses its best immediate split without searching every possible future tree. Later splits refine that choice but do not guarantee a globally optimal structure.
How does min_samples_leaf differ from max_depth?
max_depth limits how many tests a path can contain. min_samples_leaf limits how few training observations may support each leaf, directly restricting small isolated groups.
Why does the example report predictions but no accuracy?
Its probe rows have no known labels. Accuracy requires comparing predictions with reference outcomes; training purity alone cannot establish performance on unseen observations.
