Machine Learning · Chapter 25 of 40
Confusion Matrix
A CONFUSION MATRIX shows TRUE vs PREDICTED labels for each class. Four cells for binary classification: TP, FP, TN, FN.
From it you can compute accuracy, precision, recall and F1.
Example 1 (python)
from sklearn.metrics import confusion_matrix
print(confusion_matrix(y_test, y_pred))
Output
[[85 5]
[ 8 42]]
Rows: true. Columns: predicted.
Example 2 (python)
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))
Output
precision recall f1-score support...
Full metrics per class.
Key points
- Shows true vs predicted labels.
- TP/FP/TN/FN.
- Foundation for precision, recall, F1.
- Read rows as truth, columns as prediction.
💡 Note: In imbalanced problems, accuracy alone lies. Always inspect the confusion matrix and per-class metrics.
Understanding Confusion Matrix
A confusion matrix counts how often predicted categories agree with known categories, and where they disagree. Imagine a classifier that flags damaged packages for inspection. Each package has an actual label and a predicted label. The matrix groups these paired observations into cells, making different kinds of errors visible instead of combining them into one score.
In scikit-learn, rows represent actual labels and columns represent predicted labels. Cell C[i, j] counts observations whose actual class is i and predicted class is j. Always check the axis labels when reading another tool's visualization: orientation is a convention, not something the numbers themselves reveal. The diagonal contains correct predictions; off-diagonal cells contain misclassifications.
For binary classification, first define the positive class. Here, 1 means damaged and 0 means intact; positive does not mean desirable. With label order [0, 1], the first row contains true negatives (TN) and false positives (FP). The second contains false negatives (FN) and true positives (TP). A false positive sends an intact package for inspection. A false negative lets a damaged package pass without being flagged.
Several familiar metrics answer different questions about these counts. Accuracy = (TP + TN) / N measures the fraction correct overall. Precision = TP / (TP + FP) asks what fraction of flagged packages are actually damaged. Recall = TP / (TP + FN) asks what fraction of damaged packages were flagged. F1 = 2TP / (2TP + FP + FN) balances precision and recall through their harmonic mean. It excludes true negatives, so it is not a complete summary of performance.
The matrix assumes aligned observations and meaningful reference labels. It describes results on the evaluated sample, not guaranteed performance on future packages. If most packages are intact, predicting intact everywhere can produce high accuracy while detecting no damage. Inspection costs and missed-damage costs determine which errors matter most; the matrix does not choose that trade-off. It also omits prediction confidence and cannot explain why errors occurred. For multiclass tasks, use one row and column per class: the diagonal still means correct, while each off-diagonal cell identifies a specific actual-to-predicted confusion.
Practical walkthrough
Start by defining the evaluation question: how well do fixed package predictions identify damage? The example requires Python 3.10 or later and an already installed compatible scikit-learn 1.x release. It uses only in-memory lists and the confusion_matrix function. No model training, files, downloads, or random numbers are involved.
Prepare two equally long sequences. y_true contains reference labels; y_pred contains predictions for those same packages in exactly the same order. The supplied values are deliberately constructed teaching data, not evidence about a trained classifier. There are six intact packages and four damaged packages. Explicitly passing labels=[0, 1] makes both axis orders unambiguous and preserves a two-by-two shape even if a class is absent from a future sample.
Compute the matrix with actual labels first and predictions second. Before calculating metrics, check that its entries sum to ten. Check the row totals against the reference counts: six and four. The column totals instead count predictions: five intact and five damaged. These checks help catch misinterpretations, although they cannot prove that individual observations were correctly aligned.
Unpack the flattened matrix as TN, FP, FN, TP only because the specified binary label order supports that arrangement. Calculate the metrics directly to connect each denominator with its meaning. All denominators are nonzero here.
For a real evaluation, obtain predictions from untouched test data. Fit preprocessing and the model using training data only; choose settings and decision thresholds using validation data. Do not repeatedly adjust the classifier after inspecting test errors and continue calling that same test result independent.
Worked example
from sklearn.metrics import confusion_matrix
# 0 = intact; 1 = damaged. Each position is the same package.
y_true = [0, 0, 0, 0, 0, 0, 1, 1, 1, 1]
y_pred = [0, 0, 0, 0, 1, 1, 0, 1, 1, 1]
cm = confusion_matrix(y_true, y_pred, labels=[0, 1])
tn, fp, fn, tp = cm.ravel()
assert int(cm.sum()) == len(y_true)
accuracy = (tp + tn) / cm.sum()
precision = tp / (tp + fp)
recall = tp / (tp + fn)
f1 = 2 * tp / (2 * tp + fp + fn)
print('Rows: actual [0, 1]; columns: predicted [0, 1]')
print(cm.tolist())
print(f'TN={tn}, FP={fp}, FN={fn}, TP={tp}')
print(f'Accuracy: {accuracy:.3f}')
print(f'Precision: {precision:.3f}')
print(f'Recall: {recall:.3f}')
print(f'F1: {f1:.3f}')
Expected output
Rows: actual [0, 1]; columns: predicted [0, 1]
[[4, 2], [1, 3]]
TN=4, FP=2, FN=1, TP=3
Accuracy: 0.700
Precision: 0.600
Recall: 0.750
F1: 0.667
The first four packages contribute true negatives, and the next two contribute false positives. Among the four damaged packages, one is missed and three are flagged. Consequently, seven of ten predictions are correct. Precision is 3/5 because five packages were flagged, but only three were damaged. Recall is 3/4 because the classifier found three of the four damaged packages. F1 is 6/9, displayed rounded to three decimal places. Converting the matrix to a list gives a simple, predictable display. These counts demonstrate the arithmetic only; ten constructed observations cannot establish reliability for a real inspection system.
Common mistakes and how to avoid them
Swapping the function arguments or assuming the top-left cell is TP.
Pass y_true before y_pred and write down the label order. For labels=[0, 1] with 1 positive, the top-left cell is TN.
Treating normalized cells as raw counts.
Keep counts alongside proportions. In scikit-learn, normalize='true' divides by actual-class row totals; normalize='pred' divides by predicted-class column totals. State the denominator whenever reporting a percentage.
Dividing by zero when no packages are predicted positive.
Precision is undefined when TP + FP is zero. Check denominators, or use metric functions with an explicit zero_division policy, and disclose that convention rather than interpreting it as measured precision.
Choosing a threshold solely to improve the test matrix.
Tune thresholds on validation data. Lowering a positive-score threshold can increase detections but also false alarms; compare both counts against operational costs before final testing.
Review questions
What does the value 2 in the example's first row mean?
Two actually intact packages were predicted damaged. They are false positives, not missed damaged packages, because the row specifies actual class 0.
Why are precision and recall different here?
They share three true positives but use different denominators: five predicted positives for precision and four actual positives for recall.
What happens if every package is predicted intact?
The matrix becomes [[6, 0], [4, 0]]. Accuracy is 0.600 and damage recall is zero. Positive precision is undefined because there are no positive predictions.
How does a confusion matrix extend to three classes?
It becomes three-by-three with an explicit class order. Each row describes one actual class; its off-diagonal entries show which alternative classes received its incorrect predictions.
