Understanding Model Evaluation
Model evaluation asks how useful predictions are on examples the model did not learn from. A training score measures performance on familiar data; it does not establish generalization. Before calculating anything, define the prediction target, the population you expect to serve, and the action a prediction triggers. These choices determine which errors matter and whether your evaluation data represents the intended use.
Consider a system that flags shipments likely to arrive late. Let 1 mean late and 0 mean on time. A false positive flags an on-time shipment, potentially wasting a support call. A false negative misses a late shipment, potentially leaving a customer unprepared. Accuracy counts both errors equally. That can be inappropriate when their consequences differ or most shipments arrive on time.
A confusion matrix exposes these counts. With labels ordered [0, 1], scikit-learn places actual classes in rows and predicted classes in columns: [[TN, FP], [FN, TP]]. Precision, TP/(TP+FP), answers how many flagged shipments were actually late. Recall, TP/(TP+FN), answers how many late shipments were caught. F1 is the harmonic mean of precision and recall; it rewards balance but ignores true negatives and does not encode business costs. Support is the number of actual examples belonging to each class.
Always evaluate a baseline on the same held-out examples. A majority-class classifier provides a useful minimum comparison, but its majority class must come from training data. A candidate can beat that baseline on recall while losing on accuracy. Neither result alone settles the decision. If credible error costs are available, report their weighted total alongside familiar metrics, and state the assumptions behind those costs.
For numerical predictions, mean absolute error (MAE) describes average absolute error in the target's units. Root mean squared error (RMSE) gives larger errors more influence while retaining those units. R² compares squared error with prediction by the evaluation targets' mean; for nonconstant targets, it can be negative and is not a percentage of correct predictions. The scikit-learn guide treats metrics, cross-validation, threshold tuning, and leakage prevention as related evaluation concerns. Together, they remind us that a useful score requires both an appropriate formula and a trustworthy evaluation procedure.
Practical walkthrough
Start by writing an evaluation agreement: late shipments are positive, missed late shipments cost five illustrative units, and unnecessary flags cost one. These are teaching assumptions, not measured financial losses. Assume costs add independently and correct decisions have zero cost. If support capacity is limited or losses vary across shipments, this simple cost calculation needs refinement.
Next, separate training, validation, and final test data before learning anything from features. Train model parameters on training data; choose settings and decision thresholds using validation data or cross-validation within the development data. Fit imputers, scalers, and feature selection on training portions only, preferably inside a pipeline. Keep the final test set untouched until choices are fixed. For future shipments, use a time-respecting split; repeated observations of the same entity may require grouped splitting.
Run the example with Python 3.10 or newer and a compatible scikit-learn installation, version 1.3 or newer. Familiarity with Python lists and binary labels is sufficient. No files or downloads are needed. The candidate predictions are deliberately supplied rather than trained, isolating evaluation arithmetic. Treat them as frozen predictions from an earlier development process, not evidence of a real model's performance.
Check that each prediction corresponds to the truth at the same list position. Read each confusion matrix before interpreting the summary metrics. Compare candidate and baseline costs on these identical ten cases. Finally, record sample size and class support: only two cases are positive. This tiny constructed sample demonstrates trade-offs, but cannot establish deployment reliability or provide a stable estimate of future performance.
Worked example
from sklearn.metrics import (
accuracy_score,
confusion_matrix,
precision_recall_fscore_support,
)
# Separate training labels determine the baseline, not test labels.
y_train = [0, 0, 0, 0, 1, 1]
majority = max((0, 1), key=y_train.count)
# Constructed held-out outcomes and already-fixed candidate predictions.
y_test = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1]
candidate = [0, 0, 0, 0, 0, 0, 1, 1, 1, 0]
baseline = [majority] * len(y_test)
for name, predictions in [("Baseline", baseline), ("Candidate", candidate)]:
matrix = confusion_matrix(y_test, predictions, labels=[0, 1])
tn, fp, fn, tp = matrix.ravel()
precision, recall, f1, _ = precision_recall_fscore_support(
y_test, predictions, average="binary", pos_label=1,
zero_division=0,
)
cost = int(fp + 5 * fn)
print(name)
print(f"matrix={matrix.tolist()}")
print(f"accuracy={accuracy_score(y_test, predictions):.2f}")
print(f"precision={precision:.2f} recall={recall:.2f} f1={f1:.2f}")
print(f"cost={cost}")
Expected output
Baseline
matrix=[[8, 0], [2, 0]]
accuracy=0.80
precision=0.00 recall=0.00 f1=0.00
cost=10
Candidate
matrix=[[6, 2], [1, 1]]
accuracy=0.70
precision=0.33 recall=0.50 f1=0.40
cost=7
The baseline gets eight on-time shipments right but misses both late shipments. Its precision is mathematically undefined because it never predicts positive; zero_division=0 explicitly reports zero instead. The candidate catches one late shipment, misses one, and incorrectly flags two on-time shipments. Its precision is 1/3, recall is 1/2, and F1 is 0.40. Despite lower accuracy, its assumed cost is 2 + 5 = 7, versus 10 for the baseline. This preference depends on the cost assumptions: with equal error costs, the baseline wins. Fixed inputs and two-decimal formatting make the displayed output deterministic; no randomness or model fitting is involved.
