Training vs Test Data

Understanding Training vs Test Data

Training and test data answer different questions. Training data supplies examples from which a model learns a prediction rule. Test data checks that rule on examples withheld from learning. In supervised learning, each example contains features, X, available when making a prediction, and a target, y, representing the answer. Keeping those pairs together during splitting is essential: a feature row must never become associated with another row’s target.

A training score describes how well the model handles examples it already used. It is not an independent assessment. A sufficiently flexible model can memorize training details, including noise, rather than learn patterns that transfer. This is overfitting. A held-out test score provides evidence about generalization: performance on new examples resembling the evaluation population. It does not guarantee performance after conditions change, and a small test set can give an unstable estimate.

The split should reflect how predictions will be used. Random splitting is reasonable when rows represent independent examples from a comparable population. For classification, stratification can preserve approximate class proportions. If several rows belong to one patient or customer, hold out entire groups when evaluating performance on unfamiliar people. For forecasting, train on earlier observations and evaluate on later ones; features must also contain only information available at prediction time.

An 80/20 or 70/30 split is a starting choice, not a scientific requirement. More training examples may improve learning, while more test examples usually make evaluation more informative. The appropriate balance depends on dataset size, rare outcomes, and the intended use. A fixed random_state makes a randomized split repeatable for unchanged inputs; it does not make that split representative or eliminate uncertainty.

Keep model selection separate from final testing. Use a validation set, or cross-validation within the training portion, to choose features, algorithms, and settings. Once those decisions are fixed, evaluate on the test set. This separation also applies to preprocessing: averages, scaling parameters, imputation values, and selected features must be learned without test data. Scikit-learn’s guide treats evaluation, pipelines, and leakage as connected workflow concerns. A pipeline helps enforce the boundary, but cannot repair a poorly chosen split or features that reveal future answers.

Practical walkthrough

Start by defining the prediction task and evaluation unit. Are you predicting another independent purchase, a new customer’s behavior, or next month’s demand? Decide which rows must stay together and which information will exist when a prediction is requested. Choose a metric before comparing models; for numeric targets, mean absolute error measures the average absolute prediction error in target units.

Next, reserve the test portion before fitting anything. With independent rows, train_test_split can divide X and y together. With time-ordered observations, use an appropriate chronological boundary instead. Record the rule so another run evaluates the same task. Do not try several random seeds and retain whichever produces the nicest score.

The example below uses a fixed, explicit split so every calculation is inspectable. Its six synthetic observations are already ordered: the first four are training examples and the final two are test examples. This is a demonstration of data boundaries, not a recommended sample size. Prerequisites are Python 3.10 or later with compatible NumPy and scikit-learn installations; the APIs used are available in scikit-learn 1.2 and later. No external data is needed.

Fit the pipeline on X_train and y_train only. StandardScaler learns training statistics, then LinearRegression learns from the transformed training features. Predicting test targets reuses those statistics without updating them. Finally, compare predictions with y_test using the previously chosen metric.

In a larger project, perform tuning inside the training portion before this final evaluation. If the test result prompts another design change, that set has become development feedback, not an untouched final check.

Worked example

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

# Synthetic, ordered observations; no files or downloads.
X = np.array([[0.0], [1.0], [2.0], [3.0], [4.0], [5.0]])
y = np.array([1.0, 3.0, 5.0, 7.0, 9.0, 13.0])

# Reserve the last two observations before fitting.
X_train, X_test = X[:4], X[4:]
y_train, y_test = y[:4], y[4:]

model = make_pipeline(StandardScaler(), LinearRegression())
model.fit(X_train, y_train)

train_predictions = model.predict(X_train)
test_predictions = model.predict(X_test)
scaler = model.named_steps['standardscaler']

print(f'Train rows: {len(X_train)}; test rows: {len(X_test)}')
print(f'Training feature mean: {scaler.mean_[0]:.1f}')
print('Test predictions: ' + ', '.join(f'{p:.1f}' for p in test_predictions))
print(f'Training MAE: {mean_absolute_error(y_train, train_predictions):.1f}')
print(f'Test MAE: {mean_absolute_error(y_test, test_predictions):.1f}')

Expected output

Train rows: 4; test rows: 2
Training feature mean: 1.5
Test predictions: 9.0, 11.0
Training MAE: 0.0
Test MAE: 1.0

The four training pairs follow y = 2x + 1, so linear regression recovers that relationship. Scaling changes the feature coordinates, not the fitted predictions here; it is included to demonstrate safe preprocessing. The scaler’s mean is (0 + 1 + 2 + 3) / 4 = 1.5, excluding both test rows. Predictions at 4 and 5 are 9 and 11. Their actual targets are 9 and 13, giving absolute errors of 0 and 2: test MAE is 1.0. One-decimal formatting hides negligible floating-point rounding. This tiny constructed result demonstrates evaluation mechanics, not reliable real-world performance or proof of overfitting.

Common mistakes and how to avoid them

• Scaling all rows before splitting because scaling does not use targets.

  Feature distributions also carry information. Split first, then fit preprocessing only on training rows. During cross-validation, fit the entire pipeline separately inside each training fold.

• Repeatedly checking test performance while choosing model settings.

  Use validation data for those decisions. Test results influence learning indirectly when they guide choices; obtain a fresh holdout for an independent final assessment.

• Randomly separating related records or overlapping time windows.

  Split by the relevant group or chronological boundary. Check whether duplicated observations, shared subjects, or overlapping target periods allow information to cross that boundary.

• Treating model.score as a universal accuracy measure.

  Check the estimator’s scoring definition. Classifiers commonly return accuracy, while regressors commonly return R-squared. Use an explicit metric such as mean_absolute_error when that matches your task.

Review questions

Further reading

• scikit-learn user guide