Train / Test Split

Understanding Train / Test Split

A train/test split separates the observations used to learn a prediction rule from those used to assess it. Suppose you want to predict tomorrow’s demand. A model that reproduces yesterday’s records may have memorized their peculiarities rather than learned a useful relationship. Evaluating on held-out observations asks a different question: can the fitted procedure predict outcomes it did not use during learning? This ability is called generalization.

In supervised learning, X contains the input features and y contains the corresponding targets. Each feature row must remain paired with its target when splitting. The training partition supplies examples to fit the model. The test partition supplies inputs for predictions, then its targets are used to calculate an evaluation metric. Test targets must not influence feature selection, parameter tuning, or other learning decisions.

The split should resemble the situation in which predictions will be used. Random splitting is appropriate when rows are reasonably independent and drawn from the same relevant population. For classification, stratification approximately preserves class proportions, provided each class has enough examples. For forecasting, hold out later observations instead. If several rows belong to one patient or customer, separating whole groups may be necessary to evaluate predictions for previously unseen people.

Ratios such as 80/20 and 70/30 are starting points, not guarantees. More training observations can help learning, while more test observations generally make evaluation less sensitive to individual cases. Absolute counts matter: a small test partition may contain very few rare outcomes. A fixed random_state makes a shuffled split repeatable with unchanged inputs and environment; it does not make that split representative or improve the model.

Keep model development separate from final evaluation. Use a validation partition or cross-validation within the training data to compare alternatives, then evaluate the chosen procedure on the test set. Learning preprocessing statistics also counts as training: imputation values, scaling parameters, and selected features must come from training data only. Scikit-learn pipelines help enforce this boundary. Even a carefully protected test score remains an estimate for a particular population and period, not a promise about changing future conditions. These distinctions connect scikit-learn’s guidance on cross-validation, pipelines, and leakage prevention.

Practical walkthrough

Start by defining the prediction moment: what information would be available when a prediction is made? Remove features that reveal later outcomes. Then choose the split unit and ordering. For independent rows, train_test_split shuffles by default; supplying X and y together preserves their correspondence. Its returned objects are X_train, X_test, y_train, and y_test, in that order.

The example below instead assumes eight observations already ordered from earliest to latest. Its task is to predict later numerical targets from an available numerical input. Setting shuffle=False and test_size=2 reserves the last two observations. An integer test_size specifies a count; a floating-point value such as 0.25 specifies a fraction. No random seed is needed because this split involves no randomization.

Use Python 3.10 or later with a compatible scikit-learn installation, version 1.2 or later. The program creates its data in memory and needs no external dataset. Each inner list is one row with one feature, matching scikit-learn’s two-dimensional input convention. The target list has one value per row.

Build a pipeline containing StandardScaler and LinearRegression. Calling fit with training inputs and targets learns both the scaling statistics and regression coefficients. Calling predict on test inputs applies the already fitted scaler; it does not relearn anything from those inputs.

Finally, compare predictions with test targets using mean absolute error, the average absolute prediction error in target units. Printing the partitions checks the chronological boundary. The constructed relationship is deliberately exact, so this is a workflow demonstration, not evidence that real forecasting problems are easy.

Worked example

from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Rows are ordered from earliest to latest.
X = [[i] for i in range(8)]
y = [2 * i + 1 for i in range(8)]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=2, shuffle=False
)

model = make_pipeline(StandardScaler(), LinearRegression())
model.fit(X_train, y_train)
predictions = model.predict(X_test)

print("Train inputs:", [row[0] for row in X_train])
print("Test inputs:", [row[0] for row in X_test])
print("Test targets:", y_test)
print("Predictions:", ", ".join(f"{p:.1f}" for p in predictions))
print(f"Mean absolute error: {mean_absolute_error(y_test, predictions):.1f}")

Expected output

Train inputs: [0, 1, 2, 3, 4, 5]
Test inputs: [6, 7]
Test targets: [13, 15]
Predictions: 13.0, 15.0
Mean absolute error: 0.0

The first six rows train the pipeline, and the last two remain outside fitting. All targets follow y = 2x + 1, so linear regression recovers that relationship and predicts 13 and 15 for inputs 6 and 7. Scaling changes the feature representation but does not prevent a linear model from expressing this rule. One-decimal formatting suppresses insignificant floating-point roundoff, giving the exact displayed output above. The scaler learns only from inputs 0 through 5. Scaling is unnecessary for this tiny regression, but including it demonstrates safe preprocessing. Two noiseless test cases cannot establish reliability on realistic, noisy future data.

Common mistakes and how to avoid them

• Scaling or filling missing values before splitting.

  Split first, then fit preprocessing on training data only. Put preprocessing inside a pipeline so validation folds also learn their own statistics without seeing held-out observations.

• Trying different seeds until the test score improves.

  Fix the evaluation design before comparing models. Use validation results for development; selecting a favorable test split turns the test set into another tuning resource.

• Randomly separating related or time-dependent rows.

  Use a chronological boundary for future predictions or a group-aware splitter for unseen entities. Check whether overlapping measurement windows require a gap between partitions.

• Shuffling features and targets independently.

  Pass X and y into the same split call. Confirm their original row alignment first; equal lengths alone do not prove that inputs match the correct targets.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide