Understanding K-Nearest Neighbors (KNN)
K-nearest neighbors (KNN) predicts a label by comparing a new observation with labeled examples. Imagine sorting manufactured parts into categories A and B using measured dimensions. KNN assumes that parts close together in the chosen feature space tend to share a category. It does not discover what makes those categories different; its usefulness depends on whether the measurements and distance definition capture relevant similarity.
For classification, the algorithm finds the k closest training observations and combines their labels. With uniform weighting, each neighbor casts one vote, and the class with the most votes wins. A strict majority is not required when there are several classes. With distance weighting, closer observations contribute more. KNN also supports regression, where neighboring target values are averaged rather than counted as class votes.
Euclidean distance is the square root of the sum of squared feature differences. Consequently, changing a feature's units can change which observations count as neighbors. A measurement expressed in millimeters may dominate one expressed in meters simply because its numbers are larger. StandardScaler subtracts each training feature's mean and divides by its standard deviation. This is a useful starting point, not a universal rule: meaningful physical scales, outliers, and categorical features may require different treatment.
The choice of k controls how local the decision is. With k=1, one mislabeled observation can determine a prediction. Larger neighborhoods usually produce smoother boundaries, but may mix genuinely different groups and favor common classes. Choose k using validation rather than assuming five is optimal. Odd values prevent binary vote ties under uniform weighting, but do not eliminate multiclass ties or equal-distance ambiguities.
Calling KNN a lazy learner means it postpones much of its work until prediction. However, fit still validates and stores training information and may build a search structure. Prediction can become expensive with many observations. Numerous irrelevant features can also make distance less informative, so scaling alone cannot rescue poor inputs. The scikit-learn guide distinguishes neighbor classification, regression, and search algorithms; these are related tools, not interchangeable tasks. Here we focus on classification with numeric features, explicit Euclidean distance, and uniform votes.
Practical walkthrough
Use Python 3.10 or later with NumPy and scikit-learn already installed; the example uses APIs available in scikit-learn 1.2 and later. Everything runs in memory. Basic familiarity with lists, imports, and method calls is sufficient. No dataset download or random sampling is needed.
First, separate measurements from labels. X_train has six rows and two columns: each row represents one part, and each column represents one measurement. y_train contains the corresponding categories. The second measurement is deliberately ten times the first, making neighbor order easy to verify. Real measurements need not have this relationship; redundant columns can effectively give one underlying property extra weight.
Next, create a pipeline containing StandardScaler and KNeighborsClassifier. Setting n_neighbors=3 makes each prediction depend on three stored examples. Explicit uniform weighting and Euclidean distance keep the voting rule clear. Brute-force search compares against all training rows, which is appropriate for this tiny demonstration, not necessarily for a large application.
Fit the pipeline using only training rows. This learns the scaler's statistics, transforms those rows, and fits the neighbor estimator. Predict on two separate query rows. The pipeline transforms them with the same statistics; it does not relearn scaling from the queries.
Finally, inspect the selected neighbors. Because kneighbors belongs to the inner estimator, transform queries before calling it directly. Its indices refer to training-row positions, starting at zero. These queries illustrate decisions, not measured accuracy. For a real evaluation, reserve labeled test data before preprocessing, select settings through training-only cross-validation of the entire pipeline, then evaluate once on untouched test data.
Worked example
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
X_train = np.array([
[1.0, 10.0],
[2.0, 20.0],
[3.0, 30.0],
[7.0, 70.0],
[8.0, 80.0],
[9.0, 90.0],
])
y_train = np.array(['A', 'A', 'B', 'B', 'B', 'A'])
X_query = np.array([[2.2, 22.0], [7.8, 78.0]])
model = Pipeline([
('scale', StandardScaler()),
('knn', KNeighborsClassifier(
n_neighbors=3,
weights='uniform',
metric='euclidean',
algorithm='brute',
)),
])
model.fit(X_train, y_train)
predictions = model.predict(X_query)
scaled_queries = model.named_steps['scale'].transform(X_query)
indices = model.named_steps['knn'].kneighbors(
scaled_queries, return_distance=False
)
print('Predictions:', predictions.tolist())
print('Neighbor indices:', indices.tolist())
print('Neighbor labels:', y_train[indices].tolist())
Expected output
Predictions: ['A', 'B']
Neighbor indices: [[1, 2, 0], [4, 3, 5]]
Neighbor labels: [['A', 'B', 'A'], ['B', 'B', 'A']]
For the first query, the nearest training measurements in the first column are 2, 3, and 1, in that order. Their labels are A, B, and A, so A wins two votes to one. For the second query, the nearest values are 8, 7, and 9; B wins. Standardization preserves these orders because both columns encode the same progression, and both queries follow that progression. The selected distances are distinct, avoiding neighbor-order ties. Converting arrays to lists makes the printed representation explicit. These deterministic votes explain model behavior; they provide no evidence about performance on future manufactured parts.
