Support Vector Machines (SVM)

Understanding Support Vector Machines (SVM)

A support vector machine (SVM) learns a decision boundary from labeled examples. For binary classification, a linear SVM computes a score f(x) = w·x + b, where x contains the features, w contains learned weights, and b is an intercept. The boundary is where this score equals zero. Its sign determines the predicted class; its magnitude is not a probability.

Many boundaries can separate the same training points. An SVM prefers separation with a wide margin: a buffer around the boundary. With labels represented mathematically as −1 and +1, the canonical margin boundaries are f(x) = −1 and f(x) = +1. Their separation is 2/||w||, so reducing the weight vector's length widens the margin. This geometry is measured in the feature space used by the model, which makes preprocessing important.

Real datasets often overlap. A soft-margin SVM therefore minimizes a combination of weight size and margin violations: one half of ||w|| squared plus C times the sum of hinge losses. Each hinge loss is max(0, 1 − y f(x)). A correctly classified point inside the margin still has positive loss. Larger C penalizes violations more strongly; smaller C permits more violations in exchange for stronger regularization. Neither setting guarantees better predictions on unseen data.

Support vectors are training examples with nonzero coefficients in the fitted model's dual representation. They determine the decision function. In a separable example, they typically lie on the margin boundaries. With overlap, they can also lie inside the margin or be misclassified. Thus, “support vectors are the nearest correctly classified points” is too restrictive.

Kernels extend this mechanism by replacing dot products with kernel evaluations. The RBF kernel, exp(−gamma ||x − z||²), compares examples by distance. Larger gamma makes similarity decay faster, enabling more localized boundaries; smaller gamma produces broader influence. The algorithm still seeks a margin in an implicit feature space. Kernel SVC can become computationally expensive as sample counts grow. A linear model is a useful baseline, and scaling usually matters, though already suitably scaled features need not be standardized again.

Practical walkthrough

Start by defining what one row represents and which label must be predicted. Use numeric features available at prediction time; exclude identifiers or measurements that reveal the outcome. The example below assumes one numeric feature and two classes. It deliberately uses symmetric, separable data so the boundary can be checked by hand rather than inferred from a performance score.

Reserve evaluation data before learning any transformations. On real datasets, choose a split that matches deployment: grouped observations may require group separation, and forecasting requires respecting time order. Here, the training rows and probe rows are explicitly separate. The probes demonstrate predictions, not a reliable estimate of generalization performance.

Build a Pipeline containing StandardScaler and SVC. Calling fit on the pipeline learns the scaler's mean and scale from training rows only, then trains SVC on transformed rows. Calling predict or decision_function applies that same transformation to new rows without refitting it. This prevents the evaluation features from influencing preprocessing.

Begin with a linear kernel to inspect the margin mechanism. The example fixes C at 10 to make violations sufficiently expensive for this simple geometry. That value is illustrative, not a recommended universal default. For nonlinear problems, compare an RBF model and tune C together with gamma using cross-validation on training data. Put the entire pipeline inside the search so each fold learns its own scaler.

Finally, evaluate the chosen model once on untouched test data. Inspect class-specific errors when accuracy would hide costly mistakes. Decision scores support ranking, but probability estimates require additional calibration and validation.

Worked example

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC

# Requires scikit-learn; all data are defined locally.
X_train = [[-2.0], [-1.0], [1.0], [2.0]]
y_train = [0, 0, 1, 1]
X_probe = [[-0.5], [0.5]]

model = Pipeline([
    ('scale', StandardScaler()),
    ('svm', SVC(kernel='linear', C=10.0)),
])
model.fit(X_train, y_train)

print('Predictions:', model.predict(X_probe).tolist())
scores = model.decision_function(X_probe)
print('Decision scores:', [round(float(s), 3) for s in scores])
indices = sorted(model.named_steps['svm'].support_.tolist())
print('Support rows:', indices)

Expected output

Predictions: [0, 1]
Decision scores: [-0.5, 0.5]
Support rows: [1, 2]

The training mean is zero, and standardization divides each value by the square root of 2.5. Symmetry places the fitted boundary at zero. Training rows 1 and 2, containing −1 and +1, determine the margin; the outer rows do not become support vectors. The probe scores round to −0.5 and +0.5, placing both probes inside the margin while assigning opposite classes. Scores inside the margin are entirely valid predictions. They are not probabilities, and their raw magnitudes are not geometric distances unless normalized appropriately. This constructed example demonstrates the mechanism, not expected accuracy on a real application.

Common mistakes and how to avoid them

• Standardizing the full dataset before splitting it.

  Split first and fit preprocessing inside a pipeline. During cross-validation, pass the pipeline into the search, not features standardized using all folds.

• Treating C as a permitted number of classification errors.

  C weights the penalty for margin violations, including correctly classified points inside the margin. Tune it against validation performance rather than interpreting it as an error quota.

• Increasing gamma whenever training accuracy improves.

  Large gamma can produce overly localized boundaries that fail on new points. Compare validation results across C and gamma jointly, keeping the test set untouched.

• Reading a decision score of 0.8 as 80% confidence.

  Use the score as a signed decision value. If probabilities are needed, use an appropriate probability-estimation procedure and check calibration on held-out observations.

Review questions

Further reading

• scikit-learn user guide