Understanding Encoding Categorical Data
Categorical encoding changes how an algorithm sees a feature. A nominal feature, such as color, identifies groups without ranking them. An ordinal feature, such as low, medium, and high priority, has an ordering. Many scikit-learn estimators require numerical inputs, but replacing words with numbers is not enough: the numerical representation should match the relationships you intend the estimator to use.
Suppose red, green, and blue become 0, 1, and 2. A linear model using this single feature assigns the same change in its prediction score to moving from red to green as from green to blue. A distance-based algorithm treats red and blue as farther apart. Neither relationship follows from color names. A numerical decision tree also uses the codes through threshold splits, restricting which categories a single split can group together.
One-hot encoding avoids this artificial ranking by creating an indicator column for each category. For columns ordered blue, green, red, a red observation becomes [0, 0, 1]. A linear model can then learn a separate coefficient for each indicator. Distinct known categories have equal distances within this indicator block. However, one-hot encoding does not discover that some categories are semantically similar, and its contribution to distances depends on how other features are scaled.
Ordinal encoding represents ordered categories with integer codes. Explicitly specifying low, medium, high prevents alphabetical ordering from defining the meaning. Even a genuine order does not establish equal spacing: a linear model still assumes equal score increments when given codes 0, 1, 2. One-hot encoding may therefore be preferable for an ordered feature when independent category effects matter more than a compact representation. Ordinal codes alone do not enforce monotonic predictions in every estimator.
Encoding also creates a learned schema: category membership, column order, and unknown-value behavior must remain consistent after training. One-hot encoding increases width, especially for high-cardinality features with many distinct values. Sparse storage reduces memory spent on zeros, but not the number of features. The scikit-learn guide separates categorical preprocessing, composite estimators, and leakage prevention; these concerns belong together because encoding is part of the fitted learning procedure, not merely a display change.
Practical walkthrough
1. Identify feature semantics before fitting anything. In the example, color is nominal and priority is ordinal. The priority order is a domain rule supplied in advance, not inferred from held-out observations. Assume values are clean strings and there are no missing entries. In real data, decide separately whether missingness needs an explicit category or training-fitted imputation.
2. Separate training and held-out rows first. The small lists below represent an already completed split; they are not enough to evaluate predictive performance. Fit both encoders only on training rows. Even preprocessing without target labels should not learn its vocabulary or frequency thresholds from evaluation data.
3. Configure unknown-category behavior deliberately. OneHotEncoder normally rejects unseen categories. Here, handle_unknown='ignore' instead produces an all-zero color block for yellow. This preserves the trained column layout but provides no learned yellow-specific effect. We retain every known color column, so that zero block is distinguishable from each known color representation.
4. Supply the priority order explicitly to OrdinalEncoder. The example uses its default error behavior for unknown priorities: an unexpected label should trigger investigation rather than silently acquire an arbitrary rank. The declared vocabulary may include valid categories absent from training when it comes from an independent domain specification.
5. Inspect feature names and transformed rows before training a model. This example requires NumPy and scikit-learn 1.2 or later and uses dense integer output for readability. For larger datasets, consider sparse output. In a complete workflow, use ColumnTransformer for different feature groups and Pipeline to fit preprocessing within each cross-validation training fold.
Worked example
import numpy as np
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
train_color = [['red'], ['green'], ['blue'], ['red']]
test_color = [['green'], ['yellow']]
train_priority = [['low'], ['high'], ['medium'], ['low']]
test_priority = [['medium'], ['high']]
color_encoder = OneHotEncoder(
handle_unknown='ignore', sparse_output=False, dtype=np.int64
)
priority_encoder = OrdinalEncoder(
categories=[['low', 'medium', 'high']], dtype=np.int64
)
train_colors = color_encoder.fit_transform(train_color)
priority_encoder.fit(train_priority)
test_colors = color_encoder.transform(test_color)
test_priorities = priority_encoder.transform(test_priority)
print('Color columns:', color_encoder.get_feature_names_out(['color']).tolist())
print('Training colors:', train_colors.tolist())
print('Held-out colors:', test_colors.tolist())
print('Held-out priorities:', test_priorities.tolist())
Expected output
Color columns: ['color_blue', 'color_green', 'color_red']
Training colors: [[0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 0, 1]]
Held-out colors: [[0, 1, 0], [0, 0, 0]]
Held-out priorities: [[1], [2]]
The color encoder learns three categories from training and orders their output columns alphabetically. Repeated red rows produce identical vectors; frequency does not change their encoding. Held-out green activates the existing green column. Yellow activates none because it was absent during fitting and unknown categories are ignored. The priority encoder follows the supplied semantic order, so medium becomes 1 and high becomes 2. These integers express rank, not measured quantities. No target values or model are involved here: the output demonstrates the exact representation a downstream estimator would receive, without claiming that either encoding improves predictions on a particular task.
Common mistakes and how to avoid them
Encoding training and test tables independently.
Fit once on training and reuse transform. Independent dummy creation can produce different columns or meanings, even when both tables look numerically valid.
Assuming automatic ordinal codes reflect domain order.
Specify the ordered category list explicitly. For unordered inputs, prefer one-hot encoding unless the estimator explicitly interprets integer codes as categorical identifiers.
Always dropping a one-hot column.
Dropping a reference category can resolve exact collinearity with an intercept in unregularized linear models. It is not universally required and makes ignored unknowns indistinguishable from that reference category.
Replacing high-cardinality categories with full-data target averages.
Target encoding uses labels and can leak outcomes. Construct training encodings through appropriate out-of-fold estimation, keep evaluation labels out, and use smoothing for rare categories.
Review questions
Why can integer codes harm a model when every category has a unique number?
Uniqueness preserves identity, but numerical operations add relationships. Linear models impose score increments, distance methods impose closeness, and numerical tree splits impose threshold groupings that may not match category meaning.
Does a real category order guarantee ordinal encoding is the best choice?
No. Order does not imply equal spacing or a linear effect. Consider the downstream algorithm's assumptions and compare encodings using validation that fits preprocessing only within training folds.
What does the all-zero color vector mean in this example?
It means the value was not among the learned colors. It does not mean missing, a newly learned category, or absence of color. Multiple unseen colors receive this same representation.
Why place encoding inside a pipeline during cross-validation?
Each fold needs its own training-fitted preprocessing. A pipeline passed to cross-validation refits the encoder on each training fold and applies that fitted mapping to the corresponding validation fold.
