Naive Bayes

Understanding Naive Bayes

Naive Bayes is a family of supervised classifiers that compares how well each class explains an observation. For features x and class c, Bayes’ theorem gives P(c | x) = P(x | c)P(c) / P(x). The prior P(c) describes class prevalence before examining the features. The likelihood P(x | c) describes the observation under that class. The posterior P(c | x) combines both sources of information.

The important assumption is conditional independence: features are treated as independent given the class, not necessarily independent across the entire dataset. This approximates the likelihood by a product of individual feature likelihoods. Because P(x) is identical for every candidate class, prediction can compare P(c) multiplied by those likelihoods without calculating the denominator. The largest score determines the predicted class.

Implementations usually add log probabilities instead of multiplying probabilities. Products of many small numbers can become too small for floating-point arithmetic; logarithms preserve the ordering while avoiding that problem. Training is relatively simple: estimate class frequencies and the parameters of each feature’s class-specific distribution. There is no iterative search for a separating boundary in the basic estimators.

The variant specifies those feature distributions. GaussianNB models each continuous feature with a separate Gaussian distribution within each class, estimating means and variances. MultinomialNB models token counts using class-specific token probabilities; repeated words contribute repeated evidence. BernoulliNB models binary presence or absence, so an absent feature also contributes to its score. CategoricalNB handles features whose values represent discrete categories. Choose according to feature meaning, not merely the array’s numeric type.

For multinomial text classification, additive smoothing estimates a token probability as (class token count + alpha) / (total tokens in that class + alpha × vocabulary size). Positive alpha prevents a known token missing from one class from making that class’s likelihood zero. Larger alpha pulls token probabilities toward uniformity. Independence is often imperfect: related words can supply overlapping evidence. Predictions may still be useful, but posterior probabilities can become overconfident. Treat probability calibration as a separate evaluation question. TF-IDF can also work with MultinomialNB, although weighted values are not literal counts, making this an empirical modeling choice.

Practical walkthrough

Start by defining the prediction unit and label. For spam filtering, one document is one observation, and its label identifies spam or legitimate mail. Split raw documents before learning a vocabulary or any weighting statistics. If messages share authors, threads, or duplicated content, consider grouped or time-based splits so closely related observations do not appear on both sides.

Next, choose the representation. A count vectorizer builds a vocabulary and turns each document into nonnegative token counts. With MultinomialNB, each column has one consistent token meaning across training and prediction. Keep the vectorizer and classifier in a scikit-learn Pipeline: fitting the pipeline learns the vocabulary and classifier from the supplied training portion only. During cross-validation, fit the entire pipeline independently inside each fold.

Inspect the learned quantities before evaluating performance. Class priors reflect document counts, whereas multinomial likelihoods reflect token counts within each class. These are different denominators. Verify that every intended class has training examples and that preprocessing preserves useful distinctions. Lowercasing may help consolidate words, but removing punctuation or short tokens can discard meaningful signals.

Tune smoothing and representation choices using validation data or training-only cross-validation. Then evaluate once on an untouched test set. Examine per-class precision and recall alongside a confusion matrix; accuracy alone can conceal missed minority-class examples. Select thresholds using validation data if false positives and false negatives have different costs.

The example below deliberately uses an artificial vocabulary to expose the arithmetic. Its query is not a performance benchmark. Real evaluation requires representative labeled documents and an appropriate split.

Worked example

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

# Requires scikit-learn; no external data is needed.
train_text = ["meeting meeting", "meeting", "offer offer", "offer"]
train_labels = ["ham", "ham", "spam", "spam"]

model = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0))
model.fit(train_text, train_labels)

query = ["offer offer meeting"]
vectorizer = model.named_steps["countvectorizer"]
classifier = model.named_steps["multinomialnb"]

print("vocabulary:", ", ".join(vectorizer.get_feature_names_out()))
print("prediction:", model.predict(query)[0])
for label, probability in zip(classifier.classes_, model.predict_proba(query)[0]):
    print(f"P({label} | query) = {probability:.3f}")

Expected output

vocabulary: meeting, offer
prediction: spam
P(ham | query) = 0.200
P(spam | query) = 0.800

Both classes contain two documents, so each prior is 0.5. Each also contains three training tokens. With two vocabulary terms and alpha equal to one, the denominator for token probabilities is five. For spam, offer has probability 4/5 and meeting has probability 1/5; ham reverses these values. The query contains two offers and one meeting. Its unnormalized spam score is 0.5 × 0.8² × 0.2 = 0.064; the ham score is 0.016. Normalizing gives 0.800 and 0.200. The multinomial coefficient cancels between classes. These are model probabilities, not evidence that words cause spam or that real-world confidence is calibrated.

Common mistakes and how to avoid them

• Interpreting independence as a requirement that all dataset columns be uncorrelated.

  Check the conditional assumption: dependencies within each class matter. Correlation alone does not automatically disqualify Naive Bayes, but duplicated or overlapping features can exaggerate evidence. Compare held-out predictions and probability calibration.

• Standardizing word counts before MultinomialNB.

  Mean-centering can produce negative values, which MultinomialNB does not accept. Use nonnegative counts or suitable nonnegative weights. For continuous measurements, assess whether a Gaussian feature model is appropriate instead.

• Learning the vocabulary or TF-IDF weights from all documents before splitting.

  Split first and fit preprocessing on training data only. Put learned transformations inside the pipeline used for cross-validation. Transform validation and test documents without refitting; unseen words are ignored by the fitted count vectorizer.

• Treating smoothing as a solution to every unseen-word problem.

  Smoothing gives positive probability to vocabulary terms absent from a particular class. It does not add entirely new terms to a fixed vocabulary. Monitor documents with no recognized tokens: their multinomial scores depend only on class priors.

Review questions

Further reading

• scikit-learn user guide