NLP Basics

Understanding NLP Basics

Natural language processing (NLP) works with language expressed as text or speech. For a beginner text project, the immediate challenge is representation: a classifier cannot directly calculate with a sentence. We must convert documents into numerical features while deciding which distinctions to preserve. Here, a document means one input string, whether it contains a short message or an entire article.

Tokenization divides text into units called tokens. Word tokenization might turn “Cats nap!” into “Cats” and “nap”; other approaches use characters or word pieces. Rules matter: punctuation, apostrophes, emojis, and languages without spaces complicate simple splitting. Scikit-learn’s default TfidfVectorizer uses word tokens containing at least two word characters, so a standalone “I” is excluded. Inspecting tokens is therefore part of understanding your data, not merely debugging code.

Normalization makes selected variations equivalent. Lowercasing merges “Cats” and “cats,” but also loses capitalization that might identify names. Stemming removes word endings using rules; lemmatization seeks dictionary forms, often using linguistic information. Neither is automatically performed by TfidfVectorizer. These transformations are optional choices, not compulsory cleaning steps. Removing common words can reduce features, but removing “not” can erase an important sentiment distinction.

Vectorization gives each document a numerical representation. A bag-of-words matrix has one row per document and one column per vocabulary term; entries usually count occurrences. TF-IDF adjusts these counts using inverse document frequency: terms appearing in fewer training documents receive larger weights. This reflects distribution, not understanding or guaranteed predictive usefulness. Most documents use only a small fraction of the vocabulary, so sparse matrices store nonzero values without allocating every absent term explicitly.

Single-word features discard order: “dogs chase cats” and “cats chase dogs” have identical counts. Word pairs, called bigrams, preserve some local order but increase vocabulary size. Dense embeddings represent text with learned numerical coordinates; contextual transformer models can represent a word differently depending on its surroundings. They are not automatic winners: task, language, available data, memory, and latency matter. A TF-IDF baseline remains useful because its features are inspectable. The supplied scikit-learn guide places these ideas within text feature extraction, pipelines, and leakage prevention: representation and evaluation must be designed together.

Practical walkthrough

Use Python 3.10 or later with a compatible scikit-learn installation; this example uses APIs available in scikit-learn 1.2 and later. Basic familiarity with lists, imports, and matrix shapes is enough. All text is defined in memory, and running the example requires no downloads or external datasets.

First, separate the training documents from later inputs. The two training strings are deliberately tiny so every feature can be checked manually. The later strings are inspection inputs, not a meaningful benchmark. No labels or classifier are needed to demonstrate feature extraction.

Next, create TfidfVectorizer with norm=None. The usual default scales each nonempty document vector to unit Euclidean length. Disabling that scaling makes the individual TF-IDF calculations easier to inspect, but is not a recommendation for every application. Lowercasing, smoothed inverse document frequency, and raw term counts remain enabled by default.

Call fit_transform only on the training strings. This learns both the vocabulary and inverse document frequencies, then produces training features. Call transform on later inputs to reuse those learned choices. Refitting on later inputs would change the feature meanings and allow evaluation data to influence preprocessing.

Finally, print the feature names before inspecting rows: column numbers have meaning only through this mapping. The example converts tiny matrices to dense arrays solely for display. Keep large text matrices sparse. Notice that an unseen word creates no new column, and a string containing only unseen words becomes an all-zero row. In a real prediction system, monitor such cases and decide whether to request clarification or use a fallback.

Worked example

from sklearn.feature_extraction.text import TfidfVectorizer

train_texts = ["Cats nap", "Dogs nap"]
later_texts = ["Cats dance", "dance"]

vectorizer = TfidfVectorizer(norm=None)
train_matrix = vectorizer.fit_transform(train_texts)
later_matrix = vectorizer.transform(later_texts)

print("features:", vectorizer.get_feature_names_out().tolist())
print("training shape:", train_matrix.shape)
print("training rows:")
for row in train_matrix.toarray():
    print(" ".join(f"{value:.3f}" for value in row))
print("later rows:")
for row in later_matrix.toarray():
    print(" ".join(f"{value:.3f}" for value in row))

Expected output

features: ['cats', 'dogs', 'nap']
training shape: (2, 3)
training rows:
1.405 0.000 1.000
0.000 1.405 1.000
later rows:
1.405 0.000 0.000
0.000 0.000 0.000

Lowercasing produces three vocabulary terms, displayed in column order. With smoothing, inverse document frequency is ln((1 + n)/(1 + df)) + 1, where n counts training documents and df counts documents containing the term. Here n is two. Both “cats” and “dogs” occur in one document, giving ln(3/2) + 1, displayed as 1.405. “nap” occurs in both, giving 1.000. Each observed count is one, and norm=None leaves these weights unscaled. The later word “dance” is ignored because training never established its column. Formatting to three decimal places makes the displayed output deterministic; these numbers are features, not probabilities or performance scores.

Common mistakes and how to avoid them

• Learning the vocabulary from all documents before splitting.

  Split first, then fit on training text only. During cross-validation, put the vectorizer and classifier in a Pipeline so vocabulary and IDF are relearned inside each training fold.

• Treating normalization as harmless cleanup.

  Inspect examples before removing punctuation, changing case, or deleting common words. Preserve distinctions relevant to the task, and compare preprocessing choices using validation data rather than the final test set.

• Interpreting a large TF-IDF value as semantic importance.

  Check the term and its document frequency. A typo can receive a large weight merely because it is rare. TF-IDF describes corpus usage; it does not establish meaning or causation.

• Converting a realistic text matrix to a dense array.

  Use estimators that accept sparse inputs and inspect selected rows or columns. Reserve toarray() for genuinely small examples, because dense storage allocates space for every zero.

Review questions

Further reading

• scikit-learn user guide