Logistic Regression

Understanding Logistic Regression

Logistic regression estimates the probability of a categorical outcome. For binary classification, define one outcome as class 1 and the other as class 0. For example, a service might predict whether a customer cancels within the next month. The prediction is a conditional probability given the recorded features, not a guarantee about an individual customer.

The model first computes a linear score: z = b + w₁x₁ + … + wₖxₖ. Here, b is the intercept, each x is a feature, and each w is its coefficient. It then applies the sigmoid function, p = 1 / (1 + exp(−z)). A score of zero gives probability 0.5; positive scores give larger probabilities. Despite its name, this is classification rather than prediction of an unrestricted numerical target.

The linear relationship concerns log-odds, not probability. Odds are p / (1 − p), and their natural logarithm equals z. Holding other features constant, increasing a feature by one unit multiplies the modeled odds by exp(w). If w = ln(2), the odds double. That does not mean the probability doubles: changing odds from 1 to 2 changes probability from 0.5 to approximately 0.667. Coefficients describe conditional associations, not causal effects.

Training chooses coefficients that make the observed labels plausible, usually by minimizing log loss plus a regularization penalty. Log loss penalizes confident incorrect predictions strongly. Regularization discourages overly large coefficients, trading some training fit for potentially better generalization. In scikit-learn, smaller C means stronger regularization. Feature scaling matters because coefficient penalties depend on numerical scale; a coefficient for standardized data refers to a different unit than one for raw data.

Useful predictions require relevant features, reliable labels, and evaluation data representative of future use. The basic model assumes additive, linear effects on log-odds; nonlinear patterns may require transformations or interactions. Correlated features can make individual coefficients unstable. Repeated observations require careful splitting. Finally, probabilities are not automatically well calibrated: among cases assigned approximately 0.8, an observed event frequency near 80% would support calibration. A classification threshold turns probabilities into actions, but the appropriate threshold depends on error costs and available resources, not merely the model name.

Practical walkthrough

Start by defining the prediction moment, target, and positive class. For cancellation, use only information available when the prediction would actually be made. A final account-closure status would reveal the answer rather than predict it. Inspect missing values, duplicate records, class counts, and feature units before deciding how to prepare the table.

Next, reserve an untouched test set. Use a chronological split when predicting future periods and a grouped split when multiple rows belong to the same customer. For independent observations, a stratified split can help preserve class proportions. Use the remaining development data for training and validation; never select features or thresholds by repeatedly checking test results.

Build preprocessing and LogisticRegression into a scikit-learn Pipeline, using a ColumnTransformer when columns need different treatments. Impute missing numerical values, scale numerical features, and encode categories. Fit every learned transformation on training data only. During cross-validation, refit the entire pipeline inside each training fold so validation information cannot influence preprocessing.

After fitting, inspect classes_ before interpreting predict_proba: probability columns follow that class order. Select the column corresponding to your event label. Compare precision, recall, and a confusion matrix at candidate thresholds using validation data. Precision measures how often positive predictions are correct; recall measures how many actual positives are detected. Also evaluate probability quality with log loss and calibration checks.

Finally, freeze preprocessing, model settings, and the threshold, then evaluate once on the test set. Report both probability quality and decision errors. The example below isolates scoring and thresholding with explicitly supplied coefficients rather than claiming to train a useful cancellation model.

Worked example

from math import exp, log

# Illustrative coefficients, not estimates learned from customer data.
intercept = -log(4)
coefficient = log(2)
thresholds = (0.5, 0.7)

print("x p_cancel label_0.5 label_0.7")
for x in range(5):
    score = intercept + coefficient * x
    probability = 1 / (1 + exp(-score))
    labels = [int(probability >= t) for t in thresholds]
    print(f"{x} {probability:.3f} {labels[0]} {labels[1]}")

print(f"Odds multiplier per unit: {exp(coefficient):.3f}")

Expected output

x p_cancel label_0.5 label_0.7
0 0.200 0 0
1 0.333 0 0
2 0.500 1 0
3 0.667 1 0
4 0.800 1 1
Odds multiplier per unit: 2.000

This standard-library example treats x as a hypothetical count of recent service complaints. The intercept sets starting odds to one quarter, giving probability 0.2. Each additional complaint doubles modeled odds, producing unequal probability increases. At x = 3, the same probability receives different labels under the two thresholds. Raising the threshold changes the action rule, not the estimated probability. The explicit rule uses greater than or equal to, so probability 0.5 receives label 1 at threshold 0.5. Displayed probabilities are rounded only after decisions are computed. These invented coefficients illustrate arithmetic; without observed outcomes, they establish neither accuracy nor calibration.

Common mistakes and how to avoid them

• Treating a positive coefficient as proof that a feature causes the outcome.

  Describe an association conditional on included features. Check units, correlated predictors, and omitted factors before interpreting its odds multiplier.

• Scaling the whole dataset before splitting.

  Split first and fit preprocessing within a pipeline on training rows only. Apply the stored transformations unchanged to validation and test rows.

• Judging an imbalanced problem only by accuracy.

  Compare against a majority-class baseline, inspect precision and recall, and choose a threshold on validation data using the practical costs of errors.

• Assuming scikit-learn always handles multiclass logistic regression automatically through one-vs-rest.

  Replace that lesson note: supported solvers can optimize multinomial logistic regression jointly. Check solver and version compatibility; use OneVsRestClassifier when explicitly choosing separate binary models.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide