Logistic Regression

Understanding Logistic Regression

Logistic regression learns a classifier by modeling the probability of a class, rather than directly predicting a continuous measurement. For binary targets encoded as 0 and 1, it first computes a linear score: z = b + w₁x₁ + … + wₚxₚ. Each feature contributes its value multiplied by a learned coefficient; the intercept b supplies a baseline score. The sigmoid function converts that score into p = 1 / (1 + exp(−z)), interpreted as an estimate of P(y = 1 | x).

A score of zero gives probability 0.5. Positive scores give larger probabilities, and negative scores give smaller ones. The probability is nonlinear in the features, but the log-odds are linear: log(p / (1 − p)) = z. Holding other features fixed, increasing feature j by one unit changes log-odds by wⱼ and multiplies odds by exp(wⱼ). This describes the fitted model’s association, not a causal effect.

Training chooses coefficients that minimize binary cross-entropy, also called log loss. For one observation, the loss is −[y log(p) + (1 − y) log(1 − p)]. Confident mistakes receive large penalties. For an unregularized model, the gradient for coefficient j is the average of (p − y)xⱼ across training observations. An optimizer repeatedly adjusts coefficients to reduce the objective; scikit-learn provides solvers that perform this numerical optimization.

Regularization adds a penalty that discourages overly large coefficients. L2 regularization shrinks coefficients smoothly, while L1 can make some exactly zero. Solver support varies by penalty. In scikit-learn’s LogisticRegression, C controls inverse regularization strength: smaller C means stronger regularization. Scaling numeric features helps make coefficient penalties comparable across different measurement units and can improve optimization.

A classification threshold converts probabilities into labels. At threshold 0.5, the decision boundary is z = 0: a line, plane, or higher-dimensional hyperplane in the supplied feature space. Nonlinear features can create more flexible boundaries, but add complexity. Useful predictions require representative training data and appropriate features. Logistic regression does not guarantee calibrated probabilities or capture arbitrary interactions automatically. Strongly correlated features can also make individual coefficients difficult to interpret, even when predictions remain useful.

Practical walkthrough

Begin by defining the positive class and when predictions will be made. Every input must be available at that moment. For example, predicting whether an order will be returned cannot legitimately use a refund recorded afterward. Arrange X as rows of observations and columns of features, with one corresponding target in y.

Reserve a test set before learning transformations or selecting settings. For independent observations, a stratified split can preserve class proportions. For repeated customers or time-dependent records, use group-aware or chronological splits instead; random splitting can otherwise expose information that would not be available in deployment.

For a practical scikit-learn model, combine numeric standardization and LogisticRegression in a Pipeline. Encode categorical inputs and handle missing values where needed. Fit the entire pipeline only on training data. During cross-validation, each fold must learn its own preprocessing from its training portion. Validation and test observations receive those already learned transformations, never independently fitted ones.

Compare regularization settings using training-set cross-validation and a metric aligned with the task. Log loss evaluates probabilities; precision and recall describe different consequences of classification decisions. Select any custom threshold using validation data, then evaluate the finalized procedure once on the untouched test set. Inspect convergence warnings rather than assuming an unfinished optimization is acceptable.

The example below isolates the underlying arithmetic: it performs one gradient-descent update on two training observations. It deliberately omits regularization and preprocessing. Its query points demonstrate inference, not test performance. One update is not a converged fit, and two observations cannot establish real-world predictive quality.

Worked example

from math import exp


def sigmoid(z):
    # Stable evaluation for either sign of z.
    if z >= 0:
        return 1.0 / (1.0 + exp(-z))
    e = exp(z)
    return e / (1.0 + e)


# Training observations only: (feature, binary target).
train = [(-1.0, 0), (1.0, 1)]
w, b = 0.0, 0.0
learning_rate = 0.5

# Compute both gradients using the same current parameters.
errors = [(x, sigmoid(w * x + b) - y) for x, y in train]
grad_w = sum(error * x for x, error in errors) / len(train)
grad_b = sum(error for x, error in errors) / len(train)
w -= learning_rate * grad_w
b -= learning_rate * grad_b

print(f"After one update: w={w:.6f}, b={b:.6f}")
threshold = 0.6
for x in (-2.0, 0.0, 2.0):
    p = sigmoid(w * x + b)
    label = int(p >= threshold)
    print(f"x={x:.1f}, p(class 1)={p:.6f}, label={label}")

Expected output

After one update: w=0.250000, b=0.000000
x=-2.0, p(class 1)=0.377541, label=0
x=0.0, p(class 1)=0.500000, label=0
x=2.0, p(class 1)=0.622459, label=1

Initially, both training probabilities are 0.5. Their prediction errors are 0.5 and −0.5. Multiplying each error by its feature gives −0.5 for both observations, so the weight gradient is −0.5. The intercept gradient is zero because the errors cancel. Subtracting learning rate 0.5 times the gradients produces w = 0.25 and b = 0. Positive inputs now receive higher probabilities. The chosen threshold, 0.6, assigns the middle query to class 0 despite its probability being 0.5. This illustrates that learning probabilities and choosing a decision rule are separate operations. The threshold here is illustrative, not validated.

Common mistakes and how to avoid them

• Treating predict_proba columns as an undocumented fixed class order.

  Match columns to the fitted estimator’s classes_ attribute. Identify the intended positive class explicitly before computing metrics or applying a threshold.

• Standardizing the complete dataset before splitting it.

  Split first and fit preprocessing only on training observations. Use a pipeline inside cross-validation so validation-fold information cannot influence scaling, imputation, or feature selection.

• Assuming logistic regression probabilities are automatically well calibrated.

  Check calibration on held-out data. A sigmoid constrains values to a probability range; it does not guarantee that predicted risks match observed frequencies.

• Assuming multiclass logistic regression always defaults to one-vs-rest.

  Check the installed version and solver. Modern scikit-learn supports joint multinomial fitting with compatible solvers; use an explicit OneVsRestClassifier wrapper when that strategy is required.

Review questions

Further reading

• scikit-learn user guide