Activation Functions

Understanding Activation Functions

A neuron first computes a weighted sum plus a bias: z = w·x + b. Its activation function then produces a = f(z). The weights and bias are learned; common activation functions have no learned parameters. Hidden activations shape the representations passed between layers, while output activations constrain predictions to suit the task. These roles need not use the same function.

Why introduce nonlinearity? Consider two layers with identity activations: h = W₁x + b₁ and y = W₂h + b₂. Substitution gives y = (W₂W₁)x + (W₂b₁ + b₂). This is one affine transformation: linear multiplication plus a bias. Stacking such layers does not add nonlinear expressive power. This conclusion assumes the intervening operations are also affine; adding nonlinear operations changes the argument.

ReLU, f(z) = max(0, z), keeps positive values and replaces negative ones with zero. Its slope is one for positive inputs and zero for negative inputs. Although each region is linear, the bend at zero makes the overall function nonlinear. ReLU is a useful starting choice for hidden layers, not a universal winner. A neuron that stays negative across training examples receives zero gradient through its activation and may stop learning.

Sigmoid, σ(z) = 1/(1 + exp(−z)), maps real inputs into (0, 1). Tanh maps them into (−1, 1), allowing signed, zero-centered outputs. Their derivatives are σ(z)(1 − σ(z)) and 1 − tanh²(z). Both flatten at extreme inputs. During backpropagation, derivatives multiply through successive layers, so repeated small factors can make earlier gradients tiny. This is one mechanism behind vanishing gradients.

Output choices depend on the target. A single sigmoid commonly represents the positive-class probability in binary classification. Separate sigmoids support labels that can occur together. For mutually exclusive classes, softmax computes pᵢ = exp(zᵢ)/Σⱼexp(zⱼ), coupling the scores so probabilities sum to one. Unrestricted regression commonly uses an identity output. Probability-shaped outputs alone do not guarantee calibration or accuracy.

Backpropagation uses activation derivatives to connect prediction error to weight updates. Therefore, activation choice affects both representational capacity and optimization. Feature scale, initialization, loss, and learning rate still matter; changing the activation cannot repair every training problem.

Practical walkthrough

Start by specifying what one prediction means. Is it an unrestricted number, one of several exclusive classes, or several independently possible labels? Choose the output representation and loss together. For classification, check whether the loss implementation expects probabilities or raw scores, called logits. Numerically stable logits-based losses incorporate the needed transformations internally.

Next, inspect a hidden layer as two operations: calculate its preactivations, then apply its activation. For a ReLU layer, record which preactivations are positive. Those neurons pass their local gradients backward; negative ones block that route. At exactly zero, ReLU has no unique derivative, so implementations select a convention, commonly zero. One inactive example does not establish that a neuron is permanently inactive.

Run the example below in Python without installing packages. It constructs an exclusive-or mapping: the target is one when exactly one binary input is one. The hidden units measure the positive parts of x₁ − x₂ and x₂ − x₁. Their sum detects a mismatch, which a single affine score cannot separate from both matching cases using one threshold.

Follow each row from inputs to hidden activations, then to the output logit and sigmoid probability. The weights are hand-selected, not trained. Consequently, this demonstration establishes a representational possibility, not a claim about an optimizer finding these weights or generalization to unseen data.

When extending this into training, split data before fitting preprocessing. Fit scaling on training data only, apply that fitted transformation to validation and test data, and compare activations using validation results. Keep the test set untouched until final evaluation.

Worked example

import math


def relu(z):
    return max(0, z)


def sigmoid(z):
    # Equivalent branches avoid exponentiating large positive values.
    if z >= 0:
        return 1.0 / (1.0 + math.exp(-z))
    e = math.exp(z)
    return e / (1.0 + e)


print('x1 x2 | h1 h2 | logit | probability | class')
for x1, x2 in [(0, 0), (0, 1), (1, 0), (1, 1)]:
    h1 = relu(x1 - x2)
    h2 = relu(x2 - x1)
    logit = 4 * (h1 + h2) - 2
    probability = sigmoid(logit)
    prediction = int(probability >= 0.5)
    print(f'{x1}  {x2}  | {h1}  {h2}  | {logit:5d} | '
          f'{probability:.4f}      | {prediction}')

Expected output

x1 x2 | h1 h2 | logit | probability | class
0  0  | 0  0  |    -2 | 0.1192      | 0
0  1  | 0  1  |     2 | 0.8808      | 1
1  0  | 1  0  |     2 | 0.8808      | 1
1  1  | 0  0  |    -2 | 0.1192      | 0

For input (0, 1), the hidden preactivations are −1 and 1. ReLU converts them to 0 and 1, producing logit 2 and probability 0.8808. Equal inputs produce two zeros and logit −2. The threshold creates class decisions; it is not used as the training activation because hard thresholding does not provide useful ordinary gradients. Removing both hidden ReLUs makes the hidden sum (x₁ − x₂) + (x₂ − x₁) equal zero everywhere, so this particular network becomes constant. The displayed probabilities reflect chosen weights, not measured certainty. Printing four decimals makes the output reproducible without implying those digits are calibrated.

Common mistakes and how to avoid them

• Applying softmax to labels that can coexist.

  Use separate sigmoid outputs for multilabel classification. Softmax forces competition and a total probability of one, which models exclusive alternatives.

• Applying sigmoid before a loss that expects logits.

  Check the loss contract. Supply raw scores to logits-based losses; apply sigmoid separately when probabilities are needed for reporting or decisions.

• Treating ReLU as protection against every gradient problem.

  Inspect preactivations and gradient magnitudes. Check scaling, initialization, and learning rate; consider leaky ReLU if persistent inactivity is a problem.

• Computing softmax with unshifted exponentials.

  Subtract the largest logit before exponentiating. This leaves probabilities unchanged mathematically and avoids overflow for large finite scores.

Review questions

Further reading

• scikit-learn user guide