Machine Learning · Chapter 31 of 40
Neural Networks Basics
A NEURAL NETWORK stacks layers of neurons. Each neuron computes `activation(w·x + b)`.
Deep networks learn hierarchical features and power state-of-the-art vision, speech and language systems.
Example 1 (python)
# Simple feed-forward net with Keras
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(10, activation='softmax')
])
Two dense layers.
Example 2 (python)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=5)
Compile, then fit.
Key points
- Layers of neurons compute w·x + b.
- Non-linear activations (ReLU, sigmoid).
- Trained with gradient descent + backprop.
- Deep = many layers.
💡 Note: For tabular data, tree-based models (XGBoost, LightGBM) often beat neural nets. Neural nets shine on images, audio and text.
Understanding Neural Networks Basics
A feed-forward neural network is a parameterized function: numbers enter an input layer, pass through hidden layers, and produce outputs. “Feed-forward” means information moves toward the output without loops. For supervised learning, each input has a target, and training adjusts the function to reduce prediction error. The architecture specifies which calculations are possible; learned parameter values determine the particular function used.
A neuron first forms a weighted sum plus a bias: z = w·x + b. Each weight controls one input’s contribution, while the bias shifts the neuron’s response even when inputs are zero. An activation function then transforms z. ReLU returns max(0, z), suppressing negative values while leaving positive values unchanged. A dense layer gives every neuron access to every output from the preceding layer.
Nonlinearity is essential. Stacking only affine transformations—weighted sums with biases—still produces a single affine transformation, regardless of layer count. Nonlinear activations let successive layers construct more flexible relationships. Hidden layers are intermediate representations, not additional target labels. Their values need not correspond to recognizable concepts, and adding depth does not automatically produce useful features or better predictions.
The output layer depends on the task. Regression often uses an unrestricted numerical output. Binary classification commonly uses a sigmoid output between zero and one. Classification among mutually exclusive categories often uses softmax, which produces nonnegative outputs summing to one. These numbers can be interpreted as estimated probabilities, but they are not guarantees of calibrated confidence. The loss must match the output representation and target encoding.
Training alternates forward computation, loss calculation, backpropagation, and parameter updates. Backpropagation applies the chain rule to compute loss derivatives; an optimizer uses those derivatives to change weights and biases. An epoch means one pass through the training examples, usually divided into batches. Training loss alone cannot establish performance on unseen inputs.
The supplied scikit-learn guide separates multilayer perceptrons, regularization, preprocessing, and evaluation: these concerns belong together. Larger networks offer capacity but cost computation and can overfit. Compare against simpler baselines, particularly for small tabular datasets. Keep validation data for model choices and reserve test data for final evaluation.
Practical walkthrough
Start with a network small enough to inspect without a framework. The example requires Python 3.8 or newer and basic familiarity with functions, tuples, and loops. It uses no third-party packages. Unlike the lesson’s Keras sketch, it performs only a forward pass with deliberately chosen parameters; it does not claim to learn them.
The task is exclusive OR, or XOR. Each input contains two bits. The desired output is one when exactly one bit is one, and zero otherwise. These four cases illustrate a relationship that cannot be separated by one straight decision boundary in the original two-dimensional input space.
Build two hidden ReLU neurons. The first computes max(0, x1 + x2); its weights are (1, 1) and its bias is zero. The second computes max(0, x1 + x2 − 1), using the same weights but bias −1. Although their weights match, their different biases make them respond differently.
Combine their outputs using weights (1, −2) and output bias zero. Leave the output activation linear. For bit inputs, the result is exactly zero or one, so no probability conversion or threshold is needed. That exactness applies to this constructed circuit, not arbitrary real-valued inputs.
Run all four inputs and inspect both hidden activations before checking the prediction. These are truth-table checks, not an independent test set. In a learned version, split data before fitting any scaler or imputer, fit preprocessing on training data only, and apply that fitted transformation unchanged to validation and test inputs. Never tune architecture using the final test results.
Worked example
def relu(value):
return max(0, value)
def forward(x1, x2):
h1 = relu(1 * x1 + 1 * x2 + 0)
h2 = relu(1 * x1 + 1 * x2 - 1)
prediction = 1 * h1 - 2 * h2 + 0
return (h1, h2), prediction
cases = ((0, 0, 0), (0, 1, 1), (1, 0, 1), (1, 1, 0))
for x1, x2, target in cases:
hidden, prediction = forward(x1, x2)
print(f"x=({x1}, {x2}) hidden={hidden} prediction={prediction} target={target}")
assert prediction == target
Expected output
x=(0, 0) hidden=(0, 0) prediction=0 target=0
x=(0, 1) hidden=(1, 0) prediction=1 target=1
x=(1, 0) hidden=(1, 0) prediction=1 target=1
x=(1, 1) hidden=(2, 1) prediction=0 target=0
For input (1, 1), the first hidden neuron produces two, while the second produces one. The output therefore becomes 2 − 2 × 1 = 0. For either mixed input, the second neuron stays at zero, leaving an output of one. All calculations use integers, with no randomness or training, so the printed output is deterministic. The assertions check the intended truth table and print nothing when successful. This demonstrates representational ability, not generalization or optimizer behavior. Outside the bit domain, outputs need not be probabilities: input (2, 2), for example, would produce −2 rather than a meaningful XOR class.
Common mistakes and how to avoid them
Treating a defined or compiled network as already trained.
Separate architecture, training configuration, and fitting. In Keras, compile selects training machinery; fit updates parameters. This example instead assigns parameters manually.
Combining an output activation with an incompatible loss.
Check whether the loss expects logits or probabilities and whether targets are integer class IDs or encoded vectors. Configure all three consistently.
Scaling the complete dataset before splitting it.
Split first. Estimate preprocessing parameters from training data only; reuse them for other splits. During cross-validation, fit preprocessing separately within each training fold.
Adding layers whenever training performance disappoints.
First inspect data quality, feature scales, loss compatibility, and learning rate. Compare training and validation behavior before increasing capacity or adding regularization.
Review questions
What does the bias change in the second hidden neuron?
Its bias of −1 delays activation until the input sum exceeds one. This creates a different response from the first neuron despite identical input weights.
Why would removing both ReLU activations break this XOR construction?
The output would simplify to (x1 + x2) − 2(x1 + x2 − 1), or 2 − x1 − x2. That is one affine function and does not match the XOR truth table.
How do backpropagation and gradient descent differ?
Backpropagation computes derivatives of the loss with respect to parameters. Gradient descent uses those derivatives and a learning rate to update parameters. Computing gradients alone does not change the network.
Do the four successful assertions demonstrate learning?
No. They verify a manually specified function on its complete bit-input domain. Demonstrating learning requires fitting parameters from data; assessing generalization additionally requires evaluation that did not influence fitting or model selection.
