Intro to Deep Learning

Understanding Intro to Deep Learning

Deep learning uses neural networks to learn layered representations of data. A representation is a numerical description useful for a task: image pixels might become patterns associated with edges, textures, and objects. These patterns are learned from a training objective rather than all being specified manually. However, automatic feature learning does not remove the need to define the target, check data quality, and evaluate predictions on genuinely unseen observations.

A dense layer computes weighted sums of its inputs and adds biases. Weights control how inputs contribute; biases provide adjustable offsets. An activation function then transforms those values. ReLU, defined as max(0, z), keeps positive values and replaces negative values with zero. Without nonlinear activations between layers, a stack of affine transformations remains one affine transformation. Nonlinearity lets networks represent relationships that a single linear model cannot capture.

The input holds measured features, hidden layers construct intermediate representations, and the output supplies a prediction. Depth describes successive transformations; width describes the number of units within a layer. The lesson’s 10-to-32-to-1 PyTorch network has one hidden layer, so it illustrates neural-network building blocks rather than substantial depth. Its single output is not automatically a probability: interpretation depends on the output transformation and training objective.

Training compares predictions with targets using a loss, such as mean squared error for regression. Backpropagation computes how parameters affect that loss, and an optimizer updates them. A learning rate controls update size. Repeating this process can reduce training error without improving performance on new data. That distinction is generalization; memorizing training-specific patterns is overfitting. Regularization and validation-based early stopping can help, but neither repairs an inappropriate evaluation split.

Architecture should reflect the data and task. Convolutional networks exploit local spatial structure; recurrent networks and Transformers can model sequences. Large models trained from scratch may require substantial data and computation, while adapting pretrained models can reduce those requirements. For structured tables, compare against linear models and gradient-boosted trees rather than assuming a winner. The supplied scikit-learn guide organizes neural networks alongside preprocessing, evaluation, and leakage prevention: these remain essential parts of the workflow. Predictive associations alone do not establish causal effects.

Practical walkthrough

Begin with a concrete prediction question. For example, estimate a machine’s remaining operating hours from sensor readings available now. Define when a prediction is made and exclude measurements collected afterward. Otherwise, the network may learn information that would not exist when deployed. Confirm that the target is measured consistently and that missing records do not systematically exclude difficult cases.

Next, choose training, validation, and test partitions that reflect intended use. For future predictions, separate observations chronologically; for generalization to new machines, keep each machine within one partition. Fit imputation rules and scaling statistics using training data only, then reuse them unchanged for validation and test data. The pandas guide’s missing-data and indexing topics are useful preparation references; scikit-learn’s pipeline and leakage sections address organizing learned transformations safely.

Establish a baseline before increasing model complexity. For operating hours, a constant prediction provides a minimal comparison, while a linear model offers a stronger starting point. Choose an evaluation metric with understandable units, such as mean absolute error in hours. Train a small network, monitor validation error, and use validation results—not test results—to choose architecture and stopping time.

Finally, inspect errors by operating condition and machine type. An acceptable average can conceal serious failures in uncommon conditions. Evaluate the selected workflow on the untouched test set and report its limitations, including whether deployment data may differ. The example below isolates a forward pass: run it with standard Python and inspect every intermediate value. It deliberately omits training, so its outputs demonstrate computation, not predictive usefulness.

Worked example

def relu(z):
    return max(0.0, z)


def predict(x):
    # Hidden layer 1: weights +1 and -1; both biases zero.
    h1 = [relu(x), relu(-x)]
    # Hidden layer 2: weights [1, 1]; bias -1.
    h2 = relu(h1[0] + h1[1] - 1.0)
    # Linear output: weight 2; bias 0.5.
    y = 2.0 * h2 + 0.5
    return h1, h2, y


for x in [-2.0, 0.0, 3.0]:
    h1, h2, y = predict(x)
    print(f"x={x:.1f} h1=[{h1[0]:.1f}, {h1[1]:.1f}] "
          f"h2={h2:.1f} prediction={y:.1f}")

Expected output

x=-2.0 h1=[0.0, 2.0] h2=1.0 prediction=2.5
x=0.0 h1=[0.0, 0.0] h2=0.0 prediction=0.5
x=3.0 h1=[3.0, 0.0] h2=2.0 prediction=4.5

This network has one input, two hidden layers, and one linear output. Its parameters are hand-chosen, not learned. The first hidden layer separates positive and negative input components; their sum equals the input’s absolute value. The second subtracts one and clips negative results to zero. The output doubles that activation and adds 0.5. Thus, the complete function is 2 × max(0, |x| − 1) + 0.5. Inputs between −1 and 1 receive the same output. This illustrates nonlinear representation building, not a validated forecasting model. No preprocessing is fitted, no targets are supplied, and no accuracy claim follows from these calculations.

Common mistakes and how to avoid them

• Treating any single network output as a probability.

  Check the task, loss, and output transformation. A linear regression output is unrestricted; a binary probability estimate commonly uses a sigmoid and still requires calibration checks.

• Scaling the complete dataset before splitting.

  Split first, fit scaling and imputation on training data only, and apply those stored transformations elsewhere. Refit preprocessing within each training fold during cross-validation.

• Adding layers whenever training error remains high.

  First check labels, feature availability, scaling, and optimization. Compare training and validation errors; more capacity can increase cost or overfitting without improving useful predictions.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide