Neural Networks Basics

Neural networks are the machine learning technique behind almost every headline AI result of the last decade. The core building block is simpler than the name suggests — the sophistication comes from stacking a lot of them together.

A single neuron

A neuron in a neural network does one small, mechanical thing: it takes some numeric inputs, multiplies each one by a weight, adds them all up along with an extra adjustable number called a bias, and passes that total through an activation function to produce its output. The weights and bias are exactly the "parameters" training adjusts, from the previous lesson. Here's the whole calculation for a neuron with two inputs, worked through in real code:

import math

inputs = [1.0, 0.5]
weights = [0.8, -0.6]
bias = 0.1

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

weighted_sum = sum(i * w for i, w in zip(inputs, weights)) + bias
output = sigmoid(weighted_sum)

print(f"weighted sum: {weighted_sum:.2f}")
print(f"neuron output: {output:.4f}")

weighted sum: 0.60
neuron output: 0.6457

That's genuinely the entire computation one neuron performs. 1.0 × 0.8 plus 0.5 × -0.6 plus the bias 0.1 gives 0.6, and the sigmoid activation function squashes that into a value between 0 and 1 — here, roughly 0.65. Nothing about a single neuron is intelligent on its own; it's a fixed arithmetic formula.

Stacking neurons into layers

A neural network arranges many neurons like the one above into layers: an input layer that receives the raw data, one or more hidden layers that each take the previous layer's outputs as their own inputs, and an output layer that produces the final prediction. Each connection between neurons has its own independent weight, so even a modestly sized network has thousands of adjustable parameters — this is what "training a model" is actually adjusting, at scale.

Why the activation function matters

If every neuron just computed a weighted sum with no activation function, stacking layers would be pointless — mathematically, a chain of weighted sums collapses down to a single, equivalent weighted sum, no matter how many layers you stack. The activation function's nonlinearity is what lets a multi-layer network represent far more complex patterns than a single layer ever could — it's not a minor detail, it's the entire reason "deep" (many-layered) networks are more capable than shallow ones.

Training via backpropagation, at a high level

Training a neural network means repeatedly: running an example through the network, comparing its output to the correct answer to get an error score, and then adjusting every weight in the network slightly in the direction that would have reduced that error — a process called backpropagation. Done over millions of examples, with tiny adjustments each time, the weights gradually settle into values that make accurate predictions. No calculus is required to understand the shape of the idea: it's automated, repeated trial-and-error, precisely directed by how wrong the last guess was.