Machine Learning from Scratch · Lesson 01 · Part 1

Your First Neuron, by Hand

Why does a model learn? Work it out by hand with five examples, two adjustable numbers and one correction at a time.

Before you begin

You need arithmetic and a pen. No machine learning, calculus or programming is assumed: code comes in Part 2. Plan for about an hour. This is an estimate: stop to calculate rather than skim, even if it takes longer.

By the end you should be able to calculate a prediction and its loss, derive a gradient, apply one update, and recognise a learning rate that is too large.

1. Theory — what does “learning” mean?

From a fixed rule to an adjustable rule

In ordinary programming you write a rule and feed it inputs. In machine learning you choose a family of rules and let examples pick the best member. A model is the calculation that turns an input into an output. Training adjusts the model’s numbers to reduce its mistakes on examples.

Each sample pairs an input (hours studied) with a known target (exam points). Learning from input–target pairs is supervised learning; predicting a number is regression. Our five pairs are (1, 20), (2, 40), (3, 60), (4, 80) and (5, 100). They are invented and perfectly regular, so you can check every calculation by hand.

Parameters, weight, bias and prediction

Picture a straight line through the five points. Two knobs control it: the slope and the vertical offset. A parameter is a number inside the model that training adjusts. The weight $w$ sets how much the output changes when the input grows by one. The bias $b$ adds the same offset to every output.

The prediction is the model’s answer. We write the input as $x$ , the target as $y$ and the prediction as $\hat{y}$ (“y hat”). Keep them apart: a guess is not the target.

\hat{y} = wx + b

With $w = 10$ points/hour and $b = 5$ points, three hours gives $10 \cdot 3 + 5 = 35$ points. Raising $w$ by 1 adds 3 points to this prediction; raising $b$ by 1 adds 1 point. The weight’s effect grows with the input.

This model is linear regression. We call it a “neuron” because a neuron in a neural network computes the same thing, a weighted input plus a bias, before passing it through a nonlinear function. That function comes in a later lesson.

Loss: turn five mistakes into one score

An error is prediction minus target, so a negative error means the model predicted too low. Averaging signed errors hides mistakes: −10 and +10 cancel out. Instead, square each error and average the squares. The result is the loss: one number saying how badly the model fits. Smaller is better.

This loss is mean squared error (MSE). Squaring stops errors from cancelling and makes big errors count more: an error of 20 contributes four times as much as an error of 10. MSE is measured in squared points, not points or percent.

L(w, b) = \frac{1}{n} \sum_{i=1}^{n} \left( w x_i + b - y_i \right)^2

$L$ is the loss, $n$ the number of samples (5), and $i$ numbers the samples. $\sum$ is a reminder that the data stay fixed and only the parameters change the score.

Derivative: how sensitive is the loss to a change?

Turn one knob slightly. Does the loss rise or fall, and how fast? A derivative answers that: it is the rate of change of a function at a point. For a function $f(t)$ , the ratio

\frac{f(t + h) - f(t)}{h}

is the average change over a small step $h$ . The derivative is the value this ratio approaches as $h$ shrinks to zero.

For example, let $f(t) = t^2$ . At $t = 3$ , moving to $t = 3.01$ changes $f$ from 9 to 9.0601. The ratio is 6.01, and smaller steps bring it closer to 6. Algebra shows why:

\frac{(t + h)^2 - t^2}{h} = 2t + h \;\xrightarrow{\;h \to 0\;}\; 2t

So the derivative of $t^2$ is $2t$ .

Our loss depends on two numbers, $w$ and $b$ . A partial derivative changes one and holds the other fixed. We write them $\frac{\partial L}{\partial w}$ and $\frac{\partial L}{\partial b}$ (“partial L by w”, “partial L by b”). A negative $\frac{\partial L}{\partial w}$ means a small increase in $w$ lowers the loss. It says nothing about the loss itself being negative.

Gradient and gradient descent

The gradient collects all the partial derivatives into one vector:

\nabla L = \left( \frac{\partial L}{\partial w}, \; \frac{\partial L}{\partial b} \right)

Picture the loss as the height of a landscape whose map coordinates are $w$ and $b$ . The gradient points uphill, and its negative points most steeply downhill.

Gradient descent is the algorithm that uses it: compute the gradient, take a small step against it, repeat. The learning rate $\eta$ (“eta”) sets the step size. It is a hyperparameter: a setting you choose, not something the model learns.

\begin{aligned} w_{\text{new}} &= w - \eta \, \frac{\partial L}{\partial w} \\[6pt] b_{\text{new}} &= b - \eta \, \frac{\partial L}{\partial b} \end{aligned}

Compute both derivatives from the same old values before changing either parameter. A gradient is a local guide, not a promise about a long jump. Too large a rate overshoots the low region and the loss grows; too small a rate makes progress slow; zero changes nothing.

Every update in this lesson uses all five samples. One full pass over the training data is called an epoch, so here one update is one epoch.

Computing the gradient: the chain rule

The loss is built in stages: parameters give predictions, predictions give errors, and errors are squared and averaged. To find how a parameter affects the loss, multiply the rates of change along that chain. This is the chain rule.

For one sample, call its error $e = wx + b - y$ . The square $e^2$ changes at rate $2e$ as $e$ changes. The error changes at rate $x$ as $w$ changes, and at rate 1 as $b$ changes. Multiplying along the chain:

\begin{aligned} \frac{\partial (e^2)}{\partial w} &= 2e \cdot \frac{\partial e}{\partial w} = 2ex \\[6pt] \frac{\partial (e^2)}{\partial b} &= 2e \cdot \frac{\partial e}{\partial b} = 2e \end{aligned}

So each sample contributes $2ex$ to the weight’s derivative and $2e$ to the bias’s. Averaging over the five samples gives:

\begin{aligned} \frac{\partial L}{\partial w} &= \frac{2}{5} \sum_{i=1}^{5} e_i x_i \\[6pt] \frac{\partial L}{\partial b} &= \frac{2}{5} \sum_{i=1}^{5} e_i \end{aligned}

In Part 2, PyTorch does this calculation for you. Doing it by hand first means you know exactly what it computes.

2. Worked example — the first correction

Start at $w = 0$ and $b = 0$ , so every prediction is zero. Before calculating, predict the signs of both partial derivatives: all targets are positive, so would increasing each knob help?

All five samples at $w = b = 0$
$x$ (hours)	$y$ (points)	$e = \hat{y} - y$	$e^2$	$ex$
1	20	−20	400	−20
2	40	−40	1600	−80
3	60	−60	3600	−180
4	80	−80	6400	−320
5	100	−100	10000	−500

The squared errors sum to 22000, the errors to −300, and the error-times-input terms to −1100. So:

\begin{aligned} L &= \frac{22000}{5} = 4400 \\[6pt] \frac{\partial L}{\partial w} &= \frac{2 \cdot (-1100)}{5} = -440 \\[6pt] \frac{\partial L}{\partial b} &= \frac{2 \cdot (-300)}{5} = -120 \end{aligned}

With $\eta = 0.001$ :

\begin{aligned} w_{\text{new}} &= 0 - 0.001 \cdot (-440) = 0.44 \\[6pt] b_{\text{new}} &= 0 - 0.001 \cdot (-120) = 0.12 \end{aligned}

Subtracting a negative number increases the parameter. The two corrections differ because the inputs multiply the weight’s effect.

The new predictions are [0.56, 1.00, 1.44, 1.88, 2.32] and the new errors [−19.44, −39.00, −58.56, −78.12, −97.68]. Their squares sum to 20972.304, giving

\text{MSE} = \frac{20972.304}{5} = 4194.4608

The loss went down, but only a little: one correction is not training.

Check the derivative without the formula

Hold $b = 0$ and raise $w$ from 0 to 0.001. The loss becomes 4399.560011, so the ratio

\frac{4399.560011 - 4400}{0.001} = -439.989

is close to −440. It isn’t exact because the step is not infinitely small.

Why does the bias rise, when the perfect fit has

b = 0

At the start, both parameters help raise the predictions, so both grow. Later, once the slope is nearly right, a positive bias makes the small inputs overshoot, and it has to shrink again. The perfect fit is $w = 20$ , $b = 0$ , but a limited number of small steps need not reach it. The loss surface is a long, tilted valley: steep across, almost flat along it.

3. Interactive visualization — compare corrections

The simulation below runs the same equations in JavaScript. Before you move the sliders, predict what happens at learning rates 0, 0.001, 0.05 and 0.1. Each change restarts from $w = b = 0$ ; the step slider shows the model after that many updates.

Learning rate η: 0.001 Completed updates: 1

Horizontal axis: completed updates (0–100). Vertical axis: MSE in points², linear scale. The vertical maximum adjusts to each run; compare the numbers as well as the shape.

At update 1 with η = 0.001: w = 0.44, b = 0.12, MSE = 4194.4608.

Fit a line to the five points

First choose manual mode: set w = 10 and b = 5. Predict the effect of increasing only bias by 5, then restore b = 5 and increase only weight by 5. Which changes the slope and which changes the offset? Then select training and move from update 0 to 1, then to 100. Connect the line’s movement to decreasing loss.

Model shown in the plot

Manual parameters (do not change training) Weight w (points/hour): 0 Bias b (points): 0

Circles show the five training targets; the solid line is ŷ = wx + b. Fixed axes: 0–6 hours and −60–240 points. Line segments outside this range are not drawn; values remain available in the summary and table. Rescaling therefore cannot hide divergence.

Model after update 1: w = 0.44, b = 0.12; MSE = 4194.4608 points². Predictions for hours 1–5: 0.56, 1, 1.44, 1.88, 2.32.

Targets and predictions for the plotted model
Hours	Target (points)	Prediction (points)
1	20	0.56
2	40	1.00
3	60	1.44
4	80	1.88
5	100	2.32

At rate 0 the loss stays at 4400. At 0.001 it falls slowly; at 0.1 it explodes within 100 steps. Watch the numbers as well as the curve: when a run diverges, the vertical scale grows so much that the early losses look like zero. These thresholds hold for this dataset only; there is no universally good learning rate.

4. Exercises

Write your prediction before you calculate, and use the simulation to check your arithmetic.

Start at $w = 10$ , $b = 5$ . Calculate the five predictions, the MSE, both partial derivatives and one update with $\eta = 0.001$ . Check the MSE in the manual mode of the plot.
Multiply all targets by two. Before calculating, predict how the initial loss and gradients at $w = b = 0$ change. Explain why the loss and its derivatives scale differently.
In the simulation, find roughly the largest learning rate at which the loss still falls after 100 updates. What does the line in the plot do just above it?
Change the targets to [25, 45, 65, 85, 105]. Which values of $w$ and $b$ fit them exactly? Why could a model without a bias never fit these points?

Hints

Reuse $e = wx + b - y$ and sum $e^2$ , $ex$ and $e$ separately. At $w = b = 0$ , doubling the targets doubles every error; think about what squaring does. For exercise 3, try rates between 0.05 and 0.1. The new targets in exercise 4 follow $y = 20x + 5$ .

5. Assessment — explain why learning happens

Answer without scrolling back. Back each answer with a calculation or an observation, not only a definition.

A colleague says derivative, gradient and gradient descent are the same thing. Use this two-parameter model to show the difference.
At the start, why is the weight’s derivative (−440) more negative than the bias’s (−120)? Why must both be computed before either parameter changes?
In the simulation the bias rises at first, although the perfect fit has $b = 0$ . Why? Do 100 updates guarantee an exact fit?

Reasoning guide — open after writing your answers

A partial derivative measures the sensitivity to one parameter; the gradient collects them; gradient descent uses the gradient to update the parameters. The weight’s derivative sums $e \cdot x$ , so larger inputs amplify it. Updating w first and then computing the bias derivative from the new w would be a different algorithm. The bias helps reduce the early underprediction even though its final value is zero; a finite number of steps makes progress but need not converge exactly.

Completion checklist

Keep your hand calculations and a short note connecting errors → gradient → update → lower loss. The checkboxes are reminders for you, not a test.

I can calculate the first loss, gradient and update by hand. I can explain the difference between a derivative, a gradient and gradient descent. I have compared learning rates in the simulation and can explain slow progress and divergence.

Saved only in this browser.

Next: Part 2

Part 2 turns these calculations into a program and lets PyTorch compute the gradient for you.

Continue to Part 2: in PyTorch →