# Coding a neural network The model is a function that makes predictions. The training program feeds it data, measures a loss, calculates derivatives, and updates its parameters. We will write that program first with explicit array mathematics, then replace particular pieces with PyTorch. The important question is not just “which function do I call?” but **“which job does that function take over?”** ## Overall goal Run, explain, and verify a small digit classifier in three forms: manual NumPy, PyTorch tensors with automatic differentiation, and PyTorch layers with an optimizer. Be able to identify the inputs, parameters, intermediate results, gradients, and actual update in each version. **Starting point:** you can program and multiply matrices. No Python ML library, environment, notebook, or GPU experience is assumed. We explain the Python notation and the meaning of a gradient as they become necessary. Our running model is a fully connected **784 → 16 ReLU → 10** network with 12,730 trainable numbers. Its inputs are genuine 28 × 28 MNIST images. We keep the architecture and starting parameters fixed while comparing implementations. Cross entropy is the main route; MSE on probabilities is an alternative objective for the same model. The route is **tools → arrays → forward calculation → derivatives → updates → autograd → layers → verification**. Read in order the first time. Each section ends with questions whose answers are available locally. **What runs where?** Reading this webpage requires no ML installation. Its live arithmetic is a checked JavaScript equivalent, not a Python interpreter. Framework-specific tables are explicitly labeled recordings from actual Python runs. The downloadable project contains the runnable Python programs. No browser control installs a package or executes a terminal command. Displayed numbers are rounded to seven significant digits. Calculations and numerical checks use the full float64 values. ## 1. Know the tools before using them ### Goal Know where your code lives, what executes it, and where its libraries are installed. Be able to distinguish a Python statement from a terminal command before trying to train anything. We choose **Python** because it provides a short path from array mathematics to PyTorch. Python is a general-purpose programming language, not a special neural-network environment. The language does not make the model learn; the calculations in our program do. | Tool | Its job in this project | What it does not do | | --- | --- | --- | | Editor, such as VS Code | Write and read `.py` text files. | It is not the interpreter. | | Terminal or shell | Launch programs using commands. | It does not interpret ordinary Python statements by itself. | | Python interpreter | Execute the Python program. | It does not include every third-party library. | | `venv` and `pip` | Create an isolated environment; install packages into it. | Neither implements a neural network. | | NumPy | Store numerical arrays and perform efficient array operations. | In this program, it does not derive gradients or choose updates. | | PyTorch | Provide tensors, automatic differentiation, layers, and optimizers. | It does not choose the data, objective, or experimental design for us. | A **virtual environment** is a directory, here called `.venv`, with a Python entry point and its own installed packages. It separates this project's dependencies from another project's dependencies. It is not a virtual machine and does not create a GPU. The library called PyTorch is installed and imported under the name **`torch`**. NumPy uses **`numpy`**. Package names and product names are not always identical. You may have seen **Jupyter notebooks**: documents containing text and executable code cells. They can be useful, but execution order can hide state. We will start with ordinary scripts that run from top to bottom. An editor and terminal are enough. CPU execution is deliberate; no notebook, cloud account, or GPU is required. ### Experiment: which tool executes this line? **Predict:** does `import numpy as np` install NumPy, or load an already-installed package? Change the selection and read where the line belongs. The choice changes only the explanation; it does not run the command. Command to inspect - `python -m pip install …` - `import numpy as np` - `python python/train_numpy.py …` Run this in the **terminal**. It launches a specific Python interpreter, which runs pip and installs packages into that interpreter's environment. It changes the environment, not the model's weights. **What to notice:** installing a package, importing it, and running a training program are three different actions. This control is an explanation, not an embedded terminal. ### Make a small, reproducible working folder [Download the Python project](../coding_lesson_code.zip), extract it, and open a terminal in the extracted `coding_lesson_code` directory—the folder containing `experiment.json`. Python 3.13 is the tested interpreter. If needed, obtain it from [python.org](https://www.python.org/downloads/). Command format - macOS / Linux - Windows PowerShell ```bash python3 --version python3 -m venv .venv .venv/bin/python -m pip install -r requirements-numpy.txt ``` `-m` asks that interpreter to run a module such as `venv` or `pip`. `-r` tells pip to read package versions from a requirements file. We call the environment's Python by its path, so **activation is not required**. If you use an editor's Run button, configure it to use that same interpreter. Create a file named `hello.py` containing these Python statements: ```python import numpy as np print(np.__version__) ``` Then launch it from the terminal: ```bash .venv/bin/python hello.py ``` If the version prints, your interpreter can import NumPy. This establishes that the environment works; it says nothing yet about a network. Installation normally needs internet access. Once packages are installed, this project's images and calculations are local. If installation succeeds but you see `ModuleNotFoundError`, check **which interpreter** ran the file. Repeatedly installing into the wrong environment will not solve that mismatch. ### Questions to ask ourselves Think through the question before revealing its answer. #### Why can an import fail in the editor after installation succeeded in the terminal? The installer and editor may use different Python environments. Run pip through the intended environment's Python and select that interpreter in the editor. A package is not automatically shared with every Python installation. #### Do we need PyTorch or a GPU to implement gradient descent? No. Gradient descent is an algorithm: calculate derivatives and subtract a scaled gradient. We can implement it ourselves on the CPU. PyTorch makes important parts more convenient; it does not make the mathematics possible in the first place. #### What changed when you selected another command in the experiment? Only the displayed explanation changed. No command ran, no package was installed, and no model parameter changed. The real commands must be run in your own terminal or Python file, as indicated. ### What we now know We can run a Python file with the intended package. We still need to understand the numerical objects that file will manipulate. Next we put images, labels, and parameters into arrays with explicit shapes. [Continue to 2. Put data into arrays](#arrays) ## 2. Put data into arrays We have an interpreter and NumPy. Before implementing a layer, we need one consistent answer to “what does each dimension mean?” Many apparent calculus bugs are actually shape or axis mistakes. ### Goal Read array shapes, distinguish integer labels from one-hot targets, and predict whether an operation produces the intended result. Use a **batch-first** convention throughout this lesson. A NumPy `ndarray` is an N-dimensional numerical array. A vector, matrix, and stack of images are different shapes of this basic object. `shape` gives the size of each dimension; `dtype` describes the numerical type stored in its entries. Our first image is a handwritten **0**. It contains 784 brightness values, originally integers from 0 to 255. We flatten the 28 × 28 grid into one row and divide those intensities by 255, putting brightness between zero and one. A **batch** is a group of examples processed together. Stack B images as B rows: | Object | Shape | Meaning | | --- | --- | --- | | `X` | `(B, 784)` | One image per row; one pixel per column. | | `y` | `(B,)` | Integer class labels, such as 0 or 7. | | `Y` | `(B, 10)` | One-hot targets, constructed when needed. | | `W1`, `b1` | `(16, 784)`, `(16,)` | First-layer weights and biases. | | `W2`, `b2` | `(10, 16)`, `(10,)` | Second-layer weights and biases. | Only W1, b1, W2, and b2 are trainable parameters. X and y are fixed data. Intermediate arrays, such as activations and predictions, are recalculated from those inputs and parameters; they are not additional independent parameters. The comma in `(B,)` denotes a one-element Python tuple. This is a one-dimensional array, not a two-dimensional column `(B, 1)`. The value `y[0] = 0` identifies the first image's class. Its one-hot row has `Y[0, 0] = 1` and zeros in the other nine positions. The class label and the one-hot target are two representations of the same known answer, not predictions. We store weights as **(output, input)**. Each row of W1 holds one hidden neuron's 784 weights. Some mathematical explanations store examples as columns and write `W @ X`. Here examples are rows and the equivalent operation is `X @ W.T`. This changes the representation, not the network. ### Just enough NumPy syntax ```python import numpy as np # give the imported module a short name X.shape # a tuple, not a function call X[0] # first image: shape (784,) X[0:1] # a batch containing it: shape (1, 784) W1.T # transpose this two-dimensional array X @ W1.T # matrix multiplication X * X # element-by-element multiplication ``` Indices start at zero. The slice `start:stop` includes `start` but excludes `stop`. Keeping the batch dimension with `X[0:1]` lets the same code handle one or many images. Later, `params['W1']` accesses a named array in a Python dictionary; the dictionary simply groups the four parameter arrays. ### Experiment: make the shapes agree **Predict:** which dimension of the result changes when B goes from 1 to 8? Change the batch, then inspect the wrong transpose and the wrong multiplication operator. Observe both the shape and the explanation. Preview batch size B - 1 image - 8 images First-layer expression to inspect - `X @ W1.T + b1` - `X @ W1 + b1` - `X * W1.T + b1` Known label: 0 The actual MNIST images in the preview batch; captions identify their known labels. **Valid expression.** `(1, 784) @ (784, 16)` produces `(1, 16)`. The `(16,)` bias broadcasts across the rows. There is one row of 16 hidden weighted sums for each image. The first entry is `Z1[0, 0] = -0.4108369`. A larger B adds rows, not trainable weights. **What changes:** B selects the first B real images for the later numerical previews and resets the learning-step walkthrough. **What stays fixed:** all starting weights. An incorrect expression is an isolated inspection; it does not replace the supplied forward function. The browser starts with B = 1; the terminal project defaults to B = 8. Select 8 here to compare its starting values. Browser controls do not edit `experiment.json` or execute Python. ### Broadcasting and axes are part of the algorithm Adding `b1`, shape `(16,)`, to the result `(B, 16)` uses **broadcasting**: the same bias vector is applied to every row. There is not a separate learned bias for every image. NumPy aligns trailing dimensions; corresponding sizes must be equal or one must be 1. An **axis** is a dimension. Axis 0 is the batch; axis 1 is the features or classes. A reduction such as `sum(axis=0)` removes that axis by summing over it. `keepdims=True` retains a length-one dimension, which can be important when broadcasting the result back. We will use this in softmax and in bias gradients. The supplied `load_data` helper decodes a compact local file and returns `X, y` in these formats. File decoding is not a hidden network operation. The file contains 200 real MNIST training images and 100 real test images, with original split indices—not a substitute synthetic dataset. ### Questions to ask ourselves #### If B doubles, must we allocate twice as many trainable weights? No. The same parameter matrices act on more rows. Activations gain rows, but W1 remains 16 × 784 and W2 remains 10 × 16. All examples share those parameters. #### Why might X[0:1] be easier to use than X[0]? It retains the batch dimension. Our functions can continue treating X as a two-dimensional array and reducing over the same axes. X[0] removes that dimension. #### If broadcasting succeeds, is the formula necessarily correct? No. A scalar can broadcast across a bias vector even when we need a different derivative for each bias. Shape compatibility is a useful check, not proof of mathematical meaning. Section 4 demonstrates that mistake. ### What we now know Images and parameters have concrete array representations. Next we compute one neuron's weighted sum explicitly, then use matrix operations to repeat it for the entire network. [Continue to 3. Write the forward calculation](#forward) ## 3. Write the forward calculation We have chosen batch-first arrays and know which dimensions must match. NumPy can perform the arithmetic, but we still decide which operations make up the network. ### Goal Connect an ordinary multiply-and-add loop to a complete vectorized forward pass. Distinguish **parameters**, **intermediate results**, and the **cache** needed for backward propagation. “From scratch” here means we specify the model, loss, derivatives, and update rule ourselves. NumPy supplies array storage and efficient arithmetic. Reimplementing an optimized matrix-multiplication library would be a different lesson. ### One entry, with actual numbers For hidden neuron 0, multiply each pixel by its corresponding weight, add all 784 products, and add its bias. Two terms from our first image are: | Pixel index | Normalized brightness | Weight in W1 row 0 | Product | | --- | --- | --- | --- | | 402 | 0.9921569 | -0.0003825838 | -0.0003795831 | | 403 | 0.4470588 | 0.01253557 | 0.005604137 | A black pixel contributes zero to this sum. These are only two of the 784 terms, not a reduced two-input network. Including **all** products and the zero bias gives a weighted sum of **-0.4108369**. ReLU returns the larger of that value and zero, so this neuron's output is **0** for this image. Here is the weighted sum in ordinary Python. `def` defines a function, indentation groups its body, and `return` sends a value back to its caller. ```python def one_neuron(x, weights, bias): total = float(bias) for i in range(len(x)): total += float(x[i]) * float(weights[i]) return total ``` This is the actual function in [network_numpy.py](../python/network_numpy.py). Calling it with `X[0]`, `params['W1'][j]`, and `params['b1'][j]` calculates one entry of Z1. The expression `X @ W1.T + b1` performs the collection of dot products for every image and hidden neuron. Vectorizing the arithmetic does not create a different learning algorithm. ### Experiment: inspect one entry of the matrix result **Predict:** if a neuron's weighted sum is positive, what will ReLU return? Select neuron 1, then return to neuron 0. You are inspecting another entry; you are not changing any weight or turning a neuron on manually. Hidden neuron to inspect - Neuron 0 - Neuron 1 - Neuron 2 - Neuron 3 - Neuron 4 - Neuron 5 - Neuron 6 - Neuron 7 - Neuron 8 - Neuron 9 - Neuron 10 - Neuron 11 - Neuron 12 - Neuron 13 - Neuron 14 - Neuron 15 For the first image, digit **0**, inspect hidden neuron **0**. The loop includes all 784 products and its bias, giving **-0.4108369**. The matrix operation gives `Z1[0, 0] = -0.4108369`. ReLU gives `A1[0, 0] = 0`. This neuron is **inactive for this image**. The other 15 neurons still participate in the full network. **What to notice:** one matrix entry has the same meaning as the explicit loop. An inactive hidden unit does not mean the whole model outputs zero. The browser recomputes the equivalent arithmetic; the Python loop was also checked against NumPy. ### Initialize once, then reuse the parameters Weights begin as small random floating-point values. Biases begin at zero. Different initial weights keep the hidden units from being identical copies. The scale factors and seed are explicit settings in `experiment.json`; they are initialization choices, not learned values. ```python def initialize(cfg): D, H, K = cfg['input_size'], cfg['hidden_size'], cfg['classes'] rng = np.random.default_rng(cfg['seed']) return { 'W1': (rng.standard_normal((H, D)) * np.sqrt(cfg['hidden_variance'] / D)).astype(cfg['dtype']), 'b1': np.zeros(H, dtype=cfg['dtype']), 'W2': (rng.standard_normal((K, H)) * np.sqrt(cfg['output_variance'] / H)).astype(cfg['dtype']), 'b2': np.zeros(K, dtype=cfg['dtype']), } ``` Initialization happens **before** the training loop. Reinitializing at every step would erase the previous step's progress. A forward pass reuses the current parameter values; it does not draw a new random model. ### The full forward function ```python def forward(params, X): Z1 = X @ params['W1'].T + params['b1'] A1 = np.maximum(Z1, 0) logits = A1 @ params['W2'].T + params['b2'] shifted = logits - logits.max(axis=1, keepdims=True) logP = shifted - np.log(np.exp(shifted).sum(axis=1, keepdims=True)) P = np.exp(logP) return {'X': X, 'Z1': Z1, 'A1': A1, 'logits': logits, 'P': P, 'logP': logP} ``` The first two lines implement the hidden layer. ReLU is elementwise. Without a hidden nonlinearity, the two affine layers could be collapsed into one affine mapping to the output scores. `logits` contains ten unconstrained scores per image, not probabilities. Softmax normalizes each row. Subtracting the row's largest score does not change its softmax probabilities, but avoids exponentiating a large positive number. The row maximum and row sum retain shape `(B, 1)` so they broadcast along the class dimension, not across unrelated images. `logP` stores log-probabilities directly. This is useful because computing the logarithm of a probability that has underflowed to zero would fail numerically. The stable log-probability calculation avoids that problem for our cross-entropy loss. The returned dictionary is a **cache**. Backpropagation will need X, the ReLU inputs Z1, and hidden activations A1 from this exact forward pass. Saving them is bookkeeping, not another learned layer. Notice that y does not appear: predictions must be calculated without knowing the answer. For our starting image, the probability of digit 0 is **0.1351491**. It happens to be the largest of the ten probabilities, so this randomly initialized model already guesses this image's class correctly. That coincidence is not evidence that training has occurred, and it does not make the loss zero. ### Questions to ask ourselves #### Does replacing the Python loop with @ train anything? No. Both calculate weighted sums. Learning still requires a loss, derivatives, and a parameter update. Faster arithmetic is not itself training. #### Why save Z1 rather than only the final prediction? The ReLU backward rule needs to know which pre-activations were positive. A1 and X are also needed for the two weight gradients. The cache preserves these values from the forward pass that will be differentiated. #### If the largest probability belongs to the right class, can we conclude that the model has learned? No. A random model can guess an example correctly. Here its correct-class probability is only about 13.5%, and no update has happened. We need to distinguish the current prediction from evidence of learning or generalization. ### What we now know We can calculate a prediction from pixels, and we know what each line contributes. Next we compare that prediction with its target and calculate how the loss responds to the parameters. [Continue to 4. Write the derivatives](#backward) ## 4. Write the derivatives Forward propagation produced predictions and a cache. It did not measure their mismatch with the labels or change a parameter. We now add a scalar objective and its derivatives. ### Goal Read every line of the manual backward pass. Know **which variable a derivative is with respect to**, why its array has its particular shape, and where batch averaging happens. ### Begin with one image and one output bias For the first image, the label is 0 and the model assigns it probability 0.1351491. Cross entropy for this one-hot target is the negative natural logarithm of that probability: $$ C_{\mathrm{CE}}=-\log(p_0)\approx 2.001377. $$ This is a **scalar loss**. It is different from the derivative telling us how the loss changes when a particular value changes. For softmax followed by this cross entropy, the derivative with respect to logit 0 is: $$ \dfrac{\partial C}{\partial z^{(2)}_0}=p_0-1\approx-0.8648509. $$ For one image, increasing output bias 0 increases logit 0 by the same amount. Its gradient is therefore also -0.8648509. This is one component of the whole parameter gradient, not a new one-parameter model. We will use this component to inspect an update in section 5. For a batch, we average the per-image losses. The initial logit derivatives inherit that average. In code, `dZ2` means the derivative of the scalar loss with respect to the output-logit array; it has shape `(B, 10)`. In the formulas, $D_2$ denotes this same array, and $G_P$ denotes the gradient with respect to the probability array. ### Two loss choices, one backward interface With one-hot target array Y, the mean cross-entropy loss and its logit signal are: $$ C_{\mathrm{CE}}=-\dfrac{1}{B}\sum_{n=0}^{B-1}\log P_{n,y_n},\qquad D_2=\dfrac{P-Y}{B}. $$ The compact difference in the second expression is a **derivative**, not a linear cost function. The scalar objective still uses logarithms. For MSE on probabilities, average the squared differences over B × K entries, with K = 10. Differentiating each square supplies the factor 2; the mean supplies division by B × K. Then pass that probability derivative through softmax. Softmax couples the classes, so the logit gradient is not just ten independent sigmoid derivatives: $$ C_{\mathrm{MSE}}=\dfrac{1}{BK}\sum_{n,k}(P_{n,k}-Y_{n,k})^2,\qquad G_P=\dfrac{2(P-Y)}{BK}. $$ The row-wise dot-product subtraction in the MSE branch below accounts for that coupling. Both branches return the same kind of object: a scalar loss and the gradient with respect to logits. ```python def loss_signal(cache, y, kind): P = cache['P'] B, K = P.shape Y = np.eye(K, dtype=P.dtype)[y] if kind == 'ce': loss = -cache['logP'][np.arange(B), y].mean() dZ2 = (P - Y) / B elif kind == 'mse': error = P - Y loss = (error ** 2).mean() dP = 2 * error / (B * K) dZ2 = P * (dP - (dP * P).sum(axis=1, keepdims=True)) else: raise ValueError('Loss must be ce or mse') return float(loss), dZ2 ``` `np.eye(K)[y]` selects rows of an identity matrix to construct one-hot targets. The paired indices `np.arange(B), y` select the true class's log-probability in each row. `** 2` squares each error. The code returns an ordinary floating-point loss because NumPy is not tracking an automatic-differentiation graph here. Objective for the numerical previews - Cross entropy (CE) - MSE on probabilities Preview: **B = 1, objective = CE**. The loss is **2.001377**. At the unchanged starting weights, `P[0, 0] = 0.1351491`. The logit-gradient array has shape `(1, 10)`. Changing the objective changes this signal, not the prediction or the starting parameters. This choice affects the later derivative previews and resets the step walkthrough. It does not edit the terminal project's `experiment.json`. There, `loss` is the setting to change when you want to run the other objective. ### Continue through the two layers Consider the weight linking hidden unit 1 to output logit 0. For our fixed one-image CE example, its input activation is 0.1643927. A small increase in this weight changes the logit in proportion to that activation, so the chain rule multiplies the logit sensitivity by the activation: $$ \dfrac{\partial C}{\partial W^{(2)}_{0,1}} =\dfrac{\partial C}{\partial z^{(2)}_0}\,a^{(1)}_1 \approx(-0.8648509)(0.1643927)\approx-0.1421752. $$ For a batch, sum one such product per image. Matrix multiplication collects the products for every second-layer weight: `(10, B) @ (B, 16)` produces `(10, 16)`, exactly W2's shape. Biases use the corresponding sum without an activation multiplier, because each bias enters its weighted sum with coefficient one. ```python def backward(params, cache, dZ2): dW2 = dZ2.T @ cache['A1'] db2 = dZ2.sum(axis=0) dA1 = dZ2 @ params['W2'] dZ1 = dA1 * (cache['Z1'] > 0) dW1 = dZ1.T @ cache['X'] db1 = dZ1.sum(axis=0) return {'W1': dW1, 'b1': db1, 'W2': dW2, 'b2': db2} ``` Each output bias was reused across all B rows, so its derivative adds those image contributions. Summing axis 0 leaves one entry per class. To reach the hidden activations, `(B, 10) @ (10, 16)` produces `(B, 16)`. Then the elementwise ReLU mask passes a derivative where the cached pre-activation was positive and blocks it otherwise. We choose derivative zero at exactly zero. Finally, `dZ1.T @ X` applies the same dense-layer rule to the first layer. The gradient has shape `(16, 784)`, matching W1. We did not derive a new kind of calculus for “layer 1” versus “layer 2”; the operations repeat with different arrays. There is **no second division by B**. The loss's averaging factor is already in `dZ2`. Later products and sums carry it through the graph. Dividing the weight gradients again would shrink them by an unintended extra factor. ### Experiment: a legal-looking bias bug **Predict:** b2 contains ten trainable entries. Should its gradient be a scalar, B entries, or ten entries? Change the reduction below, inspect the shape and meaning, then restore axis 0. Candidate bias reduction - `dZ2.sum(axis=0)` - `dZ2.sum(axis=1)` - `dZ2.sum()` Result shape: **(10,)**. ```text -0.8648509, 0.0828975, 0.08910043, 0.09760746, 0.1243975, 0.07767895, 0.07609172, 0.09141794, 0.1070445, 0.1186149 ``` **Correct:** sum across images and retain one derivative for each output bias. Different classes receive different values. The batch average is already present in `dZ2`. Only this candidate reduction changes. The supplied backward function and the next walkthrough still use the correct axis 0. This selector does not edit the program or update any parameters. ### Questions to ask ourselves #### What changes in backward when we replace CE with MSE? The starting logit signal changes because the objective changed. The dense-layer and ReLU rules do not change. This separation lets the rest of the backward function work for either objective. #### Why use dZ2 @ W2 rather than W2.T @ dZ2? Our examples are rows. We want a `(B, 16)` result, so `(B, 10) @ (10, 16)` is the appropriate arrangement. Column-example notation expresses the equivalent calculation in transposed order. Check the representation rather than copying a transpose by appearance. #### If dZ2.sum() is almost zero, are all output biases insensitive to the loss? No. Positive and negative class contributions can cancel. Adding one common offset to every logit leaves softmax unchanged, explaining the zero total. Individual biases still have different sensitivities and require a ten-entry gradient. #### Why would another division by the batch size be a bug here? The starting signal already differentiates a mean loss. Averaging again changes its scale. For B = 8, an accidental second division makes the gradient eight times smaller than the intended mean-loss gradient. ### What we now know We can produce a gradient array with the same shape as each parameter array. None of those derivatives has moved a parameter yet. Next we perform that separate update and assemble the training loop. [Continue to 5. Assemble the learning step](#loop) ## 5. Assemble the learning step Forward propagation gave us predictions. Backpropagation gave us derivatives. We now add the operation that changes the stored parameters, using gradients calculated from the old forward pass. ### Goal Explain the order **forward → loss → backward → update**. Identify the line that actually learns, distinguish old and new measurements, and distinguish a batch, an update, and an epoch. Plain **stochastic gradient descent**, or SGD, subtracts the learning rate times each parameter's gradient. For the one-image CE example, the inspected bias starts at zero and its derivative is -0.8648509. With learning rate 0.2: $$ b^{(2)}_{0,\mathrm{new}}=0-0.2\,(-0.8648509)\approx0.1729702. $$ The learning rate controls the step size; backpropagation does not choose it. An excessively large step can raise the loss even when the derivative is correct. In `network_numpy.py`, `PARAMS` lists the four dictionary keys: W1, b1, W2, and b2. The update modifies their arrays: ```python def step(params, grads, learning_rate): for name in PARAMS: params[name] -= learning_rate * grads[name] ``` Wait until **all** derivatives have been calculated. The first-layer backward computation still needs the old W2. Updating it early would mix old and new parameters instead of differentiating one consistent forward pass. The actual step function in [train_numpy.py](../python/train_numpy.py) joins these pieces: ```python def train_step(params, X_batch, y_batch, cfg): cache = forward(params, X_batch) loss, dZ2 = loss_signal(cache, y_batch, cfg['loss']) grads = backward(params, cache, dZ2) step(params, grads, cfg['learning_rate']) return loss ``` The returned loss is the value **before** the update. Its Python variable does not recompute itself when parameters change. To measure the new cost, run forward and score the new predictions. ### Experiment: follow the code one operation at a time **Predict:** after the backward line, has the bias changed? Advance through the operations and identify when a prediction, a gradient, an update, and a new measurement become available. Preview arguments: **B = 1, objective = CE, learning rate = 0.2**. We inspect output bias `b2[0]`, but every parameter participates. The last two lines below are added post-update checks. The saved configuration file is unchanged. ```python cache = forward(params, X_batch) loss, dZ2 = loss_signal(cache, y_batch, cfg['loss']) grads = backward(params, cache, dZ2) step(params, grads, cfg['learning_rate']) cache_after = forward(params, X_batch) loss_after, _ = loss_signal(cache_after, y_batch, cfg['loss']) ``` - Show next operation - Reset this walkthrough **Starting point.** The model copy has `b2[0] = 0`. No walkthrough operation has been selected. Predict which line will first change the parameter. This is a walkthrough of checked JavaScript-equivalent calculations, not execution of Python. It operates on a **copy of the fixed starting model**. Reset restores that copy; the earlier examples and later framework comparison stay at the same baseline. The final cost change includes updates to all 12,730 parameters. It is not the effect of changing only the displayed bias. ### Repeat the step without reinitializing With 200 training images and a batch size of 8, one pass supplies 25 updates. That full pass is an **epoch**. Eight epochs give 200 updates, not eight. If the size is not divisible by B, the last batch is smaller; our loss uses its actual size. ```python for epoch in range(cfg['epochs']): order = rng.permutation(len(X)) if cfg['shuffle'] else np.arange(len(X)) for start in range(0, len(X), B): indices = order[start:start + B] train_step(params, X[indices], y[indices], cfg) updates += 1 ``` The same indices select X and y, preserving each image–label pair. Parameters are initialized before these loops and reused after each update. The full file contains the setup and imports; the extracts show the operations we are studying, not independent scripts to paste without their definitions. Before using the entire dataset, repeatedly train on **one fixed small batch**. This deliberate overfitting test checks whether the pipeline can learn a tiny problem. It is not a generalization test: ```bash .venv/bin/python python/train_numpy.py --config experiment.json --mode overfit ``` In the checked default CE run, both implementations take the same eight-image batch from loss about 2.28867 to 0.001583 in 300 updates, classifying all eight correctly. These are measured results for these settings, not promised outcomes for arbitrary data. If a small batch cannot be learned, inspect shapes, normalization, label pairing, the learning rate, and gradients before increasing the model's size. ### Questions to ask ourselves #### Which operation changes the model, and which one merely calculates its gradient? The subtraction in `step` changes the parameter arrays. `backward` computes derivatives from the cached forward pass. The next forward pass then sees the new weights. #### Does printing the variable loss after step() give the new loss? No. That variable still contains the earlier measurement. Compute a new forward pass and loss to evaluate the updated model. Printing later is not the same as recomputing later. #### What does perfect accuracy on the repeated batch establish—and what does it not establish? It establishes that this implementation can learn or memorize that tiny batch. It does not establish useful performance on unseen images. Numerical correctness, successful small-batch learning, and generalization are different checks. ### What we now know We own the entire training calculation. The most error-prone part is the collection of backward rules. Next we keep the matrix operations visible while asking PyTorch to compose those derivatives for us. [Continue to 6. Let autograd calculate the derivatives](#autograd) ## 6. Let autograd calculate the derivatives The NumPy implementation exposed every derivative. We can replace that manual differentiation without immediately hiding the forward operations inside layer objects. ### Goal Explain what tensors and a computation graph add to arrays. Distinguish **calculating**, **accumulating**, and **clearing** gradients from actually updating parameters. A **PyTorch tensor** is an array-like numerical object with a shape, dtype, and device. The file starts with `import torch`. Here we use CPU float64 tensors to match NumPy closely. Many training projects use float32; GPU execution and lower precision are separate later topics. For a parameter marked `requires_grad=True`, PyTorch records relevant tensor operations during forward computation. This record is a **computation graph**. Each supported operation has a local backward rule. Calling `loss.backward()` combines those rules in reverse with the chain rule and stores derivatives in the parameters' `.grad` fields. Autograd is not an AI guessing derivatives. It is not perturbing each of 12,730 parameters with finite differences either. It executes implemented differentiation rules for the actual operations in the graph. You still choose the forward program and the scalar objective. ### Keep the explicit matrix operations The one-step bridge in [torch_tensors.py](../python/torch_tensors.py) keeps our matrix expressions. `objective` is the PyTorch loss helper explained in section 7; on the CE route it calls `F.cross_entropy(logits, y_tensor)`. For now, its needed contract is simple: it returns one differentiable scalar tensor. ```python def tensor_step(params, X, y, cfg): weights = {name: torch.tensor(value, requires_grad=True) for name, value in params.items()} X_tensor, y_tensor = torch.from_numpy(X), torch.from_numpy(y) Z1 = X_tensor @ weights['W1'].T + weights['b1'] A1 = torch.relu(Z1) logits = A1 @ weights['W2'].T + weights['b2'] loss = objective(logits, y_tensor, cfg['loss']) loss.backward() gradients = {name: p.grad.detach().numpy().copy() for name, p in weights.items()} with torch.no_grad(): for p in weights.values(): p -= cfg['learning_rate'] * p.grad after = {name: p.detach().numpy().copy() for name, p in weights.items()} return loss.item(), gradients, after ``` The opening dictionary comprehension is a compact loop: for each name–array pair, create a tensor under the same name. `torch.tensor` makes a copy of each parameter array. `torch.from_numpy` can share the input array's memory; we do not modify those input data. `torch.relu` provides a PyTorch operation with a known backward rule. Keep NumPy conversions out of the differentiable forward path. The NumPy library is not automatically added to PyTorch's graph merely because the values look similar. `loss.backward()` replaces the manual starting derivative and backward function. It does **not** change the weights. We still implement the subtraction ourselves here. `torch.no_grad()` keeps that housekeeping update from becoming part of the graph being differentiated. The `detach().numpy().copy()` expressions export independent arrays for verification, not training. `loss.item()` converts a scalar tensor into an ordinary Python number for reporting, after backward has used the tensor. This demonstration creates fresh parameter tensors for one step. A repeated training loop should retain its tensors and clear old gradient buffers before accumulating the next batch's derivatives. ### Experiment: the gradient buffer is not the parameter **Predict:** run forward and backward twice on the same batch without an intervening parameter update. Does the second `.grad` contain the new derivative alone, or the sum of both? Toggle clearing and compare the recorded results. Clear gradients before the second backward pass | Moment | `b2[0]` | `b2.grad[0]` | | --- | --- | --- | | After clearing, before the first forward/backward | 0 | None | | After the first backward | 0 | -0.8648509 | | After the second fresh forward/backward | 0 | -0.8648509 | | After one SGD step using that buffer | 0.1729702 | -0.8648509; still stored | Recorded PyTorch case: **B = 1, objective = CE**. The second pass starts with an empty gradient buffer, so it stores **-0.8648509** again. Clearing gradients did not reset the bias. The data and weights are unchanged between the two passes. That is why the newly calculated derivative equals the first derivative in this controlled experiment. These are **recorded actual PyTorch runs**, not live PyTorch execution in the browser. Both backward calls follow fresh forward passes. No parameter update occurs between them; the final table row shows a single subsequent SGD update. Without clearing, this repeated batch at identical weights accumulates twice the same gradient. With different batches or updated weights, the sum would contain **different** derivatives—not generally twice the latest one. Accumulation can be intentional, but forgetting to clear is not the one-batch SGD algorithm we specified. **What stays fixed:** the common starting model and every other experiment. This checkbox selects between recorded cases; it does not train the live preview. Do not repeatedly reuse an already-consumed backward graph to implement the ordinary training loop. Compute a fresh forward pass after each update; saved backward intermediates are normally released after backward. This loop does not require `retain_graph=True`. ### Questions to ask ourselves #### What did autograd replace, and what are we still choosing? It replaced the manual derivative calculations and their bookkeeping. We still choose the data, forward operations, initialization, objective, and update rule. The tensor-only bridge even keeps the explicit SGD subtraction. #### Why can a second backward call affect learning if backward does not move parameters? It changes the stored gradient buffers. A later update uses those buffers. Old contributions can therefore change that update even though the backward calls themselves left the parameters unchanged. #### Why not call backward on loss.item()? `item()` returns a Python number without the graph. Backward belongs on the scalar loss tensor. Use the ordinary number for reporting, not differentiable computation. #### If we add a differentiable layer, must we derive the entire network again? No. Autograd composes local rules for the operations executed by the new forward program. We still must connect compatible dimensions and choose the objective. A custom operation without a supported differentiable implementation is a separate case. ### What we now know Automatic differentiation can replace our manual backward function while leaving the forward math visible. Next, layer objects will package parameters plus forward operations, and an optimizer will package the update. Those jobs should now be familiar. [Continue to 7. Use layers and an optimizer](#modules) ## 7. Use layers and an optimizer Tensors and autograd already implement the whole algorithm. PyTorch's higher-level building blocks now remove repetitive storage and update code—not the need to understand the calculation. ### Goal Read a small standard PyTorch model and training loop. Map `Linear`, `Sequential`, the loss API, and `SGD` to the explicit operations we already implemented. ### A layer owns its parameters `from torch import nn` gives us PyTorch's neural-network module. `nn.Linear(D, H)` owns weights shaped `(H, D)` and a bias shaped `(H,)`. Calling it computes `X @ weight.T + bias`. `nn.ReLU()` performs the elementwise gate and has no trainable parameters. ```python def build_model(cfg): D, H, K = cfg['input_size'], cfg['hidden_size'], cfg['classes'] model = nn.Sequential( nn.Linear(D, H), nn.ReLU(), nn.Linear(H, K), ) return model.to(device=cfg['device'], dtype=getattr(torch, cfg['dtype'])) ``` `Sequential` runs these modules in order. `model(X)` returns logits. `model.parameters()` exposes the registered trainable tensors to the optimizer. The final `.to` selects the configured device and dtype; `getattr(torch, 'float64')` means `torch.float64`. No custom model class is necessary for this stack. A more complicated forward computation can use a subclass of `nn.Module`, but that is not a prerequisite for the present network. ### The loss API has a contract Why is there no Softmax module at the end? The common cross-entropy API expects **raw logits** and combines stable normalization with the logarithmic loss. For display, use `logits.softmax(dim=1)`. To pick the winning class, `logits.argmax(dim=1)` already has the same result. In [network_torch.py](../python/network_torch.py), `from torch.nn import functional as F` gives the functional API a short name. `F.cross_entropy(logits, y)` provides the same standard calculation as calling an `nn.CrossEntropyLoss()` object. Our CE case uses `(B, 10)` logits and `(B,)` integer `torch.long` class indices. Probability targets are also supported by that API, but are not our CE representation here. ```python def objective(logits, y, kind): if kind == 'ce': return F.cross_entropy(logits, y, reduction='mean') if kind == 'mse': P = logits.softmax(dim=1) Y = F.one_hot(y, num_classes=logits.shape[1]).to(dtype=P.dtype) return F.mse_loss(P, Y, reduction='mean') raise ValueError('Loss must be ce or mse') ``` MSE here is deliberately **on probabilities**: apply softmax and construct floating-point one-hot targets. Mean reduction averages all B × 10 squared errors, matching NumPy. Applying MSE directly to the logits would define a different objective. ### Experiment: code can run while scoring a different function **Predict:** does passing probabilities to `cross_entropy` produce a type error? Switch its input and compare actual loss and bias-gradient measurements at the same initial weights and batch. Input to cross entropy - Raw logits - Probabilities from `logits.softmax(dim=1)` ```python loss = F.cross_entropy(logits, y) ``` **Intended CE input.** For B = 1, the loss is **2.001377**, and the derivative for output bias 0 is **-0.8648509**. PyTorch treats the input as unnormalized scores and performs its stable log-softmax-based calculation internally. This is a **recorded actual PyTorch comparison of two CE API calls**, even if MSE is selected for the earlier previews. It does not alter that objective or any parameters. **What to notice:** the extra-softmax call usually runs. It interprets the supplied probabilities as if they were logits, changing both the function and its derivative. A smaller resulting number would not establish a better prediction, because the scoring function changed. ### Give the optimizer its own job ```python optimizer = torch.optim.SGD(model.parameters(), lr=cfg['learning_rate']) ``` Then the actual function from [train_torch.py](../python/train_torch.py) performs our familiar sequence: ```python def train_step(model, optimizer, X_batch, y_batch, kind): optimizer.zero_grad(set_to_none=True) logits = model(X_batch) loss = objective(logits, y_batch, kind) loss.backward() optimizer.step() return loss.item() ``` `zero_grad(set_to_none=True)` clears previous gradient buffers by setting them to None. It does not zero parameters. `backward()` calculates and accumulates derivatives. `step()` applies the specified plain SGD update. The optimizer does not read labels or compute the loss for us, and it does not automatically clear gradients afterward. ### Compare actual numbers, not matching seed labels `nn.Linear` has its own default initialization. A shared seed does not impose identical random draws across NumPy and PyTorch. Our comparison therefore copies the NumPy arrays into the layers: ```python def copy_parameters(model, params): with torch.no_grad(): model[0].weight.copy_(torch.from_numpy(params['W1'])) model[0].bias.copy_(torch.from_numpy(params['b1'])) model[2].weight.copy_(torch.from_numpy(params['W2'])) model[2].bias.copy_(torch.from_numpy(params['b2'])) ``` `model[0]` is the first Linear layer; `model[2]` is the second. `copy_` writes into their existing storage; the underscore denotes an in-place operation. This copy is initialization, not learning. The models now start from the same actual values. ### Questions to ask ourselves #### Does Sequential add a loss or start training automatically? No. It composes the forward operations and owns their registered parameters. The program must still calculate a loss, run backward, and apply an update. #### Why is Softmax appropriate for our MSE branch but not before cross_entropy? We explicitly defined MSE on probabilities. Cross entropy's logits API already includes the normalization through log-softmax. Applying softmax first makes that API compute a different function. The correct choice depends on the loss's contract, not a rule that every model must end with the same displayed activation. #### What must be aligned before a framework comparison is meaningful? Use identical parameter values, inputs, labels, objective normalization, precision, and update rule. Compare logits, scalar loss, every gradient, and then an update. Similar shapes or matching random seed numbers are not sufficient. ### What we now know The manual, tensor-only, and module-based programs should perform the same calculation under matched conditions. Next we test that claim and use the runnable project to make a controlled change ourselves. [Continue to 8. Run, verify, and extend the program](#practice) ## 8. Run, verify, and extend the program We can name each library's responsibility. The final step is not trusting the shortest implementation: it is checking that each implementation calculates the intended thing. ### Goal Run the supplied code, diagnose mismatches using specific checks, and separate numerical correctness from predictive usefulness. Transfer the program to a changed hidden width and batch size. ### Use the complete project, not disconnected fragments [Download the complete Python project](../coding_lesson_code.zip). Work in its extracted copy so your experiments do not alter this lesson's fixed browser examples. The command format below follows your earlier OS selection. ```text experiment.json # architecture, objective, and training settings requirements-numpy.txt # install NumPy first requirements-torch.txt # add PyTorch afterward data/mnist.json # local real MNIST images python/ common.py # configuration and file decoding network_numpy.py # forward, loss, backward, and SGD train_numpy.py # complete NumPy training program torch_tensors.py # explicit tensor math plus autograd network_torch.py # the same model as layers train_torch.py # complete PyTorch training program verify.py # controlled numerical comparisons ``` A useful first implementation sequence is: 1. Run the supplied NumPy overfit check once to verify the environment. 2. In a separate file, type your own forward calculation, then the loss and backward functions. Reuse the data loader. Check shapes after each operation. 3. Check selected derivatives numerically before trusting a long run. 4. Read the tensor-only bridge and identify exactly what backward work disappeared. 5. Compare every parameter gradient with PyTorch, then compare one update. 6. Only then run multiple batches and evaluate unseen images. For one selected parameter, a centered finite difference compares the analytic derivative with two nearby losses: $$ \dfrac{\partial C}{\partial\theta}\approx \dfrac{C(\theta+\varepsilon)-C(\theta-\varepsilon)}{2\varepsilon}. $$ Use copies or restore the parameter afterward, and avoid crossing a ReLU kink. This is an approximation for checking a derivative, not the algorithm we use to train all parameters. The checker warns if its perturbation crosses a gate boundary. ```bash # Install PyTorch in the same environment .venv/bin/python -m pip install -r requirements-torch.txt # Inspect the bridge and verify all three implementations .venv/bin/python python/torch_tensors.py --config experiment.json .venv/bin/python python/verify.py --config experiment.json # First learn one batch, then try the teaching training split .venv/bin/python python/train_torch.py --config experiment.json --mode overfit .venv/bin/python python/train_numpy.py --config experiment.json --mode train .venv/bin/python python/train_torch.py --config experiment.json --mode train ``` These programs were executed with **Python 3.13.5, NumPy 2.4.4, and PyTorch 2.14.0**, on an Apple Silicon CPU using float64. Windows command formatting is provided, but Windows and Linux execution were not directly tested. The Python project needs no notebook, Node.js, or running webpage. ### Experiment: inspect an actual verification run **Predict:** should only the scalar loss agree, or every parameter-gradient entry too? Select a checked batch/objective pair. The values below are measured maximum absolute differences from actual Python executions. Recorded comparison case - B = 1, CE - B = 1, MSE - B = 8, CE - B = 8, MSE | Comparison | Maximum absolute difference | | --- | --- | | NumPy versus layer-model logits | 0 | | Scalar loss | 4.440892e-16 | | Every parameter-gradient entry | 6.938894e-17 | | Every updated parameter entry | 2.775558e-17 | | Tensor-only bridge: loss, gradients, updates | 4.440892e-16 | | Batched versus averaged individual calculations | 0 | | Selected finite differences versus manual derivatives | 1.363853e-11 | Recorded case: **B = 1, CE**. NumPy and PyTorch both give loss **2.001377** at the displayed precision. Backward alone changes parameter values by **0**. After one matched SGD update, the cost is **0.003867566**. The same parameters, inputs, objective, dtype, and learning rate were used. This establishes local calculation agreement, not unseen-image accuracy. The framework checks compare all 12,730 parameter-gradient entries and updated parameters. Finite differences check one selected entry from each of the four parameter arrays. Small discrepancies reflect floating-point arithmetic; exact bitwise equality is not required. This selector only displays recorded verification cases. It does not change the earlier preview batch, its objective, or its weights. ### Let a DataLoader deliver batches A `TensorDataset` pairs input and target tensors along their first dimension. A `DataLoader` iterates over those pairs in batches and can shuffle them. It delivers data; it does not predict, differentiate, or train. ```python dataset = TensorDataset(X, y) generator = torch.Generator().manual_seed(cfg['shuffle_seed']) loader = DataLoader(dataset, batch_size=B, shuffle=cfg['shuffle'], generator=generator) model.train() updates = 0 for epoch in range(cfg['epochs']): for X_batch, y_batch in loader: train_step(model, optimizer, X_batch, y_batch, cfg['loss']) updates += 1 ``` The two full-data programs use their libraries' own seeded shuffles. Those orders need not match across libraries, so final training trajectories can differ despite matching same-batch gradients. To compare a whole trajectory, also supply the exact same batch sequence. ### Keep prediction separate from learning For held-out images, run forward and score the results without backward or optimizer updates. The supplied PyTorch evaluation helper makes that boundary explicit: ```python def scores(model, X, y, kind): model.eval() with torch.no_grad(): logits = model(X) loss = objective(logits, y, kind).item() accuracy = (logits.argmax(dim=1) == y).double().mean().item() return loss, accuracy ``` `model.eval()` selects evaluation behavior for modules such as dropout and batch normalization. `torch.no_grad()` disables gradient tracking for the enclosed calculation. They are different things. Our Linear/ReLU model has no mode-dependent layers, but the distinction matters when extending it. Conversely, `model.train()` selects a mode; calling it does not train the model by itself. Our 200-training / 100-test image subset is a reproducible exercise, not a full-MNIST benchmark. Repeatedly using held-out scores to choose settings makes that set validation data rather than an untouched final test. The scripts print results and exit; **trained weights are not saved**. ### Synthesis: what did each tool take over? | Version | You specify | The library supplies | | --- | --- | --- | | Manual NumPy | Forward operations, loss, all backward rules, updates, and data loop. | Array storage and numerical operations. | | PyTorch tensors | Forward operations, objective, initialization, explicit update, and data loop. | Tensor operations and graph-based differentiation. | | Layers plus SGD | Architecture, objective, initialization choice, batch loop, and experiment. | Parameter-owning layers, autograd, and the specified optimizer update. | You do not need another neural-network framework immediately. First explain the PyTorch loop in terms of the manual program. Full MNIST, a convolutional network, checkpoints, and GPU execution are later extensions—not prerequisites for understanding this calculation. ### Questions to ask ourselves #### Transfer: change the hidden width to 32. Which shapes change, and which backward rules need a new derivation? W1 becomes `(32, 784)`, b1 becomes `(32,)`, Z1 and A1 become `(B, 32)`, and W2 becomes `(10, 32)`. Outputs and label formats stay unchanged. The dense/ReLU backward rules need no new derivation; their dimensions follow the new width. Change `hidden_size` in your copied configuration and rerun verification. #### Forward values agree, but a gradient is eight times smaller. What should you inspect first? Inspect loss normalization and repeated batch averaging. An extra division by batch size 8 is a strong suspect. Also check whether one implementation sums examples while the other averages them. Diagnose the definitions before compensating with a new learning rate. #### Why can two same-seed programs end with different accuracy? A seed does not impose a universal random sequence across libraries. Initialization, batch order, dtype, and arithmetic can differ. Our local comparison removes these differences by copying actual weights and fixing a batch. A long-run comparison also needs the same sequence of batches. #### If training accuracy is high but new-image accuracy is poor, is autograd necessarily broken? No. A correct implementation can overfit or use unrepresentative data. Check arithmetic separately from generalization. Numerical agreement verifies the local calculation, not the usefulness of the learned classifier. ### A concrete next exercise In your downloaded copy, change `hidden_size` to 32 and `batch_size` to 5. Predict the shapes and the 40 updates per epoch before running it. Run verification, then the fixed-batch overfit check. Explain any mismatch before increasing the scope. The complete process remains **store parameters → predict → measure loss → calculate gradients → update parameters → repeat**. Frameworks change who implements those operations, not their roles. [Return to 1. Know the tools before using them](#tools) ## Sources and execution notes The following official sources were consulted to check the explanation and API contracts. They are optional further reading; the core lesson is local. Research checked on 21 September 2026. - [Python Packaging User Guide: pip and virtual environments](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/) — interpreters, packages, requirements files, and terminal commands. - [Python 3.13: venv](https://docs.python.org/3.13/library/venv.html) — explicit interpreter paths work without activation. - [NumPy: the absolute basics](https://numpy.org/doc/stable/user/absolute_beginners.html) and [broadcasting](https://numpy.org/doc/stable/user/basics.broadcasting.html) — arrays, shapes, indexing, axes, and compatible dimensions. - [PyTorch: tensors](https://docs.pytorch.org/tutorials/beginner/basics/tensorqs_tutorial.html) — array-like operations, dtypes, devices, and memory sharing. - [PyTorch: automatic differentiation](https://docs.pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html) — graphs, backward, accumulation, and detaching. - [PyTorch 2.14: Linear](https://docs.pytorch.org/docs/2.14/generated/torch.nn.Linear.html), [CrossEntropyLoss](https://docs.pytorch.org/docs/2.14/generated/torch.nn.CrossEntropyLoss.html), and [MSELoss](https://docs.pytorch.org/docs/2.14/generated/torch.nn.MSELoss.html) — parameter layout and objective contracts. - [PyTorch: optimizing parameters](https://docs.pytorch.org/tutorials/beginner/basics/optimization_tutorial.html) and [datasets and loaders](https://docs.pytorch.org/tutorials/beginner/basics/data_tutorial.html) — optimization and batch iteration. - [Autograd mechanics](https://docs.pytorch.org/docs/2.14/notes/autograd.html) and [reproducibility](https://docs.pytorch.org/docs/2.14/notes/randomness.html) — evaluation versus no-grad and limits of matching seeds. The images are genuine MNIST by Yann LeCun, Corinna Cortes, and Christopher J. C. Burges, retained from the [Keras-hosted archive](https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz). Source checksums and original split indices are retained. No Fashion-MNIST images or synthetic substitutes are used. See the local [README](../README.md), [validation notes](../verification/VALIDATION.md), and this [complete Markdown manuscript](lesson.md). Numerical tests are evidence about the implementation, not a guarantee that every reader will find every explanation sufficient. ## Implementation appendix — reviewed interface and state text **For maintainers, not another teaching section.** The complete initial lesson ends above. This appendix keeps all shared interface wording and alternative observations in the same source of truth. Placeholder tokens below represent computed values in an interaction; they are not unfinished lesson paragraphs. Review changes here before rebuilding the webpage. The initial text above and its matching default templates must continue to agree. ```json interface { "language": "en", "subtitle": "From NumPy to PyTorch", "position": "A worked MNIST coding lesson", "contentsLabel": "Lesson contents", "readingInstruction": "Read in order. Predict before changing a control. Try the questions before revealing their answers.", "skipLink": "Skip to the lesson", "downloadLink": "Download the Python project", "manuscriptLink": "Read the Markdown source", "sourcesLink": "Sources and execution notes", "backToStart": "Back to the overall goal", "footnote": "A local learning aid. Reloading restores the preview defaults. No account, GPU, runtime CDN, or cloud computation is needed to read the lesson.", "error": "The interactive lesson could not start or update. Open it through the local HTTP server and check the README. The complete initial explanation remains below; technical details are in the browser console.", "announcements": { "batch": "Preview batch changed. The step walkthrough is reset; starting parameters are unchanged.", "loss": "Preview objective changed. Predictions and starting parameters are unchanged; the step walkthrough is reset.", "reset": "The walkthrough is back at the original starting point.", "step": "Showing operation {{phase}} of 6." }, "figures": { "digitAlt": "MNIST image with known label {{label}}", "digitCaption": "Known label: {{label}}" }, "controls": { "command-kind": {"type":"select","values":["install","import","run"],"initial":"install"}, "os-select": {"type":"select","values":["unix","windows"],"initial":"unix"}, "batch-select": {"type":"select","values":["1","8"],"initial":"1"}, "shape-operation": {"type":"select","values":["correct","transpose","elementwise"],"initial":"correct"}, "neuron-select": {"type":"select","values":["0","1","2","3","4","5","6","7","8","9","10","11","12","13","14","15"],"initial":"0"}, "loss-select": {"type":"select","values":["ce","mse"],"initial":"ce"}, "bias-axis": {"type":"select","values":["0","1","all"],"initial":"0"}, "step-actions": {"type":"buttons","ids":["step-next","step-reset"]}, "clear-grad": {"type":"checkbox","initial":true}, "ce-input": {"type":"select","values":["logits","probabilities"],"initial":"logits"}, "verification-case": {"type":"select","values":["1-ce","1-mse","8-ce","8-mse"],"initial":"1-ce"} } } ``` Run this in the **terminal**. It launches a specific Python interpreter, which runs pip and installs packages into that interpreter's environment. It changes the environment, not the model's weights. Put this in a **Python file or Python prompt**. It loads an already-installed package and binds the name `np`. It does not install NumPy and does not train a network. Run this in the **terminal**. The interpreter executes the named file. That file's training loop defines how the model learns; the shell command does not supply the algorithm. ```bash python3 --version python3 -m venv .venv .venv/bin/python -m pip install -r requirements-numpy.txt ``` ```powershell py -3.13 --version py -3.13 -m venv .venv .\.venv\Scripts\python.exe -m pip install -r requirements-numpy.txt ``` ```bash .venv/bin/python hello.py ``` ```powershell .\.venv\Scripts\python.exe hello.py ``` ```bash .venv/bin/python python/train_numpy.py --config experiment.json --mode overfit ``` ```powershell .\.venv\Scripts\python.exe python/train_numpy.py --config experiment.json --mode overfit ``` ```bash # Install PyTorch in the same environment .venv/bin/python -m pip install -r requirements-torch.txt # Inspect the bridge and verify all three implementations .venv/bin/python python/torch_tensors.py --config experiment.json .venv/bin/python python/verify.py --config experiment.json # First learn one batch, then try the teaching training split .venv/bin/python python/train_torch.py --config experiment.json --mode overfit .venv/bin/python python/train_numpy.py --config experiment.json --mode train .venv/bin/python python/train_torch.py --config experiment.json --mode train ``` ```powershell # Install PyTorch in the same environment .\.venv\Scripts\python.exe -m pip install -r requirements-torch.txt # Inspect the bridge and verify all three implementations .\.venv\Scripts\python.exe python/torch_tensors.py --config experiment.json .\.venv\Scripts\python.exe python/verify.py --config experiment.json # First learn one batch, then try the teaching training split .\.venv\Scripts\python.exe python/train_torch.py --config experiment.json --mode overfit .\.venv\Scripts\python.exe python/train_numpy.py --config experiment.json --mode train .\.venv\Scripts\python.exe python/train_torch.py --config experiment.json --mode train ``` **Valid expression.** `({{B}}, 784) @ (784, 16)` produces `({{B}}, 16)`. The `(16,)` bias broadcasts across the rows. There is one row of 16 hidden weighted sums for each image. The first entry is `Z1[0, 0] = {{z}}`. A larger B adds rows, not trainable weights. **Wrong inner dimensions.** `({{B}}, 784) @ (16, 784)` requires 784 to equal 16. It does not, so NumPy would raise a `ValueError`. W1 stores one neuron per row; transpose it for these batch-first inputs. **Wrong operation.** `*` is elementwise multiplication, not a collection of dot products. Shapes `({{B}}, 784)` and `(784, 16)` are not broadcast-compatible because their last dimensions differ. Use `@` for this matrix multiplication. For the first image, digit **0**, inspect hidden neuron **{{j}}**. The loop includes all 784 products and its bias, giving **{{sum}}**. The matrix operation gives `Z1[0, {{j}}] = {{z}}`. ReLU gives `A1[0, {{j}}] = {{a}}`. This neuron is **inactive for this image**. The other 15 neurons still participate in the full network. For the first image, digit **0**, inspect hidden neuron **{{j}}**. The loop includes all 784 products and its bias, giving **{{sum}}**. The matrix operation gives `Z1[0, {{j}}] = {{z}}`. ReLU gives `A1[0, {{j}}] = {{a}}`. This neuron is **active for this image**: the positive weighted sum passes through unchanged. The other 15 neurons still participate in the full network. Preview: **B = {{B}}, objective = {{kind}}**. The loss is **{{loss}}**. At the unchanged starting weights, `P[0, 0] = {{p}}`. The logit-gradient array has shape `({{B}}, 10)`. Changing the objective changes this signal, not the prediction or the starting parameters. Result shape: **({{size}},)**. ```text {{values}} ``` Result shape: **() — scalar**. ```text {{value}} ``` **Correct:** sum across images and retain one derivative for each output bias. Different classes receive different values. The batch average is already present in `dZ2`. **Wrong meaning:** this sums across classes, leaving one value per image rather than one per bias. With B = 1, the length-one result can broadcast silently across all ten biases. The near-zero total does not imply that the separate class derivatives are zero. **Wrong meaning:** this sums across classes, leaving {{B}} values rather than one per bias. The `({{B}},)` result cannot broadcast against our `(10,)` bias vector. The near-zero row totals do not imply that the separate class derivatives are zero. **Wrong reduction:** both axes collapse to one scalar. NumPy can broadcast it to all ten biases, silently applying the same number everywhere. The near-zero total is cancellation, not ten separate zero derivatives. Preview arguments: **B = {{B}}, objective = {{kind}}, learning rate = {{rate}}**. We inspect output bias `b2[0]`, but every parameter participates. The last two lines below are added post-update checks. The saved configuration file is unchanged. **Starting point.** The model copy has `b2[0] = {{old}}`. No walkthrough operation has been selected. Predict which line will first change the parameter. **Forward.** Predictions and cached activations are now available. `P[0, 0] = {{p}}`. The bias is still **{{old}}**. Calculating a prediction is not a learning update. **Loss and logit signal.** The cost is **{{loss}}**, and `dZ2` has shape `({{B}}, 10)`. We have measured the mismatch and its output sensitivity. The bias is still **{{old}}**. **Backward.** All four parameter-gradient arrays are now available. The derivative $\dfrac{\partial C}{\partial b^{(2)}_0}$ is **{{g}}**. The bias itself is still **{{old}}**. Every derivative used the old parameters and cached forward values. No parameter update has happened on this walkthrough copy yet. **Update.** `b2[0]` changes from **{{old}}** to **{{after}}**, using learning rate **{{rate}}** and gradient **{{g}}**. All 12,730 parameters receive their own updates on this copy. The old variable `loss` still contains **{{loss}}**; it has not been recomputed. The displayed bias alone does not explain the whole model's subsequent cost change. **New forward pass.** At the updated parameters, `P[0, 0] = {{pAfter}}`. Score these new predictions to find the new loss. The old gradient remains the one that produced the previous update. **New measurement.** The cost changes from **{{loss}}** to **{{lossAfter}}** on the same batch and objective. This completes one step. A training loop keeps the updated parameters and repeats with a fresh forward and backward pass. Reset this preview to return to the common initial baseline; the earlier sections have not been retrained. | Moment | `b2[0]` | `b2.grad[0]` | | --- | --- | --- | | After clearing, before the first forward/backward | {{old}} | None | | After the first backward | {{old}} | {{g}} | | After the second fresh forward/backward | {{old}} | {{second}} | | After one SGD step using that buffer | {{after}} | {{second}}; still stored | Recorded PyTorch case: **B = {{B}}, objective = {{kind}}**. The second pass starts with an empty gradient buffer, so it stores **{{second}}** again. Clearing gradients did not reset the bias. The data and weights are unchanged between the two passes. That is why the newly calculated derivative equals the first derivative in this controlled experiment. Recorded PyTorch case: **B = {{B}}, objective = {{kind}}**. Without clearing, the second derivative is added to the first: the buffer becomes **{{second}}**, twice **{{g}}**. Both backward passes leave the bias unchanged. The subsequent SGD step uses the accumulated buffer, so its change is twice the intended one-batch change in this controlled case. This is not the same as calculating a new derivative at updated weights. ```python loss = F.cross_entropy(logits, y) ``` ```python loss = F.cross_entropy(logits.softmax(dim=1), y) ``` **Intended CE input.** For B = {{B}}, the loss is **{{loss}}**, and the derivative for output bias 0 is **{{g}}**. PyTorch treats the input as unnormalized scores and performs its stable log-softmax-based calculation internally. **The call runs, but the function changes.** For B = {{B}}, the loss is **{{loss}}**, and the derivative for output bias 0 is **{{g}}**. PyTorch treats the already-normalized probabilities as logits. The different loss is not a fair comparison of prediction quality with the original CE. No starting weight or label changed in this experiment. | Comparison | Maximum absolute difference | | --- | --- | | NumPy versus layer-model logits | {{forward}} | | Scalar loss | {{lossError}} | | Every parameter-gradient entry | {{gradient}} | | Every updated parameter entry | {{update}} | | Tensor-only bridge: loss, gradients, updates | {{tensor}} | | Batched versus averaged individual calculations | {{mean}} | | Selected finite differences versus manual derivatives | {{finite}} | Recorded case: **B = {{B}}, {{kind}}**. NumPy and PyTorch both give loss **{{loss}}** at the displayed precision. Backward alone changes parameter values by **{{change}}**. After one matched SGD update, the cost is **{{afterLoss}}**. The same parameters, inputs, objective, dtype, and learning rate were used. This establishes local calculation agreement, not unseen-image accuracy.