About & References

How the labs build

The 19 labs build in five stages:

1. Foundations (Labs 1-5). The linear algebra and calculus that neural networks are made of, a working Python environment, and the PyTorch basics — tensors, autograd, and the training loop.
2. Core architectures (Labs 6-12). A single neuron and its activation function, forward and back propagation done by hand, then dense networks, convolutional networks for images, recurrent networks and LSTMs for sequences, and transfer learning to reuse what others have trained.
3. Advanced topics (Labs 13-16). The techniques that decide whether a network generalises, the attention mechanism that replaced recurrence, and two families of generative model.
4. Production (Lab 17). Everything that happens after the accuracy prints.
5. Retrieval (Labs 18-19). Reusing the Transformer as a search engine, and grounding a language model in your own documents.

Ideas that recur throughout

A few ideas show up again and again, and they are worth naming:

• Additive paths preserve gradients. It is the same trick in the LSTM's cell state, in ResNet's skip connections, and in the Transformer's residual connections.
• Gating is just a sigmoid used as a valve — a learned decision about how much information passes.
• Regularization is deliberate difficulty. Every technique in Lab 13 makes training harder on purpose, in exchange for generalisation.
• model.train() and model.eval() are not cosmetic. Dropout and batch norm behave differently in each, and forgetting the switch fails silently.
• Build it by hand once, then use the library. Every lab implements the mechanism from scratch, verifies it against PyTorch's version, and then uses PyTorch's.

Each lab ends with an experiment section. Those are the point — the code is meant to be broken and rebuilt.

References

1. PyTorch Documentation — official documentation for PyTorch.
2. Machine Learning Knowledge (machinelearningknowledge.ai) — the original source of several diagrams in Labs 01 and 06-09. The site no longer resolves, so those diagrams have been redrawn and now live in Misc/.
3. Miniconda Documentation — official documentation for Miniconda.
4. Understanding LSTM Networks — Christopher Olah's explanation of LSTMs, and the source of the LSTM diagrams in Lab 11.
5. Attention Is All You Need — the original Transformer paper.
6. Auto-Encoding Variational Bayes — the paper that introduced the VAE and the reparameterization trick.
7. Generative Adversarial Networks — the original GAN paper.
8. Dive into Deep Learning — a free interactive book covering most of these topics in greater depth.

Additional resources

• 3Blue1Brown — visual tutorials for neural networks and maths.
• Deep Learning with PyTorch: A 60 Minute Blitz — a beginner-friendly tutorial on deep learning with PyTorch.
• Neural Networks and Deep Learning — online book by Michael Nielsen.
• PyTorch Tutorials — complete beginner course.
• Programming with Mosh — Python for beginners.

License

This project is MIT licensed.