In this course we shall be taking some first steps in applying and understanding deep neural networks. We'll be asking:

- What is deep learning and what are neural networks?

- How can we use them, train them, build them?

> And most importantly we'll give examples of deployment with industry standard python
> packages: ['PyTorch'](https://pytorch.org) and ['Lightning'](https://lightning.ai).

### But to warm us up...

A revolution in amongst us. Deep learning holds the keys to unlocking the reams of unstructured data which is constantly collected in the modern world. With applications in the fields of computer vision--object detection, classification--to natural language processing--knowledge from text and large language models--we have only recently started building the correct tools to approach and solve big problems in industry.

With vast reservoirs of data, the past champions in machine learning and statistics struggle to draw inference without manual engineering of the data. In _deep learning_, we develop models that can learn subtle patterns in vast datasets.

And the applications are numerous:

- Reinforcement Learning
- Computer Vision
- Natural Language Processing (inc. Large Language Models)

to name but a few. To begin developing in these fields, you need to know the fundamentals. Bother theoretical and practical. And that's what we'll cover in this course.

**In this course, you will get to grips with:**

- Neural network architecture and 'learning' algorithms.
- Training and validation approaches: loss functions, hyperparameters, tips & tricks.
- Using the frameworks built into both PyTorch and Lightning for smooth training and development.
- Applying end-to-end models in applications in the wild. We have one example looking as **optimisation in drug discovery** and another in **image classification**.

**This course a gentle introduction to the topic.**
It sits alongside [The Kernel Trick](/course/the-kernel-trick-flexible-machine-learning) and [Tree-based algorithms](/course/random-forests-and-gradient-boosted-trees). And you could warm up with Tim's intro course, [Getting started in machine learning](/course/getting-started-in-machine-learning).

Deep neural networks are now solving problems that, until recently, were out of reach. You’ve no doubt seen them pop up in the news as part of the AI revolution. But let’s take a step back. What are neural networks? What is a neuron? What makes them deep? And how can we train and use these models? In this course, we start at the beginning, explaining the fundamental concepts and introducing programming essentials to prepare you to deploy and understand deep neural networks.

Dr Andy Corbett

Artificial intelligence is the future. Most of us have a good idea about what it can do. But do we understand what AI _really_ is?

<Image
	src={"https://hackmd.io/_uploads/BJZ3xl-N6.png"}
	alt={""}
	width={1000}
	height={560}
/>

Welcome to our first course on deep learning. We are going to try to answer to fundamental questions:

<Callout type="info">

1. 'What is deep learning?'

2. and 'What _is_ deep learning?'

</Callout>

Hold your breath: it's maths! And some quite simple mathematics at that. But that's not something we need to worry about. There are brilliant tools out there with which we can assemble AI models with there component parts, like machines. And why are we interested?

> Vastly over-parameterised neural networks can solve complex problems like no other approaches.

But we aren't starting at the "most complex". In this course we'll start at the beginning. our key goal is to **learn to code neural networks and use them in the wild**. We shall start with some recent motivation; and we'll uncover all the concepts and lingo so that you can hold with a deep learning engineer.

## Course Contents

Over the next few video lessons we shall cover the following topics:

<Callout type='success'>

1. ✅ An introduction to deep learning and applications.
2. ✅ A deep dive into the prototype neural network: a multi-layered perceptron.
3. ✅ Deep learning fundamentals:
   - Forward-passing and architecture
   - Training routines and backpropagation
   - Loss functions, hyperparameters and regularisation
4. ✅ Hands-on deployment:
   - Building models with PyTorch
   - Accelerating deployment with PyTorch Lightning
5. ✅ Neural Nets in the wild: - Building surrogates for drug discovery and parameter-space search. - Building classifiers and interrogating hidden layers.

</Callout>

## In this video...

We shall take a wide-angled look at deep learning applications across the board. From computer vision; language processing; and reinforcement learning. This is our motivation All of these fields of application rely fundamentally on the basic tools that we shall introduce in this course.


An Introduction to Deep Learning

<Callout type='success' heading='In this video you will...'>

- ✅ See a first-hand graphical representation of the building blocks that compose every neural network.
- ✅ Learn how to fit components together to construct wide and deep MLPs.
- ✅ Review the back propagation process--the essential mechanism by which networks are trained.

</Callout>

We seem to be talking about AI and deep neural networks all the time. But can any of us actually write one down? That's what we shall do in this video. **We will write down the prototype neural network: a multi-layered perceptron (MLP)**.

<Image
	src={"https://hackmd.io/_uploads/ByNq6-WV6.png"}
	alt={""}
	width={960}
	height={640}
/>

The MLP is modeled on the human brain, and as such composed of many small neurons--or component models. Each of these come with parameters, and each of those needs to be trained so that the model converges on its target.

## What do we mean by component models?

Let's pull up for a moment ask ourselves, how can we interpret models? Suppose that there has been a new law forcing our bananas to be straighter--the curvy ones are getting the chop! We would need a model to predict the curvature--or alternatively the eccentricity--from easily observed factors such as the weight.

<Image
	src={"https://hackmd.io/_uploads/SJKFY-b4T.png"}
	alt={""}
	width={1000}
	height={665}
/>

So suppose I have the weights and arc lengths of a bunch of bananas

$$
\mathbf{x} = [(119g, 29cm), (124g, 32cm), (101g, 27cm), (132g, 34cm), ... ].
$$

and for these we have recorded their eccentricities: $y = [0.21, 0.30, 0.25, 0.18, ...]$. We have a supervised learning task. What is the most simple model that we could try to explore this relationship with? A linear model:

$$
y = \mathbf{m}\cdot\mathbf{x} + c.
$$

It is interpretable as we have a physical understanding of the parameters $m$ and $c$ as the scaling factor and baseline eccentricity.

But let's say the problem becomes more complicated. In that case we may need to add other functions to our model, such as a logistic function $\sigma$:

$$
y = \sigma(\mathbf{m}\cdot\mathbf{x} + c).
$$

<Image
	src={"https://hackmd.io/_uploads/Hktf1fb46.png"}
	alt={""}
	width={600}
	height={400}
/>

## Did someone mention something about features?

Okay, the features. Well they are the columns of $\mathbf{x}$, which for our slippery example is weight and arclength.

> But perhaps we need much more complexity.

A linear model supports no interaction between these features. A **deep neural network** is quite the opposite: many models of the form $y = \sigma(\mathbf{m}\cdot\mathbf{x} + c)$ are stacked on top of each other, connected in every possible way, for the purpose of combining the features in a way which might predict the answer.

In fact, since neural networks are quite robust with respect to the type of input data, we could even substitute the measurements made for a picture of the banana. That's the power of deep learning.

In this video we'll take a look at how these large conglomerate models are formed and optimised at a high level, but with all the details in plain sight.

<Image
	src={"https://hackmd.io/_uploads/HJvmHfb4a.png"}
	alt={""}
	width={1000}
	height={561}
/>


What are Deep Neural Networks?

:::success
**📂 Resources**
Download the resources for this lesson <Link href="/resources/deep-learning-and-neural-networks/coding-neurons-from-scratch/Resources.zip" download="Resources.zip">here.</Link>
:::

<Callout type='success' heading='In this video you will...'>

- ✅ Learn how to code up neurons by hand.
- ✅ Implement gradients and update parameters.
- ✅ See the linear restriction of single neurons.
- ✅ Toy with how larger MLPs are implemented.

</Callout>

The best way to explore a new idea is by doing. So let's do it! We know what a neural network is. So let's write some first-principles code to act it our. That's right, we'll code up a neural network using just `numpy` alone.

## The task

Here's a basic problem: a bakery has invested in a new machine to make its current buns. It was working well. The mixture went in and the dough came out very well proportioned as a good bun should. Until one day, someone fiddled with the knobs. Since then, then buns have all been tall and thin and very short and not so thin.

Thankfully, the bakery recorded the heights and widths of the buns for each the machine produced, before and after.

They would like to change controls on the machine and would like you to build a detector to check if the outputs are well proportioned, like 'before', or not, as they were 'after'.

### Generate the data

Here's the our codification of the problem:

```python
# Generate a table of height and width data
np.random.seed(SEED)
X = np.random.uniform(low=[3.5, 2.0], high=[7.5, 5.0], size=[200, 2])

# Assign class based on thresholds
y = np.ones([len(X),])
y[(X[:, 0] > 5.5) & (X[:, 1] > 3.5)] = 0. # Bun too tall and wide
```

<Image
	src={"https://hackmd.io/_uploads/rk__TGZNT.png"}
	alt={""}
	width={567}
	height={453}
	caption="Fig. 1. Our 2D data points spread over the array. Our goal is to predict the target colour."
/>

## Step one: encode a single neuron

Our activation on each neuron is given by the logistic function:

```python
def logistic(alpha):
    """Our activation function."""
    return 1 / (1 + np.exp(-alpha))
```

which looks as follows:

<Image
	src={"https://hackmd.io/_uploads/r1-GCGWVa.png"}
	alt={""}
	width={547}
	height={413}
	caption="Fig. 2. The logistic activation function"
/>

Now we can define a class to control a single neuron.

```python
class SingleNeuron:
    """A single percepton neuron."""

    def __init__(self, weights, bias, activation_func=logistic):
        self.weights = weights
        self.bias = bias
        self.activation = activation_func

    def forward(self, x):
        linear_sum = np.dot(x, self.weights) + self.bias
        return self.activation(linear_sum)
```

As a sanity check, let's run our problem through this model and see what it predicts.

```python
# Predictions
x_axis = np.linspace(3.5, 7.5, 1000)
y_axis = np.linspace(2.0, 5.0, 1000)
xx, yy = np.meshgrid(x_axis, y_axis)
zz = neuron.forward(np.stack((xx.ravel(), yy.ravel()), axis=1))
zz = zz.reshape(xx.shape)

# Contours
contours = plt.contour(xx, yy, zz, levels=100)
plt.colorbar(contours)

# Data
good = plt.scatter(X[y==1, 0], X[y==1, 1], color='goldenrod', zorder=2)
bad = plt.scatter(X[y==0, 0], X[y==0, 1], color='tab:blue', zorder=2)

plt.title('Bakery bun data')
plt.ylabel('Bun Height (cm)')
plt.xlabel('Bun Width (cm)')
plt.legend([good, bad], ['Accept', 'Reject'], loc='upper right', framealpha=1.)
plt.show()
```

<Image
	src={"https://hackmd.io/_uploads/SkSK0zWVp.png"}
	alt={""}
	width={552}
	height={453}
	caption="Fig. 3. An untrained neuron predicting the output."
/>

## Step 2: Train the neuron

Now things begin to get tricky with our stubborn 'by hand' approach. To assess performance, we must define what we mean by 'good'. To keep things simple, we shall apply a mean-squared loss function between the output and the target. (Not really appropriate for a classification problem, but we'll look into this in more depth later.)

```python
def mse(y_true, y_pred):
    return ((y_true - y_pred) ** 2).mean()
```

For our backpropagation routine, we shall need the derivatives of both this function and the logistic function we defined earlier.

```python
def logistic_derivative(x):
    """Derivative of the logistic function"""
    sigma = logistic(x)
    return sigma * (1 - sigma)

def mse_derivative(y_true, y_pred):
    return -2 * (y_true - y_pred)
```

Finally, we can add new methods to our Single Neuron Class so that we can train the parameters on the dataset. We shall do this using _stochastic gradient descent_, but other algorithms are available--the important point is that we are updating the parameters on each epoch.

```python
class SingleNeuronGrads:
    """A single percepton neuron with gradients."""

    def __init__(self, weights, bias, activation_func=logistic):
        self.weights = weights
        self.bias = bias
        self.activation = activation_func

    def forward(self, x):
        linear_sum = np.dot(x, self.weights) + self.bias
        return self.activation(linear_sum)

    def get_gradients(self, x, y_true):
        """Computes gradient of MSE loss with respect to weights and bias."""
        y_pred = self.forward(x)
        dL_dy = mse_derivative(y_true, y_pred)
        dy_db = logistic_derivative(np.dot(x, self.weights) + self.bias)
        dy_dw = np.matmul(
            dy_db[:, np.newaxis, np.newaxis],
            x[:, np.newaxis, :],
        )
        dL_dw = np.mean(dL_dy[:, np.newaxis, np.newaxis] @ dy_dw, axis=0)
        dL_db = np.mean(dL_dy * dy_db)
        return dL_dw.squeeze(), dL_db

    def train(self, X_train, y_train, num_epochs=100, learning_rate=0.01):
        """Optimise parameters for training data."""
        loss_record = list()
        for ep in range(num_epochs):
            dL_dw, dL_db = self.get_gradients(X_train, y_train)
            self.weights -= learning_rate * dL_dw
            self.bias -= learning_rate * dL_db
            rmse = np.sqrt(mse(y_train, self.forward(X_train)))
            loss_record.append(rmse)
            print(f'Epoch {ep}: RMSE = {rmse:2.4f}')
        return loss_record
```

Training is then as simple as

```python
loss = neuron.train(X_train=X, y_train=y, num_epochs=100000)
```

### Result

Well, how did it do? Our problem we posed in Fig. 1 is almost linear--easy to solve. But can our linear model find the answer?

<Image
	src={"https://hackmd.io/_uploads/HJbGb7WVa.png"}
	alt={""}
	width={552}
	height={453}
	caption="Fig. 4. A trained neuron (i.e. a logistic regressor) solving the problem."
/>

This could certainly be better. But at the very least, the parameters have re-oriented to point to the correct solution. To improve further, we shall need to extend the complexity of the model by adding more neurons. Let's do it!

## Adding more neurons

Let's use our single neuron class to code a complete multi-layered perceptron with two layers, containing (4, 3) neurons, respectively.

```python
class MultiLayerPerceptron:
    def __init__(self):
        self.n1 = SingleNeuron(np.random.normal(size=[2,]), 0.)
        self.n2 = SingleNeuron(np.random.normal(size=[2,]), 0.)
        self.n3 = SingleNeuron(np.random.normal(size=[2,]), 0.)
        self.n4 = SingleNeuron(np.random.normal(size=[2,]), 0.)

        self.n5 = SingleNeuron(np.random.normal(size=[4,]), 0.)
        self.n6 = SingleNeuron(np.random.normal(size=[4,]), 0.)
        self.n7 = SingleNeuron(np.random.normal(size=[2,]), 0.)

    def forward(self, x):

        # First hidden layer of width 4
        node11 = self.n1.forward(x)[:, np.newaxis]
        node12 = self.n2.forward(x)[:, np.newaxis]
        node13 = self.n3.forward(x)[:, np.newaxis]
        node14 = self.n4.forward(x)[:, np.newaxis]
        z1 = np.concatenate((node11, node12, node13, node14), axis=-1)

        # Second hidden layer of width 2
        node21 = self.n5.forward(z1)[:, np.newaxis]
        node22 = self.n6.forward(z1)[:, np.newaxis]
        z2 = np.concatenate((node21, node22), axis=-1)

        # Output layer
        return self.n7.forward(z2)
```

Now, deriving the gradients for this model is far more intricate. We shan't implement this here. But why would we? `PyTorch` has very efficient routines for implementing these algorithms. We'll explore this is the next videos.

In the meantime, let's take a look at the impact of constructing a deep MLP on the linearity of the model.

<Image
	src={"https://hackmd.io/_uploads/SJ4pf7ZE6.png"}
	alt={""}
	width={548}
	height={443}
	caption="Fig. 5. An untrained multi-layed perceptron exhibiting non-linear behaviour.
"
/>


Coding Neurons from Scratch

<Callout type='success' heading='In this video you will...'>

- ✅ Take a short mathematical tour on the ins/outs of back propagation.
- ✅ Have a go at the chain rule.
- ✅ Compute gradients.
- ✅ Update parameters.

</Callout>

This video is "extra-curricular". You are more than welcome to skip it, toss it, never look at it again. Why? We're diving into he mathematics that allows us to perform backpropagation. There are plenty algorithms out there designed to keep track of this under the hood--and we shall explore these shortly.

But if you're like me, maybe you just want to know! So pull out you pencil and paper and see if you can follow along yourself.

## A forward pass ends in computing the loss.

Here's our neural network $f\colon \mathbb{R}^n \rightarrow \mathbb{R}$; that means it is a function from our input space of vectors $\mathbb{R}^n$ to a scalar value $\mathbb{R}$.

Suppose we have a training set of $D$ input-output data points $(\mathbf{x}_d, y_d)$ for $d=1,\ldots, D$.

Then after passing $\mathbf{x}_d$ through the network our next step is to assess the quality of the answer; we want to compare $f(\mathbf{x}_d)$ against the ground truth, $y_d$.

In this example, we perform this with a least-squares (MSE) loss function. Such a loss function would be appropriate for regression problems. The function can be written as

$$
L = \frac{1}{D} \sum_{d=1}^{D} (y_d - f(\mathbf{x}_d))^2.
$$

<Callout type='info'>
## Let's talk about gradients

What is all this chatter about gradients? What is a gradient? Quite simply, it is the rate at which something changes at a particular point.

If the loss $L$ of our neural network responds to an input $\mathbf{x}$, then we can ask ourselves "what happens if we perturb $\mathbf{x}$ a little bit?", by $\mathbf{x} + \varepsilon$, say. How sensitive is $L$ to a little change by $\varepsilon$?

If $L$ is highly sensitive and a little shift in $\mathbf{x}$ shows it rapidly decreases, then that's the direction we want to follow! **We can use gradients to work out where the loss becomes smaller.**

</Callout>

## Gradients of a single neuron

A single neuron takes the form

$$
f(\mathbf{x}) = \sigma(\mathbf{w}\cdot\mathbf{x} + b)
$$

where $\sigma$ is the logistic function we saw in previous videos: $\sigma(r) = (1 + r^{-r})^{-1}$. We are looking for the gradients with respect to the neuron parameters given by

$$
\frac{\partial L}{\partial \mathbf{w}}\quad \text{and}\quad \frac{\partial L}{\partial b}.
$$

Now it's time for you pencil and paper.

### Step 1: Apply the chain rule

The chain rule allows us to differentiate compositions of functions.

> For example, $L$ is a function of $f$ and $f$ is a function of $\mathbf{w}$ and $b$.

For each component $\mathbf{w} = [w_{i}]_i$, the formula gives us

$$
\frac{\partial L}{\partial w_i}
=\frac{1}{D}\sum_{d=1}^{D}
\frac{\partial L(y_{d}, f(\mathbf{x}_{d}))}{\partial f(\mathbf{x}_{d})}\cdot \frac{\partial f(\mathbf{x}_{d})}{\partial w_{i}}
=\frac{1}{D}\sum_{d=1}^{D} -2(y_{d} - f(\mathbf{x}_{d})) \cdot \frac{\partial f(\mathbf{x}_{d})}{\partial w_{i}}.
$$

### Step 2: Apply the chain rule again

Now we need to find the second derivative above:

$$
\frac{\partial f(\mathbf{x}_{d})}{\partial w_{i}} = \frac{\partial}{\partial w_i}[\sigma(\mathbf{w}\cdot\mathbf{x} + b)] = \sigma'(\mathbf{w}\cdot\mathbf{x} + b)\cdot x_i
$$

### Step 3: Finally, differentiate the activation

The last unknown is the derivative of the logistic function itself. This is equal to

$$
\sigma'(r) = e^r\cdot(1 + e^{-r})^{-2} = \sigma(r)(1-\sigma(r)).
$$

And now we can wrap this into the code.

## Updating parameters: Stochastic Gradient Descent (SGD)

Let's finish where we started, why gradients? We have already noted that gradients point to the direction of maximum change, and we can tell whether that change is positive or negative.

There is an established algorithm which tells us that the minimum of a function, like $L$, can be found by making updates

$$
\mathbf{w} \leftarrow \mathbf{w} - \eta\cdot \frac{\partial L}{\partial \mathbf{w}}
$$

where $\eta$ is a parameter called the learning rate. Tempering $\eta$ can put the breaks on and stop the model wildly diverging.

## Challenge for you...

We computed the loss of a single neuron $f(\mathbf{x}) = \sigma(\mathbf{w}\cdot\mathbf{x} + b)$. Try writing down a larger neural network and computing the gradients for all parameters.


Bonus Example on Backpropagation

:::success
**📂 Resources**
Download the resources for this lesson <Link href="/resources/deep-learning-and-neural-networks/introducing-pytorch/Resources.zip" download="Resources.zip">here.</Link>
:::

<Callout type='success' heading='In this video you will...'>

- ✅ Build a neural network with PyTorch.
- ✅ Subclass PyTorch's `nn.Module`.
- ✅ Construct a PyTorch `Dataset` and `DataLoader`.
- ✅ Implement a training loop.

</Callout>

We'll revisit our classification problem from before, but this time we shall construct and train our neural network with **PyTorch**. We'll use the tools which are provided for in PyTorch's code base:

- Activation functions
- Loss functions
- Training algorithm

We'll see that we can move quickly using this package and still maintain a high degree of flexibility in model design.

Let's start at the beginning.

## The Dataset Class

With your data in the `scikit-learn` format, where we have two `np.arrays`, the inputs `X` and outputs `y`, we need to convert this data into a format that `PyTorch` shall understand.

```python
from torch.utils.data import Dataset

# Seed your RNGs
torch.manual_seed(SEED)

class Data(Dataset):
    def __init__(self, X, y):
        """Load data into torch tensors of dtype float32"""
        self.X = torch.from_numpy(X).type(torch.float)
        self.y = torch.from_numpy(y).type(torch.float).reshape(-1, 1)
        self.len = self.X.shape[0]

    def __getitem__(self, idx):
        """Return data sample as (input, target) pair."""
        return self.X[idx], self.y[idx]

    def __len__(self):
        """Return number of samples."""
        return self.len

n_train = int(0.8 * len(X))
n_val = int(0.1 * len(X))
n_test = len(X) - n_train - n_val

data_train, data_val, data_test = torch.utils.data.random_split(
    Data(X, y), [n_train, n_val, n_test],
)
```

We do this by subclassing the `Dataset` class (above). This tells PyTorch how to load and possibly pre-process the data (it could be loading straight from file if too large to fit in your memory).

There are two methods we have to fill in ourselves:

1. The `__getitem__` method: this tells is how to pick a given sample where the index `idx` ranges over the length of the data.
2. The `__len__` method: well this _is_ the length, which parameterises the size of the dataset.

We also make use of PyTorch's own data splitting functionality `random_split`, which, unlike `scikit-learn`, permits us to split into more than two sets. Don't forget to seed your script to maintain reproducibility!

### Data Loaders

The Dataset class describes where to find the data and how it should be retrieved. But a data loader is used to split out set into batches, shuffle the data and allocate the number of process it can be split across. This makes training much more efficient when using GPU acceleration.

We can construct a data loader from the dataset very simply:

```python
from torch.utils.data import DataLoader


BATCH_SIZE = 20

# Instantiate loaders
loader_train = DataLoader(
    data_train, batch_size=BATCH_SIZE, shuffle=True,
)
loader_val = DataLoader(data_val, batch_size=n_val)
loader_test = DataLoader(data_test, batch_size=n_val)
```

## Construct a deep neural network

Now we get to the fun bit. Let's build a model in PyTorch. We shall, as is necessary, subclass the `nn.Module` class. This contains the functionality to compute and store gradients and eventually backpropagate them with jsut a single method call.

We shall build a MLP with two hidden layers, so that we transform the input via

> Input > Hidden Layer 1 > Hidden Layer 2 > Output

We'll activate with the logistic function, as before, and

```python
import torch.nn as nn


DIM_INPUT = data_train[0][0].shape[0]  # The size of the input vector
DIM_OUTPUT = 1
WIDTH = 5  # The number of nodes in each leayer

# The multi-layer perceptron
class MLP(nn.Module):
    """Our PyTorch model for an MLP."""

    def __init__(self):
        super(MLP, self).__init__()

        # Construct linear connections
        self.layer1 = nn.Linear(DIM_INPUT, WIDTH)
        self.layer2 = nn.Linear(WIDTH, WIDTH)
        self.layer3 = nn.Linear(WIDTH, DIM_OUTPUT)

        # Pick activations
        self.act1 = nn.Sigmoid()
        self.act2 = nn.Sigmoid()
        self.act3 = nn.Sigmoid()

        # Batch-normalise layers
        self.bn1 = nn.BatchNorm1d(WIDTH)
        self.bn2 = nn.BatchNorm1d(WIDTH)

    def forward(self, x):
        """The journey forward."""

        x = self.layer1(x)
        x = self.bn1(x)
        x = self.act1(x)

        x = self.layer2(x)
        x = self.bn2(x)
        x = self.act2(x)

        x = self.layer3(x)
        x = self.act3(x)

        return x
```

The `forward` pass is the important method, which needs to be implemented in all `nn.Modules`. It is called when we call the model; that is,

```python
model = MLP()
y1 = model(x)
y2 = model.forward(x)
print(y1 == y2)
```

returns`True`. This method defines the forward pass. This is how the network functions. Whereas our model components, are defined in the `__init__` call.

## Train the network

Let's walk through a classic training routine in PyTorch. First of all, we shall need to instantiate (1) the model itself, (2) a loss function to measure success, and (3) an optimisation routine with which to update the parameters.

```python
model = MLP()  # Instantiate the netork
loss_func = nn.BCELoss()  # Loss function
optimiser = torch.optim.SGD(model.parameters(), lr=0.1)  # Optimisation method
```

These in hand, we can begin training. This shall be executed as a loop, each step in the loop referred to as an `epoch`.

```python
NUM_EPOCHS = 4000

# To store loss statistics
loss_train, loss_val = list(), list()
for ep in range(NUM_EPOCHS):
    running_loss = 0.0
    for ii, batch in enumerate(loader_train):
        inputs, targets = batch
        optimiser.zero_grad()  # Zero gradients in network
        preds = model(inputs)  # Make a forward pass
        loss = loss_func(preds, targets)  # Compute loss
        loss.backward()  # Backpropagate loss (compute gradients)
        optimiser.step()  # Update parameters

        # Record loss at this patch
        running_loss += loss.item()

    # Update epoch-level training loss
    loss_train.append(running_loss / n_train)

    # Test on validation set
    for ii, batch in enumerate(loader_val):
        inputs, targets = batch
        preds = model(inputs)  # Make a forward pass
        loss = loss_func(preds, targets)  # Compute loss
    loss_val.append(loss.item())
```

At each epoch we

1. Get the data `inputs, targets = batch`
2. Clean any past recorded gradients `optimiser.zero_grad()`
3. Make a prediction `preds = model(inputs)`
4. Compute the loss `loss = loss_func(preds, targets)`
5. Back propagate the gradients `loss.backward()`
6. Update the parameters via SGD `optimiser.step()`

Point 5 would be a highly technical feat by hand, but PyTorch automates this quickly. As an output, we can plot the loss of the training session.

<Image
	src={"https://hackmd.io/_uploads/BJa58LfEa.png"}
	alt={""}
	width={547}
	height={413}
	caption={"Fig. 1. Loss plot for our training session."}
/>

A spacing between training and loss curves such as her indicates there is a degree of over fitting. In our example, this is likely to be the presence of noise on over lapping classes.

Now that our network is trained, we can inspect the output. Quite brilliantly, our model has learnt an accurate representation of the data.

<Image
	src={"https://hackmd.io/_uploads/BktBwIM4T.png"}
	alt={""}
	width={551}
	height={455}
	caption={"Fig. 2. Output of neural network solution."}
/>


Introducing PyTorch

:::success
**📂 Resources**
Download the resources for this lesson <Link href="/resources/deep-learning-and-neural-networks/develop-with-lightning/Resources.zip" download="Resources.zip">here.</Link>
:::

<Callout type='success' heading='In this video you will...'>

- ✅ Understand the `lightning` package for PyTorch.
- ✅ Create a `lightning` module.
- ✅ Assess training with TensorBoard.
- ✅ Save and load models.

</Callout>

We pick up our code from the last tutorial and refactor it to use a powerful high-level interface `lightning`, a python package previously known as `pytorch-lightning`. This interface allows us to manipulate `pytorch` objects and train models in a more fluid and simple way, taking care of much of the behind the scenes boiler plate code.

Here's an example..

## Convert to a Lightning module

Our ground zero is to collect our datasets and data loaders built with the PyTorch framework from before. We also build our neural network in PyTorch as before.

Then, step 1 in using lightning--the activation energy--is converting all of this into a `LightningModule`.

```python
import lightning.pytorch as pl
from lightning.pytorch.utilities.types import TRAIN_DATALOADERS


class PLModel(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = MLP()

    def training_step(self, batch):
        """This defines a step in the training loop."""
        x, y = batch  # The batch comes from a torch dataloader
        y_hat = self.model(x)
        loss = nn.functional.binary_cross_entropy(y_hat, y)
        self.log('train_loss', loss)
        return loss

    def train_dataloader(self):
        return DataLoader(data_train, batch_size=BATCH_SIZE, shuffle=True)

    def validation_step(self, batch):
        """This defines a validation step."""
        x, y = batch
        y_hat = self.model(x)
        loss = nn.functional.binary_cross_entropy(y_hat, y)
        self.log('val_loss', loss)
        return loss

    def val_dataloader(self):
        return DataLoader(data_val, batch_size=n_val)

    def configure_optimizers(self):
        optimiser = torch.optim.SGD(self.parameters(), lr=0.1)
        return optimiser


model = PLModel()
```

This class give a recipe on how the `lightning` should act:

1. The `__init__` gives us room to define the PyTorch neural network.
2. The `training_step` and `validation_step` tell us what to do in each step of training/validation; i.e. in each batch of each epoch. In this method, we use `self.log` to report on the values we want to report on.
3. The `train_dataloader` and `val_dataloader` construct the respective data loaders.
4. The `configure_optimizers` method instructs us which optimiser to use and how.

With this class constructed, we have made all our choices about training and validation and need not specify anything further to plot or analyse the model.

## Training with Lightning

The hard work out the way, training becomes very simple. We include the option to checkpoint our model, which is very straight forward using the `ModelCheckpoint` API. Training is then called with `trainer.fit()`.

```python
from lightning.pytorch.callbacks import ModelCheckpoint

ckpt = ModelCheckpoint(save_top_k=2, monitor='val_loss')

trainer = pl.Trainer(
    check_val_every_n_epoch=100,
    max_epochs=4000,
    callbacks=[ckpt],
)

trainer.fit(model=model)
```

Here are some further tips.

<Callout type='info'>

### Save your best model path

Don't forget to identify the be

```python
pth = trainer.checkpoint_callback.best_model_path
model = PLModel.load_from_checkpoint(pth)
```

</Callout>

<Callout type='info'>

### Inspect training logs with TensorBoard

We never have to plot the loss or accuracy again, with TensorBoard. After logging all the quantities of interest, simply call tensorboard from your terminal.

```
%%bash

tensorboard --logdir .
```

</Callout>

<Callout type='info'>

### GPU acceleration

Here are some options to control the integrated GPU acceleration:

```python
# CPU accelerator
trainer = Trainer(accelerator="cpu")

# Training with GPU Accelerator using 2 GPUs
trainer = Trainer(devices=2, accelerator="gpu")

# Training with TPU Accelerator using 8 tpu cores
trainer = Trainer(devices=8, accelerator="tpu")

# Training with GPU Accelerator using the DistributedDataParallel strategy
trainer = Trainer(devices=4, accelerator="gpu", strategy="ddp")
```

</Callout>

<Callout type='info'>

### The API docs

Here is a resource for the trainer class: https://lightning.ai/docs/pytorch/stable/common/trainer.html#

</Callout>

## Validate your model

Once again, we can simply call the trainer to validate the model in a single call.

```python
trainer.validate(ckpt_path='best')
```

## Loading from a checkpoint

Importantly, we want to recover our trained models. This is slightly more technical as the state dictionary holds parameters for the PyTorch model rather than the lightning Module, but here is some code that can fix the renaming.

```python
# Loading from a checkpoint
trained_model = PLModel.load_from_checkpoint(pth)

# Load torhc model from path
checkpoint = torch.load(pth)

# Load torch model
torch_model = MLP()
model_weights = checkpoint["state_dict"]

# update keys by dropping `auto_encoder.`
for key in list(model_weights):
    model_weights[key.replace('model.', '')] = model_weights.pop(key)

torch_model.load_state_dict(checkpoint['state_dict'])
# Loading from a checkpoint
trained_model = PLModel.load_from_checkpoint(pth)

# Load torhc model from path
checkpoint = torch.load(pth)

# Load torch model
torch_model = MLP()
model_weights = checkpoint["state_dict"]

# update keys by dropping `auto_encoder.`
for key in list(model_weights):
    model_weights[key.replace('model.', '')] = model_weights.pop(key)

torch_model.load_state_dict(checkpoint['state_dict'])
```

Following these tips and the video, you can apply the lightning acceleration to your own projects.


Develop with Lightning

<Callout type='success' heading='In this video you will...'>

- ✅ See a variety of different _activation functions_ and learn how and when to use them.
- ✅ Observe different loss functions to measure the success of model performance.
- ✅ Discern between different metrics for different types of task.

</Callout>

## Design choices: what to choose and why

As we saw when designing neural networks, there were many 'hyperparameter' choices to make. In this video, we shall take a short tour such choices for **Loss functions** and **activation functions**.

## Picking your loss functions

Loss functions assess how close our predictions are to the ground truth values. They need to be picked bespoke to specific problems for an effective comparison. For instance, if we have a binary problem, looking at the data on a continuous scale shan't be much use. So we being by splitting the problem into two classes.

### Regression of classification

<Image
	src={"https://hackmd.io/_uploads/rydoV-BNT.png"}
	alt={""}
	width={490}
	height={600}
	caption={"Fig. 1. Do we have a classification or a regression problem?"}
/>

In this video we give a visual description of the different loss functions that can be applied to these problems.

<Callout type='info'>
**Regression Problems**
These are problems whose target is a continuous variable. We there are two popular choices of loss function to choose between for different goals:

- Mean-Squared Error:$$L_{\text{MSE}} = \frac{1}{N}\sum_{i=1}^{N}(y_{i}^{\text{true}} - y_{i}^{\text{pred}})^{2}$$ where $N$ is the number of samples. We use this function to which deviate far from the ground truth.
- Mean Absolute Error:$$L_{\text{MAE}} = \frac{1}{N}\sum_{i=1}^{N}(y_{i}^{\text{true}} - y_{i}^{\text{pred}})$$This function is chosen to penalise noisy outputs which are oscillating around the true values.

</Callout>

<Callout type='warning'>

**Classification Problems**
These are problems where we have a number of distinct classes in which the target can live. We distinguish betwen two and many classes.

- Binary Cross-Entropy: $$L_{BCE} = \frac{-1}{N} \sum_{i = 1}^{N}[y_{i}^{\text{true}}\cdot\log(y_{i}^{\text{pred}}) + (1- y_{i}^{\text{true}})\cdot\log(1- y_{i}^{\text{pred}})]$$ where $y_{i}^{\text{pred}}$ is interpreted as the probability that the model is in the positive class; i.e. the activated output. The leading terms in this loss function depending on $y_{i}^{\text{true}}$ determine the class and act as an indicator to which term contributes. Then second terms are then the negative log-likely hoods predicted by the the model outputs--in face we shall see that the negative log-likelihood loss and BCE loss can be used interchangeably.
- Categorical cross entropy loss is an extension of the BCE loss for multi-class problems. It is similarly based on the negative log-likelihood loss.

</Callout>

## Choosing between activations

In the video we shall give a whilst-stop tour of the following activations, each serving different purposes in neural network architectures.

1. _Sigmoid logistic_.$$\sigma(x) = \frac{1}{1 + e^{-x}}$$This function is commonly used on output layers when predicting probabilities (values within $[0,1]$).
2. _Hyperbolic tangent_.$$\tanh(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}}$$This function has similar properties to $\sigma$ but sits in the range $[-1,1]$, symmetric about zero. For this reason it is sometimes used in intermediate layers where the nodes need to be regularised but remain unbiased, with mean zero. It can also be use on output layers to indicate where an item is for (+1), agains (-1) or neutral (0) to the outcome.
3. _Rectified Linear Unit, ReLU_.$$\text{ReLU}(x) = \max{x, 0}$$This is the most common activation to use on hidden nodes as it is cheap to evaluate and to differentiate. However regions of parameter space can be hard to explore.
4. _Leaky ReLU_.$$\text{LeakyReLU}(x) = \max{x, ax}$$This remedies the exploration bottleneck of the ReLU by allowing a slope into negative space.
5. _Softplus_.$$\text{Softplus}(x)=\frac{1}{\beta}\log(1 + e^{\beta x})$$Keep in your pocket if you want smooth derivatives: Softplus approximates ReLU as $\beta\rightarrow\infty$.
6. _Max_ (for multi-objective outputs). returns the max of the ourputs.
7. _Softmax_ (for multi-objective outputs) is a smooth way to return the max whilst normalising against the other components.


Activations and Loss Functions

<Callout type='success' heading='In this video you will...'>

Learn the key hyperparameters needed for training a neural net. These include:

- ✅ Walk through the training cycle.
- ✅ Use batches and dataloading techniques.
- ✅ See how set parameter updates.
- ✅ Choose regularisation methods in training.

</Callout>

Training a neural network can seem like an art form. Or rather a dark art. But with any luck, some of the mysteries should become a little more familiar.

The goal of this video is to explain the following boiler plate code. This is the template for a nerual network training routine. Once commited to muscle memory, it'll feel like riding a bike... looping over and over again! :bike:

```python
model = MLP()  # Instantiate the netork
loss_func = nn.BCELoss()  # Loss function
optimiser = torch.optim.SGD(model.parameters(), lr=0.1)  # Optimisation method

NUM_EPOCHS = 4000

# To store loss statistics
loss_train, loss_val = list(), list()
for ep in range(NUM_EPOCHS):
    running_loss = 0.0
    for ii, batch in enumerate(loader_train):
        inputs, targets = batch
        optimiser.zero_grad()  # Zero gradients in network
        preds = model(inputs)  # Make a forward pass
        loss = loss_func(preds, targets)  # Compute loss
        loss.backward()  # Backpropagate loss (compute gradients)
        optimiser.step()  # Update parameters

        # Record loss at this patch
        running_loss += loss.item()

    # Update epoch-level training loss
    loss_train.append(running_loss / n_train)

    # Test on validation set
    for ii, batch in enumerate(loader_val):
        inputs, targets = batch
        preds = model(inputs)  # Make a forward pass
        loss = loss_func(preds, targets)  # Compute loss
    loss_val.append(loss.item())
```

## The Training Loop

First up is the cycle of epochs which we call training. This is by design a for loop. In a single epoch we pass through all the training data once.

But then within the epoch, we pass through all the batches. So it is a nested for loop. Lots of scope for parallelisation. But importantly, **this is the recipe for what you must do in each iteration** (i.e. upon each batch).

1. _Initialise/reset you variables_. This includes clearing the accumulated gradients that are stored on each round as well as getting the next batch and moving onto your hardware device.
2. _Forward pass_. This moves all the data through the network to obtain predictions
3. _Compute the loss_. This applies the loss function to the prediction vs. the ground truth--remember this is a supervised training routine.
4. _Backpropagate_. This means computing the gradients across the whole network.
5. _Update parameters_. Now we apply our optimisation algorithm to update the network parameters.

## Batches and data loading

In the training loop we call on batches of data. This allows us to manipulate the data on the fly during training.

- We can use pre-designed data loaders (such as PyTorch's dataloaders) for computational efficiency.
- We can shuffle the dataset each epoch as a form of regularisation (seeding the process to maintain reproducibility).
- We can apply on-the-fly augmentation to our data with random probability.
- Batch loading adds an element of stochasticity. By grouping the data in different ways each epoch, we are preventing deterministic trends from falling into local minima during optimisation.

## Updating parameters

We mentioned that that batch loading adds an element of stochasticity. Indeed, this is what seperates gradient descent from _stochastic_ gradient descent (SGD), a popular optimisation algorithm.

We can make the choice of optimiser early on in the code, chosing from SGD, Adam, or various other. And for these we are interested in at least two hyperparameters:

- The learning rate: this controls the size of the update we make each step. Setting this lower prevents the model from diverging due to random noise early on. The trade off is that you must train for longer.
- Set the regularisation. This can be of various forms. In the optimise we set the $L^2$ regularisation via a `weight_decay` parameter.

## Regularisation during trianing

- Training with $L^2$ regularisation tempers the parameters by preferring to not grow so large. This is measured in the loss function and hence effected by the parameter updates.
- Batch normalisation: this is a layer you can add to your neural network what normalises the activations themselves during training. Again, this helps prevent unforeseen divergent behaviour.
- Dropout: This is a technique that zeros node in the network at random given a specified probability. If the network is concentrating on a given pathway, this technique can help to smooth out that behaviour.

> Note: both batch normalisation and dropout are ineffective during evaluation, only being applied in training. PyTroch knows when to activate/deactivate these layers by calling `model.train()` and `model.eval()`, respectively. Don't for get to include them in your code! (Although this is automated with Lightning.)

<Callout type="info">
	**Further reading**. Applying dropout to your network can be thought of as
	swiching between an ensemble of architectures. Fire samples through this
	architecture in evaluation mode can build up a Monte Carlo-based picture of the
	distributin of model outputs, given this probabilistic view of the ensemble
	model architecture. The technique is known as 'MC Dropout' and the paper
	introducing this notion can be found
	[here](https://arxiv.org/pdf/1506.02142.pdf).
</Callout>


The Training Loop

:::success
**📂 Resources**
Download the resources for this lesson <Link href="/resources/deep-learning-and-neural-networks/dnns-in-the-wild-for-drug-discovery/Resources.zip" download="Resources.zip">here.</Link>
:::

<Callout type='success' heading='In this video you will...'>

- ✅ Train and deploy a PyTorch model to solve a real world problem.
- ✅ Analyse data which may occur in experimental drug prediction.
- ✅ Minimise a PyTorch model to predict optimal solutions outside the experimental range.

</Callout>

Time to get our hands dirty. In this video we'll be setting up a drug trial experiment and we want to use machine learning to model our data and inform our actions. Perfect time to pull out our newly devised neural network.

## Experimental data

First port of call is to load in our experimental readings and see what we're dealing with. We'll do this with `pandas`:

```python
import pandas as pd

df = pd.read_csv('discovery.csv')
df.info()
```

```
RangeIndex: 5000 entries, 0 to 4999
Data columns (total 5 columns):
 #   Column    Non-Null Count  Dtype
---  ------    --------------  -----
 0   soluble1  5000 non-null   float64
 1   soluble2  5000 non-null   float64
 2   soluble3  5000 non-null   float64
 3   soluble4  5000 non-null   float64
 4   output    5000 non-null   float64
```

Out data table contains 5 columns, each with 5000 samples. There are four solubles which constitute the input parameters to the experiment and our measurement of success is stored in the output: 'how much bacteria remains after apply our new concoction?'

> Our goal, as scientists, is to **minimise the output by choosing the correct solubles**.

How can we solve this with neural networks? We'll do two things:

1. We shall emulate the experimental findings of this table.
2. We shall then use a package `autograd-minimise` to find the minimum of our emulator.

### Inspecting the data

Before diving straight in, let's determine what we are working with. First of all, plotting histograms of the data variable will indicate if there is any skew or other anomalous behaviour.

<Image
	src={"https://hackmd.io/_uploads/B1O1RhEV6.png"}
	alt={""}
	width={569}
	height={413}
	caption={
		"Fig. 1. A histogram of the _output_ variable showing well distributed results."
	}
/>

Let's also assess if there are any linear correlations between our variables.

<Image
	src={"https://hackmd.io/_uploads/ryWU0n4Ep.png"}
	alt={""}
	width={508}
	height={428}
	caption={"Fig. 2. Correlation matrix between the variables."}
/>

To things pop out to us from this table. Firstly, soluble 4 seems to be somewhat correlated to soluble 1. This tells us that we have not conducted our experiments so far in a completely uniform way. Hey-ho, we'll have to live with that. But more interesting, there seems to be high negative correlation between soluble 4 (and hence 1) and the output.

Let's investigate this further by finding the line of least-squares best fit between them.

<Image
	src={"https://hackmd.io/_uploads/BJExH6VN6.png"}
	alt={""}
	width={571}
	height={455}
	caption={"Fig. 3. The linear relationship between the output and solution 4."}
/>

By eye, it seems that we shall get a long way with our model with this dependency. So we expect it to be picked up by the network. One option we have is to de-trend the data--subtracting this red line from the output column--and learning the residuals. However, we expect that our neural network can handle this feature.

## Building our model

Step one is to rescale our data. The purpose of this is to put each feature on a level pegging with the randomly initialised weights. The scaling shall become the first step in our model pipeline.

```python
from sklearn.preprocessing import StandardScaler


scale = StandardScaler()
X = df.drop(columns='output').to_numpy().copy()
X = scale.fit_transform(X)
y = df['output'].to_numpy().copy()

print('Means: ', scale.mean_)
print('Vars: ', scale.var_)
```

```
Means:  [ 55.21510051 242.114565    23.30306534  41.5352574 ]
Vars:  [1.65841110e+02 2.31802953e+03 1.84469320e-02 2.44658012e+02]
```

### Build `Dataset` and `DataLoader` classes

Just as before, we construct python data objects. We shuffle the data and take a train, test and validation set in the ratio 80:10:10.

```python
from torch.utils.data import DataLoader, Dataset

class Data(Dataset):
    def __init__(self, X, y):
        """Load data into torch tensors of dtype float32"""
        self.X = torch.from_numpy(X).type(torch.float)
        self.y = torch.from_numpy(y).type(torch.float).reshape(-1, 1)
        self.len = self.X.shape[0]

    def __getitem__(self, idx):
        """Return data sample as (input, target) pair."""
        return self.X[idx], self.y[idx]

    def __len__(self):
        """Return number of samples."""
        return self.len

n_train = int(0.8 * len(X))
n_val = int(0.1 * len(X))
n_test = len(X) - n_train - n_val

data_train, data_val, data_test = torch.utils.data.random_split(
    Data(X, y), [n_train, n_val, n_test],
)

BATCH_SIZE = 10

# Instantiate loaders
loader_train = DataLoader(data_train, batch_size=BATCH_SIZE, shuffle=True)
loader_val = DataLoader(data_val, batch_size=n_val)
loader_test = DataLoader(data_test, batch_size=n_val)
```

### Neural network architecture

Using the MLP architecture, we choose a very simple architecture for this problem: a single hidden layer of width 4.

> **Experiment at home**: With the provided notebook, tinker with the architecture. Add more layers, increase the width and see what effect this has on model performance.

```python
import torch.nn as nn


DIM_INPUT = data_train[0][0].shape[0]  # The size of the input vector
DIM_OUTPUT = 1
WIDTH = 4  # The number of nodes in each leayer

# The multi-layer perceptron
class MLP(nn.Module):
    """Our PyTorch model for an MLP."""

    def __init__(self):
        super(MLP, self).__init__()

        # Construct linear connections
        self.layer_hid = nn.Linear(DIM_INPUT, WIDTH)
        self.layer_out = nn.Linear(WIDTH, DIM_OUTPUT)

        # Pick activations
        self.act = nn.Sigmoid()

        # Batch-normalise layers
        self.bn = nn.BatchNorm1d(WIDTH)

    def forward(self, x):
        """The journey forward."""
        x = self.layer_hid(x)
        x = self.bn(x)
        x = self.act(x)
        x = self.layer_out(x)
        return x
```

### Train the model

We use our training code from last time. However, we include an important new feature: $L^2$ regularisation. This is inserted in the `weight_decay` parameter. $L^2$ regularisation helps to temper the growth of parameters to avoid divergence.

```python
model = MLP()  # Instantiate the netork
loss_func = nn.MSELoss()  # Loss function
optimiser = torch.optim.SGD(model.parameters(), lr=0.001, weight_decay=1e-3)

NUM_EPOCHS = 50

# To store loss statistics
loss_train, loss_val = list(), list()

model.train()

for ep in trange(NUM_EPOCHS):
    running_loss = 0.0
    for ii, batch in enumerate(loader_train):
        inputs, targets = batch
        optimiser.zero_grad()  # Zero gradients in network
        preds = model(inputs)  # Make a forward pass
        loss = loss_func(preds, targets)  # Compute loss
        loss.backward()  # Backpropagate loss (compute gradients)
        optimiser.step()  # Update parameters

        # Record loss at this patch
        running_loss += loss.item()

    # Update epoch-level training loss
    loss_train.append(running_loss / n_train)

    # Test on validation set
    for ii, batch in enumerate(loader_val):
        inputs, targets = batch
        preds = model(inputs)  # Make a forward pass
        loss = loss_func(preds, targets)  # Compute loss
    loss_val.append(loss.item())
```

Training converges quickly with little overfitting.

<Image
	src={"https://hackmd.io/_uploads/ryehPTEN6.png"}
	alt={""}
	width={560}
	height={413}
	caption={"Fig. 4. The loss plot for our training session."}
/>

### Visualise the results

With high dimensional (higher than one dimension!) we are constrained in our visualisation tools. But a one-size-fits-all plot is the predicted-vs.-truth plot which compares the models prediction with the corresponding ground-truth value. This is easy to read as the perfect model sits on the diagonal, and we are able to measure any deviance with the $R^2$ score.

<Image
	src={"https://hackmd.io/_uploads/By2qsa4Ea.png"}
	alt={""}
	width={442}
	height={453}
	caption={"Fig. 5. Predicted-vs.-truth plot"}
/>

> Task 1: Improve this outcome! With a better regularised; wider, deeper, longer trained and using; apply your new techniques to improve the $R^2$ score.

> Task 2: Can you extract the parameters (using `model.parameters()`) associated to solution 2 on the input layer to assess the weighting given to the linear relationship we found in fig. 3?

## Application: Optimise the emulator

Our goal was to build a function--we called this `model`--which can emulate the results of our experiment. Now we want to ask: which set of parameters best minimises the output variable.

There is a great package called `autograd-minimize` which wraps SciPy's minimise routine but includes the stored gradient information given by torch tensors.

Lets perform this optimisation now.

```python
from autograd_minimize import minimize

X_init = list(loader_val)[0][0][0]
res = minimize(
    model,
    X_init.numpy().reshape(1, -1),
    backend='torch',
    bounds=(0, None),
    tol=1e-8,
)
```

The output from this routine gives the results of the minimisation:

```
  message: CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH
  success: True
   status: 0
      fun: 82.90983581542969
        x: [[ 7.159e-08  1.261e-08  1.923e+01  3.175e+01]]
      nit: 4
      jac: [-6.233e-06 -3.515e-06  1.863e-06 -2.502e-05]
     nfev: 19
     njev: 19
 hess_inv: <4x4 LbfgsInvHessProduct with dtype=float64>
```

And we can pick out the optimal parameters as follows (taking care to back-project through our scale transformation):

```python
scale.inverse_transform(res.x)
```

```
array([[ 55.21510143, 242.11456561,  25.9146459 , 538.17894934]])
```

Low and behold, the `soluble4` parameter has indeed been picked at the base of the linear slope given in fig. 3. But we also have values for the other parameters which were not apparent from this one-dimensional relationship.


DNNs in the Wild for Drug Discovery

:::success
**📂 Resources**
Download the resources for this lesson <Link href="/resources/deep-learning-and-neural-networks/dnns-in-the-wild-for-classification/Resources.zip" download="Resources.zip">here.</Link>
:::

<Callout type='success' heading='In this video you will...'>

- ✅ Import and process image data.
- ✅ Build a simple neural network to classify the images.
- ✅ Inspect the weights to visually observe the feature-patterns learnt.

</Callout>

Image classification is big business, and we are now toe-stepping on one of the big applications of deep neural networks. But we are just dipping our toes in lightly here. our goals are twofold:

1. To use PyTorch lightning on an image-data application.
2. To visually unpack the network weights themselves.

## Aquiring the data

Learning how to use PyTorch's in-built data sources can be very useful for quick development and testing of new models. One such benchmark task is known as the MNIST dataset, which contains a series of handwritten digits from 0, ... , 9 and the task is to identify the correct class.

<Image
	src={"https://hackmd.io/_uploads/ryb0fR4V6.png"}
	alt={""}
	width={416}
	height={435}
	caption={"Fig. 1. An example of the handwritten digit data provided."}
/>

It is a simple problem, hence we can get away with using a simple model, without throwing the full weight of computer vision at the task.

We can acquire the data through PyTorch directly:

```python
from torchvision import datasets, transforms


transform=transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))  # Mean/Std of MNIST data.
])

train_kwargs = {'batch_size': 64}
test_kwargs = {'batch_size': 32}

# Torch datasets
train = datasets.MNIST('data/', train=True, transform=transform, download=True)
test = datasets.MNIST('data/', train=False, transform=transform)

# Torch dataloaders
loader_train = torch.utils.data.DataLoader(train,**train_kwargs)
loader_test = torch.utils.data.DataLoader(test, **test_kwargs)

data_dict = {
    'training_loader': loader_train,
    'validation_loader': loader_test,
}
```

## Constructing a Deep Neural Network

As mentioned, we wish to use Lightning to train the model, but this does not preclude us from first writing our `nn.Module`.

```python
import torch.nn as nn


DIM_INPUT = np.prod(train[0][0].shape)  # The size of the input vector
DIM_OUTPUT = 10
WIDTH = 20  # The number of nodes in each leayer

# The multi-layer perceptron
class MLP(nn.Module):
    """Our PyTorch model for an MLP."""

    def __init__(self):
        super(MLP, self).__init__()

        # Construct linear connections
        self.layer1 = nn.Linear(DIM_INPUT, WIDTH, bias=False)
        self.layer2 = nn.Linear(WIDTH, DIM_OUTPUT, bias=False)

        # Pick activations
        self.act = nn.Sigmoid()
        # self.dropout = nn.Dropout(0.2)

        # Batch-normalise layers
        self.bn1 = nn.BatchNorm1d(WIDTH)

    def forward(self, x):
        """The journey forward."""

        x = self.layer1(x.flatten(start_dim=1))
        x = self.bn1(x)
        x = self.act(x)
        # x = self.dropout(x)

        x = self.layer2(x)

        return nn.functional.softmax(x, dim=1)
```

This is a relatively small model. Just downscaling to a single hidden layer of dimension `20` from the input images. With this assumption, we are postulating that there are 20 _feature_ in the images which we want to extract.

> Task: Augment this architecture. Try increase/decreasing these nodes and see how performance varies. Also try adding more layers whilst keeping the number of hidden neurons (20) constant.

## Train with Lightning

We specify our training routine as before with the `pl.Module` subclass.

```python
import lightning.pytorch as pl


class PLModel(pl.LightningModule):
    def __init__(self):
        super().__init__()
        self.model = MLP()

    def training_step(self, batch):
        """This defines a step in the training loop."""
        x, y = batch  # The batch comes from a torch dataloader
        y_hat = self.model(x)
        loss = nn.functional.nll_loss(y_hat, y, reduction='sum')
        y_pred = y_hat.argmax(dim=1, keepdim=True)  # get number index
        correct = y_pred.eq(y.view_as(y_pred)).sum().item()
        self.log('train_loss', loss, prog_bar=True)
        self.log('train_acc', correct / 64)
        return loss

    def train_dataloader(self):
        return loader_train

    def validation_step(self, batch):
        """This defines a validation step."""
        x, y = batch
        y_hat = self.model(x)
        loss = nn.functional.nll_loss(y_hat, y, reduction='sum')
        y_pred = y_hat.argmax(dim=1, keepdim=True)  # get number index
        correct = y_pred.eq(y.view_as(y_pred)).sum().item()
        self.log('val_loss', loss)
        self.log('val_acc', correct / 32)
        return loss

    def val_dataloader(self):
        return loader_test

    def configure_optimizers(self):
        optimiser = torch.optim.SGD(self.parameters(), lr=0.01)
        return optimiser

# init the autoencoder
model = PLModel()
```

Training can then be executed in a few lines of code.

<Callout type="warning">
	Go and make yourself a hot drink. This may take some time! :timer_clock:
</Callout>

```python
from lightning.pytorch.callbacks import ModelCheckpoint

ckpt = ModelCheckpoint(save_top_k=2, monitor='val_loss')

trainer = pl.Trainer(
    check_val_every_n_epoch=5,
    max_epochs=200,
    callbacks=[ckpt],
)

trainer.fit(model=model)
pth = trainer.checkpoint_callback.best_model_path
print('Best model path:\n', pth)
```

## Inspection of the model weights

To glean intuition of this simple model, we shall print out the matrix of weights used on the incoming data for _each_ hidden node. This means we shall have 20 matrices, each of whose parameters correspond to a single pixel in the image data.

First of all, the parameters of our neural network are stored here:

```python
params = list(model.model.parameters())
sizes = [p.shape for p in params]
```

We can illustrate the plots with the following code:

```python
fig, axes = plt.subplots(10, 2, figsize=(5, 10))

scale_max, scale_min = params[0].max(), params[0].min()  # Scale factors

for param, ax in zip(params[0], axes.ravel()):
    ax.matshow(param.reshape(28, 28).detach().cpu().numpy(),
               cmap=plt.cm.magma,
               vmin=scale_min,
               vmax=scale_max)
    ax.set_xticks(())
    ax.set_yticks(())

plt.show()
```

<Image
	src={"https://hackmd.io/_uploads/rJrlGkHV6.png"}
	alt={""}
	width={290}
	height={990}
	caption={
		"Fig. 2. A depiction of the model weights transforming the inputs into the 20 hidden feature nodes."
	}
/>

<Callout type="info">
	From the images, it is apparent that much of the network is formed by noise.
	Outside the spherical region in the centre, there is no differentiation of the
	weights. However, there are some notable bright and dark areas in the image
	which indicate features being extracted from the input. **Remarkable!**
</Callout>


Deep Learning 101: Getting Started with Neural Networks

Course Description

But to warm us up...

Lessons

1. An Introduction to Deep Learning

2. What are Deep Neural Networks?

3. Coding Neurons from Scratch

4. Bonus Example on Backpropagation

5. Introducing PyTorch

6. Develop with Lightning

7. Activations and Loss Functions

8. The Training Loop

9. DNNs in the Wild for Drug Discovery

10. DNNs in the Wild for Classification