Lab 08: Dense Neural Networks¶

In this lab, you will learn how to build and train dense (fully connected) neural networks using PyTorch. Dense neural networks, also known as multilayer perceptrons (MLPs), are fundamental architectures in deep learning and are widely used for a variety of tasks such as classification and regression.

We will cover the following topics:

• Understanding the structure of dense neural networks
• Implementing a dense neural network from scratch with PyTorch
• Training and evaluating the network on a dataset
• Experimenting with different architectures and hyperparameters

1. What is a Dense Neural Network?¶

A dense neural network, also known as a fully connected network or multilayer perceptron (MLP), is a type of artificial neural network where each neuron in one layer is connected to every neuron in the next layer. These networks are composed of an input layer, one or more hidden layers, and an output layer.

Key characteristics:

• Each neuron receives input from all neurons in the previous layer.
• Each connection has an associated weight and bias.
• Non-linear activation functions are used to introduce complexity and allow the network to learn non-linear relationships.

Dense neural networks are widely used for tasks such as image classification, regression, and more.

A typical dense (fully connected) neural network architecture. Each neuron in one layer is connected to every neuron in the next layer. Diagram redrawn locally; the original was hot-linked and is no longer served.

2. Architecture of a Dense Neural Network¶

A typical dense neural network consists of:

• Input Layer: Receives the input features (e.g., pixel values, tabular data, etc.).
• Hidden Layers: One or more layers where each neuron is connected to all neurons in the previous and next layers. These layers learn complex representations of the data.
• Output Layer: Produces the final prediction (e.g., class probabilities, regression value).

Each neuron computes a weighted sum of its inputs, adds a bias, and applies an activation function such as ReLU, sigmoid, or softmax.

3. Introduction to PyTorch for Neural Networks¶

PyTorch is a popular open-source deep learning library that provides flexible tools for building and training neural networks. It is widely used in both research and industry due to its dynamic computation graph and ease of use.

In this lab, we will use PyTorch to:

• Define a dense neural network model
• Train the model on a dataset
• Evaluate its performance

In [1]:

# Import PyTorch and check version
import torch
import torch.nn as nn
import torch.optim as optim
print('PyTorch version:', torch.__version__)

# Import PyTorch and check version import torch import torch.nn as nn import torch.optim as optim print('PyTorch version:', torch.__version__)

PyTorch version: 2.13.0+cpu

4. Dataset Preparation¶

For this lab, we will use a simple synthetic dataset for binary classification. PyTorch and scikit-learn provide utilities to generate such datasets easily.

We will use the make_moons dataset from scikit-learn, which is a popular toy dataset for testing classification algorithms.

In [2]:

# Generate and visualize the dataset
from sklearn.datasets import make_moons
import matplotlib.pyplot as plt

# Generate synthetic data
X, y = make_moons(n_samples=500, noise=0.2, random_state=42)

# Visualize the data
plt.figure(figsize=(6,4))
plt.scatter(X[:,0], X[:,1], c=y, cmap=plt.cm.Set1, edgecolor='k')
plt.title('make_moons dataset')
plt.xlabel('Feature 1')
plt.ylabel('Feature 2')
plt.show()

# Generate and visualize the dataset from sklearn.datasets import make_moons import matplotlib.pyplot as plt # Generate synthetic data X, y = make_moons(n_samples=500, noise=0.2, random_state=42) # Visualize the data plt.figure(figsize=(6,4)) plt.scatter(X[:,0], X[:,1], c=y, cmap=plt.cm.Set1, edgecolor='k') plt.title('make_moons dataset') plt.xlabel('Feature 1') plt.ylabel('Feature 2') plt.show()

In [3]:

# Split the dataset and convert to PyTorch tensors
from sklearn.model_selection import train_test_split
import torch

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Convert to PyTorch tensors
X_train = torch.tensor(X_train, dtype=torch.float32)
X_test = torch.tensor(X_test, dtype=torch.float32)
y_train = torch.tensor(y_train, dtype=torch.float32).unsqueeze(1)
y_test = torch.tensor(y_test, dtype=torch.float32).unsqueeze(1)

# Split the dataset and convert to PyTorch tensors from sklearn.model_selection import train_test_split import torch # Split into train and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Convert to PyTorch tensors X_train = torch.tensor(X_train, dtype=torch.float32) X_test = torch.tensor(X_test, dtype=torch.float32) y_train = torch.tensor(y_train, dtype=torch.float32).unsqueeze(1) y_test = torch.tensor(y_test, dtype=torch.float32).unsqueeze(1)

5. Defining a Dense Neural Network in PyTorch¶

Let's define a simple dense neural network with one hidden layer using PyTorch's nn.Module.

In [4]:

# Define a simple dense neural network
class DenseNet(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(DenseNet, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_dim, output_dim)
        self.sigmoid = nn.Sigmoid()
    
    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        out = self.sigmoid(out)
        return out

# Instantiate the model
model = DenseNet(input_dim=2, hidden_dim=16, output_dim=1)
print(model)

# Define a simple dense neural network class DenseNet(nn.Module): def __init__(self, input_dim, hidden_dim, output_dim): super(DenseNet, self).__init__() self.fc1 = nn.Linear(input_dim, hidden_dim) self.relu = nn.ReLU() self.fc2 = nn.Linear(hidden_dim, output_dim) self.sigmoid = nn.Sigmoid() def forward(self, x): out = self.fc1(x) out = self.relu(out) out = self.fc2(out) out = self.sigmoid(out) return out # Instantiate the model model = DenseNet(input_dim=2, hidden_dim=16, output_dim=1) print(model)

DenseNet(
  (fc1): Linear(in_features=2, out_features=16, bias=True)
  (relu): ReLU()
  (fc2): Linear(in_features=16, out_features=1, bias=True)
  (sigmoid): Sigmoid()
)

6. Training the Dense Neural Network¶

Now, let's train our model using binary cross-entropy loss and the Adam optimizer.

In [5]:

# Training loop
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

num_epochs = 1000
losses = []

for epoch in range(num_epochs):
    model.train()
    optimizer.zero_grad()
    outputs = model(X_train)
    loss = criterion(outputs, y_train)
    loss.backward()
    optimizer.step()
    losses.append(loss.item())
    if (epoch+1) % 100 == 0:
        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')

# Plot the training loss
plt.plot(losses)
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Training Loss Curve')
plt.show()

# Training loop criterion = nn.BCELoss() optimizer = optim.Adam(model.parameters(), lr=0.01) num_epochs = 1000 losses = [] for epoch in range(num_epochs): model.train() optimizer.zero_grad() outputs = model(X_train) loss = criterion(outputs, y_train) loss.backward() optimizer.step() losses.append(loss.item()) if (epoch+1) % 100 == 0: print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}') # Plot the training loss plt.plot(losses) plt.xlabel('Epoch') plt.ylabel('Loss') plt.title('Training Loss Curve') plt.show()

Epoch [100/1000], Loss: 0.2513

Epoch [200/1000], Loss: 0.1866

Epoch [300/1000], Loss: 0.1173

Epoch [400/1000], Loss: 0.0766

Epoch [500/1000], Loss: 0.0614

Epoch [600/1000], Loss: 0.0536

Epoch [700/1000], Loss: 0.0486

Epoch [800/1000], Loss: 0.0450

Epoch [900/1000], Loss: 0.0421

Epoch [1000/1000], Loss: 0.0397

7. Evaluating the Model¶

Let's evaluate the trained model on the test set and compute its accuracy.

In [6]:

# Evaluate the model on the test set
model.eval()
with torch.no_grad():
    y_pred = model(X_test)
    y_pred_label = (y_pred > 0.5).float()
    accuracy = (y_pred_label == y_test).float().mean()
    print(f'Test Accuracy: {accuracy.item()*100:.2f}%')

# Evaluate the model on the test set model.eval() with torch.no_grad(): y_pred = model(X_test) y_pred_label = (y_pred > 0.5).float() accuracy = (y_pred_label == y_test).float().mean() print(f'Test Accuracy: {accuracy.item()*100:.2f}%')

Test Accuracy: 98.00%

8. Experiment: Try Different Architectures and Hyperparameters¶

You can experiment with:

• Changing the number of hidden units or layers
• Using different activation functions (e.g., Tanh, LeakyReLU)
• Adjusting the learning rate or optimizer
• Training for more or fewer epochs

Try modifying the model definition or training loop above and observe how the performance changes.

9. Conclusion¶

In this lab, you learned how to build, train, and evaluate a dense neural network using PyTorch. You also explored how changing the architecture and hyperparameters can affect model performance.

Dense neural networks are foundational in deep learning and serve as a basis for more advanced architectures. Continue experimenting and exploring to deepen your understanding!