Implementation of neural network from scratch using NumPy

Introduction

Building a neural network from scratch using NumPy is a valuable exercise for understanding the inner workings of deep learning models. By implementing the core components manually, we gain insights into the mechanics of neural networks, including forward propagation, backpropagation, and gradient descent optimization. In this guide, we'll walk through the process of creating a simple neural network to classify the letters A, B, and C using only NumPy.

Step 1: Preparing the Dataset

For this implementation, we'll represent the letters A, B, and C as 5x6 binary matrices, where each element corresponds to a pixel. The dataset consists of input matrices and their corresponding one-hot encoded labels:

import numpy as np

# Input matrices for letters A, B, and C
a = [0, 0, 1, 1, 0, 0,
     0, 1, 0, 0, 1, 0,
     1, 1, 1, 1, 1, 1,
     1, 0, 0, 0, 0, 1,
     1, 0, 0, 0, 0, 1]

b = [0, 1, 1, 1, 1, 0,
     0, 1, 0, 0, 1, 0,
     0, 1, 1, 1, 1, 0,
     0, 1, 0, 0, 1, 0,
     0, 1, 1, 1, 1, 0]

c = [0, 1, 1, 1, 1, 0,
     0, 1, 0, 0, 0, 0,
     0, 1, 0, 0, 0, 0,
     0, 1, 0, 0, 0, 0,
     0, 1, 1, 1, 1, 0]

# One-hot encoded labels
y = [[1, 0, 0],  # A
     [0, 1, 0],  # B
     [0, 0, 1]]  # C

We can visualize these matrices using Matplotlib:

import matplotlib.pyplot as plt

# Visualize letter A
plt.imshow(np.array(a).reshape(5, 6))
plt.title("Letter A")
plt.show()

Step 2: Defining the Neural Network Architecture

Our neural network will consist of:

• Input layer: 30 neurons (5x6 grid flattened)
• Hidden layer: 5 neurons
• Output layer: 3 neurons (corresponding to A, B, and C)

We'll initialize weights and biases with small random values to break symmetry:

np.random.seed(42)

# Initialize weights and biases
input_size = 30  # 5x6 grid
hidden_size = 5
output_size = 3

W1 = np.random.randn(input_size, hidden_size) * 0.01
b1 = np.zeros((1, hidden_size))
W2 = np.random.randn(hidden_size, output_size) * 0.01
b2 = np.zeros((1, output_size))

Step 3: Implementing Activation Functions

We'll use the sigmoid activation function for both the hidden and output layers:

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    return x * (1 - x)

Step 4: Forward Propagation

In forward propagation, we compute the activations of each layer:

def forward(X):
    # Hidden layer
    Z1 = np.dot(X, W1) + b1
    A1 = sigmoid(Z1)
    
    # Output layer
    Z2 = np.dot(A1, W2) + b2
    A2 = sigmoid(Z2)
    
    return A1, A2

Step 5: Backpropagation

Backpropagation involves calculating the gradients of the loss function with respect to the weights and biases, and updating them accordingly:

def backward(X, y, A1, A2, learning_rate=0.1):
    m = X.shape[0]
    
    # Output layer gradients
    dZ2 = A2 - y
    dW2 = np.dot(A1.T, dZ2) / m
    db2 = np.sum(dZ2, axis=0, keepdims=True) / m
    
    # Hidden layer gradients
    dA1 = np.dot(dZ2, W2.T)
    dZ1 = dA1 * sigmoid_derivative(A1)
    dW1 = np.dot(X.T, dZ1) / m
    db1 = np.sum(dZ1, axis=0, keepdims=True) / m
    
    # Update weights and biases
    W1 -= learning_rate * dW1
    b1 -= learning_rate * db1
    W2 -= learning_rate * dW2
    b2 -= learning_rate * db2

Step 6: Training the Neural Network

We can now train the neural network using the forward and backward functions:

def train(X, y, epochs=10000, learning_rate=0.1):
    for epoch in range(epochs):
        A1, A2 = forward(X)
        backward(X, y, A1, A2, learning_rate)
        if epoch % 1000 == 0:
            loss = np.mean(np.square(y - A2))
            print(f"Epoch {epoch}, Loss: {loss}")

# Train the network
X = np.array([a, b, c])
y = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]])
train(X, y)

Step 7: Making Predictions

After training, we can use the network to make predictions:

def predict(X):
    _, A2 = forward(X)
    return np.argmax(A2, axis=1)

# Predict the class of letter A
prediction = predict(np.array([a]))
print(f"Predicted class for letter A: {prediction}")

Conclusion

Implementing a neural network from scratch using NumPy provides a deeper understanding of how neural networks function. While this approach is educational, for more complex tasks and larger datasets, it's advisable to use established libraries like TensorFlow or PyTorch. Nevertheless, building a neural network from the ground up is a valuable exercise for grasping the fundamentals of deep learning.

---