Artificial intelligence → Module 3 → A neural network and scikit-learn

A small neural network and scikit-learn

Until now we wrote everything ourselves. Real neural networks have thousands of weights and complex learning, so we don't write them from scratch – we use the library scikit-learn. In this lesson a network learns to recognize handwritten digits and reaches an accuracy of almost 98%.

Data: handwritten digits

The scikit-learn library has a built-in set of 1797 small images of handwritten digits. Each image has 8 × 8 = 64 pixels, and each pixel is a number from 0 (white) to 16 (black).

# A look at the data: handwritten digits (8 × 8 pixels) from the scikit-learn library
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits

digits = load_digits()
print("Number of images:", len(digits.images))
print("Size of one image:", digits.images[0].shape)      # (8, 8)
print("The first image as numbers (0 = white, 16 = darkest):")
print(digits.images[0])

# showing the first 10 images with their labels
for i in range(10):
    plt.subplot(2, 5, i + 1)
    plt.imshow(digits.images[i], cmap="gray_r")
    plt.title(f"label: {digits.target[i]}")
    plt.axis("off")
plt.show()

Program output:

Number of images: 1797
Size of one image: (8, 8)
The first image as numbers (0 = white, 16 = darkest):
[[ 0.  0.  5. 13.  9.  1.  0.  0.]
 [ 0.  0. 13. 15. 10. 15.  5.  0.]
 [ 0.  3. 15.  2.  0. 11.  8.  0.]
 [ 0.  4. 12.  0.  0.  8.  8.  0.]
 [ 0.  5.  8.  0.  0.  9.  8.  0.]
 [ 0.  4. 11.  0.  1. 12.  7.  0.]
 [ 0.  2. 14.  5. 10. 12.  0.  0.]
 [ 0.  0.  6. 13. 10.  0.  0.  0.]]

Program: the network learns digits

# A neural network recognizes handwritten digits (scikit-learn, MLPClassifier)
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score, confusion_matrix

digits = load_digits()
X = digits.data / 16.0        # each image is a row of 64 numbers; we divide by 16 so that they are from 0 to 1
y = digits.target             # the correct labels: 0..9

# 80 % for training, 20 % for testing
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
print("For training:", len(X_train), "images, for testing:", len(X_test))

# a network with one hidden layer of 64 neurons
network = MLPClassifier(hidden_layer_sizes=(64,), max_iter=500, random_state=0)
network.fit(X_train, y_train)                # TRAINING

predicted = network.predict(X_test)       # TESTING
print(f"Accuracy on the test data: {accuracy_score(y_test, predicted) * 100:.1f} %")

print("Error table (row = correct digit, column = what the network said):")
print(confusion_matrix(y_test, predicted))

# five images on which the network made a mistake
errors = [i for i in range(len(y_test)) if predicted[i] != y_test[i]]
for i in errors[:5]:
    print(f"image {i}: correct {y_test[i]}, the network says {predicted[i]}")

Program output:

For training: 1437 images, for testing: 360
Accuracy on the test data: 97.8 %
Error table (row = correct digit, column = what the network said):
[[27  0  0  0  0  0  0  0  0  0]
 [ 0 34  0  0  0  0  0  0  1  0]
 [ 0  0 36  0  0  0  0  0  0  0]
 [ 0  0  0 29  0  0  0  0  0  0]
 [ 0  0  0  0 29  0  0  1  0  0]
 [ 0  0  0  0  0 39  0  0  0  1]
 [ 0  1  0  0  0  0 43  0  0  0]
 [ 0  0  0  0  0  0  0 39  0  0]
 [ 0  1  1  0  0  0  0  0 37  0]
 [ 0  0  0  0  0  1  0  1  0 39]]
image 118: correct 6, the network says 1
image 124: correct 8, the network says 1
image 130: correct 9, the network says 5
image 143: correct 9, the network says 7
image 181: correct 5, the network says 9

• digits.data is a table: each image is one row of 64 numbers – these are the features. digits.target are the labels (0–9).
• train_test_split splits the data into 80% for training and 20% for testing – as in the lesson How a machine learns.
• MLPClassifier(hidden_layer_sizes=(64,)) is a network with one hidden layer of 64 neurons; there are 64 input neurons (pixels) and 10 output neurons (digits).
• fit = training, predict = guessing.
• In the error table, the numbers on the diagonal are correct answers. For example, in the second row we see that one 1 was classified as an 8.

Comparison: k-NN and the network

# Comparison: k-NN and a neural network on the same data
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.neural_network import MLPClassifier

digits = load_digits()
X, y = digits.data / 16.0, digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

models = {
    "k-NN (k = 3)": KNeighborsClassifier(n_neighbors=3),
    "network, 16 neurons": MLPClassifier(hidden_layer_sizes=(16,), max_iter=800, random_state=0),
    "network, 64 neurons": MLPClassifier(hidden_layer_sizes=(64,), max_iter=500, random_state=0),
}
for name, model in models.items():
    model.fit(X_train, y_train)
    print(f"{name:20} accuracy: {model.score(X_test, y_test) * 100:.1f} %")

Program output:

k-NN (k = 3)         accuracy: 98.3 %
network, 16 neurons  accuracy: 97.2 %
network, 64 neurons  accuracy: 97.8 %

Interesting: on these small images the simple k-NN is just as good as the network (even slightly better). Neural networks show their advantage on large images and complex tasks – where k-NN becomes too slow and unreliable. The lesson: always compare with a simple model before reaching for a complex one.

Common misconceptions

• “98% means the network almost never makes mistakes” – on these images; on digits written differently (for example in your handwriting) it can make mistakes much more often – you will see this in lesson 18.
• “More neurons are always better” – after a certain point accuracy grows only a little, and learning takes longer; too many neurons can also lead to overfitting.

Practice tasks

Try it yourself first, and only then open the solution.

1. Try networks with 4, 8, 32 and 128 neurons in the hidden layer. How does the accuracy change?
2. Show the images on which the network makes mistakes, with the correct label and the network's answer.
3. How accurate is the network if it learns from only 50, 100 or 500 images?

# Task: how the number of neurons in the hidden layer affects accuracy
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(digits.data / 16.0, digits.target, test_size=0.2, random_state=0)

for n in [4, 8, 32, 128]:
    network = MLPClassifier(hidden_layer_sizes=(n,), max_iter=1000, random_state=0)
    network.fit(X_train, y_train)
    print(f"{n:4} neurons: accuracy {network.score(X_test, y_test) * 100:.1f} %")

   4 neurons: accuracy 92.8 %
   8 neurons: accuracy 94.4 %
  32 neurons: accuracy 97.8 %
 128 neurons: accuracy 98.6 %

Solution 2: images of the mistakes

# Task: show the images on which the network makes mistakes
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(digits.data / 16.0, digits.target, test_size=0.2, random_state=0)
network = MLPClassifier(hidden_layer_sizes=(64,), max_iter=500, random_state=0).fit(X_train, y_train)
predicted = network.predict(X_test)

errors = [i for i in range(len(y_test)) if predicted[i] != y_test[i]]
for k, i in enumerate(errors[:6]):
    plt.subplot(1, 6, k + 1)
    plt.imshow(X_test[i].reshape(8, 8), cmap="gray_r")      # 64 numbers back into an 8 × 8 image
    plt.title(f"{y_test[i]} → {predicted[i]}")
    plt.axis("off")
plt.suptitle("correct → what the network said")
plt.show()

# Task: how good is the network if it learns from only 100 images?
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(digits.data / 16.0, digits.target, test_size=0.2, random_state=0)

for count in [50, 100, 500, len(X_train)]:
    network = MLPClassifier(hidden_layer_sizes=(64,), max_iter=1000, random_state=0)
    network.fit(X_train[:count], y_train[:count])
    print(f"learns from {count:4} images: accuracy {network.score(X_test, y_test) * 100:.1f} %")

learns from   50 images: accuracy 80.8 %
learns from  100 images: accuracy 90.6 %
learns from  500 images: accuracy 96.7 %
learns from 1437 images: accuracy 97.8 %

Check your knowledge