TechVidvan

Handwritten Digit Recognition with Python & CNN

Hello friends, ‘Digits’ are a part of our everyday life, be it License plate on our cars or bike, the price of a product, speed limit on a road, or details associated with a bank account. In the case of a text which is unclear, it is easier to guess the digits in comparison to the alphabets

Machine Learning and Deep Learning are reducing human efforts in almost every field. Moreover, a solution achieved using ML and DL can power various applications at the same time, thereby reducing human effort and increasing the flexibility to use the solution. One such solution is a handwritten digit recognition system that can be used in postal mail sorting, bank check processing, form data entry, etc.

Convolution Neural Network

A Convolutional Neural Network or CNN is a Deep Learning Algorithm which is very effective in handling image classification tasks. It is able to capture the Temporal and Spatial dependencies in an image with the help of filters or kernels.

cnn sliding kernel

The kernel is just like a small window sliding over the large window in order to extract the spatial features and in the end, we get feature maps.

MNIST Dataset

mnist dataset

Source: MNIST

We are going to use the famous MNIST dataset for training our CNN model. The MNIST dataset was compiled with images of digits from various scanned documents and then normalized in size. Each image is of a dimension, 28×28 i.e total 784 pixel values.

You do not need to download the dataset from any external source as we will import it from keras.datasets

Layout of the basic idea

  1. Firstly, we will train a CNN (Convolutional Neural Network) on MNIST dataset, which contains a total of 70,000 images of handwritten digits from 0-9 formatted as 28×28-pixel monochrome images.
  2. For this, we will first split the dataset into train and test data with size 60,000 and 10,000 respectively.
  3. Then, we will preprocess the input data by reshaping the image and scaling the pixel values between 0 and 1.
  4. After that, we will design the neural network and train the model.
  5. After the model is trained, we will save it for future use.
  6. Next, we are going to use a webcam as an input to feed an image of a digit to our trained model.
  7. Our model will process the image to identify the digit and return a series of 10 numbers corresponding to the ten digits with an activation on the index of the proposed digit.

Download Handwritten Digit Recognition Code

Please download project source code: Handwritten Digit Recognition in Python

File Structuring

1. Train.py

We utilize the MNIST dataset to train our CNN model and then save the model in the current working directory.

2. RecognizeDigit.py

We load the saved model and use appropriate functions to capture video via webcam and pass it as an input to our model. Our model produces a prediction which is displayed to the user.

Libraries Required

Make sure that the following libraries are installed on your working machine before proceeding further

  1. Keras
  2. Tensorflow
  3. OpenCV
  4. Sklearn
  5. Numpy

Training the Model (Train.py)

Before we begin training, I would suggest you to train the model on Google colab as it offers training the model on GPU if your computer does not have one. It speeds up the training process by manifold and helps you achieve the final results much quicker.

Simply open a Google Colab Notebook > Edit > Notebook Settings > Hardware Accelerator > GPU > Save and Done..!!

1. Import the necessary libraries and modules

import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras import backend as K

2. Splitting the MNIST dataset into Train and Test

(x_train, y_train), (x_test, y_test) = mnist.load_data()

3. Preprocessing the input data

num_of_trainImgs = x_train.shape[0] #60000 here
num_of_testImgs = x_test.shape[0] #10000 here
img_width = 28
img_height = 28
 
x_train = x_train.reshape(x_train.shape[0], img_height, img_width, 1)
x_test = x_test.reshape(x_test.shape[0], img_height, img_width, 1)
input_shape = (img_height, img_width, 1)
 
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

4. Converting the class vectors to binary class

num_classes = 10
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)

5. Defining the model architecture

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

6. Compiling the model

model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

7. Fitting the model on training data

model.fit(x_train, y_train,
          batch_size=128,
          epochs=12,
          verbose=1,
          validation_data=(x_test, y_test))

Output :

model training

8. Evaluating the model on test data

score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])

Output :

test accuracy

You might have observed that with the training, our test loss decreased significantly as we ran our model for 30 epochs and accuracy improved to over 89%. I may not sound like a good figure but let’s test out our model on the real-world input.

9. Saving the Model

model.save('trained_model.h5')

Note : If you trained your model on Google Colab, then make sure you download the model in the project directory.

Digit Recognition

1. Importing the necessary libraries

import numpy as np
import cv2
from skimage import img_as_ubyte    
from skimage.color import rgb2gray
from keras.models import load_model

2. Setting up the videoCapture

width = 640
height = 480
cameraNo = 0
 
cap = cv2.VideoCapture(cameraNo)
cap.set(3,width)
cap.set(4,height)

3. Loading our pretrained model

model = load_model('trained_model.h5')

Note : Steps D to N will be in the infinite while loop

4. Reading the Image

while True:
  success, im_orig = cap.read()

5. Converting the image to grayscale

img_gray = rgb2gray(img_original)

6. Converting the result to uint8 range

img_gray_u8 = img_as_ubyte(img_gray)

7. Thresholding

(thresh, im_binary) = cv2.threshold(img_gray_u8, 128, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)

8. Resizing the image

img_resized = cv2.resize(im_binary,(28,28))

9. Inverting the image colours

im_gray_invert = 255 - img_resized
  cv2.imshow("invert image", im_gray_invert)

10. Reshaping the image for final transmission

im_final = im_gray_invert.reshape(1,28,28,1)

11. Transmitting the image to our model

ans = model.predict(im_final)

12. Extracting the result from the array returned and printing the predicted value

ans = np.argmax(ans,axis=1)[0]
  print(ans)

13. Putting the predicted value as a text on webcam feed

cv2.putText(img_original,'Predicted Digit : '+str(ans),
                    (50,50),cv2.FONT_HERSHEY_COMPLEX,
                    1,(0,0,255),1)
  cv2.imshow("Original Image",img_original)

14. Handling the exit

if cv2.waitKey(1) and 0xFF == ord('q'):
    break

15. Releasing the camera control and destroying all the windows

cap.release()
cv2.destroyAllWindows()

Plotting the Collage of Images of Digits from Dataset

Just in case, if you are curious and do not know how I made the above collage of images from the train dataset, let me show

(x_train, y_train),(x_test, y_test) = mnist.load_data()
 
import matplotlib.pyplot as plt
fig, axes = plt.subplots(10, 10, figsize=(8, 8), subplot_kw={'xticks':[], 'yticks':[]}, gridspec_kw=dict(hspace=0.1, wspace=0.1))
for i, ax in enumerate(axes.flat):
    ax.imshow(x_train[i], cmap='binary', interpolation='nearest')
    ax.text(0.05, 0.05, str(y_train[i]),transform=ax.transAxes, color='green')
plt.show()

Explanation:

Just after we load our data via mnist.load_data(), we need to import matplotlib. The image we see is the collection of various subplots hence we define a 10×10 subplot, meaning there are 100 images to be accommodated in the plot. You can see we have disabled both the xticks and yticks. In order to relate the image to its target we value, we also put a small text in the bottom left corner of the image. Gridspec_kw basically helps specify the gaps in between the plots, both horizontally and vertically. In the end, we display the plot using plot.show() method.

Summary

Hooray..!! You have successfully made a handwritten digit recognition system. Honestly, the intention was to make it work on real-life data, apart from the test dataset. Hence, you built something different from the usual tutorials. You can extend this project by adding the functionality of multi-digit recognition or you can completely create a new project from scratch. In this new project, you can ask the user to draw the digits with gestures and then detect them. Happy coding and all the best for great projects ahead.