Keras – 9.1 Build&Train AlexNet With CIFAR10 Dataset (Keras)

Welcome to CS with James

In this tutorial I will discuss about how to build the AlexNet and train the AlexNet with CIFAR10 Dataset

Before talking about the code and implementation let’s visualize it.

The Original AlexNet uses two path, that is solely because of not enough GPU Memory back in the 2012, in 2018 we don’t have to use two path strategy.

You can see from the image that the first layer have 224*224*3 which is standard input shape for the image-net challenge. But we are going to use CIFAR10 which has 32*32*3 input shape, so we have to modify some of the hyper parameters in the Network Design.

AlexNet became famous on 2012 after the Image-Net challenge. It was the first implementation to solve the image classification problem in Image-net challenge and the AlexNet was the winner of the 2012 and decrease error rate by 9.4%

The sequential Design architecture of the Deep Neural Network became traditional design.

ZFNet share same design with the AlexNet and it works better, but AlexNet is more famous because the ZFNet simply changed some of the hyper parameters from the AlexNet.

Here is the published paper for AlexNet

Here is full code for building AlexNet

As you can find out from the Code The only difference from the previous code is there it defines the Model.

# Define the Model
model = Sequential()
# model.add(Conv2D(96, (11,11), strides=(4,4), activation='relu', padding='same', input_shape=(img_height, img_width, channel,)))
# for original Alexnet
model.add(Conv2D(96, (3,3), strides=(2,2), activation='relu', padding='same', input_shape=(img_height, img_width, channel,)))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
# Local Response normalization for Original Alexnet
model.add(BatchNormalization())


model.add(Conv2D(256, (5,5), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=(2,2)))
# Local Response normalization for Original Alexnet
model.add(BatchNormalization())


model.add(Conv2D(384, (3,3), activation='relu', padding='same'))
model.add(Conv2D(384, (3,3), activation='relu', padding='same'))
model.add(Conv2D(256, (3,3), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(3, 3), strides=(2,2)))
# Local Response normalization for Original Alexnet
model.add(BatchNormalization())


model.add(Flatten())
model.add(Dense(4096, activation='tanh'))
model.add(Dropout(0.5))
model.add(Dense(4096, activation='tanh'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))

There are few modifications from the Original AlexNet design. The Most important modification is from the first layer. The original AlexNet uses the filter size of (11,11) and strides of (4,4), however, if you try to apply it to the CIFAR10 Dataset you will encounter the error. The reason is input shape. Filter size of (11,11) and strides of (4,4) will decrease the tensor size by a lot but CIFAR10 dataset is not big enough for it.

Second modification is Normalization layer. The original AlexNet uses the Local Response Normalization. However, I couldn’t find the Local Response Normalization from the Keras so I replaced with the batch normalization which works better and uses as standard normalization layer in nowadays.

As I have mentioned previous introduction post I will train 1k epochs and here is the result.

The result is not so good, 73.87%, however we will increase accuracy by building next neural network designs. If I increase the epoch there is possibility for the improvement but it is not worth for the educational purposes.

It took me about 3.5 hours to train 1k epochs from the single 1080 Ti, so if you have something smaller than that decrease the epoch just for educational purpose or good luck with that especially if you try to train with the CPU.

The 1k epoch might sounds too much but actually in the real competition they train about 300k epochs.

Leave a Reply

Your email address will not be published. Required fields are marked *