Keras – 5. NN go Wider & Deeper

Welcome to CS with James

In this tutorial I will discuss how to build NN go wider and deeper.

Wider Network Code
Deeper Network Code

If you take closer look at the code, everything is same except the code that define the Network

Wider Network

# Define the Model
model = Sequential()
model.add(Dense(1024, activation='sigmoid', input_shape=(784,)))
model.add(Dense(1024, activation='sigmoid'))
# 32 -> 1024 Neurons, Wider Network
model.add(Dense(num_classes, activation='softmax'))

Increased the number of neurons on the each layer.

After train this network on the MNIST Dataset

The result is 95.73% accuracy. compare to 91.41% accuracy improved a lot.

 

Deeper Network

Similar to the wider Network only the code that define the network changed.

# Define the Model
model = Sequential()
model.add(Dense(32, activation='sigmoid', input_shape=(784,)))
model.add(Dense(32, activation='sigmoid'))
model.add(Dense(32, activation='sigmoid'))
model.add(Dense(32, activation='sigmoid'))
model.add(Dense(32, activation='sigmoid'))
# 3 -> 6 Layers, Deeper Network
model.add(Dense(num_classes, activation='softmax'))
 

The result is 86.56% accuracy. Compare to previous Network which resulted 91.41% it get worse. 

Is it true to say deeper network works worse than wider network?

In this case “yes” but researchers found out if the network get too deep then it is hard to train. Therefore, they came with different activation function to solve the problem.

I will talk about why the deeper network works worse than the wider network and how to solve the issue.

 

Leave a Reply

Your email address will not be published. Required fields are marked *