How Dropout works on Neural Network

Overview

Dropout is one of the good techniques to make good neural network model. The system of it is very simple.

What is Dropout?

Dropout is the technique to prevent neural network from over-fitting. At the time of training, it deactivates some rates of nodes, leading to prevent the model from over-fitting.
Usually, neural network has awesome capacity to trace data. It means it can easily come front of over-fitting. So, some techniques like regularization and dropout can fulfill important role to solve that.
You can imagine how dropout works.

The black nodes means deactivated nodes. At the training, the parameters are updated with some nodes deactivated.

Check concretely

By dropout, we can get two following effects.

• easing over-fitting
• improving accuracy by model combination

---

Let’s think the neural network below. It has input, hidden, output layer.

If you set dropout on the hidden layer, you can express that as the diagram below. The nodes, $Be\left( p \right )$, means dropout. As you can see, each node on the specific layer has possibility to be activated.

To think it mathematically, let's check more accurate diagram.

This is not that the specified rate of nodes is activated but that each node has the specified rate of being activated.
By equation, you can express as below.
$r_j^{\left( l\right) }\sim Bernoulli\left( p\right)$
$\tilde{y^{\left( l \right)}} \sim r^{\left( l \right)} \times y^{\left( l \right)}$
$z_i^{\left( l + 1 \right)}=w_i^{\left( l+1 \right)}\tilde{y^l}$
$y_i^{\left( l+1 \right)}=f\left(z_i^{\left( l + 1 \right)} \right)$
The $r$ expresses the dropout. Those are set to each nodes and follow Bernoulli distribution. It means the value becomes $1$ with probability $p$ and $0$ with $1-p$.
In this case, $y_{i}^{\left( l\right) }$ becomes the input of the next nodes with probability $p$ as it is and with probability $1-p$, it gives $0$ to the next nodes.
At the training, the parameters are updated with activated parameters and deactivated parameters. At the test, the updated parameters are multiplied by $p$ and the values are used.
$W_{test}^{\left( l\right) }=pW^{\left( l\right) }$
“model combination” means that at the training step, with activated nodes some different networks construct combination of the models. When the architecture has $n$ nodes, $2^{n}$ networks can appear.

Adjust dropout parameter

Dropout has one parameter $p$. To fix this $p$, two types of setting was tried in the thesis.

1. adjust $p$ with fixed number of nodes of specified layer
2. adjust $p$ and node number $n$ with fixing $p \times n$

The first one means that you can change the activated nodes number rate $p$ and check accuracy and loss. The second one means that you can change $p$ with fixing the activated nodes number and check accuracy and loss.
Actually, if the number of activated nodes is not enough, the training doesn’t go well. If the ratio of activated nodes to deactivated ones is high, the over-fitting is easily observed.
So, it is efficient to adjust parameters from the viewpoint of those two settings.
Roughly, just after input, more or less 80% of nodes should be activated and on hidden layers 50% is favorable.

---

Personally, following points I cared about.

• the parameter $p$ is set per nodes not on layer
• on thesis, $p$ means activated nodes rate but on Keras, $p$ means deactivated nodes rate

Reference

Mr.Geoffrey E. Hinton’s page has the link for the thesis about dropout.
On the thesis, some experiments to many types of data are written.