How to write Dense block of DenseNets: understanding and coding with Keras

Abstract

This article covers basic understanding and coding of Dense block of DenseNets. DenseNets is one of the convolutional neural network models. If you have an experience of using fine-tuning or frequently tackle with image recognition tasks, probably you have heard that before.
DenseNets is composed of Dense blocks. It is expressed as the image below, which is quoted from https://arxiv.org/abs/1608.06993.

On the context of the history of convolutional neural network, ResNet helps the network to be deeper without degradation problem by the shortcut path to the output of the Residual module. DenseNets and Dense block is near concept from the different approach.

This article is to help to understand the basic concept of Dense block of DenseNets and how to write that. For coding, I’ll use Python and Keras.
About the ResNet and Residual module, please read the article below.

• How to write Residual module: understanding and coding with Keras

If you want to know the detail of DenseNets and Dense block, I recommend you read the article below.

• https://arxiv.org/abs/1608.06993

When you find a mistake, please let me know by comment or mail.

How to write Residual module: understanding and coding with Keras

Abstract

This article covers basic understanding and coding of Residual module. If you have experience of using fine tuning or frequently tackle with image recognition tasks, probably you have heard the network name, ResNet. ResNet is composed of Residual module, whose structure is expressed as below.

The image above is from https://arxiv.org/abs/1512.03385.
Basically, deeper neural network contributes to the better outcome. If you have enough computational resource(unfortunately, I don't have), for difficult task, you can approach it with really deep neural network. However, with deeper neural network, the problem of degradation comes, which makes it difficult to train the model. Residual module offers one of the solutions to this problem, meaning that with this, we can make deeper neural network by softening the difficulty of training.
For precise and better understanding, I recommend that you read the paper below. Here, I'll just show summary for simple and concise understanding and coding with Keras.

• https://arxiv.org/abs/1512.03385

If there are strange or wrong points, please let me know by comment or message.

How to write Inception module: understanding and coding with Keras

Abstract

This article covers the basic understanding and coding of Inception module.
GoogLeNet, which is composed by stacking Inception modules, achieved the state-of-the-art in ILSVRC 2014. And probably, many people already touched the models which have the name “Inception” by fine-tuning. Here, on this article, I'll deal with the Inception module.
To write the model, I'll use Keras with Python.
To deepen your knowledge, you can use the following paper.

• https://arxiv.org/abs/1409.4842

Convolutional Neural Network with Julia: Flux

Abstract

Here, I'll make a convolutional neural network model by Flux with Julia. In the article, Deep learning with Julia: introduction to Flux, I made simple neural network with Flux. Neural network, especially convolutional neural network, is quite efficient in image classification area. So, this time, I'll make the convolutional neural network model to image classification.

Object detection by CAM with Keras

Abstract

On this article, I'll try CAM(Grad-CAM) to high resolution images. Cam has the potential for object-detection. So, I will make CNN model and by CAM, check if it really works.
About CAM(Grad-CAM) itself, I'll recommend the theses below.

• Learning Deep Features for Discriminative Localization
• Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization

Class Activation Map with Keras

Abstract

On this article, I'll try CAM, Class Activation Map, to mnist dataset on Keras.

Similar image finder by CNN and Distance

Overview

On this article, I’ll show one of the methods to find similar images to some specific target image.

Usually, when we try to make the system to find some similar items, we have some choices and should choose one or some of them in response to the purpose. Here, I’ll adapt distance-based method using supervised learning model’s prediction.

For example, when we try to find the images which are similar to the leftmost image, the other images below are picked up by this.

VGG16 Fine-tuning model

Overview

On the article, VGG19 Fine-tuning model, I checked VGG19’s architecture and made fine-tuning model. On the same way, I’ll show the architecture VGG16 and make model here.

There are some image classification models we can use for fine-tuning.
Those model's weights are already trained and by small steps, you can make models for your own data.

About the fine-tuning itself, please check the followings.

• How to make Fine tuning model by Keras
• VGG19 Fine-tuning model

How to make Fine tuning model by Keras

Overview

Fine-tuning is one of the important methods to make big-scale model with a small amount of data.

Usually, deep learning model needs a massive amount of data for training. But it is not always easy to get enough amount of data for that. To be added, in many cases, it takes much time to make model from the viewpoint of training. I know you don’t like to see one epoch of training using the time from sunrise to sunset. In some areas like image classification, you can use fine-tune method to solve this situation.

For example, when you try to make image classification model, very deep CNN model works well(sometimes and other time not). To make that kind of model, it is necessary to prepare a huge amount of data. However, by using the model trained by other data, it is enough to add one or some layers to that model and train those. It saves much time and data.

Here, I show this type of method, fine-tuning, by Keras.

Re-try CNN + KNN model

Overview

When I tried CNN + KNN model before, the training epoch was not enough(50) to check the characteristics. This time I trained 200 epoch on the CNN phase.

CNN + KNN model accuracy

Overview

On the contest site like Kaggle, we can see many trials and good scores by the combination of some methods.
For example, you can get scores by logistic regression and lasso regression. You can make xgboost model by using those scores.
This time, about cifar-10, I make CNN model. And by using the score, I check KNN scores.

The pragmatic procedure of making CNN model

Overview

On the image classification modeling, you need to understand how good your model is, meaning not absolute accuracy itself but relative meaning of the accuracy.

This is the example. Your first trial model's validation accuracy is 0.6. How do you think about it?
Without knowing the unique label number, ratio, and the data's difficulty, only answer you can return is "I don't know".
To evaluate the model, not only the absolute accuracy but also the base score to compare with are necessary.

If the modeling trial and error don't take much time, you can feel the scale of goodness and accuracy by many trial samples. But usually image classification model takes much time to make themselves.
How do we do the shortcut?

How to write diverged type neural network by keras

How to write Diverged neural network

Overview

Simple style neural network as below is easy to write by deep learning frame work. 

This time, I make diverged neural network whose route to output is diverged and merged. The image of this is like below.

The purpose of this article is following two points.

• see how to write diverged type neural network
• see how accurate and good this type of model is

I used keras to write. Although it is bit annoying to write this kind of neural network compared with simple type, keras makes diverged type of model in relatively easy manner.
About the model's characteristics and accuracy, it’s difficult to judge, because there is no simple model which is relevant to the diverged model. So, we can just check roughly.

How to use Inception v3

Low cost image classification by CNN, convolutional neural network

Overview

These days, CNN(convolutional neural network) is almost regarded as the best answer to classify images.
But it has many rules.

• it needs huge amount of images
• it takes much time to train
• it needs slow and gradual steps to find good network model to attein the goal
• it needs high spec environment to do try-and-error

Of course, there are free data sets and kinda sample network which works easily in short time. But at the case you practically make model to solve some practical problem personaly or oficilally, those restrictions can face with you.
The combination of Inception v3 model and fine tune can solve the point.

Convolutional neural network scale experiment by keras

Overview

It is not easy to understand about convolutional neural network how the goodness changes when the nodes each layer has, layer’s number and other factors change.

For practical use of convolutional neural network, I experimented some types of convolutional neural network.

Convolutional neural network by keras

Make convolutional neural network model for mnist in keras

Overview

Convolutional neural network is one of the best solutions about image classification. In keras, it is relatively easy to make model.