Overview
On the image classification modeling, you need to understand how good your model is, meaning not absolute accuracy itself but relative meaning of the accuracy.
This is the example. Your first trial model's validation accuracy is 0.6. How do you think about it?
Without knowing the unique label number, ratio, and the data's difficulty, only answer you can return is "I don't know".
To evaluate the model, not only the absolute accuracy but also the base score to compare with are necessary.
If the modeling trial and error don't take much time, you can feel the scale of goodness and accuracy by many trial samples. But usually image classification model takes much time to make themselves.
How do we do the shortcut?
Basic Knowledge
Usually, when we make classification model, we can judge the model by comparing with base score. The base score is one of the check point and by the difference between the base score and accuracy score, we can check if the model's network is at least preferable or not.
The unique number of the teacher labels and the ratio of data which belongs to each teacher label makes one of the base score.
For example, cifar-10, one of the most popular color image data set, has ten categories and each category has same amount of images. In this case, even if you randomly give labels to images, the accuracy gathers around 0.1. So, you at first can judge your model's accuracy compared with 0.1.
This is the image which is composed of some parts of the cifa-10 data set.
Practical hack
As I wrote, the accuracy at the time you randomly give label can be one of the base score to compare with.
But what we hope to CNN is not that scale. When we use CNN, we often set high score such as 0.8 as goal accuracy.(This depends on what users want model to do and data.) Base score based on random selection is useful just as start point.
So, when you try to make CNN model by yourself, it is better to use inception-v3 fine tune model's accuracy as base score. About fine tune, it becomes bit long to write here. So, please check the article, How to make Fine tuning model.
The procedure is like this.
- make inception-v3 fine tune model by your data
- check accuracy by that and think the score as base score
- make your original model
- compare the accuracy score with the base score
Base score based on random selection works as start point. And inception-v3 fine tune model can be work as second check point.
In many cases, what we hope to deep neural network model is accuracy. If after some trials, our original model does't reach the inception-v3 base score, it is better just to use inception-v3 fine tune not our own.
Summary
When we make our own model, it is important to set some check points which are relatively easy to set. There are very good model and by fine tune, we can adjust those to our own data.
If it is difficult to surpass the check point not only from the aspect of accuracy but from time, the amount of data, money, you can adapt the check point model.