Overview

On this article, I'll try four image classification models, vgg16, vgg19, inception-v3 and xception with fine tuning. With Keras, we can easily try this.
About fine tuning itself, please check the article below.

How to make Fine tuning model by Keras

The purpose of this article is to understand the points of fine tuning by doing some fine tuning models at once.

Data

I'll use cifar10 data set, which is composed of ten class color images.
enter image description here

Anyway at first, we need to prepare the data for fine tuning. By the code below, it imports the necessary libraries and prepares the data.

About the data preparation, there are some phases, limiting the amount of data and resizing. Here, I’ll just try fine-tuning. For that, few amount of data is enough. So, I limited the amount of the data. And pre-trained models have the restriction about the input data size. We need to resize the data to proper ones.

You can check the restrictions about the input data size on the Keras’s official documents.

Keras document: Applications

import copy
import random
import cv2
from keras.applications import VGG16, VGG19, InceptionV3, Xception
from keras.datasets import cifar10
from keras.layers import Dense, Dropout, GlobalAveragePooling2D
from keras.models import Model
from keras.optimizers import SGD
from keras.utils import to_categorical
import numpy as np

# read data
(x_train, y_train), (x_test, y_test) = cifar10.load_data()

# limit the amount of the data
ind_train = random.sample(list(range(x_train.shape[0])), 2000)
x_train = x_train[ind_train]
y_train = y_train[ind_train]

ind_test = random.sample(list(range(x_test.shape[0])), 1000)
x_test = x_test[ind_test]
y_test = y_test[ind_test]


def resize_data(data, size_list):
    data_upscaled = np.zeros(tuple(size_list))
    for i, img in enumerate(data):
        large_img = cv2.resize(img, dsize=tuple(size_list[1:3]), interpolation=cv2.INTER_CUBIC)
        data_upscaled[i] = large_img

    return data_upscaled


# resize data
# vgg16
vgg16_x_train = resize_data(x_train, [x_train.shape[0], 48, 48, 3])/255
vgg16_x_test = resize_data(x_test, [x_test.shape[0], 48, 48, 3])/255

# vgg19
vgg19_x_train = copy.deepcopy(vgg16_x_train)
vgg19_x_test = copy.deepcopy(vgg16_x_test)

# inception v3
inception_x_train = resize_data(x_train, [x_train.shape[0], 139, 139, 3])/255
inception_x_test = resize_data(x_test, [x_test.shape[0], 139, 139, 3])/255

# xception
xception_x_train = resize_data(x_train, [x_train.shape[0], 71, 71, 3])/255
xception_x_test = resize_data(x_test, [x_test.shape[0], 71, 71, 3])/255

# explained variable
y_train_hot_encoded = to_categorical(y_train)
y_test_hot_encoded = to_categorical(y_test)

Model

I'll try four pre-trained models, vgg16, vgg19, inception-v3 and xception. What we need to do is the followings.

load pre-trained model
add some layers
re-train the added layers with the training data

The code below is for those.
If you want to know those points more, please check How to make Fine tuning model by Keras.

The number of freezed layers depends on the models. Here, I decided to re-train the last blocks of the models.

# vgg16
def vgg16_fine_tune(x_train, y_train):
    vgg16_model = VGG16(weights='imagenet', include_top=False)
    x = vgg16_model.output
    x = GlobalAveragePooling2D()(x)
    x = Dense(128, activation='relu')(x)
    x = Dropout(0.2)(x)

    predictions = Dense(10, activation='softmax')(x)

    model = Model(inputs=vgg16_model.input, outputs=predictions)
    for layer in vgg16_model.layers:
        layer.trainable = False

    model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
    model.fit(x_train, y_train)

    for i, layer in enumerate(model.layers):

        if i < 15:
            layer.trainable = False
        else:
            layer.trainable = True

    model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metrics=['accuracy'])
    history = model.fit(x_train, y_train, batch_size=256, epochs=50, shuffle=True, validation_split=0.1)
    return history


# vgg19
def vgg19_fine_tune(x_train, y_train):
    vgg19_model = VGG19(weights='imagenet', include_top=False)
    x = vgg19_model.output
    x = GlobalAveragePooling2D()(x)
    x = Dense(128, activation='relu')(x)
    x = Dropout(0.2)(x)

    predictions = Dense(10, activation='softmax')(x)

    model = Model(inputs=vgg19_model.input, outputs=predictions)
    for layer in vgg19_model.layers:
        layer.trainable = False

    model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
    model.fit(x_train, y_train)

    for i, layer in enumerate(model.layers):

        if i < 17:
            layer.trainable = False
        else:
            layer.trainable = True

    model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metrics=['accuracy'])
    history = model.fit(x_train, y_train, batch_size=256, epochs=50, shuffle=True, validation_split=0.1)
    return history


# inception_v3
def inception_fine_tune(x_train, y_train):
    inception_model = InceptionV3(weights='imagenet', include_top=False)
    x = inception_model.output
    x = GlobalAveragePooling2D()(x)
    x = Dense(128, activation='relu')(x)
    x = Dropout(0.2)(x)

    predictions = Dense(10, activation='softmax')(x)

    model = Model(inputs=inception_model.input, outputs=predictions)
    for layer in inception_model.layers:
        layer.trainable = False

    model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
    model.fit(x_train, y_train)

    for i, layer in enumerate(model.layers):

        if i < 249:
            layer.trainable = False
        else:
            layer.trainable = True

    model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metrics=['accuracy'])
    history = model.fit(x_train, y_train, batch_size=256, epochs=50, shuffle=True, validation_split=0.1)
    return history

# xception
def xception_fine_tune(x_train, y_train):
    xception_model = Xception(weights='imagenet', include_top=False)
    x = xception_model.output
    x = GlobalAveragePooling2D()(x)
    x = Dense(128, activation='relu')(x)
    x = Dropout(0.2)(x)

    predictions = Dense(10, activation='softmax')(x)

    model = Model(inputs=xception_model.input, outputs=predictions)
    for layer in xception_model.layers:
        layer.trainable = False

    model.compile(optimizer='rmsprop', loss='categorical_crossentropy')
    model.fit(x_train, y_train)

    for i, layer in enumerate(model.layers):

        if i < 115:
            layer.trainable = False
        else:
            layer.trainable = True

    model.compile(optimizer=SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metrics=['accuracy'])
    history = model.fit(x_train, y_train, batch_size=256, epochs=50, shuffle=True, validation_split=0.1)
    return history

Execute and Evaluation

By following code, we can execute the re-training.

hist_vgg16 = vgg16_fine_tune(vgg16_x_train, y_train_hot_encoded)
hist_vgg19 = vgg19_fine_tune(vgg19_x_train, y_train_hot_encoded)
hist_inception = inception_fine_tune(inception_x_train, y_train_hot_encoded)
hist_xception = xception_fine_tune(xception_x_train, y_train_hot_encoded)

We can check how the trainings went on by visualizing those. But we need to know that this is not comparison between pre-trained models. It is because with the consistent manner, I chose the re-training target blocks and added layers, meaning I chose to re-train the last block, added the layers and the added layers are the same between models. To be added, I didn’t fix the random number. So, this is just that I made fine-tuning models with same data. This is not the comparison of how sophisticated the pre-trained models are.

import matplotlib.pyplot as plt
def show_history(history, title):
    plt.title(title)
    plt.plot(history.history['acc'])
    plt.plot(history.history['val_acc'])
    plt.ylabel('accuracy')
    plt.xlabel('epoch')
    plt.legend(['train_accuracy', 'test_accuracy'], loc='best')
    plt.show()


show_history(hist_vgg16, 'vgg16')
show_history(hist_vgg19, 'vgg19')
show_history(hist_inception, 'inception')
show_history(hist_xception, 'xception')

To the test data, we can check the test accuracy.

vgg16_eval = hist_vgg16.model.evaluate(vgg16_x_test, y_test_hot_encoded, verbose=1)
vgg19_eval = hist_vgg19.model.evaluate(vgg19_x_test, y_test_hot_encoded, verbose=1)
inception_eval = hist_inception.model.evaluate(inception_x_test, y_test_hot_encoded, verbose=1)
xception_eval = hist_xception.model.evaluate(xception_x_test, y_test_hot_encoded, verbose=1)

print('vgg16:{}'.format(vgg16_eval))
print('vgg19:{}'.format(vgg19_eval))
print('inception:{}'.format(inception_eval))
print('xception:{}'.format(xception_eval))

vgg16:[1.160641137123108, 0.597]
vgg19:[1.218008373260498, 0.577]
inception:[1.0930208473205567, 0.663]
xception:[1.8237217445373535, 0.458]

Data Science Struggle

Thursday, March 1, 2018

Some Fine tuning models with Keras: vgg16, vgg19, inception-v3 and xception

Overview

Data

Model

Execute and Evaluation

Blog Archive