Class Activation Map with Keras

Abstract

On this article, I'll try CAM, Class Activation Map, to mnist dataset on Keras.

CAM(Class Activation Map)

About CAM and Grad-CAM, please read the following theses for detail. Here, I'll just write roughly. And as a premise, on this article, I don’t distinguish CAM and Grad-CAM.

• Learning Deep Features for Discriminative Localization
• Grad-CAM: Why did you say that? Visual Explanations from Deep Networks via Gradient-based Localization

CAM is a technique to identify regions in an image by using CNN for a specific class. To say more precisely, it will show the weighted linear sum of the last convolutional layer's output.

Code

At first, we need to make usual convolutional neural network with Global Average Pooling. The data is MNIST and the network architecture is two convolutional layers and one global average pooling layer. This experiment is just to see how CAM works. So, about model's detail, I don’t touch.

from keras.datasets import mnist
from keras.layers import Conv2D, Dense, GlobalAveragePooling2D
from keras.models import Model, Input
from keras.utils import to_categorical

(x_train, y_train), (x_test, y_test) = mnist.load_data()

x_train_resized = x_train.reshape((60000, 28, 28, 1))
x_test_resized = x_test.reshape((10000, 28, 28, 1))
y_train_hot_encoded = to_categorical(y_train)
y_test_hot_encoded = to_categorical(y_test)

inputs = Input(shape=(28,28, 1))

x = Conv2D(64, (3,3), activation='relu')(inputs)
x = Conv2D(64, (3,3), activation='relu')(x)
x = GlobalAveragePooling2D()(x)
predictions = Dense(10, activation='softmax')(x)

model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['accuracy'])
model.fit(x_train_resized, y_train_hot_encoded, epochs=30, batch_size=256, shuffle=True, validation_split=0.3)

CAM is not difficult to write. But here, I'll use the keras-vis, which has function for CAM.

• Keras Visualization Toolkit

With this library, we can get the heat map of the CAM output by one line. The function for CAM is visualize_cam().
By the following code, I'm picking up one image from each classes and showing the outcome of CAM.

from vis.visualization import visualize_cam
import matplotlib.pyplot as plt
import numpy as np

for i in range(10):
    ind = np.where(y_test == i)[0][0]
    plt.subplot(141)
    plt.imshow(x_test_resized[ind].reshape((28,28)))
    for j,modifier in enumerate([None, 'guided', 'relu']):
        heat_map = visualize_cam(model, 4, y_test[ind], x_test_resized[ind], backprop_modifier=modifier)
        plt.subplot(1,4,j+2)
        plt.imshow(heat_map)

    plt.show()

The outcome is the following image. Although because I didn't keras-viz code, I'm not sure it is right or not, it seems to be working well.