Friday, February 9, 2018

Slack bot for image stylization by magenta

Overview

With magenta and Slack’s Bots application, I made an image style change bot.
This works as following gif.

enter image description here

When we post the image to the bot, it adds the style to the image and posts the images to the slack channel.



Reason of making this

Probably, some of the readers of this article already have husband or wife. And some of you have maybe experienced the following conversation with the partner.

enter image description here

I'm getting married and as possible as I can, I want to avoid the situation above.

enter image description here

Approach

As a data scientist and a machine learning engineer, the adventure to get fiancee’s understanding to my area just started. Anyway at first, to reach the understandings, I re-thought about my fiancee.
She is
  • art-oriented
  • love museum
  • illustrator/digital artist?
  • on some services, selling drawings
This is her instagram account.

art heronita (@heronita.art) * Instagram photos and videos

67 Followers, 79 Following, 58 Posts - See Instagram photos and videos from art heronita (@heronita.art)


Okay. There are some approaches.
This time, I decided to try to do something to her drawings by machine learning knowledge. And by the combination of the word “art and machine learning”, magenta hit on me. About magenta, later I’ll show the detail. Anyway, with magenta, I can add some styles to the drawings.

Next, I thought about the flow. If whenever she wants to do style change, she needs to send the image to me and wait for my response, this is not nice. This kind of dull-witted flow can lead to a misunderstanding to the technology. So, as a preparation, I adopted Slack between us. By using Slack as front-end, we can make clean flow.

magenta

As I wrote above, I used magenta to add style. magenta is machine learning library for art area. From the official page,
Magenta is a research project exploring the role of machine learning in the process of creating art and music.
There is a nice presentation about magenta.


When I checked the information about magenta, I found more information about music than about image. In a day, I want to try the function about music.
Anyway, this time, I used the image style changer. The image below is one of the examples.(This image is from the fiancee’s instagram account, with her permission.)

enter image description here
This is official page link.

How to make

At first, I thought for the purpose, I could use Slack's outgoing webhook and incoming webhook. But actually, Slack already has Bots application and I used this. On the web page, we can make configurations and get API token.

On the server side, I used Python’s library slackclient and slacker. Actually, I can probably make just by slackclient. But about image posting to slack, I was stuck and from the viewpoint of fast prototype, I chose to use the both to avoid the problem for now. In a day, I’ll fix.(I know on this pattern, I sometimes won’t fix.)
About magenta’s stylization, I checked the official page and the following link. On my bot, I changed the following example way to the class and used it.
This is the code about magenta part. I need to fix some points. Anyway, as a prototype, it is enough.

import os
import random
import urllib2
import tensorflow as tf
from magenta.models.image_stylization import model


class Style():

    def __init__(self):
        self.original_image = None
        self.style = 'varied'
        self.checkpoints = 'check_points/multistyle-pastiche-generator-varied.ckpt'
        self.num_styles = 32

    def change_style(self, style):
        self.style = style
        if self.style == 'monet':
            self.checkpoints = 'check_points/multistyle-pastiche-generator-monet.ckpt'
            self.num_styles = 10

    def generate_image(self, image):
        styles = range(self.num_styles)
        random.shuffle(styles)
        which_styles = styles[0:6]

        with tf.Graph().as_default(), tf.Session() as sess:
            stylized_images = model.transform(tf.concat([image for _ in range(len(which_styles))], 0),
                                              normalizer_params={'labels': tf.constant(which_styles),
                                                                 'num_categories': self.num_styles, 'center': True,
                                                                 'scale': True})

            model_saver = tf.train.Saver(tf.global_variables())
            model_saver.restore(sess, self.checkpoints)
            stylized_images = stylized_images.eval()

            return stylized_images

    @staticmethod
    def download_checkpoints(checkpoint_dir):
        """
        This method makes a directory and downloads checkpoints data to that.
        """

        # check directory
        if not os.path.isdir(checkpoint_dir):
            os.mkdir(checkpoint_dir)

        url_prefix = 'http://download.magenta.tensorflow.org/models/'
        checkpoints = ['multistyle-pastiche-generator-monet.ckpt', 'multistyle-pastiche-generator-varied.ckpt']
        for checkpoint in checkpoints:
            full_checkpoint = os.path.join(checkpoint_dir, checkpoint)
            if not os.path.exists(full_checkpoint):
                print 'Downloading', full_checkpoint
                response = urllib2.urlopen(url_prefix + checkpoint)
                data = response.read()
                with open(full_checkpoint, 'wb') as fh:
                    fh.write(data)

And the prototype bot is the image below.

enter image description here

Later I'll upload whole code to my github account. The code is too big for blog and is still messy.
Little by little, I want to add the functions to the bot.