On our first Friday hack we decided to look into image style transfer using a neural network.
Our test case you ask? Exploring how artwork styles can be transferred from a famous artist image into some mugshots of the guys in the office.
We found a great open source project at this link. This library uses a convolutional neural network to create a style representation of any image which is input for training. This would allow us to feed it our own images which are then transformed using the styles of famous artists.
The most difficult part of getting going was actually setting up the library to run on our work stations. So we cracked open some beer and spent the installation time (~ 1 hour) coming up with strategies on what artists artwork would yield the best results.
Unfortunately, once installed, this library would take 4 – 6 hours to train the network and that doesn’t work well within the confines of a hack day.
Luckily for us, there is an excellent free online resource called Deepart. This allowed us to start testing immediately.
A machine learning algorithm is only a tool. In order to get good results, its down to the strategies of the person using the tool.
The Inputs
So we took some headshots from some of the guys in the office. These would work as our inputs.
To train the algorithm we chose artwork that we thought would work well. We chose an Andy Warhol styled portrait of James Dean, a Monet gardenscape and two portraits by Picasso.
We then decided that a few wild cards would add a bit more fun to the experiment. So we added a painted cartoon cat from google images and Vigo the Carpathian from Ghostbusters 2.
Andy Warhol
We found that for the most part the algorithm did capture the style of the artists. For the Andy Warhol portrait, it really captured the image with only a slight extension of things in some areas, for example, our teams hair grew a little longer and a few extra shoulders where added in places.
It got the colors mostly right, but seemed to think that the hair color was a good fit for clothing and that the background was an amalgamation of every color in the piece.
Finally the style was lost a little in the piece as pop art features block colours and dot gradients, whereas the algorithm used colour gradients and a mesh of colours rather than a blocked style.
Monet
For the Monet gardenscape we found that the algorithm may need to brush up on it’s impressionist skills a bit, but would do well in the pixel art industry.
It rendered our team members in extreme detail within an impressionist style, which seems like a real paradox.
It was confined to the colours of the piece, but really ingeniously used the flowers pinks, reds and purples of the piece to capture skin tone, and shading.
Picasso
The Picasso-ized photos of the team are the most realistic to the style by far, they mimic the artists style really well with block colors and features, skewed aspects as well as a slight change in expression over all of our team members faces.
I think Picasso’s style really plays to the algorithms strengths as an artist, anything with a very defined pattern is very well portrayed.
Cartoon
The final two wild card styles gave us a very unexpected result.
The images that we got where very good, the cartoon cat gave us all a really nice painted look and really followed the blocky paint stroke style that the artist has.
The algorithm followed this style very closely using the correct colors and style for the face, the only difference that I can pick out, and I really have to try, is that the background has some slight differences in color in all of our pictures.
Vigo the Carpathian
Finally, the best image we got in the experiment was from the Vigo the Carpathian photo style.
Although the algorithm confused the subject, color and style. It has produced one of the scarier portraits that anyone here has seen to date.
Find out some interesting facts about our good friend Vigo over at Trivia Happy.
Donald the Carpatian
In conclusion, we found that the best and most enjoyable results came from adding famous people to the images of the algorithm and choosing evil and creepy images as the style. If you have some free time I would recommend having a go yourself.
Head over to Deep Art for a ready built solution.
Or if you fancy rolling your own there’s a github repository here.