So I thought I would take the inaugural post on charlesearl.blog to commemorate the great dearly departed Tupac while also dipping my foot into Deep Learning Neural Style Transfer. What??
Style transfer[1] is the AI that powers apps like Pikazo and Prisma — it extracts the style in from one image then re-images a second image in that style to create a third novel image.
When a deep neural network is trained to classify images, something really interesting happens. During the training process, the network is discovering features that can be used to accurately identify images — millions of images might be analyzed. The feature-specific knowledge gets organized in a hierarchy. This knowledge is split across the layers of the network roughly according to complexity — the lower level layers tend to capture relations at the pixel level whereas the units in higher levels of the network capture the relations between abstract entities in the image.
In the neural original neural style paper [1] the authors use a pre-trained deep neural network in kind of the same way that a doctor uses an X-ray machine. That is, as an instrument that can discover structure not immediately apparent. While the doctor can get some idea of the condition of your bones by careful use of the X-ray source, the researchers use the network to extract both “style” and “abstract content” from the artwork and the photograph. In style transfer, spatial correlations between image features — explicitly defined by the Gram matrix — are used as a proxy for style. Cleverly reconstructed images corresponding to the output of each convolutional layer of the network are used as a proxy for abstract content.
Constructing the composite image then is a matter of overlaying the style (Gram matrix representation ) and content images for selected layers. Through some experimentation, the authors of the original algorithm [1] arrived a set of layers that seem to work best for style and content respectively. This is still an active area of experimentation. Prisma uses clever custom algorithms to speed the stylization process so those hip images can appear in your Facebook post in reasonable time.
After experimenting with a few implementations, I made some headway (given quirkiness with my NVIDIA 1060 graphics card and the various deep learning libraries ) with an implementation of neural style transfer by Justin Johnson at https://github.com/jcjohnson/fast-neural-style. It is further described in his paper[2].
Now on to the image making! Given that Tupac’s mother Afeni Shakur also passed away this year (her memorial in the New York Times is a stirring testament) I thought it would be appropriate to look at the art of the 1970’s Black Power movement as a source of style. The painting Revolutionary by Wadsworth Jarrell seems to capture the urgency of the lives that Afeni and Tupac lived.
What an image! Both the rich structure of the image and the fact that I hadn’t come across many examples of the Black Arts movement work in style transfer seemed to make this especially gratifying project.
So here’s what I came up with.
On the left are the original photos of Afeni Shakur (taken in the ’70s) and Tupac at the height of his art. On the right Revolutionary. The new images are in the center. To my naive eyes, the new image of Afeni works a bit better. Sorry Tupac. But the other thing that has me intrigued is the textures in Revolutionary. Jarrell has embedded a lot of symbolism: you can pick out “Black”, “Revolution”, “Unrest”, and “Black is Beautiful” in the painting. Are these symbolic elements recognized at any layer in the network? There are clearly some interesting questions at the intersection of art, computer vision, and deep learning.
To end my post, I’m going to leave with an experiment based on Romare Bearden‘s Train Whistle Blues. I think this works a little better for Tupac.
Now I’m really interested in seeing how clever we can get with identifying interesting textures for the transfer.
[1] Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. A Neural Algorithm of Artistic Style(2015).
[2] Johnson, Justin, Alahi, Alexandre, and Fei-Fei, Li Perceptual Losses for Real-Time Style Transfer and Super-Resolution in The 14th European Conference on Computer Vision – Amsterdam, The Netherlands, ECCV 2016.
Charles write here 