This project was done through VAKA. Goethe Institut approached us and wanted to create a "interactive" art piece
This Blog outlines the work I did to create a painted portait looking video, to be deployed outside of the german language center in wellington
Firstly, I wanna show how it worked. Johann Wolfgan Von Goethe is essentially the mascot, and they wanted him to do a welcoming piece as people come to the language center. There's just 1 problem... He died in 1832... However we have technoology, and I can make it happen...somehow...
Video of it in action
NOTE: The portait triggers with hidden ultra sonics sensors, that I installed, along with with a custom video player that I built to run this experience in Unity
The Beginings
We sat down with the client (Judith, who is the Head of the Language Department), and we ran through the possible options for what image of Goethe we wanted to use. We essentially ruled out any close up images, so that left the middle one, because we definitly wanted to capture that "painting on a wall" feeling
The Voice Actor
Once we had decided on the routine for Goethe, we needed a voice actor, so let me introduce Torsen, who will be our voice actor for the portait.
We went through and recorded some lines that we may want to use, but now that we have the audio for the frame, next came figuring out how to make everything work
What are you looking at?
So the main issue we faced right away was that Goethe was painted looking off into the distance. The main problem with this is that the software we planned to use (which was crazy talk), probably couldnt compensate for the off angle of the head.
And with some simple testing it confirmed my theory that it couldnt handle it well. So, the next step was to figure out how to turn his head. I would like to make a note here, that I only had a week to get this "Shot" done and deployed.
-
I tried a few things, ranging from making a simple model in Autodesk Maya, and mapping the face onto it, with a green screen plate in the background, to even considering frame by frame animation to manually rotate the head.
In the end I actually came up with a cool solution, and its all cause of facebook!
-
While experimenting, I had remembered that facebook had a 3D photo sytem, and that it allowed the user to pivot in small amounts...well thats all I really need!
I found out that you can force facebook to create a 3D image, but pharsing it 2 images. the first image is the main texture, and the second is essentially a deformation map, and for the purpose of what I needed, I knew I could get away with a simple gradient map.
From there I knew I had the head turn down. I also knew that I had to do some minor clean up, but that was something that was easy to do
So here is the video I used to create the head turn for Goethe.
Crazy Talk
With the hard part of the project done, now came the part where Goethe speaks, and for this we used CrazyTalk 8.
I decided to use it because it essentially allows me to 2D Rig a image, and then puppet it, like I would with a 3D rig.
Crazy Talk also has a feature that allows for auto puppeting with a audio source, so that means that I can feed it a previously recored line, and have it mimic (or close to it)
I was playing to the "Anime effect", where when someone watches Anime, technically the lip actions arent perfect to the verbal words, so I figured that effect would work the same with Goethe.
Once I exported out the seqence I knew there would also be clean up, so I did some smoothing and clean up in DaVinci Resolve, once all the footage and audio was sorted
Final Deployment
And yeah from there, that is a simplified version of how I brought a 200+ year old painting "back to life". and below if one of the final outputs for the lines that Goethe says.
Following on from the first one, we have actually launched another 7 lines for Goethe as well.
Software Used
