The backgrounds were created from a set of 3D assets and tileable background patterns, which were randomly swapped around to the beat of the music.
Due to NDA shenanigans, we couldn’t get access to the face-tracking tech Intel developed for their app. It fell to me to duplicate the performance of the app: full facial feature tracking, only it also needed to work in the lower light levels common on concert stages.
I started by researching cameras sensitive to near-infra-red light. The heat of the concert lighting meant that there’d be a fair bit of infrared energy to pick up, even if the light levels were too low for a normal camera to perform well. I settled on a high-performance monochrome research camera from Point Grey.
For the face tracking itself, we used a two-pronged approach. Jason Saragih’s FaceTracker provided the fine-grained feature tracking of the lips and eyebrows, and head rotation was handled by a library from Seeing Machines.
To add some extra mouth movement, I mapped the microphone’s volume level to the avatar’s mouth opening ammount, in addition to the mouth movement detected by the cameras. A compressor and mixer provided an even audio signal across a wide range of singing volumes.
The XBox controller gave me live control of changing the background colors + designs, and the avatar’s rotation + position in case the cameras lost track of the face during a particularly vigorous performance.
Shoutouts to Point Grey for providing outstanding example code and documentation for their C++ libraries, and for free tech support over the phone.
Intel wanted a fun way to introduce their Pocket Avatar app, which showcased their latest face-tracking research, so we thought, what better than a live karaoke party, with the 3D avatars singing along?
Using multiple face-tracking technologies and Unity3D, we showcased the avatars on a huge curved projection screen with each performer’s facial movements mapped onto the avatars.
- Agency: Sid Lee
- Studio: Bossa Digital
- CD: Andrezza Valentin
- Producer: Talia Greene
- Asst. Producer: Nick Santana
- Developer: Jeremy Abel
- Developer: Jeramy Morrill
- Design: Jeremy Abel
- Fabrication: The Villian