photo

Background Designs

photo

The backgrounds were created from a set of 3D assets and tileable background patterns, which were randomly swapped around to the beat of the music.

photo

Process


Due to NDA shenanigans, we couldn’t get access to the face-tracking tech Intel developed for their app. It fell to me to duplicate the performance of the app: full facial feature tracking, only it also needed to work in the lower light levels common on concert stages.

I started by researching cameras sensitive to near-infra-red light. The heat of the concert lighting meant that there’d be a fair bit of infrared energy to pick up, even if the light levels were too low for a normal camera to perform well. I settled on a high-performance monochrome research camera from Point Grey.

photo

For the face tracking itself, we used a two-pronged approach. Jason Saragih’s FaceTracker provided the fine-grained feature tracking of the lips and eyebrows, and head rotation was handled by a library from Seeing Machines.

photo

To add some extra mouth movement, I mapped the microphone’s volume level to the avatar’s mouth opening ammount, in addition to the mouth movement detected by the cameras. A compressor and mixer provided an even audio signal across a wide range of singing volumes.

photo

The XBox controller gave me live control of changing the background colors + designs, and the avatar’s rotation + position in case the cameras lost track of the face during a particularly vigorous performance.



Shoutouts to Point Grey for providing outstanding example code and documentation for their C++ libraries, and for free tech support over the phone.

Intel wanted a fun way to introduce their Pocket Avatar app, which showcased their latest face-tracking research, so we thought, what better than a live karaoke party, with the 3D avatars singing along?

Using multiple face-tracking technologies and Unity3D, we showcased the avatars on a huge curved projection screen with each performer’s facial movements mapped onto the avatars.


Credits:

  • Agency: Sid Lee
  • Studio: Bossa Digital
  • CD: Andrezza Valentin
  • Producer: Talia Greene
  • Asst. Producer: Nick Santana
  • Developer: Jeremy Abel
  • Developer: Jeramy Morrill
  • Design: Jeremy Abel
  • Fabrication: The Villian