Jeremy Abel

Background Designs

The backgrounds were created from a set of 3D assets and tileable background patterns, which were randomly swapped around to the beat of the music.

Process

Due to NDA shenanigans, we couldn’t get access to the face-tracking tech Intel developed for their app. It fell to me to duplicate the performance of the app: full facial feature tracking, only it also needed to work in the lower light levels common on concert stages.

I started by researching cameras sensitive to near-infra-red light. The heat of the concert lighting meant that there’d be a fair bit of infrared energy to pick up, even if the light levels were too low for a normal camera to perform well. I settled on a high-performance monochrome research camera from Point Grey.

For the face tracking itself, we used a two-pronged approach. Jason Saragih’s FaceTracker provided the fine-grained feature tracking of the lips and eyebrows, and head rotation was handled by a library from Seeing Machines.

To add some extra mouth movement, I mapped the microphone’s volume level to the avatar’s mouth opening ammount, in addition to the mouth movement detected by the cameras. A compressor and mixer provided an even audio signal across a wide range of singing volumes.

The XBox controller gave me live control of changing the background colors + designs, and the avatar’s rotation + position in case the cameras lost track of the face during a particularly vigorous performance.

Shoutouts to Point Grey for providing outstanding example code and documentation for their C++ libraries, and for free tech support over the phone.