Active Appearance Models

For our vision final project we explored Active Appearance Models. To simplify the problem we used a training data set consisting of 12 pictures of Jim. We then hand marked the correspondence points between the images to generate a mesh. To aid us in marking future training data we succeeding in creating a heuristic for assisting with the correspondence point marking. It looked back one frame and calculated a corresponding point based on spacial orientation of previous points and surrounding texture.

Using Matlab, the 12 training pictures and their correspondence points were run through Principle Component Analysis to generate 6 eigenfaces and 6 eigenshapes. Using these eigenfaces we are able to model a surprising number of expressions and lighting conditions on Tucek. We double checked the validity of our eigenfaces by using them to reconstruct our training data as seen in the images below.

We then attempted to fit the eigenfaces to the rest of the video sequences of Jim. Unfortunately, we were unable to find a suitable description of the Procrustes Iterative Fitting algorithm. As a result, we made do with our own matching algorithm which used gradient descent over sum of squares difference. This, of course, was understandably quite a bit slower then the algorithm used in the paper.

We also explored taking advantage of the graphics card's image processing abilities to quickly transform the geometric mesh of our eigenfaces. In particular, we used this to generate the geometric normalization which was then used to calculate our eigenfaces. Attempts to use Open GL's alpha blending capability to quickly render a face from the eigenfaces were partially successful, but not used in the demo for compatibility reasons.

In order for this to do full blown face tracking we need to train it against a much larger training set. It would not be difficult for our training data format to be extended to match standards sets such as those available here and elsewhere on the net. We also need to extend it to use the iterative procrustes fitting algorithm, as opposed to our own gradient ascent.