# Non-Rigid Face Tracking (2)

In this post, we continue to consider the Shape Model of the Non-Rigid Face Tracking principle. Previously, we illustrated how to construct a Shape Model. Now we emphasize on the Shape Model analytical expression.

Edit: a flash-based demonstraction is available here.

A shape $x$ is expressed as a linear function:

$$\mathbf{x} = \overline{\mathbf{x}} + Pb$$

where $\overline{\mathbf{x}}$ refers to the mean shape, $P$ is a matrix constructed using the few first  eigen vectors obtained from the Principal Component Analysis on Shapes, and $b$ are the corresponding modes of variation.

This formulation is our Shape Model also known as the Point Distribution Model (PDM). By varying values of $b$ (usually between $+\pm3\sigma$), we can model different face deformations as depicted here-under.

Resulting PCA mean (center) and main rigid basis deformations

The PDM model is nice, however faces can appear at any place (left, right, including fat or close to the camera) and at any orientation (even non facing ones) in images. That’s a problem because the PCA analysis on such a database will fail to deliver a correct PDM. In this case, few modes of variations (after truncation) will not be able to explain a large enough variance over the data (see left image in the following illustration).

Left, unaligned face landmarks. Center and right, aligned face landmarks. Picture extracted from the book: "Mastering OpenCV with Practical Computer Vision Projects".

So we need to align the faces before building our PDM with the face databse. To do that, a transformation is used:

$$\mathbf{x} = s.R(\overline{\mathbf{x}} + Pb) + t$$

The procrustes alignment procedure gives a correct estimation of the PDM (see Cootes paper to understand how to align a set of shapes).

# A Flash Sample

Non rigid face-tracking. It’s in flash so not as robust as the c++/mobile application.

## 4 thoughts on “Non-Rigid Face Tracking (2)”

1. Dear Sir.
I am pleasure due to your face tracking example.
Now, I am analyzing the source code of book of “Mastering OpenCV with Practical Computer Vision Projects”.
And I simulated the book source.
However, I can’s obtain a sucessful face tracking result compared to your flash example.
I wonder several different points or algorithms between book source and your flash example.

Sincerely Dong-Ju Kim

• Hi,

Flash example is not based on the approach you mentionned above. We rather use Support Vectors to construct the classification of face pixels.

Good luck!

One more,
Can you teach me the reference papers to implement your example?

Sincerely Dong-Ju Kim

• Unfortunately, I can’t tell you too much since the technology is being part of a commercial product.

However, I’m sure you’ll find out by yourself interesting papers on the topic.