Miss Daegu 2013 Contestants Face Morphing

Recently, there is a debate on plastic surgery from the post on Reddit titled "Korea's plastic surgery mayhem is finally converging on the same face. Here are the miss Korea 2013 contestants." 

To better see the similarity among them, I first took the images from one Japanese blog. After simple normalization and registration, I can get an aligned, animated GIF looping through the 20 contestants as shown below.

                                       

These images remind me of a course project in computational photography I did two years ago. I wonder how the "averaged" face of these 20 contestants look like. Thus, I morph all the face images into a mean face shape, then average over the picture values using the code I developed in the course project. Here is the result.

Using the shape of the face images, I can generate a short movie of face morphing from contestant number 1 to 20. We can now see how one face is smoothly transformed into another.

Another application is to synthesize faces with new shapes. For example, I can morph each face a bit toward the mean shape. This results in a stabilized version of animated GIF looping from contestant 1 to 20 as shown below. 

So far, the above visualization figures are qualitative, we may be able to get more insights by quantitative analysis. 

First, we construct the eigenspaces from these contestants faces. However, the standard Principle Component Analysis cannot be directly applied due to the slight pose variation and occlusion by the hair. Instead, we perform a robust version of PCA, e.g., [Robust PCA, Low-Rank Matrix Recovery], to factor the low rank part and the sparse errors. Specifically, I use the implementation provided in [Low-Rank Matrix Recovery].
[The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices, Z. Lin, M. Chen, L. Wu, and Y. Ma (UIUC Technical Report UILU-ENG-09-2215, November 2009)]
Here are three sample results.

Second, by performing singular value decomposition, we get the eigenfaces and the corresponding eigenvalues. Here we can see that the eigenvalues vanish after 7, suggesting that the rank of the image data is 6.

Here we visualize the largest 6 eigenfaces. These eigenfaces encodes the variation within the 20 contestants.

Third, by projecting faces onto the eigenspace, we can analyze how these faces are distributed in the eigenspace. We show here the coefficients of the 20 contestants. Note that most of the energies are concentrated on first two eigenvalues.

Thus, we plot the coefficients corresponding to the first two eigenvectors. We can now see how similar the appearance of these contestants are.

 Fourth, we can try to construct the affinity matrix which encodes the pairwise similarity. (blue indicates similar and red indicates dissimilar).

This affinity matrix facilitates many interesting applications. For example, by summing up over columns or rows, we get a measure of how a contestant is different from the rest. In the plot below, x-axis denotes the contestant index (from 1- 20). The y-axis indicates how distinct the contestant is. This measure can also be interpreted as the salient object among the 20 contestants using the surround center information divergence.

(If you are interested in saliency, please see our paper on ICPR or slide.)

From the plot above, we can see that contestant 1, 2 and 6 are more distinct (or more salient) than the rest of the contestants. On the other hand, contestant 7, 12, and 15 are more common (i.e., closed to the averaged face) than others. 

Salient contestants (1, 2, 6)

It is interesting that one actually used these most salient contestants (1, 2, 6) to support their opinions on how they are dissimilar [link]. (I am not sure whether they did the same eigen analysis or not.)

Common contestants (7, 12, 15)

Here are the pictures of all contestants.

Another application of using the affinity matrix is to come up with a displaying order with maximized or minimized similarity from one to another. You may use one of them to support your opinion on whether they are similar or not.

Consider a complete graph of 20 vertices with edges encode the pairwise similarity, the problem of finding a displaying order is the problem of finding a (weighted) Hamiltonian circuit in an undirected graph [see Hamiltonian path problem]. As the problem is known to be NP-hard, we could use a greedy heuristic to find approximated solutions. 

The heuristic algorithm works as follows. Start from a random contestant, choose the next contestant (that has not been selected before) with maximum or minimum similarity with the previous contestant. Repeat the process until we find the ordering.

Here is the result of ordering with maximized pairwise similarity. Note that the transition between contestants are more smooth.

Here is the result of minimized pairwise similarity. They look more dissimilar in this displaying order. 

Resources: 

• If you would like to try it out on your own, you can download my code (in Matlab) here (12.7 MB)

• You can also just download the cropped, aligned, and mean faces here (12.6 MB)

Have fun! :D



Further readings:

• Average faces
• Face of tomorrow project by Mike Mike. [link]
• Face Research has a online demo for you to make an average face [link]
• Face morphing [wiki]
• Principle component analysis
• PCA [wiki]
• Eigenfaces [scholarpedia] [wiki]
• Low-Rank Matrix Recovery and Completion via Convex Optimization [link]
• Saliency
• Salience (neuroscience) [wiki]
• Hamiltonian path/circuit [wiki]

Updates (May 4, 2013)

1. The title of the original post was wrong. They are in fact contestants for Miss Daegu. I had corrected the title. Sorry for the confusion.

2. The poll results are out and is available on the official website

The top candidates are contestant number 5, 1, 15, 20, and 19.  We may be able to "guess" how the quantitative analysis of similarity is related to the result.

The first thing I observed is that one of the most salient (No. 1) and most common contestants (No. 15) won the 2nd and the 3rd places, respectively. This result seems to coincide with the psychological study on averaged faces. 

Averaged Faces Are Attractive, but Very Attractive Faces Are Not Average 

Thomas R. Alley and Michael R. Cunningham, Psychological Science, 1991

We then plot the coefficients corresponding to the first two eigenvectors for top 5 contestants (marked in red). It seems that the selection process somehow avoided choosing both candidates with similar appearances. 

Update (July 20, 2013)

One of the most salient contestant (number one), Yoo Ye-bin, crowned the Miss Korea 2013.