HW3: Faces morphing

Overview

In this homework, we are going to morph an image into another. We also create a morph video sequence for the morph processing. To morph an image A to image B, we need to define some corresponding feature points on two images, then warp the image shape and disslove the colors. This homework is divided into two parts. In part A we manually select corresponding points from two images, then we morph a face to another face and we can also morph an object to another. In part B we automatically select facial features for many faces, then an average face is calculated by applying the morphing algorithm. In this part, we also masculinize and feminize a face with the mean man and woman faces.

Morphing algorithm

The morphing algorithm is composed of the 3 steps: 1. Defining correspondences 2. Computing a triangulation 3. Create the morph. For the face images, the correspondences should map eyes to eyes, mouth to mouth, chin to chin, ears to ears, etc., to get the smoothest transformations possible. In part A we use 43 points to define a face, in order to process the background, we also define several points on the image border. Then we use the Delaunay triangulation algorithm to divide the face image into several parts. The triangulation of different faces should be the same, we compute the triangulation for the mean of the two point sets to decrease the potential deformations.

Résultats
Average Triangulation Average Triangulation
Once we got the triangulation for two faces, we will define the mapping between two triangles. We use a 6-DOF affine matrix for triangulation transformation. These matrices must be computed independently for each pair of triangles. For each triangle, we compute the inverse affine transform, use it to look up the color associated to each pixel in both images, and compute their weighted average. To smoothly generate the morphing animation, we use a sigmoid function to control the level of dissolution according to the warp degree.

Result

PART A

Morphing Animation

Class Photo
 cat to tiger
me to caricaturebean fish to diver

The feature points define correspondence between two images. Generally, the more points, the better the morph. We create a morphed hand by morph left hand and right hand to their mid-shape and color. We also morphed a low-pass filtered image with a high-pass filtered image. The images are aligned and warped by the sampled points, the morphed hybrid looks better than the hybrid image, notice eyes and mouth.

More result

Left Hand
 Right Hand
 Morphed
lowpassImg
 highpassImg
 Morphed Hybrid
 Morphed Hybrid Small Size
 Hybrid
 Hybrid Small Size

PART B

In this part, we calculated the average faces for our class photos and a face dataset using morphing. By computing the average face for female and male faces, we can masculinize and feminize a face.
Instead of using the points selected in part A, we use a face feature detection method to find the correspondences among faces.
For the class photos, we show the average face by using different facial points. The selected points correspond to the whole face including hair. The detected points mainly choose the facial points, compare to the selected facial points there are more sample points around the mouth and eyes. We get a smooth hair region by using the selected points. By using the detected points we get a better result around the mouth and eyes.

Mean face using the selected points

Mean Face
 Average shape
Triangulation for mean face

Mean face using the detected points

Mean Face
 Average shape
Triangulation for mean face
Notice the difference between the mean faces, e.g. look at the detail of the mouth and hair.

For the Utrecht photos, there are 20 women and 49 men, we show the mean shape and mean face. The mean face tends to a male face, because the dataset contains more male photos. Furthermore, we generate the average face for woman faces and the average face for man faces. We will masculinize and feminize a face by morphing it to the average man face and average woman face separately.

Mean face for the Utrecht Dataset

Mean Face
 Average shape
Triangulation for mean face

Average face with/without smile

Just smile faces
 Without smile faces

Masculinizing and feminizing my own face

Average Female Face

Average Male Face

 Female Triangulation
Male Triangulation
Feminization Masculinization Original
Notice the difference in chin and beard part between the results. The chin and cheek of the average male face is broader than the female face.

PCA

Principal Component Analysis (PCA) is applied to the Utrecht Dataset, PCA will help us to understand the dataset and find the important components in the face space. A mean of zero is needed to find the PCA basis, thus we align the Utrecht faces based on the facial points, then we extract the mean from the dataset to compute the bases. In the following figure, we show the top 10 components of the zero mean data.
PCA basis

Note the bases are normalise to 0-1.