TP 2 - Texture Synthesis and Transfer

By Ali Hajiabadi

website : http://ali.hajiabadi.net/en/

 

Texture Synthesis

Texture synthesis is the process of stitching small texture images together in order to obtain larger images containing the texture. The basic idea is to clip several patches from the seed image and put them together, side by side, to generate larger image. The first drawback of this approach is the different between adjacent blocks along the borders which produces noticeable edges between them. In order to avoid this, we use the method described in “Image Quilting for Texture Synthesis and Transfer” (Efros et al., 2001). The general steps of this algorithm, as mentioned in the article are:

  1. Go through the image to be synthesized in raster scan order in steps of one block (minus the overlap).
  2. For every location, search the input texture for a set of blocks that satisfy the overlap constraints (above and left) within some error tolerance. Randomly pick one such block.
  3. Compute the error surface between the newly chosen block and the old blocks at the overlap region. Find the minimum cost path along this surface and make that the boundary of

the new block. Paste the block onto the texture. Repeat.
I am going to implement the mentioned algorithm in 3 phases. The first phase is just to select patches from the source image randomly and put them together to generate the output. I use MATLAB and for this phase I create a function called “quilt_random” with this declaration:
quilt_random(sample, outsize, patchsize)

Where sample is the input seed image (small texture image), outsize is the desired size for the output image, and patchsize is the size of the square patches.
The results for this phase are shown in the following figures. (Patchsize = 50 px)

 

Source (variant sizes)
Result (500 x 500 pixels)

The next phase is to improve the previous function to choose the patches more wisely. In order to do that, we select patches with a little overlap on each other. The new patch is selected due to the similarity of its overlap parts with the previous patches. Overlap parts are on left and top sides of the new patch.
The similarity of the overlaps is estimated by calculating Sum of Squared Difference (SSD) between them and all possible blocks in the source image. The result of this calculation is a matrix that each of its elements shows the SSD of its surrounding block.
The overlapped parts are not supposed to match each other exactly, therefore we choose a block with SSD lower than a specified threshold. This threshold is calculated by this equation:
Threshold = 1 + min_SSD x tol
Where min_SSD is the minimum value in the calculated SSD matrix and tol is the desired tolerance. Usually there are several blocks with SSD lower than the threshold; in that case one the acceptable blocks will be selected randomly.

For this part the following function is implemented:
quilt_simple(sample, outsize, patchsize, overlap, tol)

Where sample, outsize and patchsize is exactly like the first function (quilt_random). overlap is the size of the overlap part in pixel, and tol is the tolerance described above.

The results for this phase are shown in the following figures. (patchsize = 50 px, overlap = 10 px, tolerance = 0.1)

 

Source (variant sizes)
Result (500 x 500 pixels)

As can be seen, the result has improved significantly using similarity check approach. Now to get even better results, the last part of the method is implemented. In this part after selecting a new patch, its overlap parts are compared to the corresponding parts of the previous patch in order to generate a mask to eliminate least-similar parts of the overlaps. Technically, this mask determines how two patches are supposed to stitch together.
To generate this mask, squared difference of the parts are calculated, then a minimum cost path is calculated to be used as the border between two patches. This way, the edge between two patches has the minimum difference possible, thus improving the smoothness of the result along the edges.

For this part another function is declared like this:
quilt_cut(sample, outsize, patchsize, overlap, tol)

Where inputs are exactly the same as the last function (quilt_simple). The following figures show you the result obtained using this algorithm. (patchsize = 50 px, overlap = 10 px, tolerance = 0.1)

Source (variant sizes)
Result (500 x 500 pixels)

The last 4 textures are downloaded from cgtextures.com.

A sample of masking done between overlaps is shown in the following figure:

From left to right:

a) Right overlap part of the previous patch

b) Minimum cost path calculated

c) Left overlap part of the new patch

d) Result obtained applying the generated mask

Here are some comparison between the results obtained using three described functions.

Random Quilting
Overlap Quilting
Cut Quilting

Texture Transfer

As mentioned before, this algorithm searches for similar square patches inside a small texture image and after selecting best matches, it puts them together to create a larger image with the same texture. Therefore, we can use this algorithm as a texture transfer algorithm too. We just have to change the similarity check part of the algorithm to evaluate the intensity similarity between the texture patches and an image of our choice.

This function is implemented to do so:
quilt_transfer(sample, input, patchsize, overlap, tol, alpha)

The inputs are just like the previous function, however instead of outsize, the input image is sent as an argument.
The following images show the input picture, texture and the obtained result.

Source
Texture
Result

Click for bigger version

Click for bigger version

Click for bigger version

 

Iterative approach

In texture transfer, sometimes synthesising the texture once is not going to generate acceptable results therefore, we iterate over the synthesized image over and over again (N times), reducing the patch size on each iteration. We select N between 3 and 5 times and calculate alpha parameter in each iteration separately.
alpha is the mixing coefficient we use to mix spatial SSD and intensity SSD to choose the best match for texture transferring. In iterative approach we can calculate this coefficient using this equation:
alpha = 0.8 * ((itr-1)/(N-1)) + 0.1;

We also select overlap parameter 1/6 of the patchsize, reducing patchsize by the scale of 3 in each iteration.
Here are some results for iterative approach: (N = 3, initial patchsize = 21, tol = 0.2)

Texture & Source
Iterations (from left to right: 1st, 2nd and 3rd)

References:

  1. Efros, Alexei A., and William T. Freeman. "Image quilting for texture synthesis and transfer." Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 2001.
  2. Cgtextures.com,. '[CG Textures] - Textures For 3D, Graphic Design And Photoshop!'. N.p., 2015. Web. 17 Feb. 2015.