Abstract

Image stitching defines as the process of combining some pictures, which have overlapping region, to produce a single wide (panorama) picture. In this assignment, I attempted to implement two algorithms for image stitching (called manual and automatic). Generally, both algorithms try to wrap images into a base image by finding some correspondence points between them, and then blend them to produce one image.

Manual image stitching

I try to explain the manual algorithm here for two input images. It can be extended simply in order to get more than 2 images

  • 1) two input images which have some common features (one of them as base image, and other as input image )
  • 2) select at least 5 correspondence points from the two images
  • 3) now, we are able to compute a homography matrix in order to wrap the input image into the base image.
  • 4) After image alignment (by wrapping we try to align the two images in respect to their overlap region), blend images. I did not average the two images to produce a panorama image, but used max function to blend the two images, since when I average the two images, my result has edge artifact.

    • I should notice that if the correspondence points are selected carelessly, the resultant image will have many ghosty things. Plus, if the input images have moving parts, like moving human, car, and so on, the moving objects dont appear clearly in the resultant image.

    How do I compute the Homography matrix

    Let p1 =(x1, y1) from image 1 and p2 =(x2, y2) from image 2, where p1 and p2 are correspondence points. Point p2 can be transformed to p1 by a homography matrix. This matrix is 3*3:

    Make p1 a homogonous vector p1= (x1 ,y1, 1). By mutilplying H, and p1, we obtain p2 =(X2, Y2 , Z2), which x2 = X2/Z2, and y2 = Y2/Z2. We dont know H, but know the correspondence points. Since we have 8 unknowns, we have to have at least 8 points from the two images, 4 from image 1, and 4 from image 2. For solving the linear equation, I do as follow:

    Now, we are looking for H that it can compute simply by mldivide. So H = A\b

    Results







    Disscution

    I could not use all pictures that are provided for each panorama due to the out of memory error that I got from Matlab. Instead I used a few of them to generate a signle panorama picture from the university's environment. It is reasonable that I can not take all the input images, which their count is 10 or more, since I used rectilinear projection. In other words, when th angles of view become greater than 120 degrees, this projection greatly exaggerates perspective, and then the resultant image becoming large and larger[cambridgeincolour]. In the above panorama images, you can see that some objects appearing skewed at some edges. One solution for this problem is using cylindrical projection.

    Automatic image stitching

    Manually finding the correspondence points from several related images is not desirable, when we can find the correspondence points automatically. In this section, I focus on the algorithm that finds the correspondence points automatically.

  • 1)Harris corner detection helps me to find many interesting points in the input image and the base image.
  • 2)Adaptive Non-Maximal Suppression:

    • As the paper indicated, we should select some points among the found interesting points where their distribution is more uniform. The author suggested a novel way to restrict the interesting points (called Adaptive non-Maximal suppression). In a nutshell, some interest points are allowed to remain in the image that their corner strength is locally maximum. In other words, consider a circle with radius r, then for each interest points, find its neighbors inside of the circle, then if its corner stretch is larger than its neighbors, then this interest point remain. In the below images, I show the result of Adaptive Non-Maximal Supression on Vachon Pavillon image.

    As you can see in the right image, Harris corner detection locates a lot of interesting points, which is indicated by red points. But we should suppress many of the points, and let to remain the points that have locally large corner strength and evenly distributed in the image, the yellow diamonds are the selected interesting points by Non-Maximal suppression.

  • 3) I have some interest points (for example 500 points) from the input image, and 500 interest points form the base image. Now I should find match points between the two sets of points. But before, a feature descriptor for each point should be extracted. As the paper indicated, I used a 8*8 patch for each interest point. Then the feature vectors are normalized to mean zero and standard deviation 1, in order to become invariant to affine changes in intensity. For finding matching among points from the two images, we used min-squared error. In other words, two points match if their descriptors are more similar.
  • 4) Computing Homography matrix from these points done by RANSAC. It is a simple but powerful approach for curve fitting. If there exist a lot of points for curve fitting, we can ignore some points that are outlier (noisy points) from the process of curve parameter estimation. We are suggested to use this approach for estimating the Homography matrix by removing the outlier points. In the following, you can see the result of RANSAC algorithm on two images. The interesting points that used for computing Homography matrix, are indicated by red circle, while the blue stars in the images are matched points. As you can see, just one pair (blue star) don’t participate in computing Homography matrix.
    • base image input image
  • 5) by the homography matrix, we are able to align the input image to the base image. Then, blend these images.

    • Results

    5 images
    3 images
    3images

    More Results from automatic stitching

    manually cropped



    manually cropped



    manually cropped

    Disscution

    As you can see in the above image, automatic stitching can not correctly align the images to produce a final panorama image beacuse just one correspondence point is not correctly chosen from the images. The one pair of correspondence points is shown in the following images by yellow circle around them. While manually selecting the correspondence points works very well for these images. Therefore, whether automatically or manually image stitching, the quality of the resultant image highly dependent on the match (correspondence) points.

    Manually selecting correspondence points Manually cropped

    More Results

    I appear mutilple times next to HaftSin table. Happy Iranian new year.





    day and night in the same picture!

    cylindrical projection

    As I mentioned the rectilinear projection not only skewed the edges that are near the angle of view, but also it reduce the quality (resolution) of the panorama image. While by the cylindrical projection, we are able to stitch images when the angel of view is more than 120 degree, and try to preserve the quality of image.

    For the cylindrical projection, I need the focal length, which I extracted it from jhead application. I set the focal length 1278.56 pixel. Also, distortion coefficients (k1 , and k2) for the images are needed to be compute, but I was not able to compute it. So in my experimental result, I changed k1 and k2 utile get desirable results. The next step is projecting the input images on a cylinder. The cylindrical projection are computed as follows:

    For finding the correspondence points, and computing homography matrix, I used the previous algorithms (Non-Maximal suppression and RANSAC). Finally, blend the images. I tested the cylidrical projection on three sets of images. Here, I had a big problem with finding the distortion coefficients, and in the final result (my own images) this problem appears since I did not know the values of k1 and k2 exactly, but approximated them by reapting experiment.

    6 images
    7 images
    3 images