TP4
Image Stitching
by Saeed Sojasi
Goal of Assignment
The goal of this assignment is to implement an application of image mosaicing. Image mosaicing or image stitching is the process of combining multiple images
with overlapping fields of view to produce a segmented panorama or high-resolution image. Image mosaicing not only allow you to create a large field of view using normal camera,
the result image can also be used for texture mapping of a 3D environment such that users can view the surrounding scene with real images.
Most approaches to image stitching require overlaps between images.
For doing this assignment we need to learn how to compute homographies, and how to use them to warp images.
We should to compute the homography that maps each image into the base image and then stitches all the mapped images together to form a whole panoramic scene.
we will define some corresponding points between the images and then we will find a warp homography matrix and register the two images together on a mosaic image, and finally we will blend them together.
This homework consists of two parts: 1) At first, we should use the manually selected correspondence points to compute homography 'H'.
2) Afterwards, we will have to implement a way to automatically detect and match interesting features in images.
1. Manual Matching
The image stitching algorithm can be summarized as: 1) Features matching. 2) Compute homography. 3)Projective warping.
4)Blending the images into a mosaic. (see the code 'manual_matching.m')
We need pairs of correspondences for calculation of homography. (see the code 'select_points.m').
The calculation of a homography requires at least 4 pairs of correspondences.
For this purpose I select manually these interest points in both images.
We should select very carefully these correspondences, because an error of a couple of pixels can produce huge changes in the recovered homography.
Homography (H) is a 3x3 matrix with 8 degrees of freedom. I calculated the homgraphy that will project each image onto the base image
(see the code 'compute_homography.m').
Aftere calculation homography, in the third step,
we need to warp our images using this homography (see the code 'warp_image.m').
The input image is frist mapped into the base image (by 'H') then we use the invese of 'H' to find the color of each mapped pixels.
Finally, we blend the warped image (see the code 'blend.m').
1.1 Results
The results of this part are shown below:
Panorama1
Manually Cropped
Panorama2
Manually Cropped
Panorama3
Manually Cropped
Discussion
My implementation has a problem, It dose not have good results for large images. I could not get results for more than 5 images.
For more than five images, my computer return error. It returns out of memory.
2. Automatic matching
Automatic features matching
o Detecting corner features in an image
o Extracting a feature descriptor for each feature point
o Matching these feature descriptors between two images
o Use a robust method (RANSAC) to compute a homography
The automatic matching (see the code 'automatic_mosaic.m') is the same as manual matching except that all the correspondences are identified automatically.
Find interest points
First of all we need to detect corners features.
A corner feature in an image is a location where the horizontal and vertical gradients are both very strong.
I used harris detector for detect corner features. Harris detector (or harris corner detector) is an classic algorithm for corner detection.
there is a large number of Harris corners and must be reduced down. For this purpose we use an adaptive non-maximal suppression
(see the code 'adaptive_non_maximal_suppression.m').
Adaptive Non-maximal suppression
There are a lot of interest points.
It is important that interest points are spatially well distribute over the image and for image stitching applications
the area of overlap between a pair of images may be small. We use adaptive non-maximal suppression (ANMS) strategy
to select a fixed number of interest points from each image. We select maximum in neighborhood of radius r.
r is changeable. First we set r = 0 and then we increase this amount until obtain desired number of interest points.
For each Harris point, the minimum suppression radius is the minimum distance from that point to a different point with a higher corner strength
For this homework I selected 500 desired interest points with highest radius.
Results
The results of this part are shown below:
We see the base image (left) and input image (right). The red points are found by harris corner detection and the green circle around the red points
are found by Adaptive Non-maximal suppression.
Feature descriptor
After the determined interest points place, we need to extract a description of the local image such local feature vectors.
(see the code 'descriptor.m').
These vectors have been developed including local intensity patches, Gaussian derivatives, shift invariant feature transforms, and affine-invariant
descriptors. We sample a 8 × 8 patch of pixels, which is taken from a 40x40 pixel window centered on the point sampled by every 5 pixels.
These sampling are performed at a higher pyramid level for avoid aliasing.
We should normalize the descriptor vector after sampling. For this purpose we put mean is 0 and standard deviation is 1.
Feature matching
We will need to find pairs of features that look similar and are thus likely to be good matches (see the code 'find_match.m').
The similarity between two points are compared in terms of sum of squared error based on the feature descriptor.
The points with the least error are considered for matching.
Also, we need to calculate threshold for compare the ratio between the lowest and second lowest errors.
Point pairs that are less than threshold are consider for matching.