As-Projective-As-Possible Image Stitching with Moving DLT

Julio Zaragoza*, Tat-Jun Chin*, Michael Brown and David Suter

*Corresponding authors

railtracks
Figure 1: Result of APAP warp for image stitching.
Quick links: Papers - Source codes - Datasets

Abstract

We investigate projective estimation under model inadequacies, i.e., when the underpinning assumptions of the projective model are not fully satisfied by the data. We focus on the task of image stitching which is customarily solved by estimating a projective warp - a model that is justified when the scene is planar or when the views differ purely by rotation. Such conditions are easily violated in practice, and this yields stitching results with ghosting artefacts that necessitate the usage of deghosting algorithms. To this end we propose as-projective-as-possible (APAP) warps, i.e., warps that aim to be globally projective, yet allow local non-projective deviations to account for violations to the assumed imaging conditions.

railtracks
Figure 2: 1D analogy of APAP warp to fit a set of 1D correspondences (x,x'). (a) A normal projective warp cannot capture the local variations due to violations to the imaging assumptions (pure rotational camera motion). (b) An APAP warp is globally projective to respect the effects of a pin-hole projection, but can vary locally to account for deviations to the imaging assumptions.
Based on a novel estimation technique called Moving Direct Linear Transformation (Moving DLT), our method seamlessly bridges image regions that are inconsistent with the projective model; see Figure 1. The result is highly accurate image stitching, with significantly reduced ghosting effects, thus lowering the dependency on post hoc deghosting.

Papers

Source codes

*Kindly cite one of the above papes if using our code in your work.

Results

Pairwise stitching - see papers and supp materials above for more results

Baseline method with single homography.
Smoothly varying affine (SVA) method of Lin et al., CVPR 2011.
Content Preserving Warp (CPW) method of Liu et al., SIGGRAPH 2009.
Dual Homography Warp (DHW) method of Gao et al., CVPR 2011.
Results from Autostitch software by Matthew Brown.
Results from Microsoft Photosynth.
APAP stitching using MDLT estimation.

Panoramic stitching - see papers and supp materials above for more results

Results from Autostitch using single homography for each image and bundle adjustment for simultaneous refinement.
APAP stitching using Bundle Adjusted MDLT (BAMDLT).

Evaluation data

*Kindly cite one of the above papers if using Dataset 1 in your work.

Runtime and memory consumption statistics

Processing pipeline of our MDLT code (pairwise stitching without bundle adjustment):

  1. Read pair of images to stitch (image I and image I').
  2. Obtain SIFT keypoint matches between I and I'.
  3. Remove incorrect keypoint matches (outliers) by means on RANSAC.
  4. Obtain matrix A from the set of N correct keypoint matches ({x_i,x_i'} for i=1,…,N) after RANSAC.
  5. Generate mesh (of size C1 x C2) over image I.
  6. (perform Moving DLT) For each vertex x* on the mesh:
    1. Obtain matrix W* which contains the weights between x* and all x_i for i=1,…,N.
    2. Perform SVD on matrix repmat(W*,1,9).*A.
    3. Obtain homography H* from the least significant right singular vector of W*A.
  7. Composite the images:
    1. Obtain offset between I' and warped I.
    2. Make "blank" canvas (the canvas will contain image I' and the warped image I).
    3. Copy I' to the canvas in the position determined by the offset.
    4. For each pixel coordinate p in image I, obtain the nearest mesh vertex x*.
      1. Use the homography H* from the nearest x* to warp p in I to p' in I'.
      2. Obtain the average of the intensity of pixels I(p) and I'(p').
      3. Put averaged pixel in canvas (in coordinate p' + offset).
On step 6(b) of the pseudo-code, the size of matrix A is 2N x 9 and the size of W* is 2N x 1, the ".*" operator means element-wise multiplication and repmat is MATLAB's replicate and tile array function.

In the following sections we present the running time and the memory consumption statistics of our implementation of the previous pseudo-code.

Runtime statistics

The following plots present the size of the images (in pixels) and the number of positive keypoint matches (inliers after RANSAC) per image pair (the stitching results for each one of these image pairs are available above.

The runtime of most of the image stitching methods increases with the number of keypoint matches. However, even when the number of keypoint matches is high our method is able to stitch images in reasonable time (less than 7 seconds in our experiments) as shown in the next figure.

Note that the time shown in the previous plot (the green line) includes the As-Projective-As-Possible warp estimation time and the Compositing time (which includes image warping and image average blending).

As can be seen from the previous plot, the warp estimation time increases with the number of keypoint matches and the compositing time increases with the size of the images. However, our As-Projective-As-Possible image stitching method is able to maintain low running times even with large images (> 2000 x 1500 pixels) that contain thousands of keypoint matches (> 5000).

These running times were obtained on a PC with an Intel i7 950@3.07Ghz CPU and 12GB of RAM running windows 8 (64 bits).

Memory consumption

Our method requires of two main matrices for operating, namely the matrix A and the matrix W* (please refer to the paper for details). The size of these matrices depends on the number of positive keypoint matches (inliers). The following table shows the number N of positive keypoint matches per image pair and the size of the A and W* matrices. The size of the matrix A is 2N x 9 and the size of W* is 2N x 1.

The number of required Megabytes consists of the number of Megabytes required for the matrix A + number of megabytes for the matrix W* (assuming 8 bytes per entry in the matrices).

As can be seen, our method does not make use of a large amount of memory. These statistics assume that the weights for each one of the cells (W*) are obtained before performing SVD on repmat(W*,1,9).*A and then disregarded after use (i.e., disregarded after SVD on repmat(W*,1,9).*A).

EOF