Accurate Multiple View 3D Reconstruction Using Patch-Based Stereo for Large-Scale Scenes
Stereo Pair Selection
- Given a reference image, we need to find out neighboring images to update the depth map of reference image.
- The angle between principal view diretions of camera \(i\) and \(j\). (\(5^{\circ} < \theta_{ij} < 60^{\circ}\))
- The distance between optical centers.
- We compute multiple \(d_{ij}\) and their median \(\bar{d}\). Then, we discard images whose \(d_{ij}>2\bar{d}\) and \(d_{ij}<0.05\bar{d}\).
- At last, we choose the top \(10\) images whose \(\theta_{ij}d_{ij}\) are smallest.
Depth-Map Computation
- The core idea is that for each pixel in the input image, we try to find a good support plane that has minimal aggregated matching cost (NCC) with the reference image.
- How do we initialize a plane for a pixel?
- If neighboring images have already depth maps, their dpeth maps could be warped into the reference image and used to help initialize plane.
- Otherwise, we firstly select a random depth for a pixel. Then, we assign the normal of the plane randomly in camera’s spherical coordinate.
- Why do we initialize a plane for each pixel?
- Once the plane is determined, we can compute the homography for the plane. The homography will be used to project pixels from one image into another. By that, we can compute NCC in a sqaure window (\(7\times7\)).
- Plane refinement (3 iterations)
- At odd iterations, we start from the top-left pixel and travers in row wise order until we reach the bottom-right pixel.
- At even iterations, we reverse the order to visit the pixel from the bottom-right to the top left pixel, also in row wise order.
- At each iteration, each pixel has two operations, called spatial propagation and random assignment.
- Spatial propagation
- Purpose: Compare and propagate the planes of neighboring pixels to that of the current pixel.
- Assumption: Neighboring pixels are very likely to have similar 3D planes especially for high resolution images.
- In odd iterations,check left, upper, and upper-left neighbors.
- In even iterations, check right, lower, and lower-right neighbors.
- If neighbor’s plane parameters lead to a lower NCC, replace the current ones with them.
- Random assignment
- Purpose: Reduce the matching cost by testing several random plane parameters.
- We remove unreliable points in the depth-map whose aggregated matching costs are above a certain threshold (0.3).
Depth-Map Refinement
- Purpose: Enforce consistency over neighboring views.
- Project each pixel into neighbor images, then check if their depth values are close. We must find at least 2 consistent neighbors to retain the depth, otherwise discard it.
Depth-Map Merging
- Purpose: Remove redundant points and potential occluded points.
- If a viewray of pixel 1 is blocked by pixel 2’s ray from another image, pixel 1 is potentially occluded.
- If the viewrays’ end points (3D points) are close enough, they probably represent the same 3D point. Thus, we need to remove one of them from their depth map.
- All depth-maps are back projected into 3D, and merged into a single point cloud.
