Feb 26, 2012 - Analytical Results. Results1 - Outdoor Scene 1 at a temple in Hampi, India. Low Resolution and corresponding high resolution point cloud.
High Resolution Point Cloud Generation from Kinect and HD Cameras using Graph Cut SUVAM PATRA Indian Institute of Technology Delhi
February 26, 2012
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Authors: Suvam Patra Brojeshwar Bhowmick Subhashis Banerjee Prem Kalra
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Agenda
Problem Definition What are the alternatives? Graph Cut as an optimization tool Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior 4 Results Results Comparing our results with traditional Epipolar Geometry based method using NCC Verification of correctness of Geometry Analytical Results 5 References SUVAM PATRA High Resolution Point Cloud Generation from Kinect and HD Cam References 1 2 3
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Problem Definition Problem Statement To obtain a High resolution point cloud from Kinect and multiple HD cameras The point cloud captured by Kinect is of a very low resolution corresponding to 640x480 images. We can use HD cameras which gives RGB images at a much higher resolution (i.e 1440x1080) Use stereo along with graph cut to generate a much a higher resolution point cloud.
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
What are the alternatives?
Normalized Cross Correlation based approach Epipolar transfer of points between HD images Rectify the images to have row wise search for the correspondences Find out the correspondence points in the two HD rectified images using NCC Use the calibration to get the triangulated 3D points to fill the holes
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior
Why use Graph Cut? Disadvantages of NCC based Approach No neighborhood interactions considered. Scale of the estimated 3D point is approximated Error in least squares based triangulation Advantages of using graph cut The graph based optimization technique takes neighborhood interaction into consideration Can adjust weights on each edge according to the order of importance of the most important to least important criterion i.e. consistency, distance metric, etc. SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior
Representation of the scene using a Graph
1
2
3
Each point of the point cloud represents a node in the two-terminal graph. Assign a data cost to each node which is the weight of the edge between each node and the terminals. Assign a smoothness cost to each node which is the weight of the edges between that node and its neighbors. SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior
Assigning the Data Term A Photo consistency Metric A Distance Metric Why choose these particular metrics ? Photo consistency assures the correctness of the geometry in the high resolution point cloud. Using photo consistency alone gives a thick block where though geometry is correct but thickness increases. A Distance metric in the data cost of each node assures that the point cloud thickness can be controlled by the nearby Kinect points. SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior
Comparison between resolution enhancement without and with distance measure
Front View of the Resolution Enhanced Image
Top view of the high resolution image without and with consideration of the distance metric SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior
The Photo consistency Metric Used Adaptive photo consistency measure proposed by Slabaugh et. al. Calculated the per image and inter-image standard deviation. Designed the photo consistency cost based on the equation:
S(i) < τ1 + τ2 ∗ ¯s (i) where, S(i): the standard deviation of the intensity values in the projection neighbourhoods calculated over all N HD images. ¯s (i):the average of the standard deviation for each image projection. τ1 , τ2 :global and local thresholds based on the scene. SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior
The Photo consistency Metric Accordingly the assigned Photo consistency cost is given by: Dphoto (i) = photocost ∗ exp(−
S(i) ) τ1 + τ2 ∗ ¯s (i)
(1)
with the SOURCE and Dphoto (i) = photocost ∗ (1 − exp(−
S(i) )) τ1 + τ2 ∗ ¯s (i)
(2)
with the SINK where, S(i) and τ1 + τ2 ∗ ¯s (i) is the standard deviation and the adaptive threshold respectively for the i th voxel and photocost is a scale factor. SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior
The Distance metric
The distance metric exercises a control on the thickness of the high resolution point cloud generated. Assign a cost on the basis of distance from a local surface constructed at this point.
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior
The Distance Metric Take a KxKxK neighborhood around each voxel . Calculate an approximate surface around each voxel in two steps: 1 First project all the points onto an approximate plane. 2 In the second step use the projection in step 1 to project the points to an order ’n’ surface. Accordingly assign a data cost as:
Ddist (i) =
P(ri ) − ri dist threshold
where P(ri ) : The projected point of the actual point ri on the local surface and distt hreshold is determined experimentally. SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior
MLS Surface Creation
First, a local reference domain H for the purple point r is generated. Then, a local polynomial approximation g to the heights fi of points pi over H is computed based on the method proposed by Alexa et.al. SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Why Use Graph Cut? Representation of the scene using a Graph Assigning the Data Term The Photo consistency Metric The Distance metric Total Data cost and smoothness prior
Total Data cost and smoothness prior Total data cost is thus: Dp (i) = Ddist (i) ∗ Dphoto (i) Assigning smoothness prior to the graph: TABLE:Assignment of Vp,q based on Potts Model Vp,q (fp, fq) 0 100
Condition fp = fq(Both are Kinect voxels) Otherwise
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Results Comparing our results with traditional Epipolar Geometry based me Verification of correctness of Geometry Analytical Results
Results1 - Outdoor Scene 1 at a temple in Hampi, India
Low Resolution and corresponding high resolution point cloud generated of a sculpture at Hampi SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Results Comparing our results with traditional Epipolar Geometry based me Verification of correctness of Geometry Analytical Results
Results2 - Outdoor Scene 2 at a temple in Hampi, India
Low resolution and corresponding high resolution point cloud generated of another sculpture at Hampi
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Results Comparing our results with traditional Epipolar Geometry based me Verification of correctness of Geometry Analytical Results
Results3 - Indoor Scene
Low resolution and corresponding high resolution point cloud generated of a typical indoor scene.
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Results Comparing our results with traditional Epipolar Geometry based me Verification of correctness of Geometry Analytical Results
Comparing our results with traditional Epipolar Geometry based method using NCC
High resolution point cloud of a typical indoor scene genertaed by NCC and our method respectively.
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Results Comparing our results with traditional Epipolar Geometry based me Verification of correctness of Geometry Analytical Results
Verification of correctness of Geometry
Low resolution and its corresponding high resolution depth map of a dummy dog and their overlapping respectively showing that the geometry have not been distorted after resolution enhancement.
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Results Comparing our results with traditional Epipolar Geometry based me Verification of correctness of Geometry Analytical Results
Analytical Results1- Comparison between original and projected image
One of the original images and its corresponding projected image compared using the difference image showing the geometry has been preserved.
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
Results Comparing our results with traditional Epipolar Geometry based me Verification of correctness of Geometry Analytical Results
Analytical Results2 - Comparison between our result and that obtained by interpolation
The marked region shows loss of depth information in the interpolated surface as compared to our result. SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
References
References Boykov, Y. and Kolmogorov, V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in vision. In IEEE Transactions on PAMI 2004. Diebel, J. and Thrun, S. An application of markov random fields to range sensing. in advances in neural information processing. In Advances in Neural Information Processing Systems Jean-Michel Morel, G. Y. Asift: A new framework for fully affine invariant image comparison. In SIAM Journal on Imaging Sciences. Volume 2 Issue 2 2009. S. Schuon, C. Theobalt, J. D. and Thrun, S. Highquality scanning using time-of-flight depth superresolution.In IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
References
References
S. Schuon, C. Theobalt, J. D. and Thrun, S. Lidarboost depth superresolution for tof 3d shape scanning.In CVPR 2009. Slabaugh, G. and Schafer, R. Methods for volumetric reconstruction of visual scenes. In IJCV 2003. M Alexa, J. B. et al. Computing and rendering point set surfaces. In IEEE Transactions on Visualization and Computer Graphics.
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
Problem Definition What are the alternatives? Graph Cut as an optimization tool Results References
References
THANK YOU
SUVAM PATRA
High Resolution Point Cloud Generation from Kinect and HD Cam
