Panoramic Vision

0 downloads 0 Views 324KB Size Report
14.4 Experimental Results. 279. 14.4.1 Experiments with Synthetic Data. 279. 14.4.2 Experiments with Real ... S. Peleg, B. Rousso, A. Rav-Acha, and A. Zomet.

Ryad Benosman Sing Bing Kang Editors

Panoramic Vision Sensors, Theory, and Applications Foreword by Olivier Faugeras With 267 Illustrations

Springer

Contents

Foreword Preface Contributors 1 Introduction R. Benosman and S.B. Kang 1.1 Omnidirectional Vision in Nature 1.2 Man-Made Panoramic Vision 1.3 Organization of Book 1.4 Acknowledgment 2 A Brief Historical Perspective on Panorama R. Benosman and S.B. Kang 2.1 Panorama in the Beginning 2.2 Prom Panorama Exhibits to Photography 2.3 Panorama in Europe and the United States 2.3.1 Panorama in Britain 2.3.2 Panorama in France 2.3.3 Panorama in Germany 2.3.4 Panorama in the United States 2.4 From Panoramic Art to Panoramic Technology 2.4.1 Panoramic Cameras 2.4.2 Omnidirectional Vision Sensors 2.5 The Use of Mirrors in Paintings 2.5.1 The Evolution of Mirrors 2.5.2 Mirrors in Paintings 2.5.3 Anamorphosis 2.6 Concluding Remarks 2.7 Additional Online Resources 2.8 Acknowledgment

v vii xix 1 1 3 3 4 5 5 6 9 9 9 11 12 13 14 14 15 15 16 18 18 19 19

x

Contents

Section I: Catadioptric Panoramic Systems

21

3 Development of Low-Cost Compact Omnidirectional Vision Sensors

23

H. Ishiguro 3.1 Introduction 3.2 Previous Work 3.2.1 Omnidirectional Vision Sensors 3.2.2 Omnidirectional Images 3.3 Designs of ODVSs 3.3.1 Designs of Mirrors 3.3.2 Design of a Supporting Apparatus 3.4 Trial Production of C-ODVSs 3.4.1 Eliminating Internal Reflections 3.4.2 Making Mirrors from Metal 3.4.3 Focusing in an ODVS 3.4.4 Developed C-ODVSs 3.5 Applications of ODVSs 3.5.1 Multimedia Applications 3.5.2 Monitoring Applications 3.5.3 Mobile Robot Navigation 3.6 Conclusion 4

Single Viewpoint Catadioptric Cameras S. Baker and S.K. Nayar 4.1 Introduction 4.2 The Fixed Viewpoint Constraint 4.2.1 Derivation of the Fixed Viewpoint Constraint Equation 4.2.2 General Solution of the Constraint Equation . . . 4.2.3 Specific Solutions of the Constraint Equation . . 4.2.4 The Orthographic Case: Paraboloidal Mirrors . . 4.3 Resolution of a Catadioptric Camera 4.3.1 The Orthographic Case 4.4 Defocus Blur of a Catadioptric Camera 4.4.1 Analysis of Defocus Blur 4.4.2 Defocus Blur in the Orthographic Case . . . . . . 4.4.3 Numerical Results 4.5 Case Study: Parabolic Omnidirectional Cameras 4.5.1 Selection of the Field of View 4.5.2 Implementations of Parabolic Systems 4.6 Conclusion

23 24 24 25 26 26 29 30 31 31 32 34 34 34 36 37 38 39 39 41 41 44 45 53 54 57 59 59 62 63 65 67 .67 70

Contents 5 Epipolar Geometry of Central Panoramic Catadioptric Cameras T. Pajdla, T. Svoboda, and V. Hlavdc 5.1 Introduction 5.2 Terminology and Notation 5.3 Overview of Existing Panoramic Cameras 5.3.1 Stereo and Depth from Panoramic Images . . . . 5.3.2 Classification of Existing Cameras and Comparison of Their Principles 5.4 Central Panoramic Catadioptric Camera 5.5 Camera Model 5.5.1 Hyperbolic Mirror 5.5.2 Parabolic Mirror 5.6 Examples of Real Central Panoramic Catadioptric Cameras 5.7 Epipolar Geometry 5.7.1 Hyperbolic Mirror 5.7.2 Parabolic Mirror 5.8 Estimation of Epipolar Geometry 5.9 Normalization for Estimation of Epipolar Geometry . . . 5.9.1 Normalization for Conventional Cameras 5.9.2 Normalization for Omnidirectional Cameras . . . 5.10 Summary

xi 73 73 74 75 77 77 79 81 82 85 87 88 92 95 97 98 98 99 102

6 Folded Catadioptric Cameras S.K. Nayar and V. Peri 6.1 Introduction 6.2 Background: Single Mirror Systems 6.3 Geometry of Folded Systems 6.3.1 The General Problem of Folding 6.3.2 The Simpler World of Conies 6.3.3 Equivalent Single Mirror Systems 6.4 Optics of Folded Systems 6.4.1 Pertinent optical effects 6.4.2 Design Parameters 6.4.3 System Optimization 6.5 An Example Implementation

103

Section II: Panoramic Stereo Vision Systems

121

7 A Real-time Panoramic Stereo Imaging System and Its Applications

123

A. Basu and J. Baldwin 7.1 Introduction 7.2 Previous Applications

103 104 105 105 106 108 112 113 114 115 115

123 125

xii

Contents 7.3

Stereo Design 7.3.1 Vertical Extent of Stereo Field of View 7.3.2 Effective Eye Separation 7.3.3 Orientation of Eye Separation 7.4 Device Calibration 7.4.1 Analog Approach 7.4.2 Digital Approach 7.5 Hardware Design and Implementation 7.6 Results Produced by System 7.7 The Mathematics of Panoramic Stereo 7.8 Experimental Results 7.9 Further Improvements 7.10 Acknowledgment

126 127 127 128 129 130 131 133 134 136 139 141 141

8 Panoramic Imaging with Horizontal Stereo S. Peleg, M. Ben-Ezra, and Y. Pritch 8.1 Introduction 8.1.1 Panoramic Images 8.1.2 Visual Stereo 8.1.3 Caustic Curves 8.2 Multiple Viewpoint Projections 8.3 Stereo Panoramas with Rotating Cameras 8.3.1 Stereo Mosaicing with a Slit Camera 8.3.2 Stereo Mosaicing with a Video Camera 8.4 Stereo Panoramas with a Spiral Mirror 8.5 Stereo Panoramas with a Spiral Lens 8.6 Stereo Pairs from Stereo Panoramas 8.7 Panoramic Stereo Movies 8.8 Left-right Panorama Alignment (Vergence) 8.9 Concluding Remarks 8.10 Acknowledgment

143

9 Panoramic Stereovision Sensor R. Benosman and J. Devars 9.1 Rotating a Linear CCD 9.2 System Function 9.3 Toward a Real-time Sensor? 9.4 Acknowledgment

161

,

10 Calibration of the Stereovision Panoramic Sensor R. Benosman and J. Devars 10.1 Introduction 10.2 Linear Camera Calibration using Rigid Transformation . 10.2.1 The Pinhole Model 10.2.2 Applying the Rigid Transformation

143 143 144 145 145 145 147 148 148 151 157 158 159 160 160

161 164 166 167 169 169 169 169 171

Contents

10.3

10.4 10.5 10.6 10.7

10.2.3 Computing the Calibration Parameters 10.2.4 Reconstruction 10.2.5 Experimental Results Calibrating the Panoramic Sensor using Projective Normalized Vectors 10.3.1 Mathematical Preliminaries 10.3.2 Camera Calibration Handling Lens Distortions Results Conclusion Acknowledgment

xiii 171 172 172 173 173 175 177 178 180 180

11 Matching Linear Stereoscopic Images R. Benosman and J. Devars 11.1 Introduction 11.2 Geometrical Properties of the Panoramic Sensor 11.3 Positioning the Problem 11.4 A Few Notions on Dynamic Programing 11.4.1 Principle 11.4.2 The Family of Dynamic Programming Used . . . 11.5 Matching Linear Lines 11.5.1 Principle 11.5.2 Cost Function 11.5.3 Optimal Path Retrieval and Results 11.5.4 Matching Constraints 11.6 Region Matching 11.6.1 Introduction 11.6.2 Principle of Method 11.6.3 Computing Similarity between Two Intervals . . . 11.6.4 Matching Modulus 11.6.5 Matching Algorithm 11.6.6 Adding Constraints 11.6.7 Experimental Results

181

Section III: Techniques for Generating Panoramic Images

201

12 Characterization of Errors in Compositing Cylindrical Panoramic Images

205

S.B. Kang and R. Weiss 12.1 Introduction 12.1.1 Analyzing the Error in Compositing Length . . . 12.1.2 Camera Calibration 12.1.3 Motivation and Outline 12.2 Generating a Panoramic Image

181 181 183 184 184 184 185 185 186 186 187 193 193 193 194 195 196 196 198

205 206 206 207 208

xiv

Contents 12.3 Compositing Errors due to Misestimation of Focal Length 12.3.1 Derivation 12.3.2 Image Compositing Approach to Camera Calibration 12.4 Compositing Errors due to Misestimation of Radial Distortion Coefficient 12.5 Effect of Error in Focal Length and Radial Distortion Coefficient on 3D Data 12.6 An Example using Images of a Real Scene 12.7 Summary

13 Construction of Panoramic Image Mosaics with Global and Local Alignment H.-Y. Shum and R. Szeliski 13.1 Introduction 13.2 Cylindrical and Spherical Panoramas 13.3 Alignment Framework and Motion Models 13.3.1 8-parameter Perspective Transformations 13.3.2 3D Rotations and Zooms 13.3.3 Other Motion Models 13.4 Patch-based Alignment Algorithm 13.4.1 Patch-based Alignment 13.4.2 Correlation-style Search 13.5 Estimating the Focal Length 13.5.1 Closing the Gap in a Panorama 13.6 Global Alignment (Block Adjustment) 13.6.1 Establishing the Point Correspondences 13.6.2 Optimally Criteria 13.6.3 Solution Technique 13.6.4 Optimizing in Screen Coordinates 13.7 Deghosting (Local Alignment) 13.8 Experiments 13.8.1 Global Alignment 13.8.2 Local Alignment 13.8.3 Additional Examples 13.9 Environment Map Construction 13.10 Discussion 13.11 Appendix: Linearly-constrained Least-squares 13.11.1 Lagrange Multipliers 13.11.2 Elimination Method 13.11.3 QR Factorization

209 210 215 218 222 223 226

227 227 230 233 234 237 239 240 240 241 242 243 244 245 245 248 250 250 252 253 255 258 260 263 265 266 266 267

Contents 14 Self-Calibration of Zooming Cameras from a Single Viewpoint L. de Agapito, E. Hayman, I.D. Reid, and R.I. Hartley 14.1 Introduction 14.2 The Rotating Camera 14.2.1 Camera Model 14.2.2 The Inter-image Homography 14.2.3 The Infinite Homography Constraint 14.3 Self-calibration of Rotating Cameras 14.3.1 Problem Formulation 14.3.2 Constant Intrinsic Parameters 14.3.3 Varying Intrinsic Parameters 14.4 Experimental Results 14.4.1 Experiments with Synthetic Data 14.4.2 Experiments with Real Data 14.5 Optimal Estimation: Bundle-adjustment 14.5.1 Maximum Likelihood Estimation (MLE) 14.5.2 Using Priors on the Estimated Parameters: Maximum a Posteriori Estimation (MAP) . . . . 14.5.3 Experimental Results 14.6 Discussion

xv 269 269 270 270 271 272 275 275 275 276 279 279 281 282 283 284 285 286

15 360 x 360 Mosaics: Regular and Stereoscopic S.K. Nayar and A.D. Karmarkar 15.1 Spherical Mosaics 15.2 360° Strips 15.3 360° Slices 15.4 Slice Cameras 15.5 Experimental Results 15.6 Variants of the Slice Camera . . . 15.7 Summary

291

16 Mosaicing with Strips on Adaptive Manifolds S. Peleg, B. Rousso, A. Rav-Acha, and A. Zomet 16.1 Introduction 16.2 Mosaicing with Strips 16.3 Cutting and Pasting of Strips 16.3.1 Selecting Strips 16.3.2 Pasting Strips 16.4 Examples of Mosaicing Implementations 16.4.1 Strip Cut and Paste 16.4.2 Color Merging in Seams 16.4.3 Mosaicing with Straight Strips 16.4.4 Mosaicing with Curved Strips: Forward Motion . 16.5 Rectified Mosaicing: A Tilted Camera

309

291 292 295 296 297 298 299

309 313 314 314 315 317 317 318 318 319 320

xvi

Contents 16.5.1 Asymmetrical Strips 16.5.2 Symmetrical Strips 16.6 View Interpolation for Motion Parallax 16.7 Concluding Remarks

321 322 323 325

Section IV: Applications

327

17 3D Environment Modeling from Multiple Cylindrical Panoramic Images

329

S.B. 17.1 17.2 17.3 17.4 17.5

Kang and R. Szeliski Introduction Relevant Work Overview of Approach Extraction of Panoramic Images Recovery of Epipolar Geometry 17.5.1 8-point Algorithm: Basics 17.5.2 Tracking Features for 8-point Algorithm 17.6 Omnidirectional Multibaseline Stereo 17.6.1 Reconstruction Method 1: Unconstrained Feature Tracking and 3D Data Merging 17.6.2 Reconstruction Method 2: Iterative Panoramic Structure from Motion 17.6.3 Reconstruction method 3: Constrained Depth Recovery using Epipolar Geometry 17.7 Stereo Data Segmentation and Modeling 17.8 Experimental Results 17.8.1 Synthetic Scene 17.8.2 Real Scenes 17.9 Discussion and Conclusions 17.10 Appendix: Optimal Point Intersection 17.11 Appendix: Elemental Transform Derivatives 18 N-Ocular Stereo for Real-Time Human Tracking T. Sogo, H. Ishiguro, and M.M. Trivedi 18.1 Introduction 18.2 Multiple Camera Stereo 18.2.1 The Correspondence Problems and Trinocular Stereo 18.2.2 Problems of Previous Methods 18.3 Localization of Targets by N-ocular Stereo 18.3.1 Basic Algorithm 18.3.2 Localization of Targets and Error Handling . . . 18.3.3 False Matchings in N-ocular Stereo 18.4 Implementing N-ocular Stereo 18.4.1 Simplified N-ocular Stereo

329 330 331 332 333 334 336 337 338 339 341 343 343 343 345 347 349 350 359 359 361 361 362 363 363 364 365 366 366

Contents

xvii

18.4.2 Error Handling in the Simplified N-ocular Stereo 18.5 Experimentation 18.5.1 Hardware Configuration 18.5.2 Detecting Azimuth Angles of Targets 18.5.3 Precision of N-ocular Stereo 18.5.4 Tracking People 18.5.5 Application of the System 18.6 Conclusion

367 369 369 369 370 372 373 374

19 Identifying and Localizing Robots with Omnidirectional Vision Sensors H. Ishiguro, K. Kato, and M. Barth 19.1 Introduction 19.2 Omnidirectional Vision Sensor 19.3 Identification and Localization Algorithm 19.3.1 Methodology 19.3.2 Triangle Constraint 19.3.3 Triangle Verification 19.3.4 Error Handling 19.3.5 Computational Cost 19.4 Experimental Results 19.4.1 Simulation Experiments 19.4.2 Real-world Experiment 19.5 Conclusions

377 377 378 378 379 380 381 382 384 384 384 387 390

20 Video Representation and Manipulations Using Mosaics 393 P. Anandan and M. Irani 20.1 Introduction 20.2 From Frames to Scenes 20.2.1 The Extended Spatial Information: The Panoramic Mosaic Image 20.2.2 The Geometric Information 20.2.3 The Dynamic Information 20.3 Uses of the Scene-based Representation 20.3.1 Visual Summaries: A Visual Table of Content . . 20.3.2 Mosaic-based Video Indexing and Annotation . . 20.3.3 Mosaic-based Video Enhancement 20.3.4 Mosaic-based Video Compression 20.4 Building the Scene-based Representation 20.4.1 Estimating the Geometric Transformations . . . . 20.4.2 Sequence Alignment and Integration 20.4.3 Moving Object Detection and Tracking 20.5 Conclusion

Bibliography

393 395 396 397 398 399 399 400 408 411 416 416 420 423 424

425