Do ambiguous reconstructions always give ambiguous images

Do ambiguous reconstructions always give ambiguous images? M. Pollefeys and L. Van Gool ESAT-PSI, K.U.Leuven Leuven, Belgium

Abstract

motion the knowledge of one intrinsic camera parameter is sufficient to allow for successful self-calibration [14, 8].

In many cases self-calibration is not able to yield a unique solution for the 3D reconstruction of a scene. This is due to the occurrence of critical motion sequences. If this is the case, an ambiguity is left on the reconstruction. In this paper it is derived under which conditions correct novel views can be generated from ambiguous reconstructions. The problem is first approached from a theoretical point of view. It is proven that novel views are correct as long as the inclusion of the new view in the sequence yields the same ambiguity on the reconstruction. The problem is therefore much related to the problem of critical motion sequences since the virtual camera can be arbitrarily moved within the smallest critical motion set that contains the recovered camera motion without distortions becoming visible. Based on these result a practical measure for the expected ambiguity on a novel view based on the recovered structure and motion is derived. As an application a viewer was built that indicates if a specific novel view can be trusted or not by changing the background color.

However, in practice the motion of the camera is often restricted and there remains an ambiguity on the reconstruction. This is known as the problem of critical motion sequences (CMS). It was introduced by Sturm [15] and further studied in [9, 10, 17, 12]. Depending on the constraints available for self-calibration different classes of motions can be identified as critical. For each of these classes a specific ambiguity remains on the reconstruction. It depends on the application if some ambiguity is acceptable or not. There are two main classes of applications for 3D reconstructions from images. The first one consists of metrology applications and in most cases no ambiguity can be tolerated. The second type of applications consists of visualization. In this case the goal is to generate novel views based on original images. Over the last years this second type of applications has received more and more attention. Image-based modeling of 3D objects or scenes has become a major topic in both computer vision and computer graphics. Considering this application the important point is not the correctness of the reconstruction, but the correctness of the novel views that are generated from it. This problem was already partially addressed in [10], but only theoretically, for constant intrinsics and using a more restricted case by case analysis.

1. Introduction One of the important applications in computer vision is to retrieve the 3D structure of a scene from a collection of images. However, depending on the available knowledge and the images at hand, it is not always possible to obtain a unique solution for this problem. One well known ambiguity is when the observed features and the projection centers of the camera are all located on a special type of surface, called a critical surface [11]. Another well-known ambiguity is that when totally uncalibrated cameras are used, it is only possible to recover the structure of the scene up to an arbitrary projective transformation [1, 3]. It is possible to reduce this ambiguity by imposing constraints on the intrinsic parameters of the camera. This is in general understood as self-calibration. In recent years many different methods were proposed. Some are based on the assumptions that the intrinsics do not change during acquisition [2, 4, 13, 18]. Other method relax the constraint for constant intrinsics but require the knowledge of one or more intrinsic parameters [5, 7, 14]. It was proven that for sufficiently general

In this paper a general theorem is derived that allows to determine which motion a virtual camera can undergo to generate unambiguous novel views given the recovered (ambiguous) motion of the original camera. Further on, a practical algorithm is presented that allows to characterize the ambiguity on a novel view. This was used in a number of synthetic experiments to verify the validity of the theorem on some restricted motion sequences and to derive some more insight into this problem. This algorithm was also included into a 3D viewer that tells the user in how far he can trust a specific view based on the poses of the original cameras (and the applicable constraints on the intrinsics). This could for example be used to optimize fly-throughs of virtual worlds containing visual 3D reconstructions. 1

NM L

2. Background Some familiarity with the projective formulation of vision geometry is assumed [6]. A perspective camera is modeled through the projection equation where represents the equality up to a non-zero scale factor, represents a 3D world point, represents the corresponding 2D image point and is a projection matrix. In a metric or Euclidean frame can be factorized as follows


 
        +,- . - /1 23  & ( % '  0  (1)  "!$# -) where !* contains the intrinsic camera parameters, # is a rotation matrix representing the orientation and ) is a 3-vector representing the- position of the camera. The intrinsic camera measured in width parameters represent the focal length /6 57198 represents of pixels, 0 is the aspect ratio of pixels,. 4 the coordinates of the principal point and is a term accounting . for the skew. In general can be assumed zero. In practice, the principal point is often close to the center of the image and the aspect ratio 0 close to one.

The problem is, however, that for a specific set of selfcalibration constraints, not all motion sequences will yield a unique solution for the AC. In this case there is more than one potential absolute conic and the motion sequence is termed critical with respect to the set of constraints. The . In this set of potential absolute conic is defined as case an ambiguity will persist on the reconstruction.

QR4SL 8

3. Theoretical analysis The classification of all possible CMS for a specific set of self-calibration constraints can be used to avoid critical motions when acquiring an image sequence on which one intends to use self-calibration. In some cases, however, an uncalibrated image sequence is available from which a metric reconstruction of the recorded scene is expected. In this case, it is not always clear, at first, what can be achieved nor if the motion sequence is critical or not. It can be shown that the recovered set of cameras also has to satisfy the self-calibration constraints. This result is also valid for CMS, where the recovered motion sequence would be in the same CMS class as the original sequence. In [16] a proof was given for the case of constant intrinsic camera parameters. Here a simpler and more general proof based on the disc quadric representation is given. It is valid for all possible types of self-calibration constraints.

Projective geometry only encodes cross-ratios and incidence. The affine structure (parallelism and ratios of parallel lengths) is encoded by defining the plane at infinity . Euclidean structure (lengths and angles) is encoded by a proper virtual conic on . The simplest way to represent this absolute conic is by its envelope, i.e. a disc-quadric represented by a symmetric positive semidefinite rank3 matrix . In a metric frame . is the plane at infinity and thus The null-space of . The similarities or metric transformations (i.e. Euclidean plus a global scale-factor) are exactly the transformations that leave the absolute conic unchanged. The following abbreviations will be used repeatedly throughout the text AC for Absolute Conic and IAC for Image of the Absolute Conic. The AC is the central concept for self-calibration. Localizing the AC in a projective frame allows to upgrade this frame to a metric one. Since it is invariant to rigid displacements, the IAC is only depending on the intrinsic calibration and not on the extrinsic parameters (i.e. camera pose). Constraints on the intrinsic camera parameters can thus be translated to constraints on the IAC. These can then be back-projected to constraints on the AC. In general it is then possible to single out the absolute conic by combining sufficient constraints from different views, i.e. at least 8 equations are needed. It was shown that this was possible imposing only the rectangularity of pixels [14]. The self-calibration approach can be formulated as follows. If represents the set of camera projection matrices for an image sequence, then the AC, represented by its envelope , can be found as the proper virtual conic for which

:9;

?A@ = >?  @ ? @ :K;  I

?@

! satisfying the self ? @  
"!$! PO (2)

for every there exists a valid calibration constraints so that

: