post-demolition landscape assessment using ...

Proceedings of the 16th International Conference on Construction Applications of Virtual Reality, 11-13 December 2016, HK

POST-DEMOLITION LANDSCAPE ASSESSMENT PHOTOGRAMMETRY-BASED DIMINISHED REALITY (DR)

USING

Kazuya Inoue, Tomohiro Fukuda, Nobuyoshi Yabuki, Ali Motamedi, and Takashi Michikawa Division of Sustainable Energy and Environmental Engineering, Osaka University, Osaka Japan. ABSTRACT: Recently, Augmented Reality (AR), which extends a visible scene with virtual objects, has gained popularity for landscape assessment. It is used in urban planning to facilitate reaching consensus on a design. However, existing AR approaches cannot correctly simulate views after the demolition and removal of outdoor buildings. Tanemura et al. (2014) applied Diminished Reality (DR) for this application. DR is a technique for removing an object from a scene by overlaying an appropriate background image on top of the object’s area, in real time. They proposed a system that facilitates landscape simulation before removal of outdoor buildings by using point cloud data. Their system creates point cloud data using a 3D laser scanner and registers video camera’s position and orientation manually. However, this method is expensive due to the use of laser scanners. Moreover, some parts of acquired point cloud data have low density or low quality. This paper proposes a photogrammetry-based DR system for landscape assessment. It uses photographs to obtain point cloud data of buildings and to calculate the video camera’s position and orientation. The proposed system is advantageous to the previous work as it is higher in quality due to the use of polygon data instead of point cloud data to overlay the background image. Additionally, this system is more robust in the augmentation step as the computation of camera’s position and orientation is done using automatic image matching, whereas the conventional method uses manual registration between point clouds and real-time video feed. The effectiveness of the proposed method is successfully demonstrated in a case study performed in a real environment. KEYWORDS: Diminished Reality, Augmented Reality, Photogrammetry, Visual simulation for community planning, Construction Planning

1. INTRODUCTION Augmented Reality (AR), which extends a visible scene with virtual objects, has gained popularity for exterior construction applications (Klinker et al. 2001). AR can facilitate reaching consensus on a landscape design due to its ability to simulate views of full scale new structures at the design stage. Unlike Virtual Reality (VR) technology, AR does not require 3D virtual models of every object around the desired target, such as buildings, roads, lands, and terrains. Hence, AR has a potential to drastically reduce time and cost required to produce 3D virtual models of the surrounding environment (Fukuda et al. 2014). However, existing AR approaches cannot correctly simulate the view after the demolition and removal of exterior structures. If simulation of new structures is carried out while old existing structure is still present, a 3D virtual model of the new structure will overlap the existing to-be-renewed structure. As a result, part of it, if not all, will be still visible and displayed. To solve this problem, Diminished Reality (DR) can be employed. DR is a technique that removes the image of an existing object from a scene, replacing it with the background image of the area.

689


Enomoto and Saito (2007) proposed a DR method for diminishing the occluding objects from each camera’s view by employing different cameras that capture the same scene at different viewpoints. This method used AR-Tag markers to calibrate multiple cameras. However, the markers should constantly remain in the camera’s viewpoint which makes it impractical for outdoor structures. Cosco et al. (2009) proposed a DR method which simulates a view without using a visuo-haptic device by using Image Based Rendering (Buehler et al. 2001; Debevec et al. 1996). In this method, to-be-removed target objects must be moved freely because the background image was prerecorded without the target objects. If the target objects is movable, this method is effective. However, this method is not effective for fixed objects such as exterior structures. Tanemura et al. (2014) proposed a DR system by using point cloud data. This system called Point Cloud Diminished Reality Simulation (PCDR) creates point cloud data of a target structure and background structures by using a 3D laser scanner. This system overlays the point cloud data of parts of background structures on the target structure and simulates the view as if the target structure is removed. In this system, a user registers natural feature points of images that the web camera captured to correspond to the point cloud data for the alignment of new structures (Yabuki et al. 2012). However, this system is expensive due to the use of laser scanners which limits its usability. Moreover, some parts of acquired point cloud data have a low density or a low quality. Additionally, this system requires natural feature points to be continuously captured by the web camera. Once natural feature points go out of the view, a user must register the points again. This research focus on photogrammetry as an alternative measurement technology. Photogrammetry is a technique which computes camera’s position and orientation and creates point cloud data of target objects. In comparison with laser scanners, photogrammetry enables creating inexpensive point cloud data. Recently, Sato et al. (2016) proposed a marker-less AR system. Using automatic image matching technology, their system matches camera’s position and orientation, computed by Structure from Motion (SfM) method (Hartley and Zisserman 2000), to the web camera’s position and orientation. This system does not need special equipment such as GPS and gyroscope sensors, allowing users to move freely when the target structure is visible. This paper proposes a novel DR method for landscape assessment using photogrammetry. A prototype application called PhotoDR-2015 is also developed, which simulates views after the demolition and removal of exterior structures with a lower cost and a higher quality. Moreover, PhotoDR-2015 automatically computes the web camera’s position and orientation based on the method proposed by Sato et al. (2016). This paper evaluates the accuracy of the proposed method and demonstrates its applicability in a case study performed in a real environment.

2. PHOTOGRAMMETRY-BASED DIMINISHED REALITY Our proposed system creates DR scenes from video images acquired by a camera for the post-demolition landscape assessment. As explained in Section 1, DR is a technique for removing an unwanted object from a scene by overlaying an appropriate background image on the object. DR requires using computational techniques for tasks, such as estimation of the video camera’s position and orientation, computation of the background image, and recognition and tracking the object. Our method estimates the video camera’s position and orientation based on the method proposed by Sato et al. (2016). Based on the estimated position and orientation, our method computes 690


the background image of the target area and recognizes and tracks the target. Our method has two main phases; preprocessing and the real-time processing (Figure 1).

Fig.1: Overview of the proposed method In the preprocessing phase, several photographs of the target structure which is to be diminished and background structures which are hidden behind the target structure are needed as input data. Photographs of the target structure are used for reconstructing point cloud data and estimating the position and orientation of the web camera. Point cloud data of the target structure is reconstructed using SfM method. From its point cloud data, a mask polygon is made, which is used for determining the removal region. The green mesh of the mask polygon (Figure 1) is to distinguish the removal region clearly. Additionally, local features of each photographs of the target structure are extracted to be used for the image matching. Moreover, point cloud data and polygon model of background structures are reconstructed from their photographs by using SfM method. In the real-time processing phase, local features, mask polygon, and, background structures polygon, which are calculated and created in the preprocessing phase, are used as input data. First, local features of a live video image are extracted. The extracted features are compared with the features of stored images, which are calculated in the preprocessing phase. The automatic image matching method finds the most similar image and the position and orientation of the camera for that image are chosen as the current position and orientation of the web camera. By using the estimated positon and orientation and the web camera, the mask polygon and background structures polygons are rendered. The rendered mask polygon determines the removal area. The area of the rendered background structures polygon is overlaid on top of the live video image. As a result, the target structure on the 691


live video image seems to be diminished.

3. VERIFICATION AND THE CASE STUDY 3.1

Experiment for Verification of System’s Accuracy

A comparative verification was applied in order to determine the accuracy of the developed system (i.e., PhotoDR2015). The verification test consists of a physical model made of two sub-models; (1) the target structure (Figure 2 (a)), and (2) background structures (Figure 2(b)). The physical model is modular and each part is removable. Additionally, grid lines are printed on the texture of background structures and on the ground. The accuracy is evaluated using these grid lines. If PhotoDR-2015 accurately diminished the target structure, the DR image should look similar to the ideal image with continuous and matching grid lines.

(a) Model of target structure

(b) Model of background structures

Fig.2: Physical model of structures for verifying of the accuracy of PhotoDR-2015 In the preprocessing phase, 42 photos of the model were used to reconstruct point cloud data of the model and for the image matching (Figure 3 (a)). To reconstruct point cloud data, OpenMVG (Open Multiple View Geometry. Ver. 0.7) was used, which is an open source library for SfM. From the point cloud data, the mask polygon was then manually created (Figure 3 (b)). By using SfM, polygons for background structures were made from 40 photos in which only the model of background structures could be seen (by physically removing the model of target structure) (Figure 3 (c)). PhotoScan (by Agisoft LLC ver. 1.2.3) was used, which can reconstruct 3D polygon with photorealistic textures. The orientation, position, and size of both polygons were manually adjusted based on point cloud data of the model.

692


(a) Point cloud data of models

(b) mask polygon

(c) Background structures polygon

Fig.3: Point cloud data and polygons for the accuracy verification In the real-time processing phase, the model of the target structure was placed on its position (Figure 4 (a)) and DR operation was carried out with a web camera at three viewpoints. The target structure is then removed from the model and three pictures were taken from the same viewpoints (Figure 4 (b)) for comparison. Figure 4 (c) and (d) show the resulting DR images after placing the target structure back to its position and executing DR.

(a) Captured image

(b) Ideal image

(c) DR image

(d) The diminished area on DR image (Inside the red frame)

Fig.4: Results for the verification model (from Viewpoint1)

693


The accuracy of PhotoDR-2015 was evaluated using a comparative verification method. As shown in Table 1, measuring points on the ideal image and measuring points on a DR image were compared and the horizontal and vertical axis errors were measured. Table 1: Comparing measuring points from three different viewpoints Captured Image

Measuring points on the ideal image Measuring points on the DR image

Viewpoint 1 Viewpoint 2 Viewpoint 3 Results presented in Table 2 are the average error for all measuring points. They show that the ratios of the average error to the image size were less than 5 %. Background Structure A (Figure 2 (b)), which contains measuring points a, b, c, and d (shown in Table 1), was vertical to the direction of the web camera from viewpoint 1. Therefore, the pixel errors are easily translated to distance errors regarding Background Structure A. Table 3 shows both pixel and distance errors of measuring points a, b, c, d from viewpoint 1. The distance errors of Background Structure A, which is 280 millimeters in height and 170 millimeters in width, is 21.78 millimeters in the horizontal axis, 4.66 millimeters in the vertical axis.

Table 2: The average error for the accuracy verification experiment Average Error (pixel)

694

Error Ratio (%)


Horizontal Axis Vertical Axis Horizontal Axis Vertical Axis Viewpoint 1

30.2

9.25

3.35

1.54

Viewpoint 2

21.5

7.67

2.39

1.28

Viewpoint 3

8.41

11.2

0.93

1.86

Table 3: The pixel and distance error Pixel Error (pixel)

Distance Error (mm)

Horizontal Axis Vertical Axis Horizontal Axis 39.56

3.2

8.63

Vertical Axis

21.78

4.66

Case Study

To verify the applicability of the proposed method, a verification experiment was conducted in an outdoor environment. Restaurant, 2F, Welfare Bldg., Poplar Street in Osaka University, Suita Campus was selected as a target of the experiment. In the preprocessing phase, 21 photos of the target structure were used to construct point cloud data and for image matching (Figure 5 (a)). The number of photos is less than that of the verification experiment (Subsection 3.1). This is because it was difficult to take photos of the whole target structure due to narrowness of the street. Although the density of the resulting point cloud was not very high, it was possible to create the mask polygon (Figure 5(b)). The polygon of the background structures were made from the point cloud data of 75 photos by using PhotoScan (Figures 5 (c)). The orientation, position and size of both polygons were adjusted based on the point cloud data of the target structure. Table 4 shows the results of PhotoDR-2015 for diminishing the target structure from the scene in the real-time processing phase.

(a) Point cloud of the target structure

(b) Mask polygon

(c) Background structures polygon

Fig.5: Point cloud data and polygons for case study

Table 4: Results of PhotoDR-2015

695


At the start

20 seconds later

40 seconds later

Captured image DR image Table 4 confirms that PhotoDR-2015 can simulate views after the demolition with high-quality in an outdoor environment. Moreover, it shows that the web camera’s positions and orientations were correctly calculated while the web camera was moved during 40 seconds. PhotoDR-2015 was able to maintain the proper position of DR images while changing the viewpoint during this time period. Additionally, a verification test was designed to verify an AR scenario in which a new structure replaces the target structure. Figure 6 shows the plan plot of the new structure and the camera’s position and orientation in preprocessing phase. The red line and arrows (Figure 6 (a)) show the photography route and camera’s direction for the target structure. Figure 6 (b) shows the photography route and camera’s direction for taking pictures from the background structures. Table 5 shows the result of AR visualization both with PhotoDR-2015 and without PhotoDR-2015. It shows that the 3D virtual model of the AR images of the new structure is overlapped the target structure when DR function was not used (first row of Table 5). Hence, it is hard to assess the landscape of the new structure considering the landscape of the background structures. However, as shown in the second row of Table 5, DR have solved this problem which can and facilitate reaching a consensus on a landscape design more easily. Table 5 also confirms that PhotoDR-2015 can maintain the correct position of DR images while changing the camera’s viewpoints.

696


Fig.6: A plan plot of new structure and the camera’s position and orientation in preprocessing phase

Table 5: Results of a new structure simulation using PhotoDR-2015 At the start

20 seconds later

40 seconds later

AR Image without AR Image with

4. CONCLUSIONS AND FUTURE WORK In this paper, a novel DR method for landscape assessment using photogrammetry and geometric registration methods was proposed. It demonstrated that augmenting the scene with a new structure using the DR function is of higher quality than that of without using the DR function. A prototype application (i.e., PhotoDR-2015) was developed to verify the proposed method. Through the experiments for the verification of the accuracy, the errors of overlaid background image were 21.78 mm in the horizontal axis and 4.66 mm in the vertical axis. In the case study, PhotoDR-2015 proved to be able to simulate views after the demolition with the removal of unwanted target structures. In comparison to the previous study, PhotoDR-2015 can simulate views with a higher quality, it is less

697


costly due to the use of camera instead of a laser scanner, and it doesn’t require manual input for inputting the web camera’s position and orientation. It is only possible to use this method on the route in which photos of the target structure are taken, hence, this system has a limitation in simulating from multi-viewpoints. Moreover, this method is sensitive to the light condition. It is not possible to use when there is not enough light or it is raining. Additionally, this system requires having photographs of the background structure in order to create their polygons. If it is not possible to capture photographs that cover the entire building or if the building is occluded by trees, signs or eaves, high-quality polygons cannot be created. The use of UAVs for taking photographs can be considered for certain cases. In the future, it is necessary to adapt image inpainting technique (Bertalmio et.al. 2000; Herling and Broll 2010), which automatically generate semi-realistic textures using information from around the target area, to improve the quality of visualization.

ACKNOWLEDGMENT This work was supported by JSPS KAKENHI Grant Number JP25350010 and JP26-04368.

REFERENCES Bertalmio M., Sapiro G., Caselles V., and Ballester C. (2000). Image inpainting, In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, 417-424. Buehler C., Bosse Michael., McMillan L., Gortler S., and Cohen M. (2001). Unstructured lumigraph rendering, In Proceedings of SIGGRAPH’01, 425-432. Cosco F. I., Garre C., Bruno F., Muzzupappa M., and Otabuy M.A. (2009). Augmented touch without visual obtrusion, In Proceedings of ISMAR2009, 99-102. Debevec P. E., Taylor C. J., and Malik J. (1996). Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach, In Proceedings of SIGGRAPH’96, 11-20. Enomoto A. and Saito H. (2007). Diminished reality using multiple handheld camera, In Proceedings of ACCV 2007, 130-150. Fukuda T., Zhang T., and Yabuki N. (2014). Improvement of registration accuracy of a handheld augmented reality system for urban landscape simulation, Frontiers of Architectural Research, Vol. 03, No. 04, 386-397. Hartley R. and Zisserman A. (2000). Multiple view geometry in computer vision, Cambridge University Press. Herling J., and Broll W. (2010). Advanced self-contained object removal for realizing real-time diminished reality in unconstrained environments, In Proceedings of ISMAR 2010, 207-212. Klinker G., Stricker D., and Reiners D. (2001). Augmented reality for exterior construction applications, Augmented Reality and Wearable Computers, 379-382. 698


Sato Y., Fukuda T., Yabuki N., Michikawa T., and Motamedi A. (2016). A marker-less augmented reality system using feature-based image registration and structure from motion technologies for building and urban environment, In Proceedings of the 21st International Conference on Computer-Aided Architectural Design Research in Asia (CAADRIA 2016), 713-722. Tanemura T., Yabuki N., and Fukuda T. (2014). Diminished reality for ar simulation of demolition and removal of urban objects, Journal of Japan Society of Civil Engineers, Ser. F3 (Civil Engineering Informatics). Vol. 70, No. 2, I_16-I_25. Yabuki N., Hamada Y., and Fukuda T. (2012). Development of an accurate registration technique for outdoor augmented reality using point cloud data, In Proceedings International Conference on Computing in Civil and Building Engineering.

699