Interactive, immersive visualization for indoor ... - IEEE Xplore

Interactive, Immersive Visualization for Indoor Environments: Use of Augmented Reality, Human-Computer Interaction and Building Simulation Ali Malkawi1, Ravi Srinivasan1, Benjamin Jackson2, Yun Yi1, Kin Chan1, Stanislav Angelov2 1 Building Simulation Group, Department of Architecture, School of Design 2 Department of Computer and Information Science, School of Engineering and Applied Science University of Pennsylvania, Philadelphia, PA, USA {[email protected], [email protected], [email protected], [email protected], [email protected], [email protected]}

Abstract We present an interactive gesture recognition-based, immersive Augmented Reality system visualizing Computational Fluid Dynamics [CFD] datasets of indoor environments. CFD simulation is used to predict the indoor environments and assess their response to specific internal and external conditions. To enable efficient visualization of CFD datasets in actual-space, an Augmented Reality system was integrated with a CFD simulation engine. To facilitate efficient data manipulation of the simulated post-processed CFD data and to increase the user-control of the immersive environment, a new intuitive method of HumanComputer Interaction [HCI] has been incorporated. A gesture recognition system was integrated with the Augmented Reality-CFD structure to transform handpostural data into a general description of hand-shape, through forward kinematics and computation of hand segment positions and their joint angles. This enabled real-time interactions between users and simulated CFD results in actual-space. Keywords: Augmented Reality, HCI, building simulation, visualization

1. Introduction Computer simulations of physical processes aid the scientific research community by modeling and simplifying real-world systems. One such simulation is CFD that provides a method for solving Navier-Stokes equations governing heat and mass transfer. They are extensively used in aerospace, nuclear automotive, biomedical, environmental, microelectronics, building design-construction etc. Building simulation is gaining widespread acceptance in the building designengineering community for simulating thermal energymass behavior. In building design, these simulations

allow expert users, among other things, to evaluate a series of environmental decisions in buildings. In addition to simulation, visualization of CFD is critical to enhance the comprehension of simulated results by the user. Rapid research and development in hardware and software has enabled visualization of complex data, both static and dynamic, in Virtual Environments [VE], including virtual reality [1-4] and augmented reality [5-7]. While Virtual Reality [VR] systems aid in complete immersion, Augmented Reality [AR] systems lead to partial immersion of the user [8]. One of the earlier applications of VEs for CFD visualization is the virtual wind tunnel allowing the visualization of particles as streamlines, path-lines, volume arrows, etc. [9]; others applications include immersive visualization for structural analysis [10], immersive real-time fluid simulation [11], building performance visualization [12-14], etc. Such immersive building simulation [15] enables the integration of VEs with building data such as CFD. Demand for data manipulation during immersive visualization has led to the integration of intuitive HCI techniques to immersive building simulation. Early HCI developments were used for interacting within VE [16]. Moreover, as computers are becoming more pervasive in everyday life, natural human-computer interactions through speech, gestures, eye-movement, aural, and other modalities are being researched and integrated with other technologies. For example, gestures provide a more natural interface [17,18]; they are simpler to use compared to typical Windows, Icons, Menus, and Pointing interfaces, and have been widely used by molecular biologists [19], computer games [20], exploring cyberspace [21], sign languages [22], etc. Although building performance data is becoming more readily available, no research has been established to enable the visualization or interaction with this information. Such interactions between buildings and their occupants will dramatically enhance the way

Proceedings of the Eighth International Conference on Information Visualisation (IV’04) 1093-9547/04 $ 20.00 IEEE

buildings are experienced, managed and operated. This paper presents an interactive, immersive visualization pipeline that enables efficient exploration of CFD simulation datasets in actual-space, (Figure 1). IMMERSIVE VISUALIZATION

CFD SIMULATION

CFD DATA

generation. This is followed by Fluent 6.02 execution to perform the CFD simulation. Once the simulation converges, the results are stored in Virtual Reality Modeling Language [VRML] format represented in isoplanes and iso-surfaces for AR visualization. Unique identifiers are attached to these VRML slices for quick display onto HMD based on the user’s hand gestures.

2.2 VRML MODEL CALIBRATION HC I

TRACKER DATA SMOOTHING

HUMAN-COMPUTER INTERACTION

Figure 1. Interactive, immersive visualization of CFD datasets. Through the utilization of advancement in humancomputer interaction, AR visualization and building simulation, this interactive, immersive visualization system will permit new ways of interacting with buildings.

2. Interactive, Immersive Visualization Pipeline The interactive, immersive visualization pipeline allows for efficient processing and transfer of data (Figure 2). It encompasses three routines: (a) CFD analysis routine that generates CFD datasets, (b) HCI routine comprises of a library of standardized glossary of a priori gesture recognition tasks, and (c) AR visualization routine aids in tracking user’s movement in real-time, visualize CFD results, and provides an immersive environment in actual-space. CFD ANALYSIS ROUTINE HCI ROUTINE AR VISUALIZATION ROUTINE

Figure 2. Immersive visualization pipeline.

2.1

CFD Analysis Routine

CFD simulation enables designers to fine-tune their designs based on performance results, thus speeding up the design process through comparison of a broader range of design variants. CFD analysis involves, modeling the space to be investigated, setting the boundary conditions and specifying the algorithms to be used. Gambit 2.0.6 [23] is used for modeling and mesh

HCI Routine

HCI routine enables better accessibility and user participation with the immersive environment. In this routine, hand gestures are transformed into a set of functions that facilitate better manipulation of CFD datasets. In addition of being an intuitive approach, gestures are space-related modality that supports communication of concrete and spatial content. In this project, CyberGlove [24] is used to transform posture data into a general hand-shape description. Forward kinematics is used to compute hand segment positions from given joint angles. Gestures were studied to determine the minimum number of joint angles necessary to ensure the uniqueness of the gesture. The model needed to account for individual variations in grip strength, resting thumb position, and palm arch. In addition, the model needed to be effective within a standard deviation that was tight enough to ensure that there was no confusion as to which gesture was being performed. For example, for the grab model, only the metacarpophalangeal [MP] and proximal interphalangeal [PI] joints of the first four fingers were measured, leaving the thumb free to assume any position. This is based on the assumption that the grab motion is the only closed-fist motion in the system; if we were to add a thumbs-up gesture, the model would have to be altered to allow for it. The distal interphalangeal [DI] joints were not measured to allow for a variety of different grips. Similarly, the chop motion was implemented by measuring the MP, PI and DI joints, ensuring that all four fingers were fully extended. The finger abduction was also measured, as the gesture requires that all four fingers be together. Similar to the grab motion, the thumb joints were not measured. The benchmark measurements for each gesture were determined by computing the mean values from a set of measurements taken during calibration. Twenty sample readings were taken for each gesture in the system. Each gesture consists of 22 values that represent the measurements of the bi-metal bending sensors that characterize the hand-shape and hand-internal movements; the results were analyzed to determine the mean joint angles and statistical standard deviation (an example is shown in Figure 3). Some of the hand shapes developed in this project are, ‘Closed_Fist’, ‘Open_Flat_Palm’, ‘Touch_Finger’, Shoot_Hand’, Inc_Thumb’ etc. Table 1 presents various hand gestures employed and their respective joint angles and significance.


For example, to create an iso-plane at a distance of 2.25 ft along the X-axis, the user ‘moves’ to that location by reading his/her position displayed onto the HMD, and ‘chops’ along the X-axis specifying the required isoplane. Table 2 provides visual representation of each of the recognized gestures along with their respective actions. Table 2. Visualization commands and their respective actions.

VISUAL REPRESENTATION

ACTIONS

Figure 3. Closed_Fist measurement calibration. Table 1. Gestures, joint angles and gesture significance.

GESTURE

FINGER

JOINT ANGLES MPJ PIJ DIJ

TI ANGLE

CHOP X

SIGNIFICANCE CHOP Y

Closed_Fist

Open_Flat_ Palm

Touch_Finger

Shoot_Hand

I M R P T I

180 178 183 205

185 192 185 177

98

72

M R P T I M R P T I M R P

75 96 96

73 64 68

NC

NC 105 139 157 167

75 172 175 174 NC

121 111 158 186

67 76 185 186

156 171 158 185 NC 77 71 67 86 NC 80 154 132 132 NC 70 75 43 80

NC

NC NC

NC NC

C: Move, Information A: Grab & Drop, Point

CHOP Z

C: Move, Create A: Chop X,Y,Z, Grab & Drop

GRAB

C: Information A: Point

MOVE

ACTION(S) Chop X: Create isoplane along X axis. Chop Y: Create isoplane along Y axis. Chop Z: Create isoplane along Z axis. CONSTRAINT(S) Isoplanes can be created either along X, Y or Z axis only.

ACTION(S) GRAB & DROP: Grab an isoplane from one location; move to a new location; drop at the new location. CONSTRAINT(S) Isoplanes can be moved either along X, Y or Z axis only.

NC C: Zoom-in / out A: None [Shoot_Hand precedes Inc_Thumb]

DROP

T 142 NC 124 Inc_Thumb I 132 65 73 C: Zoom-in / out A: Snap-in / out M 114 74 77 R 158 185 42 P 185 186 80 T 81 NC 181 I – Index finger; M – middle finger; R – ring finger; P – pinkie finger; T – thumb; MPJ – Joint where the finger meets the palm; PIJ – Second joint from the finger tip; DIJ – Joint closest to finger tip; (For thumb, MPJ = PIJ since there is only one joint unlike the other fingers); NC – Non critical; TI angle – angle between thumb and index finger; c – Component; A – Action.

When the system needs to check for a match, the current joint angle values are measured using the GetState method of the CyberGlove object. To accommodate slight variation in hand gestures, a standard deviation is introduced to the gesture recognition scheme. For each gesture in the system, joint angles are tested to see if they fall within this standard deviation of the values taken from the glove. If all of the relevant joint angles for a gesture fall within the acceptable range, a match is reported and the system responds accordingly. Similarly, hand position and orientation information is obtained with the aid of magnetic trackers attached to the hand glove.

ACTION(S) SNAP-IN & SNAP-OUT: This action enables the number of zooming increments, either zoom-in or zoom-out. ZOOM IN / OUT CONSTRAINT(S) The axis along the palm is considered the axis of zoom. SNAP-IN

SNAP-OUT ACTION(S) POINT: Point index finger to obtain information. POINT

Depending on a user’s hand gesture, VRML surfaces are drawn, calibrated and transferred to HMD for visualization. Tracking global hand movement using magnetic trackers, and local fingers motion using bend sensors enables on-the-fly data manipulation and control.


2.3

AR Visualization Routine

AR visualization routine assists in the visualization of CFD datasets with the support of magnetic positionorientation trackers and a HMD. This routine involves two steps (a) Tracker data smoothing and (b) VRML model calibration (Figure 4).

through transformations: (a) Scaling: one scale factor is used to scale VRML model along XYZ axes, (b) Rotation: based on the orientation of the created isoplane, the VRML model is rotated, and (c) Translation: this is the same as rotation above (Figure 6).

VRML

GAUSSIAN FILTER

6 DOF DATA

VRML

TRACKER DATA SMOOTHING

CALIBRATED VRML

CALIBRATED VRML MODEL

SCALING

ROTATION

TRANSLATION

AR VISUALIZATION

Figure 6. VRML model calibration. Figure 4. AR visualization routine. Flock-of-Birds magnetic trackers [25] were employed to track the user’s head movement in 6 Degrees-of-Freedom (DoF). These trackers have a static accuracy of 1.8mm (position-RMS) and 0.5deg (orientation-RMS) over a verified range of 20.3cm to 76.2cm. A catadioptric CRT-based HMD [26] was utilized to display virtual objects in actual-space. Due to the possible presence of metal in the indoor spaces, high degree of noise can propagate while tracking user movement using the magnetic trackers. To remove the noise in the tracker data, a one-dimensional Gaussian filter was used (Figure 5). Į

The 6DoF real-time tracker data (Rt) = {t1(x,y,z,a,b,c)} 1 Where x,y,z are sensor positions; a, b, c are sensor orientations along X,Y,Z at time t=1. Since the tracker data consists of 6 DoF values in single direction (forward time-related), a ID Gaussian filter is used in this computation. The Gaussian filter is represented as,

-x2 2 e 2ı

1 G(x) =

ı – Standard Deviation Mean of the distribution is zero [centered].

¥2ʌ ı

The convolution kernel selected for smoothing, in this case, is shown in sketch below. Since tracker data is relative to time, and for real-time visualization purposes, only the second half of the kernel is used for computation. FOR REAL-TIME VISUALIZATION PURPOSES, UNSHADED PORTION IS NOT USED IN SMOOTHING PROCESS.

0.553

0.350

When the VRML models are positioned in the actual-space for AR visualization, the smoothened tracker data values are applied to further calibrate the modules further. A final calibrated VRML model is imported into a Java3D program using a VRML Loader package.

3.

System Interface

Building this environment involved a wide range of hardware and software. The hardware included a catadioptric HMD, a hand-glove and magnetic 6-DoF trackers. C++ and Java3D APIs were employed to construct the AR software. The software consisted of two parts (a) visualization component and (b) gesture recognition component. The visualization software component encompasses VRML file loader, Gaussian functions and motion-capture. The gesture software component developed allowed for selective joint model that ensures the uniqueness while allowing wide variation of individual hand gestures. The interactive immersive AR system maps the CFD output onto the actual space through the HMD. The user can navigate through the space, inquire about its thermal conditions, manipulate the results of data displayed as well as interact with the actual environment (Figures 710).

REAL-TIME 6 DoF DATA IS SMOOTHENED WITH THE SHADED PORTION.

0.088 0.009 -3

-2

-1

0

1

2

3

The convolution is performed by sliding the above kernel G(x) over R, in real-time to obtain smoothened, calibrated tracker data, Į Calibrated tracker data (R) = R(t-x).G(x) t=1

Figure 5. Calibration of tracker data. The second step involves VRML model calibration to accurately map the VRML slices to actual-space


Figure 7. Iso-plane as seen by the user using AR system.

4.

Figure 8. User obtaining information from isoplane using hand-glove.

The current system implements an interactive, immersive visualization of CFD building simulation in actual-space. The research demonstrated that constructing an AR system is a challenge due to issues related to AR hardware such as registration and latency. The sensitivity of human eyes to detect registration errors is extremely high and poses a challenge in building interactive AR systems. Moreover, due to eye anatomy, each user can perceive the view slightly differently and various viewing parameters contribute to the registration problem. Latency could be overcome by using more specialized hardware or by predicting future viewpoints. This work demonstrated the possibility of a larger development that can take into account on the fly simulations and interactions. In order to accomplish such a task, issues of data visualization that can provide real-time simulation need to be researched. This includes developing a building representation for an AR based simulation system that will support the real-time interaction and developing rapid data visualization techniques that will enable the user to see the effect of operational changes in buildings, conduct changes, run simulation and visualize their results in real-time. In addition, a multi-modal HCI that includes both speech and gesture recognitions is being researched for enhanced joint-performance of tasks.

5. [1]

[2]

Figure 9. User moving the isoplane using handglove.

[3]

[4]

[5]

[6]

[7]

Figure 10. Dynamic particle visualization as seen by the user through HMD.

Conclusions

[8]


References W. Ribarsky, E. Ayers, Eble, J., and S. Mukerjee, Using Glyph-maker to create Customized Visualizations of Complex Data. Computer 27(7): 57-64, 1994. T.V. Teylingen, W. Ribarsky, , and Charles van der Mast. Virtual Data Visualizer. IEEE Transactions on Visualization and Computer Graphics, 3(1): 65-74, 1997. van Dam, Andries, Laidlaw, R.M. Simpson, Experiments in immersive VR for Sci Viz, Computers & Graphics 26: 535-555, 2002. T. Wilde, J.A. Kohl, and R.E. Flanery, Jr,. Immersive and 3D viewers for CUMULVS: VTK/CAVE and AVS/Express, Future Generation Computer Systems 19: 701-719, 2003. W. Piekarski, D. Hepworth, V. Demczuk, B. Thomas, and B. Gunther. A mobile AR user interface for terrestrial navigation. In Proceedings of 22nd Australasian computer science conference, pp: 122-133, 1999 M. Pilgrim, D. Bouchlaghem, D. Loveday and M. Holmes. A Mixed Reality System for Building Form and Data Representation, IEEE, pp 369-375, 2001. S. Vogt, A. Khamene, F. Sauer, A. Keil, and H. Niemann. A High performance AR system for Medical Applications. In Proceedings of second IEEE and ACM International Symposium on Mixed and AR, 2003. R. Azuma A Survey of Augmented Reality. Presence: Teleoperators and Virtual Environments 6 (4): 355-385, 1997.

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23] [24] [25]

[26]

Bryson S., 1993. Virtual Wind Tunnel: A highperformance virtual reality application, 1993. In Proceedings of IEEE Annual Virtual Reality International Symposium, pp:20–25, 1997. T. Impelluso. Physically Based Virtual Reality in a Distributed Environment, Computer Graphics, 30(4): 6061, 1996. T.M. Wasfy, and A.K. Noor. Visualization of CFD results in immersive Virtual Environments. Advances in Engineering Software 32 (2001) pp 717-730, 2001 A. Malkawi, and R. Choudhary. Visualizing the Sensed Environment in the Real World, Journal of HumanEnvironment Systems, 3(1): 61-69, 1999. V. Shahnawaz, J. Vance, S. Ktti. Visualization of Post Processed CFD data in a Virtual Environment. In the Proceedings of DETC99, ASME Design Engineering Technical Conference, September 1999, Las Vegas, Nevada, pp 1-7, 1999. E. Linden, J. Hellstrom, M. Cehlin, and M. Sandberg. Virtual Reality Presentation of Temperature Measurements on a Diffuser for Displacement Ventilation. In Proceedings of 4th International Conference on Indoor Air Quality and Energy Conservation in Buildings, Changsha, Hunan, V.II., pp 849-856, 2001. A. Malkawi. Immersive Building Simulation, chapter in: (eds) A. Malkawi, G. Augenbroe. Advanced Building Simulation, Taylor Press, 2004. H.A. Sowizral. Interacting with VE using augmented virtual tools”, in Stereoscopic Displays and VR systems, 1994. V.K.Pavlovic, R. Sharma, and T.S. Huang. Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review. In: IEEE PAMI, Vol. 19(7): 677695, 1997. R. Cipolla, and A. Pentland, (eds) Computer Vision for Human-Machine Interaction. Cambridge: Cambridge University Press, 1998. V.K. Pavlovic, R. Sharma, and T.S. Huang, Gestural interface to a visual computing environment for molecular biologist. In Proceedings of International Conf. Automation in Face Gesture Recognition, pp: 3034, 1996. W.T. Freeman, K. Tanakan, J. Ohta, and K. Kyuma. Computer vision for computer games. In Proceedings of International Conf. Autom. Face Gesture Recognition, pp: 100-105, 1996. R. Kadodayashi, K. Nishimoto, and K. Mase. Design and evaluation of an immersive walk-through application for exploring cyberspace. In Proceedings of Third International Conf. Autom. Face Gesture Recognition, pp: 534-539, 1998. R. Liang, and M. Ouhyoung. A real-time gesture recognition system for sign language. In Proceedings of Third International Conf Autom. Face Gesture Recognition, pp: 558-567, 1998. Fluent.2003. Fluent Inc., website accessed on June 10, 2003. URL: http://www.fluent.com VTI 2003. Virtual Technologies Inc, website accessed on August, 2003. URL: http://www.immersion.com Ascension-Tech. 2003. Ascension Technologies Inc., website accessed on August 10, 2003. URL: http://www.ascension-tech.com NVIS 2003. Nvision Inc., website accessed on June 24, 2003. URL: http://www.nvisionindustries.com
