A Software Framework for Multimodal Interactive ...

9 downloads 27873 Views 398KB Size Report
The. SoFMIS design allows plugging or removing functional ... components can be easily extended by custom .... http://bulletphysics.org/wordpress/. [9] “Havok ...
A Software Framework for Multimodal Interactive Simulations (SoFMIS) Tansel Halic*, Sreekanth A. Venkata*, Ganesh Sankaranarayanan*, Zhonghua Lu +, Woojin Ahn°1, Suvranu De* *Department of Mechanical, Aerospace and Nuclear Engineering, Rensselaer Polytechnic Institute + Intelligent Manufacturing and Control Institution, Wuhan University of Technology °Department of Mechanical Engineering, Korea Advanced Institute of Science and Technology { halict, arikav, sankag, luz4, des }@rpi.edu, [email protected] Abstract The development of a multimodal interactive simulation is a very elaborate task due to the various complex software components involved, which run simultaneously at very high rates with maximum CPU load. In this work, we propose a multimodal parallel simulation framework called SoFMIS to create rapid interactive simulations such as surgical simulations. Our framework offers great flexibility and customization allowing simulation developers and researchers to concentrate on the simulation logic rather than component development.

Introduction Highly demanding features of physics-based interactive simulations, such as plausible and robust physics simulations, high quality visual rendering, collision detection and multimodal sensation as haptic feedback, constitute very challenging tasks in software development. Furthermore, the interactivity and the support of multimodality impose performance constraints on each individual component as well as the overall system. In addition to performance constraints, the components need to communicate (e,g, one component output is required for other component input for execution) in order to achieve multimodality and seamless interaction. All these difficulties require particular foci from different disciplines. In surgical simulation development, crucial components such as real-time tissue simulation, tool-tissue interaction, and realistic rendering could be similar in two different surgical simulations, given usually developers and researcher groups write their own software from scratch and then customize it. Their initial efforts normally become redundant due to wasting time solving software problems or implementing and designing a physics simulation that is already addressed by some other research group. However, a framework with the support of multimodalities and built-in components can drastically minimize the development time of a surgery. This allows more time to be invested on the actual challenges of the simulation scenario rather than in the development of the necessary components. Several research efforts have been carried out to build a software framework for rapid development of surgical simulation. SOFA [1] is a highly active group which

provides numerous components such as deformable bodies, collision detection, and fluid dynamics. Their architecture uses separate descriptions of objects, e.g. as physics models, rendering models, collision models, etc. These objects are stored in the scene graph for both simulation and rendering. Another framework, GiPSi [2], includes mass spring models, Finite Element Methods (FEM) along with a collection of time integrators and explicit solvers that use BLAS and LAPACK functions. Similarly, SPRING [2] is a framework for creating collaborative environments for real time surgical simulation. Its library supports multiple input devices through developed sensor abstraction which is intended to read the data from underlying hardware. Open tissue [3] is an open source toolkit that is composed of a collection of libraries varying from collision detection, matrix libraries, medical imaging and scientific visualization to physics simulation, while Tuchsmid et al.[4] support an object oriented modular simulator framework consisting of soft tissue deformation library, graphics engine, audio for effects, etc. Likewise, Chai3D [5] is a framework that integrates both haptic and visual rendering at the same time and supports various haptic devices with a unified API. There are also other frameworks that target game physics such as ODE [6], PhysX [7], Bullet [8], and Havok [9] that can be adopted for surgical simulations [10],[11]. Methods and Materials One of the essential motivations of the SoFMIS is to allow users to utilize only the components that they need through a modular and extendible framework. The SoFMIS design allows plugging or removing functional

modules by its modular structure. Moreover, the components can be easily extended by custom implementation. Therefore, SoFMIS can be described as a module oriented framework. The entities in our framework could be separated into three major groups: objects, modules and interfaces. In general, the objects in the framework only store the information about each object’s current state. For instance, the rigid body or FEM (Finite Element Method) is an object that indicates its classification, encapsulates every detail that FEM requires such as material property and geometry boundary information, but not its functionality. A scene is an abstraction which includes the objects and attached objects iterators in the environment. The modules represent main functionalities of the framework. For instance, the simulation engine is a module that coordinates the physics processes in the scene. Similarly, the viewer is dedicated to all the rendering tasks of the scenes. Both modules and objects can be also regarded as conceptual definitions, even though they have implementation correspondence. The general outline of the framework and modules can be seen in the Figure 1. The modularity of the SoFMIS is also achieved through the loose connection between the modules. In order to assure component independency, an event mechanism is adopted into our design. It simply takes the responsibility for communication between modules. The events and event manager handles both content and context events. Moreover, it allows custom events to be generated by the developers. The communication backbone is entirely managed by the event manager module. The events are categorized into two groups: context and content based. For example, the events triggered by the haptic device for updated positions are considered to be content-based whereas the bleeding of a tissue in the surgical simulation is regarded as a context-based event. Context-based events are dealt with easily through our simulation language called SoFMIS Meta Language (SML). This language simplifies the development of simulations and also allows semantic embedding during the simulation development. The SoFMIS architecture also allows the use of other physics engines such as PhysX and Bullet in addition to built-in physics modules. The physics of the external library is scheduled by our simulation module. Thus, the threading and synchronization are handled by the SoFMIS framework without any additional programming effort by the developer. SoFMIS Modules SoFMIS has numerous components to create interactive simulations, such as rendering, simulation components,

and profiler components etc. The viewer module is responsible of rendering the objects in the scenes. In each rendering frame, the viewer module traverses the scenes and, based on the type; it calls the appropriate rendering function for its corresponding objects. The viewer module handles a full range of functions, from low-level to higher level such as basic windowing functions, stereo viewing and shadow mapping. Shader support is provided in the framework for flexible and high-quality realistic rendering. Similarly, a texture manager is available for texture loading, storing or other operations over images like several filters. The simulator module is the master process of the particular simulators [12]. We support multiple simulators such as rigid body dynamics, finite element, and position based dynamics [13]. The simulator module parallelizes and controls the simulation through its thread pool. The simulator scheduler is controlled by our built-in profiler module that tracks the simulation performance and tunes the thread priorities and the size of the thread pool. There are also other components in the SoFMIS, such as a collision detection and contact response module which computes and resolves the collision of scene objects. There is a device interface manipulation module as well, which supports various hardware, and a visual effects module that allows lowcost non-physics based simulation. In addition, framework provides a scripting engine for real-time configuration of the scene and an authoring module that designs the simulation environment and manipulates scene files, importing and exporting. Results We successfully employed our SoFMIS effects module in a Laparoscopic Adjustable Gastric Band (LAGB) simulator [14] . This module is responsible for realistic effects with low computation cost. In our simulator, it was used for realistic smoke generation and bleeding during an eloctrocautery procedure. This module is a GPU based non-physical approach that aims to remove the computational burden from CPU through SoFMIS shaders. The snapshots of the effects can be seen in Figure 2. Conclusions We presented the overall design of our SoFMIS framework. Our framework provides a parallel architecture with various components for interactive simulations, particularly surgical ones. Its flexibility and easy-to-use design makes it an effective software platform for surgical simulation development. Furthermore, its modular structure allows adopting the framework components in existing simulators. Our LAGB simulator validation studies demonstrate the competency of this framework.

Acknowledgements The authors gratefully acknowledge the support of this work by the NIH/NIBIB through grant # R01EB005807. References [1]

[2]

[3]

[4]

[5]

[6] [7]

[8] [9] [10]

[11]

[12] [13]

[14]

J. Allard, S. Cotin, F. Faure, P.J. Bensoussan, F. Poyer, C. Duriez, H. Delingette, and L. Grisoni, “Sofa-an open source framework for medical simulation,” Studies in health technology and informatics, vol. 125, 2006, p. 13. M.C. Cavusoglu, T.G. G\öktekin, F. Tendick, and S. Sastry, “GiPSi: An open source/open architecture software development framework for surgical simulation,” Medicine meets virtual reality 12: building a better you: the next tools for medical education, diagnosis, and care, 2004, p. 46. K. Erleben, J. Sporring, and H. Dohlmann, “OpenTissue-An Open Source Toolkit for PhysicsBased Animation,” 2005. S. Tuchschmid, M. Grassi, D. Bachofen, P. Fr\üh, M. Thaler, G. Szekely, and M. Harders, “A flexible framework for highly-modular surgical simulation systems,” Biomedical Simulation, 2006, pp. 84–92. F. Conti, F. Barbagli, D. Morris, and C. Sewell, “CHAI: An open-source library for the rapid development of haptic scenes,” IEEE World Haptics, 2005. “Open Dynamics Engine - home.”, http://www.ode.org/ Physx, “NVIDIA PhysX Physics Simulation for Developers.”, http://developer.nvidia.com/object/physx.html Bullet, “Game Physics Simulation.”, http://bulletphysics.org/wordpress/ “Havok - Home.”, www.Havok.com A. Maciel, T. Halic, Z. Lu, L.P. Nedel, and S. De, “Using the PhysX engine for physics-based virtual surgery with force feedback,” The International Journal of Medical Robotics and Computer Assisted Surgery, vol. 5, 2009. Z. Lu, G. Sankaranarayanan, D. Deo,, D. Chen,, and S. De, “Towards Physics-based Interactive Simulation of Electrocautery Procedures using PhysX,” Boston: IEEE Haptics symposium, 2010. J. Andrews, Designing the Framework of a Parallel Game Engine, February, 2009. M. M\üller, B. Heidelberger, M. Hennix, and J. Ratcliff, “Position based dynamics,” Journal of Visual Communication and Image Representation, vol. 18, 2007, pp. 109–118. G. Sankaranarayanan, J. D. Adair, T. Halic, M. A.

Gromski, Z. Lu, W. Ahn, D. B. Jones, and S. De, “Validation of a Novel Laparoscopic Adjustable Gastric Band Simulator,” will appear in Surgical Endoscopy, 2010.

Illustrations

Figure 1. SoFMIS Overall Architecture.

(a)

(b)

Figure 2. SoFMIS Module for (a) Smoke and (b) Bleeding employed in LAGB simulation.