Projecting Wonderment: Magic through AR

Projecting Wonderment: Magic through AR Richard Snow

Matt Jones

Parisa Eslambolchilar

Dept. of Computer Science Swansea University, Singleton Park, Swansea, SA2 8PP Wales, UK



[email protected]

[email protected]

[email protected]

ABSTRACT Augmented reality artwork is an emerging field of art, including grand transformations of entire building facades that have been carried out by use of detailed 3D models and customised to the building’s unique surface. The extension of projectors into the realm of small mobile devices affords an opportunity to extend such AR art into the personal scale of the individual, allowing the user to customise and transform their surroundings on an ad hoc basis. In this paper, a system is proposed for the modification and augmentation of a mobile user’s surroundings is put forward, together with the technical challenges such a system raises.

1. INTRODUCTION In recent years, a great many uses have been proposed for projector based augmented reality, from the augmentation of objects with additional labels and information [9], through the visual enhancement of static displays such as maps [15], into providing users with the kind of practical x-ray vision that is priceless to engineers, builders and the like who are abruptly able to see the wiring and structure of objects at a glance [14]. It is the contention of this paper that the miraculous properties of these abilities be seized on and exhibited, to explore ways of blurring the line between technology and performance. Consider the following scenario... In a solitary office, the occupant looks up from their work to watch as tiny animals crawl over the walls, opening doors in uprights, climbing down steep drops with ropes, projected from their momentarily unused mobile device lying to one side across the desk. A touch of a key and the figures disappear, to be replaced by the calming image of a green jungle beginning to grow from flat shelves, waterfalls pouring silently from protruding ridges. In a busy coffee shop, one wall is alive with geometric distortions, sprites, creatures, flowing water and all manner of odd things, interacting with the shape of their surroundings, moving around each other, interacting, forming an image based on the half a dozen individuals with active AR projectors contributing to the communal illusion. One individual picks up their device and walks away, the projection reconfiguring itself with swift wireless negotiation between the remaining projectors, the loss of their particular set of imagery tempered by someone new directing their own AR device at the wall and adding their own visual elements. Spaces are the backdrop to our lives, passively unresponsive to those who pass through them. A different world is envisaged where everyone that enters a space contributes to it, where sprites dart in and out of cups, where jungles spread over the walls, and oceans decorate tabletops. We have the ability and technology to

each become magicians, commanding powers unseen with flair as well as simple efficiency. Embracing the concept of magic encourages the kind of sights that will produce inspiration and delight in others, sights and experiences that prompt new thoughts and ideas, prompting new ways of thinking about things that have grown too familiar. A system is described for the projection of augmented reality art on arbitrary surfaces, and the interactions between separate units, giving rise to a description of the desired user experience. Finally, a list of research challenges is laid out to chart a path to such a potential system. The goal of this work is to suggest an avenue for future development of mobile camera-projectors to enhance and transform the surroundings of their users. This paper is intended to encourage the melding of augmented reality and picoprojection with the idea of wonderment (as defined by Paulos et al.) [11,12], exploring a way to allow projections to form a vibrant ecology centred around their users. This position paper outlines a proposed system that illustrates a larger research agenda, inviting feedback from the greater community to shape its further development.

2. RELATED WORK This paper rests on a foundation of wonderment and AR, reaching out to recent papers which have described methods to bind disparate camera-projection units together, calibrating their displays despite variations in geometry, texture and colour of the projection surface and compensating for motion. These concepts are necessary to construct the ideal of the proposed system. Wonderment. As technology has progressed throughout society since the industrial revolution, the daily lives of the developed countries has come to revolve around it. Increasingly, the small tyrannies of the office computer, the photocopier, and the transport networks have come to shape our lives. As more and more computing power and connectivity is packed into smaller, more mobile systems, they become the de facto shapers of the modern working life. Paulos et al. argued strongly that these systems must be designed with a view to encouraging curiosity and inquisitiveness [11,12], a view which this paper embraces as a driving goal. Augmented Reality (AR). Halfway between unfiltered reality and the entirely artificial vistas of Virtual Reality, AR is the discipline of enhancing real world objects and surfaces with virtual information. AR systems may be defined as having the following three traits: that the system visually combines real objects and virtual ones, that the system is interactive in real time, and that the visual elements are registered with the real objects that they share a visual field with [1]. AR systems may make use of three main display methods; transparent displays which allow direct viewing

of real objects while overlaying virtual ones, screen based displays which use real time camera images to display both real and virtual objects, and projector based displays with project the virtual images directly onto the real objects [6]. Projector based displays have the unique property of altering the appearance of the actual object for all observers, a property that can be leveraged to transform a space for all within it. Greaves et al. demonstrated this with use of a single projection field to shape the communal experience of a shifting, arbitrary group of contributing individuals [7]. Real objects must be tracked to maintain registration of images projected upon them, which can be accomplished either by the use of sensors to detect motion directly, or vision based tracking. Vision based tracking itself breaks down into the use of previously built 3D models of the scene, and ad hoc systems which attempt to categorise their surroundings in real time [6,9,10]. Visually, this paper extends the work of large scale AR art displayed on building surfaces such as that deployed by the company UrbanScreen (see figure 1) [17] to the mobile device scale, but seeks the goal of ad hoc augmentation without the prior construction of 3D models that UrbanScreen carries out.

Figure 1. AR projected building scale artwork. 1 Ad-hoc projector clustering. As projectors become smaller, cheaper and more energy efficient, they will be incorporated into a growing number of mobile devices. These devices in turn will be carried by many people who share a communal space, be it an office, a restaurant or other public buildings. Rather than possessing a single coordinating authority, these conglomerations of projection devices will shift as individuals join and eventually leave the space, and no one projection/sensor unit will have an overview of the entire structure. A system has been demonstrated which can successfully register disparate projectors and unite them into a single display, adding and subtracting units as required [14]. By use of multiple projectors with different focal depths, Bimber et al. demonstrated techniques to create images with minimal defocus, which are tolerant of heterogenous projectors and the properties of the projection surface [2]. Imperceptible registration. In a system that allows projectors to join and leave at arbitrary intervals, with no regard for the current state of the whole, it is extremely undesirable that registration patterns need to be displayed and the field of view as a whole 1

555 Kubik photographed by Ken Mccown available under a Creative Commons Attribution-Noncommercial license http://www.flickr.com/photos/kenmccown/3785511445/ http://creativecommons.org/licenses/by-nc-nd/2.0/deed.en_GB

replaced until the calibration sequence has finished. With the potential for frequent changes in projector layout due to multiple participants arriving and leaving with no vested interest in the long term stability of the projection, it is possible the system would never leave test patterns at all! However, Raskar et al. described a potential solution to such a problem, wherein registration codes are embedded in a projection that are imperceptible to the human eye by interplexing the visible test pattern with the main projection too swiftly for the eye to detect [13]. Grundhöfer et al. proceeded to demonstrate a prototype of such a system. While this technique is not as swift as overt registration, it will not interrupt the displayed image [8]. An alternative approach was put forth by Shirae et al. who proposed the use of an infrared projection grid that the human eye cannot pick up, but the camera can read. This would in theory take less time to calibrate, if the hardware for infrared projection is available in the unit in question [16]. Motion-stabilised augmentation. Not all objects in a scene will remain stationary relative to the projector, and it is often desirable to maintain continuity of an augmented reality projection on that surface despite the movement of the object or the projector. By use of a marker which faces the camera-projector system, Ehnes et al. demonstrated a system which tracked a moving object for the purpose of augmenting its surroundings [4]. Building on this work, Ehnes et al. proceeded to devise a system of overlapping camera-projector units which enable the marked object to move more freely between different camera fields of view, maintaining a consistent augmentation more robust to occlusion [5]. Schöning et al. tackled a slightly different problem, in which the projection surface is a map, and a mobile camera-projection unit is moved over it, projecting labels and information stationary relative to the projection surface despite the movement of the mobile unit. The map itself is not explicitly marked to allow tracking, but by its very nature it provides a wealth of known features to enable tracking relative movement and adjusting the projected image to compensate [15]. Raskar et al. have also demonstrated motion stabilised content relative to the projection surface, using explicit markers to calibrate the projector’s current position [14]. Shape adaptive display. In ad hoc environments, the available surfaces for projection will frequently be of forms other than smooth, planar spaces. When the images will be viewed from multiple angles, the projection cannot be of a format in which only one location sees the projection correctly. This problem has been approached by the generation of a projected image which is warped to the properties of the surface in such a manner that it appears as desired simultaneously from multiple angles without prior knowledge of the surface geometry [3,14]. Radiometric compensation. When projecting upon arbitrary surfaces, not all will be so obliging as to possess smooth, unpatterned textures. When a surface is of an obvious colour or pattern, if uncompensated for, these background factors will heavily influence the perceived projection. Compensating entails modulating the projected image to minimise any visual features of the projection surface. In The Visual Computing of ProjectorCamera Systems Bimber et al described the work done in this field to adapt in real time to changes in projection surface [3].

3. PROPOSED SYSTEM The envisaged system entails using paired cameras and projectors built into mobile devices to carry out on a small scale augmented

reality art such as geometric distortion, moving lights, spurious additional and removal of surface features etc on an ad hoc basis with no prior knowledge or models of the surrounding area. By taking an image of the unmodified projection space with the camera, using radiometric compensation to smooth out the appearance of projections atop it, then projecting the image of the original space, the surface would appear little changed, but would now be much more malleable by manipulation of the projected image. Examples of large scale augmented reality art has been demonstrated on buildings by companies such as UrbanScreen who use 3D models of the façade and design appropriate animations for each building [17].

3.1 Unit Properties In the proposed system, the camera-projector device (hereafter referred to as the unit) would require the following properties: • The unit would possess a camera and projector with a shared field of vision (see figure 2). • The unit would possess an accelerometer. • The unit would possess a wireless means of communicating with other units surrounding it.

• Once the unit has registered the approximate geometric distortion of its surroundings, appropriate animations for each geometric shape would be chosen automatically. Specific animations would be included or absent from the possible animations projected according to pre-selected user preferences, customising the overall feel of the projected space. • Once an object has been assigned an augmentation, the unit will track any movements of the object, maintaining the augmentation’s position relative to the object until the animation is complete or the object passes beyond the unit’s field of vision. • Other objects abruptly intruding on the line of sight to a currently augmented object would generate occlusion, and after a few seconds of stationary presence, the unit would augment the occluding object separately to the originally augmented object.

3.3 Unit Behaviour (Ad Hoc Cluster) When the unit’s field of projection overlaps, with compatible devices, it would behave as in stand alone mode with the following changes and additions:

• The unit would be capable of recognising primitive three dimensional shapes via distortion of a projected registration field.

• When registration begins, if the unit detects the presence of an existing registration grid, the device may then wirelessly negotiate joining the ad hoc projection cluster its visual field is overlapping, and the entirety of the display may gain in size, resolution and clarity.

• The unit would be capable of tracking those three dimensional units throughout its field of vision, in real time with sufficiently small latency to maintain the augmentation’s relative position to the object in question, without explicit markers.

• If the unit is moved, once the five second waiting period has ended, the ad hoc cluster will be notified of the projector’s new relative position if the unit is once more stationary, or notified of the unit’s withdrawal from the ad hoc cluster if not.

• The unit would contain a library of three dimensional effects which are designed for different primitive geometric shapes, which can be scaled, stretched and resized as appropriate to fit onto the objects in the unit’s field of vision.

• Projections originating from multiple users will be permitted to mingle, the individual units negotiating on the fly for free spaces in the communal visual field.

3.4 User Experience

Figure 2. Device layout.

3.2 Unit Behaviour (Stand Alone) When the unit’s field of projection does not intersect any other, it would behave as follows: • Using the accelerometer, the unit would determine when it has been stationary for a small period of time, and begin registration of its surroundings. When the unit is moved, it would suspend projection. If movement ceases within five seconds, suggesting a momentary disturbance, the unit would immediately resume registration and projection. Else, the unit would wait for a sustained period of stillness as before. • The unit would project a registration field throughout all periods when augmentation is active. This would take the form of either temporally encoded imperceptible registration codes, or if the camera is capable of perceiving IR, and the projector capable of generating it, a full time registration grid that is invisible to the human eye. This would be used to register local geometry and surface textures appropriate for projection.

The user would begin by entering a space in which AR visual art is appropriate. Placing a mobile camera-projector device such as a mobile phone down on a surface, AR projection would be enabled by the user. After a moment for the unit to register its surroundings, nearby surfaces suitable for projection begin to be augmented with a selection of the user’s chosen animations. The user possesses the tools to explore their surroundings, transforming them as they see fit to suit their mood, trying out possibilities that change a single backdrop into myriad places to explore. In public spaces, communal visual fields create an overall composite that reflects the people whose mobile devices are helping create the projected illusion. If multiple communal visual fields are visible, the user may choose to change the position of the unit to leave one communal space and enter another, exploring different communities and their influence upon them. The user may find it easier for others to strike up a conversation based on the shared visual experience, and directing a unit at an existing visual display to allow the animations of both users to mingle may noninvasively act as the first step in social contact. Whether in concert with others or merely with their own choice of projected display, each user is helping themselves and others view their surroundings in new ways, making each space a transitive, changing gestalt of those who are present and wish to contribute to it, stimulating fresh interest and imagination.

4. RESEARCH CHALLENGES The system laid out in this paper is not yet possible, but builds on many relevant fields of work which are steadily advancing the capabilities of both individual camera-projector units and their ability to function in concert. The following challenges are particularly important to the eventual realization of such a system, and can be divided into preparation and performance stages.

4.1 Preparation Authoring of AR art. In order for the system to have a large and varied range of expression, it is necessary that as many potential artists as possible are given the tools to create AR animations, customized to as wide a range of surfaces and geometric shapes as can be managed. A toolkit for the production of such AR art would be an important spur in extending the range of the system from the work of a handful of professional AR designers to a wider creative base, and simultaneously spread interest in such projected artwork.

4.2 Presentation Ubiquitous tracking. The recognition of objects with no prior knowledge by visual acquisition is a difficult problem. The challenge is to construct a system which is extremely robust to differences in lighting conditions, materials and surface geometries, recognising an object moving with very little latency. The recognition and handling of sudden occlusion of the tracked object by other objects is an important problem for scenarios which are more unconstrained in the movements and actions of the actors. Gestalt processing. A net of camera equipped devices could possibly handle complex situations with more reliability. Further research is required into ways for a shifting population of devices to form an emergent communal process which is capable of coordinating projections and synthesising varying sensor inputs into a coherent model of the surrounding environment, without being dependent on the presence of any single device and tolerant of sudden changes in device population. Accelerated registration. Acceleration of the registration process for melding multiple projections fields into a singular communal field is another goal which would make the proposed system more flexible and reactive to a shifting population of users. Finding ways to lessen the burden of wireless communications between devices occupied in such communal projections is necessary if the projection process is not to saturate the device’s communication capability. Interaction styles. Whatever the technical strengths of a system, it’s quality is bounded by the user’s ability to operate it. A fluid and intuitive interface for the controlling, shaping and selection of AR projections to transform the user’s space would be essential to allow users of all levels of experience to explore the system to its full potential.

5. CONCLUSION This position paper presented a specific system for the augmentation and transformation of spaces, both personal and communal, as a specific embodiment of a larger research agenda into the use of projection to produce magical effects. A swift summary of current related research led to an exploration of future developments that will enable the vision documented here to

move closer to an implemented system. We welcome the views of the wider research community on the agenda here, and hope that they will guide the future of this work.

6. REFERENCES [1] R.T. Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments 6:4, 355-385, 1997. [2] Bimber, O. and Emmerling, A. Multifocal Projection: A Multiprojector Technique for Increasing Focal Depth. IEEE T VIS COMPUT GR 12, 4, 658-667, 2006. [3] Bimber, O., Iwai, D., Wetzstein, G., and Grundhöfer, A. The visual computing of projector-camera systems. SIGGRAPH '08, 1-25, 2008. [4] Ehnes, J., Hirota, K., and Hirose, M. Projected Augmentation - Augmented Reality using Rotatable Video Projectors. Symposium on Mixed and Augmented Reality, 26-35, 2004. [5] Ehnes, J., Hirota, K., and Hirose, M. Projected Augmentation II — A Scalable Architecture for Multi Projector Based ARSystems Based on 'Projected Applications'. Symposium on Mixed and Augmented Reality, 190-191, 2005. [6] Feng Zhou, Duh, H. B., and Billinghurst, M. Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. Symposium on Mixed and Augmented Reality, 193-202, 2008. [7] Greaves, A. and Rukzio, E. View & share: supporting copresent viewing and sharing of media using personal projection. MobileHCI '09, 1-4, 2009. [8] Grundhöfer, A., Seeger, M., Hantsch, F., and Bimber, O. Dynamic Adaptation of Projected Imperceptible Codes. Symposium on Mixed and Augmented Reality, 1-10, 2007. [9] Kim, S., DiVerdi, S., Chang, J. S., Kang, T., Iltis, R., and Höllerer, T. Implicit 3D modeling and tracking for anywhere augmentation. S. N. Spencer, Ed. VRST '07, 19-28, 2007. [10] Mistry, P. and Maes, P. SixthSense: a wearable gestural interface. SIGGRAPH ASIA '09, 1-1, 2009. [11] Paulos, E. and Beckmann, C. Sashay: designing for wonderment. R. Grinter, T. Rodden, P. Aoki, E. Cutrell, R. Jeffries, and G. Olson, Eds. CHI '06, 881-884, 2006. [12] Paulos, E., Jenkins, T., Joki, A., and Vora, P. Objects of wonderment. DIS '08, 350-359, 2008. [13] Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. The office of the future: a unified approach to image-based modeling and spatially immersive displays. SIGGRAPH '98, 179-188, 1998. [14] Raskar, R., van Baar, J., Beardsley, P., Willwacher, T., Rao, S., and Forlines, C. iLamps: geometrically aware and selfconfiguring projectors. SIGGRAPH '03, 809-818, 2003. [15] Schöning, J., Rohs, M., Kratz, S., Löchtefeld, M., and Krüger, A. Map torchlight: a mobile augmented reality camera projector unit. CHI EA '09, 3841-3846, 2009. [16] Shirai Y., Matsushita M., Ohguro T.: HIEI Projector: Augmenting a Real Environment with Invisible Information. WISS ‘03, 115—122, in Japanese, 2003. [17] UrbanScreen AR Projection website (in German): www.urbanscreen.com accessed on 22nd January 2010