Foldable Interactive Displays - Semantic Scholar

3 downloads 0 Views 2MB Size Report
Oct 22, 2008 - a long way from achieving these technological visions. Many of the displays .... two IR emitters defining the top edge of a rigid square panel. The first fold is .... the implicit privacy states of the display depending on its expected ...
Foldable Interactive Displays Johnny Chung Lee, Scott E. Hudson Human-Computer Interaction Institute Carnegie Mellon University 5000 Forbes Ave., Pittsburgh, PA 15213 {johnny, scott.hudson}@cs.cmu.edu

Edward Tse Smart Technologies 1207-11 Avenue SW, Suite 300 Calgary, Alberta, Canada T3C 0M5 [email protected]

ABSTRACT

Modern computer displays tend to be in fixed size, rigid, and rectilinear rendering them insensitive to the visual area demands of an application or the desires of the user. Foldable displays offer the ability to reshape and resize the interactive surface at our convenience and even permit us to carry a very large display surface in a small volume. In this paper, we implement four interactive foldable display designs using image projection with low-cost tracking and explore display behaviors using orientation sensitivity. Author Keywords

Foldable displays, interactive, mobile, projection, augmented reality, orientation sensitivity, privacy. ACM Classification Keywords

H5.2 [Information interfaces and presentation]: User Interfaces. H5.1 [Multimedia Information Systems]: Augmented Reality.

Figure 1 – Foldable fan display with stylus input.

this paper, we explore this concept of inactive foldable displays and create a number of working prototypes such as the one shown in Figure 1.

INTRODUCTION

In the realm of science fiction, future display technology often depicted as holographic surfaces that float in thin air. Sometimes these displays can be summoned at will in proximity to a person’s body, can be changed in size and shape to fit the desired usage, can be collapsed or dismissed in an instant if the user needs to tend to some other activity, and of course, can support interactive input. While modern displays have become thinner, higher in resolution, and provide input using a stylus or touch sensitivity, we still are a long way from achieving these technological visions.

Emerging technologies such as electronic paper and organic light emitting diode (OLED) displays are expected to provide some degree of flexibility. However, current prototypes remain quite rigid and are typically rectilinear. This prevents them from becoming truly foldable in the sense that we think of paper as being foldable. Additionally, performing input on such flexible displays is an entirely separate technological hurdle. The approach we use in exploring flexible displays is by augmenting the appearance of passives surfaces with image projection [1, 2]. This allows us to combine the flexibility and minimal weight of plain paper or fabric with the dynamic content capabilities of a computer display creating a coherent and fully functional user experience. PaperWindows utilized a similar approach to explore interaction techniques with sheets of paper as if they were digital displays [4], but focused mostly on flat paper interactions in a tabletop scenario and relied on a high-cost motion tracking system for location discovery. Our work demonstrates foldable and re-shapeable semi-rigid hand-held displays using an affordable, low-cost location tracking technology.

Many of the displays we see in hand-held devices today are small LCD displays of fixed shape and size. In this respect, they are insensitive to the desires of a user or the needs of an application. Ideally, we would like displays that we can dynamically reshape or resize to suit our desired usage, similar to the way we might handle a newspaper, or simply so that we are able to fit a large display into our pocket. In

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. UIST’08, October 19–22, 2008, Monterey, California, USA. Copyright 2008 ACM 978-1-59593-975-3/08/10...$5.00.

287

Figure 2 – Foldable display shapes (left to right): newspaper, scroll, fan, and umbrella. Red dots indicate the location of the LEDs use for tracking in these prototypes. Tracking

Tracking is accomplished using infrared (IR) LEDs embedded into the foldable surface and the infrared dot tracking capabilities of the PixArt camera within the Nintendo Wii remote. This camera contains hardware blob tracking up to four points simultaneously with a resolution of 1024 by 768 at 100Hz. Since the computer vision tracking is performed by hardware in the camera, this is a low-cost, easy to implement solution that provides highresolution, low-latency tracking. When positioned adjacent to a projector, the camera tracking data can be calibrated to the image location resulting in spatially augmented reality on moving surfaces [5]. However, camera based tracking has limitations such as the number of distinct points that can be reliably tracked simultaneously, the inability to provide point identity, and manual calibration with the projected image. The four point limit of the PixArt chip reduces the complexity of our foldable geometries requiring simplifying assumptions, such as requiring shapes to have fairly high folding symmetry or equally sized square panels. We take advantage of the semi-rigid properties of our designs and place the IR emitters at strategic locations such that we can fit a geometry model to the data. This model provides estimations of the display edges and corners which may not be explicitly tracked. Assumptions are made regarding display starting orientation, point ID, and the range of movement allowing the model to fit the surface using a small number of points. The fewer points that are available, the stronger these estimations and assumptions must be. However, as the technology evolves providing larger numbers of trackable points, many of these assumptions can be eliminated resulting in more robust performance. If the projector was capable of presenting both visible application content and non-visible structured light patterns [6], large numbers of points could be tracked simultaneously without ambiguous identity further reducing the number of assumptions and spatial limitations.

Figure 3 – Foldable display prototypes at various stages of expansion (top to bottom): newspaper, scroll, fan, & umbrella FOLDABLE SHAPES

In this section, we present four foldable display designs. This is, of course, not an exhaustive list. However, we believe they present a number of expansion and collapsing behaviors likely to be used in a typical foldable display. For each description, please refer to Figure 2 for an illustration and Figure 3 for images of the working prototype.

The LEDs in the display surface run for several hours using a small rechargeable battery pack. Since infrared LEDs emit non-visible light, the LEDs appear as small black dots 5mm in diameter. The LEDs can also be placed beneath a translucent surface to hide their visual presence entirely. As with any optical based tracking system, occlusions are a problem but could be address using additional cameras or location estimation using remaining visible points.

Newspaper

One of the most common formats in which we interact with large sheets of printed material is a typical newspaper.

288

Figure 4 – Orientation sensitivity behaviors (left to right): double-sided display surfaces can react differently depending on the direction they are flipped, simulated lenticular can change the document view depending on the angle of viewing in a hand-held display, or a tabletop scenario where tilt angle may correspond to different privacy states: private, public, excluded.

Sometimes referred to as the broadsheet format, these large sheets of paper are folded in half vertically and then again horizontally allowing a variable visual area ratio of 4 to 1. Additional folds can be added to further increase the magnitude of variability in visual area. In our prototype, we use two folds to support viewing half a page up to two full pages side by side. The user can gracefully increase or decrease the viewing area simply by unfolding or folding the display. Tracking, when viewing a half-page, uses only two IR emitters defining the top edge of a rigid square panel. The first fold is defined by a third point in the bottom left corner becoming visible below. The second fold is defined by a fourth point appearing in the top right corner.

display surface minimizing distortion. Two IR emitters define the outer and inner radius of the display area. The hinge location is estimated using the known physical proportions of the fan. A third IR emitter identified the sweep of the display along an ellipsoid. Umbrella

Another common example of expanding and collapsing a large surface is a parasol or umbrella. These surfaces can frequently be operated by one hand using spring loaded designs and can produce a very large surface area very quickly. Depending on the culture of origin and intended purpose, umbrella and parasol designs vary from parabolic bell shapes, to conical, to nearly planar. Distortions due to non-planar surfaces can be compensated for if the geometry is known before hand. An umbrella design may perhaps not be the most ideally suited shape for interactivity due to the central perpendicular column of the handle. However, the surface area change ratio is very dramatic making it potentially attractive for certain applications and lends itself to rotational input. The handle also provides an optically plausible location for a projection and tracking device for true mobility. When all four IR emitters are visible, a plane is defined which orients a parabolic model.

Scroll

While less common today, large printed material was once transported and viewed in the format of scrolls that could be unrolled. This allowed individuals to not only customize the amount of visible area, but also the location of that area within a long document – hence the concept of “scrolling” a window in a typical GUI environment. By creating a digital display scroll we can change the size and aspect ratio of the viewable area quickly and easily exposing more of the application content. This design can also be collapsed into a relatively small form factor for storage. This is tracked using an IR emitter in each corner and the display is assumed to begin in an upright orientation.

ORIENTATION SENSITIVITY

Though we track gross movements for the purposes of projection and folding, we can also respond to flipping or subtle display movements to create orientation sensitive behavior. The following behaviors are illustrated in Figure 4 and images of the working prototype are shown in Figure 5.

Fan

Folding fans are perhaps one of the best examples of a device that must be very large in surface area to be effective, usable by one hand so that the other hand is free to perform a task, and collapsible for easy storage in a pocket. Coincidentally, these properties are also desirable in a mobile display technology. As a result, this design may be one of the most practical for foldable displays in a mobile scenario. In our prototype the ratio of display area from a fully expanded to fully collapsed configuration is approximately 20 to 1 ranging from 100 square inches to a small strip. The elongated strip can be used to display status messages, rolling text, or progress bars similar to a portable music player. Some folding fans designs allow full 360 degree expansion creating a circular display area. The fan format can either be used in full or partial expansion to vary the amount of screen area desired. While we used a pleated folding fan for our prototype, folding fans can also be composed of parallel slats resulting in a nearly planar

By monitoring the persistent visibility of edges, we can detect when the display has been flipped and in which direction. This is accomplished by tracking emitter visibility and motion modeling to determine probabilistic movement. Emitters are placed along the edge such that they remain visible when viewed from the side but back facing edges will result in disappearance. This provides a way to create lightweight double-sided display surfaces simply by projecting different content on each side. Flipping the surface in different directions (left, right, up, down, or diagonally) can trigger events for document navigation or view manipulation [3]. We can react to more subtle tilting movement of the surface altering the view depending on the angle at which is it held.

289

This is done by monitoring the ratio of the edges of a semrigid surface. We refer to this as a simulated lenticular screen and can respond to multiple axes of orientation change. In a multi-user scenario, horizontally tilting or turning the display to another person (assuming the computer is aware of viewer locations) can trigger different document views. If placed on a table top, vertical tilting may correspond to the implicit privacy states of the display depending on its expected visibility. Tilting the display toward yourself might indicate a private state, placing it flat would indicate a public state, and tilt it away might indicate an excluded state, possibly useful for presentation or gaming scenarios. Use of this capability will depend on the application scenario of the table top system. While it would be possible to have the system react to additional degrees of rotational freedom as well as three degrees of translational movement, having such complex display behavior would be highly application dependent. For example, six degree-of-freedom display tracking would be appropriate for creating a view portal into a virtual 3D environment but unnecessary for many 2D GUIs. Figure 5 – Orientation sensitivity prototypes: (top) doublesided flip direction, (middle) vertical and horizontal simulated lenticular in a hand-held surface, and (bottom) a tabletop scenario where tilt correlates to different privacy states.

INTERACTIVITY

By tracking additional dots over those embedded into the display surface, we can track a stylus for input shown in Figure 1. Since our fan design used only three IR emitters, we were able to use the fourth point to represent an input stylus. The stylus contains a button to activate the emitter providing a passive method of detecting clicking and dragging. The location of the cursor is mapped to the 2D location on the foldable display. Alternative style designs could utilize a tip switch, but this will vary performance when using different display materials. This technique provides an easy way to obtain interactivity on all of the surfaces described including their double-sided variants. High-speed IR data transmission could be used to transmit additional data from the stylus such as left and right click, or orientation data. While it would be possible to support multiple cursors, a camera based approach may lead to tracking confusion when the pens approach each other or the emitters defining display boundaries. However, projector-based tracking would allow many cursors to be used simultaneously without ambiguity [6].

manner. As tracking technologies improve, surface modeling assumptions can be reduced improving robustness. With the emerging flexible display and portable projection technologies, foldable designs may become practical in portable consumer electronics in the near future. ACKNOWLEDGMENTS

This work was supported in part by grants from General Motors, the Intel Research Council, and the National Science Foundation under Grant IIS-0713509 REFERENCES 1. Badyopadhyay, D., Raskar, R., Fuchs, H., “Dynamic Shader Lamps: 2. 3.

CONCLUSION

In this paper, we explored the concept of interactive foldable displays by presenting four functional prototype designs featuring different folding behaviors. The displays react to the user’s movement, expansion state, orientation, and input styli. Foldable displays offer the ability store large display surfaces in a small volume allowing users to quickly expand and collapse them when convenient.

4.

5.

6.

While the current state of projection technology is not yet practical for supporting spatially augmented reality in truly mobile scenarios, this exercise allows us to explore interacting with high-fidelity prototypes in a low-cost

290

Painting on Real Objects.” IEEE ACM Int. Symposium on Augmented Reality (ISAR), New York, 2001. Brown, L. et al, "A Widget Framework for Augmented Interaction in SCAPE" Proceedings of the ACM Symposium on User Interface Software and Technology, October 2003. Chen, N., Guimbretiere, F., Dixon, M., Lewis, C., Agrawala, M., "Navigation Techniques for Dual-Display E-Book Readers" CHI 2008. Holman, D., Vertegall, R., Altosaar, M., Troje, N., Johns, D., "Paper windows: interaction techniques for digital paper" Proceedings of the SIGCHI conference on the human Factors in computing systems, 2005. Gupta, S. Jaynes, C., "Active Pursuit Tracking in a Projector-Camera System with Application to Augmented Reality" IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005 Lee, J., Hudson, S., Dietz, P. "Hybrid Infrared and Visible Light Projection for Location Tracking" Proceedings of the ACM Symposium on User Interface Software and Technology, October, 2007.