Body-based interfaces - CiteSeerX

ARTICLE IN PRESS

Applied Ergonomics 35 (2004) 263–274

Body-based interfaces Gerard J. Kima, Sung H. Hanb,*, Huichul Yangb, Changseok Choa a

Department of Computer Engineering, Pohang University of Science and Technology, San 31, Hyoja-Dong, Nam-Gu, Pohang, Gyungbuk 790-784, South Korea b Department of Industrial Engineering, Pohang University of Science and Technology, San 31, Hyoja-Dong, Nam-Gu, Pohang, Gyungbuk 790-784, South Korea Received 9 September 2003; received in revised form 5 February 2004; accepted 20 February 2004

Abstract This research explores different ways to use features of one’s own body for interacting with computers. Such ‘‘body-based’’ interfaces may find good uses in wearable computing or virtual reality systems as part of a 3D multi-modal interface in the future, freeing the user from holding interaction devices. Four types of body-based interfaces have been identified: Body-inspired metaphor (BIM); Body-as-interaction-surface (BAIS); Mixed mode (MM); and Object mapping (OM). These four body-based interfaces were applied to a few different applications (and associated tasks) and were tested for their performance and preference. It was generally found that, among the four, the BIM exhibited low error rates, but produced relatively longer task completion times and significant fatigue. The BAIS method had the contrasting character of higher error rates, but shorter task completion times and lower intuitiveness. The OM method exhibited high error rates, longer completion times, and much fatigue. Overall, the MM was superior in terms of both performance and preference as it combined the merits of the above three methods. Thus, it is expected, for applications with many associated tasks, a careful division of tasks among those that have natural semantic links to body parts and those that do not, is necessary to design the most performing body-based interface. r 2004 Elsevier Ltd. All rights reserved. Keywords: User interfaces; Wearable computing; Virtual reality; 3D multi-modal interfaces; Human–computer interaction; Body-based interaction; Metaphors

1. Introduction While the science of designing interaction models and implementing appropriate interfaces has matured for 2D tasks in desktop environments (Preece et al., 1994; Shneiderman, 1998; Wickens et al., 1997), much research is still on-going for the 3D counter part. The 3D multi-modal interfaces are particularly important for fields such as virtual reality and wearable computing. However, no definite interaction design guideline has emerged as yet. The main reason behind the slow progress in 3D multi-modal interface research seems to lie in the complexity raised from the high number of possible interaction methods and modalities (e.g. finger/ arm/body gestures, voice, 3D motion, gaze, haptics, etc. vs. mouse-based simple drag and click), and complications with the non-standardized and unreliable tracking *Corresponding author. Tel.: +82-54-279-2203; fax: +82-54-2792870. Email-address: [email protected] (S.H. Han). 0003-6870/$ - see front matter r 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.apergo.2004.02.003

devices and recognition technologies. However, many of the lessons and principles learned from the 2D interface design would still apply to 3D interaction design as well: for instance, ergonomic design according to the Fitts’ law, use of good metaphors for user’s fast recognition and interactive response, consideration of human’s short-term memory, etc. This paper starts from a simple idea of making use of various parts of the body as metaphorical interaction objects. We use parts of our body for various purposes (including communication) everyday, and are quite familiar with context-dependent functionalities (or meanings) that can be abstracted for using them naturally as multi-purpose metaphoric objects for interaction. For instance, our eyes may represent something that is related to seeing, such as, opening/closing a file, invoking an image viewer or video player, following a hyperlink, zooming in or out, etc. Through body parts, we can take advantage of the proprioception and/or passive haptics, while eliminating the usually wired and unfamiliar intrusive devices (e.g. 3D mouse, spaceball,

ARTICLE IN PRESS 264

G.J. Kim et al. / Applied Ergonomics 35 (2004) 263–274

wand, bat). In this paper, we explore how human body parts can be used for effective interactions with computers. Wearable computing would be one of the ideal applications for such an interaction method. The idea of ‘‘wearable’’ computing, perhaps started as a novelty idea about 10 years ago, is gradually becoming a reality and attracting more attention from the industry. A number of companies including IBM and Hitachi have introduced self-contained body-wearable computers complete with a special display and interaction devices (IBM, 2002; Xybernaut, 2002; Miyake, 2002). This not only reflects the growing needs of wearable or mobile computing in fields such as military, service business (e.g. delivery, machine maintenance, insurance), but also is an acknowledgment that one day it may even enter the realm of everyday computing (like we use our PDA’s and cellular phones) (Baber and Baumann, 2002). Ockerman and Pritchett (1998) cited many possible uses of wearable computers: medical applications (e.g. surgery, EMT); aiding inspection and maintenance workers; navigation (e.g. tour guides and military applications); communication (e.g. military operations, remote collaboration); and memory aids in everyday life. Researchers are pushing to make interaction and computing devices ‘‘wearable’’ in the real sense by miniaturizing and integrating them into clothing or even planting them into our bodies using bio-interfacing technology (Kovacs, 1998). It is projected that it will become feasible to sew basic computing elements (e.g. processing, input/output, and memory units) into our clothing or body parts in the near future (within 5–10 years) (Wakefield, 2001). Virtual reality (VR) is another good application for body-based interfaces. One of the defining goals of virtual reality systems is to create the feeling of being in the environment, called the ‘‘presence’’, and one cause of breaking presence is the existence of intrusive wired sensing devices. While body-based interfaces may not increase realism (since they do not exist in the real world), they may still find good uses in imaginary virtual worlds for increasing self-awareness through self-interaction. According to Nichols (1999), the difficulties in VR hand-held input device include non-intuitive design (poor mapping) and requirements for unnatural postures. Those input devices showed a lack of satisfaction with VR use and lowered effectiveness of virtual environments. Body-based interfaces can realize intuitive mapping (especially, when using body metaphors) and natural hand movements and postures as well. In this context, this research explores different ways to use features of one’s own body for interacting with computers. Four types of body-based interfaces have been identified: Body-inspired metaphor (BIM) uses various parts of the body as metaphoric interaction;

Body-as-interaction-surface (BAIS) simply uses parts of the body as points of interaction; Mixed mode (MM) mixes the former two; Object mapping (OM) spatially maps the interaction object to the human body. In addition to classification of various ways to use our body features for interaction, the purpose of this study is to compare their relative merits and shortcomings, and thus derive an ergonomic guideline for body-based interaction and, perhaps future wearable computer design. As such, usability experiments have been conducted, having participants carry out several interaction tasks using the four styles of body-based interfaces.

2. Related work A sizable body of work has emerged in the 3D and multi-modal interaction and interface design for virtual environments recently, and a good review can be found in Bowman et al. (1999). In their study, three major tasks were classified for which interaction must be designed for 3D virtual environments, namely, selection, navigation, and manipulation (sometimes, the fourth task, system control, is added to this list). Most other tasks can be described as combinations (or composite tasks) of these three major primitive tasks. Many different interaction models and interfaces have been proposed for these three primitive tasks (Bowman et al., 1999). Designing interaction models and interfaces in the context of VR is usually difficult because there are many modalities, devices, and sensing/ recognition technologies to consider. In addition, VR adds another dimension, presence, to the problem of designing the most usable and performing interaction scheme. Standard interaction design techniques or principles are often employed, such as the use of metaphors and multi-modality, ergonomic consideration, etc. (Bowman et al., 1999). One notable direction was given by Mine et al. (1997) who exploited the sense of proprioception in improving usability by using automatic scaling to bring objects within reach for manipulation. Lindeman et al.’s (1999) work comes close to using parts of one’s body (and thus proprioceptive) as the contact surface for interaction. In fact, his work considered using a physical pad (held in one hand) on which virtual interface would be laid and manipulated by the other hand. The pad can be viewed as an extension of the body, and be replaced by a certain part of the body as well. The research showed that the effective use of proprioception and passive haptics improved the overall usability of the interfaces. A work by Fishkin considered using body-attached sensors to activate music, which naturally induced a dance like interaction for invoking music (Harrison et al., 1998).

ARTICLE IN PRESS G.J. Kim et al. / Applied Ergonomics 35 (2004) 263–274

However, these are very simple forms of body-based interactions without any use of metaphors. A study by Lehikoinen (2000) from the Nokia Research Center is similar. He suggested the concept of virtual pockets for opening and saving documents in a wearable computing setting. Virtual pockets are physical pockets augmented with pressure sensors on one’s clothing woven using a special material for tracking finger position on the clothing surface. A user can move files between different pockets, a process analogous to the dragging and dropping in the familiar desktop environment. Using finger pressure, files can be opened or saved. This can be viewed as mapping the desktop space onto the front surface of the upper body. In another study by Fishkin et al. (2000), it was proposed to design a device-embodied task in relation to some familiar analog tasks that users were skilled at performing. For instance, to embody a page-turning task in a computational device, the device must naturally represent as many of the critical properties of the analog task as possible, such as providing a way to flick the page in the upper right or upper left corner of the display. On the other hand, Zhai and Milgram (1993) have considered improving the ergonomics of interaction devices to increase usability of VR interfaces. Bodybased interfaces give new twist to ergonomic design of interfaces as the interface is the human body (or part of) itself.

3. Body-based interfaces 3.1. Why body-based interface?

It is the year 2015. Gerry is planning for a family vacation in Florida. He goes into his living room wearing the latest sensor equipped sweatshirt. He uses his wrist watch/PDA to retrieve the site address and lightly touches his ‘‘eyes’’ to bring up the 3D display of the latest attraction at the RideWorld. He goes on to make a reservation for the ride for his family by touching his back pocket to retrieve personal information and motion an approval to pass it on to the on-line reservation system. He got somewhat tired reading about the ride on the screen, so by touching his ear, the information is brought into audio mode and spoken to him. He remembers that his friend, Tom, may want to check upon this site, so he pulls down a menu by pressing his the top (shirt) button, then presses the (shirt) third button to send out the site address to Tom’s wrist PDA. This futuristic story illustrates the possible merits in taking advantages of our body features for interacting

265

with computers. The story may not be too far fetched as most of the technologies required to make this story a reality is already mostly possible, if not readily available (Shankland, 2001; Kovacs, 1998; DDD, 2002). The metaphorical body part (used as interaction objects) nicely augments the functionality of the wrist PDA obviously because the user can directly invoke the often used commands by easily remembering the mapping relationships. Even though voice/gesture recognition technologies have come a long way, it still suffers from user dependency and the burden of having to remember particular keywords or gestures. The physicality of interfaces (as offered in the body-based interfaces) has been shown to improve usability through many different studies (Ullmer and Ishii, 2000; Lindeman et al., 1999; Insko, 2001; Hinckley et al., 1997) compared to, for instance, the direct manipulation of virtual interfaces. In any case, it seems inevitable that a certain type of body-based interfaces will be employed with or without the conventional interfaces. Note that the concept of body-based interaction can be certainly extended to ‘‘everyday objects’’-based interaction, as suggested by Fishkin et al. (2000) and Ullmer and Ishii (2000), although it is expected that the ease of metaphorical or spatial mapping will decrease as the place of interaction gets farther from the body (both mentally and physically). 3.2. BIM The first type of the body-based interaction is the BIMs. Metaphors, in the context of human computer interaction, are entities that are deliberately designed to be easily manipulable (for its familiarity, concreteness, abstraction, etc.) in order to provide more intuitive control of a certain task. Metaphors we use everyday are mostly visual (e.g. icons), although other modalities such as textual, semantic, aural, and even haptics would be possible. Metaphors help users build a mental model of computer systems, and the knowledge about a familiar domain in terms of elements and their relation to each other is mapped on to elements and their relations in the unfamiliar domain or task (Preece et al., 1994). The success of using metaphors hinges upon the degrees of matching the user expectation to what the interface object should and should not do (Preece et al., 1994). It would be preferable if an interface could be designed in a way so that a metaphorical consistency exists across different applications (e.g. dragging and dropping applied both to simple file moving and invoking a program with file input). We (humans) are very familiar with our own bodies; after all, we have used them (in many ways) throughout our life times. We often make gestures out of human body parts to express ourselves (e.g. knocking one’s


266

head) and communicate ideas (e.g. placing one’s hand on the chest). Literature is full of (metaphorical) personification of non-human objects and abstract ideas. Therefore, body parts naturally lend themselves for various metaphors. The wealth of possible semantics applicable to body parts may allow its application to various tasks and domains with consistency. Table 1 shows a short list of possible applications of BIMs. Prior to the experiment, the associations between the applications and the body parts were chosen through a simple user survey to make them as intuitive as possible. As a result, the eyes, ears, mouth, head, pocket (chest), and back pocket (hip) were matched as points of invocation for applications such as the folder operations, control panel, MP3 player, voice recorder, PowerPoint, and e-wallet (See Section 5.4 and Table 4 for the details). 3.3. BAIS: ‘‘beam me up, Scotty’’ For a lack of a better term, ‘‘BAIS’’, uses parts of the body as the place of interaction, that is, where the physical contact actually occurs. Fig. 1 shows a case where the left forearm is used as the interaction surface. While Fig. 1 shows a simple case of installing a few number of switches, other variations are certainly possible, for instance, like the ‘‘tracking’’ suite introduced in Lehikoinen (2000), where a special material is used so that finger movements are actually tracked on the whole clothing surface, or as in Lindeman et al. (1999), where a user holds a tablet with a tracker as an extension of the body. In this model, we try to find the ergonomic position on the body for the location of the interface for a given task. The most plausible location seems to be the forearm of the non-dominating hand for its mobility, accessibility from the dominating hand and visibility, although other parts of the body may be considered such as on the lap (especially when sitting down), or on the chest (as in the famous ‘‘beam me up Scotty, touching the chest pin motion’’ in Star Trek), and so forth.

3.4. MM The disadvantage of the BIM is that it usually would be difficult to find a complete and intuitive set of metaphors for given tasks. Only few tasks will lend themselves to the use of body parts in a metaphorical way. For the other tasks that cannot find any bridge to the semantics of the body parts, we can resort to the BAIS, and combine the first two approaches appropriately. For example, imagine an application for seeing presentation materials like the Microsoft Wordt. The left or right side of the body can naturally and intuitively be mapped to the task of moving the cursor to the left or right side of the text. However, it would be very difficult to map a task like ‘‘Go to the 13th page’’ to a certain body part. This can be overcome by the BAIS. Therefore, when there are many tasks associated with an application, since it is likely that some of them will lend themselves to a body mapping for similarities in the semantics carried and some will not, an approach to mix the methods of OM and BAIS would be most appropriate. 3.5. OM The OM method is a special kind of BIM based on spatial mapping of the whole body. That is, the user

Fig. 1. Body as interaction surface.

Table 1 Body-inspired metaphors Body parts/clothing

Original semantics

Applicable tasks

Eye(s)/eye glass Mouth Ear(s)/ear rings Head Hand(s)/fingers Leg(s) Skeleton Buttons Pockets

Seeing, window Eating, speaking, blowing Hearing, collect info Importance, thought Hold Locomotion, support Hierarchy, structure Switch/key open/close Containment, safe

Presentation, check emails, viewers, turn on TV, activate window Beep, take items (in games), coloring (by blowing) Play sound, increase volume, join in mail list Homing, bring out notepad Temporary storage Navigation, constraining Avatar control, data traversal Activation Folders, trash can, file transfer


Fig. 2. OM: user looking at himself through a mirror and seeing that an MP3 player has been spatially mapped on his chest. The user’s virtual right hand is seen in the mirrored image clicking on the play button.

becomes the object to be interacted with or to be manipulated with, and thus the user interacts with the object by interacting with oneself. For instance, if the user was to activate a virtual cassette player located somewhere in the virtual space, the user body would be mapped to that of the virtual cassette player spatially (i.e. his position in the virtual space would be transported to and become coincident with that of the cassette player). The switches (play, stop, record, and pause buttons) would be located on one’s body through relative scaling operation between the two entities. The viewpoint could be attached anywhere (e.g. as seen from the view of the cassette player or as seen from some other convenient third person viewpoint). To activate the play button, the user would have to use one’s hand/ finger to find and touch (by looking through the virtual display) the part of his body to which the play button was mapped (see Fig. 2). This model seems particularly useful for when interacting with ‘‘far-away’’ objects. For interacting with far-away objects, a few other methods have been suggested, such as the extended hands, ray casting, World-In-Miniature (WIM), automatic scaling, and so forth (Bowman et al., 1999). OM is different in that the user ‘‘transports’’ the location of the object by becoming it, and manipulates it from the first person viewpoint both physically and mentally. This is somewhat similar to the BAIS, however different in that the user actually becomes the whole object, not just the placement for the intersection medium (like the buttons, handles and menu items).

267

the four different styles of body-based interfaces. The purpose of the experiment is not to compare the effectiveness of body-based interfaces to other conventional interfaces such as using 3D devices (e.g. 3D mouse, SpaceBall, bat, wand), voice/gesture recognition, etc. It was hypothesized that the interaction types could affect the objective performance of the tasks such as the task completion time and the number of errors, and the subjective preference such as intuitiveness, perceived speed, ease of use, and fatigue level. In addition, the effect of the interaction types on performance and preference could be different according to the extent of user’s familiarity with them. Therefore, after a week from the first experiment session, the second experiment was conducted. Before explaining the specific tasks and experimental methodologies, the experimental set-up is described as we had to resort to conventional technologies to ‘‘imitate’’ the future wearable body-based interfaces. 4.1. Implementation for experiment set up Since it was not possible to have an access to a technology to embed specially made ‘‘micro-switches’’ into our body or into the fabrics of our clothing, an ad hoc wearable interface was implemented using conventional wired trackers to imitate the true form of wearable computing in spirit. The user wore mainly three devices, a headset, an interactive shirt and long-sleeved gloves. Fig. 3 shows the headset consisted of a dummy eyeglass (with no particular optical function), headphones, and extended microphone, on which small contact wireless switches were mounted for the interaction. The contact switches were very small (approximately quarter inch) and emitted on/off events using infrared signals. Similarly, the interactive shirt was made of a simple T-shirt with the same type of contact switches mounted on various strategic locations such as on the shoulder, chest, and so forth. Two wired trackers were attached on the gloves to track the user’s hand position, and for the left side, a

4. Experiment This section describes an experiment conducted to compare the performance and subjective preference of

Fig. 3. The three devices, headset, interactive shirt and the gloves, used in the experiment.


G.J. Kim et al. / Applied Ergonomics 35 (2004) 263–274 Table 2 Three applications and associated tasks

Fig. 4. User touching one’s ear to invoke an MP3 player and hear a song.

number of contact switches were attached on the sleeves of the glove (for body-based interaction using the forearm). Upon interacting with their body parts, using these devices, a selected application was visualized on the display appropriately (see Fig. 4). In summary, for the case of BIM, contact switches were attached on various body positions like on the eyes (eyeglass), ears (earphone), mouth (microphone), chest (shirt on the pocket), shoulders, and pockets. On the other hand, for the case of BAIS, the contact switches for the all tasks were put on the left forearm. For the case of MM, the user could use both the BIM and BAIS switches. For the case of OM, an optical mouse was used to map the user’s chest to a mouse pad (that is, to track the user’s hand on the surface of one’s body) so that the tasks could be performed in the same way with a traditional mouse operation. 4.2. Experimental tasks and participants Using the devices to roughly simulate the concept of the body-based interactions, participants were asked to interact with three different applications (that boiled down to a number of tasks) in different interaction modes in the experiment. The three applications were PowerPoint (application 1), MP3 player (application 2), and a simple desktop manager (application 3). The specific tasks are shown in Table 2 for each application. The table also lists the parts of the body used when the BIM was used. For the BAIS case, the contact switches located on the forearm (with the extended glove) were used. The MM employed both the BIM and BAIS so that the participants were free to use any of them. As for the OM case, only the MP3 application and associated tasks were tried, as it was the only application that resembled to being a spatially mappable object. There were 15 participants, all students (both undergraduate and graduate whose average age was 26) and

Application

Tasks

Body-inspired metaphor

PowerPoint

Start/end Next slide Previous slide

Touch the eye Right shoulder Left shoulder

MP3 player

Start/end Play Stop Next song Previous song

Touch the ear Touch the mouth Touch the chest Right shoulder Left shoulder

Desktop manager

Open folder Log in

Touch a pocket Touch the back pocket

12 were male and 3 were female. The participants were also classified, according to their experience with virtual reality systems (e.g. exposure to 3D graphics and familiarity with special devices), into novice (first time VR users), intermediate (less than 5 exposures) and expert users, and this was factored into the analysis later. As a result, five, four, and six participants were included in each experience category, respectively. 4.3. Experimental design and procedure Three types of applications and four interaction styles, as mentioned above, were manipulated in the experiment. The ordering of the experimental conditions was randomized to avoid any systematic biases. The experiment was repeated twice in 2 weeks (1 week apart) to observe any changes in the user performance over time. During the instruction session, the participants were first briefed about the experimental tasks, and given a chance to play with the devices (e.g. interactive shirt, the long-sleeved glove, etc.) to become familiar with them. During the experimental session, the participants were instructed (through the computer display) to invoke each application (in a random order), using the four body-based interactions. Once the application was started, the participant was instructed to invoke a number of associated tasks of the application (see Table 2). Certain tasks were asked to be invoked more than once. Thus, each experiment section consisted of a total of 29 tasks from 3 applications without any break. As for the OM case, only the six tasks of the MP3 player application were examined (20 times) by the participants. As for the second experiment (1 week later), the exact same procedure was repeated except for that the participants were not briefed nor had a chance to practice with the devices. The task completion time and the number of errors were measured. The participants were given questionnaires (for both experiments) asking about the intuitiveness, perceived speed, ease of use, and fatigue level of the


interfaces. Each experiment session took approximately 40 min to complete.

5. Results The analysis of variances (ANOVA) and Chi-square tests were used to analyze the experimental data. The ANOVA was used to analyze the task completion time and subjective preference measures, while the Chisquare test was used to analyze the number of errors. All the tasks listed in Table 2 were performed using the first three interaction styles (BIM, BAIS, and MM). OM was just applied to and tested only on the MP3 player application, as it was the only application that possessed visual and physical qualities for establishing a physical mapping with the human body. Therefore, in the following analysis, when comparing ‘‘all’’ the four interaction styles, only the results (e.g. average time and error rate) from the tasks of the MP3 player were used in the ANOVA. After ANOVA, mean difference tests across conditions in the significant factor, i.e., post-hoc analysis, were performed using the Student–Newman–Keuls test (a ¼ 0:05) (Montgomery, 2001). Simple effect tests (Keppel, 1991) were employed to examine significant interaction terms. 5.1. Task completion time

completion times and the standard deviations across interaction styles. Numbers on the figures represent the mean task completion times (in milliseconds). Different shades of gray were used to represent groups resulting in significant differences in the mean task completion time by the Student–Newman–Keuls test (a ¼ 0:05). For example, the difference between BIM and BAIS was not statistically significant in terms of the task completion time as shown in Fig. 5. When the MM was used as the interaction style, it took significantly less time to complete the task. Further analysis on the interaction between experiment session and interaction style showed that the effect of interaction style on the task completion time was significant in both first experiment session (F ð2; 28Þ ¼ 9:815; po0:001) and second experiment session (F ð2; 28Þ ¼ 4:736; po0:02) (see Fig. 6). In the first session, BAIS was the slowest among the three interaction styles, while BIM was slower than any other interaction styles in the second session. Data obtained from the tasks of the MP3 player application were used to compare all the four interaction styles. This is because the OM was used only for the MP3 player application. An average task completion time for each task was used in the analysis, because the number of tasks (of the MP3 player) performed was different for each interaction style. ANOVA showed the results similar to those presented above. In other words, the session (F ð1; 14Þ ¼ 41:83; po0:0001), interaction style (F ð3; 42Þ ¼ 7:08; po0:001), and interaction between session and interaction style (F ð3; 42Þ ¼ 9:2; po0:001) were all significant at the 0.05 significance level. It also took less time to complete the task in the second experiment session. MM was the fastest among the four interaction styles. A simple effect test showed that the effect of the interaction style on the task completion time was significant in both experiment sessions. BAIS was the slowest among the four interaction styles in the first experiment session. In the second experiment, however, BIM and OM were included in the slower group, and BAIS jumped to be one of the fastest. Fig. 7 shows the

100000 80000 74572

75219 BIM: Body-inspired Metaphor

60000

65819

40000

BAIS: Body-as-interaction-surface MM: Mixed mode

20000 0 BIM

BAIS

MM

Interaction style

Fig. 5. Mean task completion times and standard deviations across interaction styles. For all tasks and three interaction styles except OM. (Different shades of gray means that differences are statistically significant at a ¼ 0:05:)

Task completion time (ms)


The results showed that the task completion time was influenced by the experiment session (F ð1; 14Þ ¼ 15:46; po0:002), interaction style (F ð2; 28Þ ¼ 6:14; po0:007), and interaction between the interaction style and experiment session (F ð2; 28Þ ¼ 7:14; po0:005) at the 0.05 significance level. OM was not included in this ANOVA because it was tested with MP3 player application only. The task completion time in the first session (78,121 ms) was significantly more than that in the second session (65,619 ms). Fig. 5 presents the mean task

269

90000

87267 Session1

80000

76756 72388

Session2 70339

70000

BIM: Body-inspired Metaphor 63171

61298

60000

BAIS: Body-as-interaction-surface MM: Mixed mode

50000 BIM

BAIS MM Interaction style

Fig. 6. Mean task completion times across sessions and interaction styles. For all tasks and three interaction styles except OM.


270

differences in mean task completion time across the two sessions and four interaction styles. 5.2. Number of errors It was found that the number of errors was significantly affected by the interaction style (Chisquare(2)=100.1, po0:0001). Note that the OM was not included in this analysis. The largest number of errors occurred with the BAIS among the three tested methods. As in the analysis for the task completion time, the data obtained with the MP3 player application were used only to compare all the four interaction styles. A Chi-square test showed that only the interaction style had a significant effect on the error rate (Chisquare(3)=68.4, po0:0001). Again, more errors occurred with the BAIS than the three other interaction styles. Fig. 8 presents the mean number of errors for each trial and standard deviations across the four interaction styles (i.e., BIM, BAIS, MM and OM). Numbers on the figures represent the mean number of errors for each trial (note that the number of trials was different from one another, because the tasks of the applications were randomly selected). 5.3. Subjective evaluation

2960

3000

Before the first experiment session, the participants were asked to name body parts most appropriate for invoking many applications. Eyes, ears, mouth, head, pocket, and back pocket were used as choices of bodyparts to name. Folder operation, control panel, MP3 player, voice recorder, invoking PowerPoint, and ewallet application (e.g. giving out credit card information for an on-line commerce activity) were given as applications to match with the available body parts. The results showed that, for certain applications, there existed overwhelming and unambiguous associations to particular body parts. These results also provided the basis for choosing particular body parts for metaphors of the applications used in the experiment (see Table 4).

Session 1

2500

2235 2002

2000

Session 2

2543

2446 2283

2234

1956

MM: Mixed mode OM: Object Mapping

1500

BIM


OM

80

82.7

73.9

60

69

BIM: Body-inspired Metaphor BAIS: Body-as-interaction-surface

46

40


20 0 BIM


OM

Fig. 9. Mean intuitiveness and standard deviations across interaction styles (Different shades of gray means that differences are statistically significant at a ¼ 0:05).

0.2 0.15


0.1


0.0686

0.05 0

100


Fig. 7. Mean task completion times across sessions and interaction styles. For tasks of MP3 Player only and all four interaction styles including OM.

Number of errors

5.4. Intuitiveness and recall test

Intuitiveness


After the second experiment, the participants were asked to rate the intuitiveness, perceived speed, ease of

use, and fatigue level for each interaction styles on a 1–100 rating scale. Significant differences were found with the intuitiveness (F(3,42)=11.93, po0:0001), ease of use (F(3,42)=13.58, po0:0001), and fatigue level (F(3,42)=5.27, po0:005) across the interaction styles at the a level of 0.05. However, no significant differences were found on the perceived speed among the four interaction styles. A post-hoc analysis showed that BAIS exhibited relatively lower intuitiveness and ease of use than other interaction styles. However, BIM and MM produced a higher level of fatigue than BAIS. Fig. 9 shows the mean differences and standard deviations across the interaction styles on the intuitiveness. Table 3 shows mean scores and standard deviations of speed, ease of use, and fatigue.

0.0076

BIM

0.0113

BAIS

MM

0.0067

Table 3 Subjective rating scores of speed, ease of use, and fatigue Interactions styles

OM

Speed

Ease of use

Fatigue

Mean

SD

Mean

SD

Mean

SD

69.2 72.5 76.9 72.7

13.9 21.0 12.7 19.4

83.9 56.7 76.9 85.1

11.4 19.1 13.1 9.9

71.7 48.0 63.7 55.1

18.9 23.7 21.0 23.5

Interaction style

Fig. 8. Mean number of errors and standard deviations across interaction styles. For tasks of MP3 Player only and all interaction styles including OM. (Different shades of gray means that differences are statistically significant at a ¼ 0:05:).

BIM BAIS MM OM


271

Table 4 User’s association of tasks and applications to body parts (15 participants)

Body parts

Folder

Eyes Ears Mouth Head Pocket Back pocket

1

Control panel

PowerPoint

MP3 Player

Voice recorder

13 (87%)

1 12 (80%) 2

3 12 (80%)

1 1 13 (87%) 1

E-Wallet

14 (93%)

For example, 13 out of 15 participants told that eyes were well associated with PowerPoint, and likewise, ears with the MP3 player. We also asked the participants to recall the functionalities of the contact switches used in various interaction styles. The recall test was carried out right before the second experiment session. The participants remembered the functionalities of, on the average, 7.5 (out of 8) contact switches used in BIM compared to that of 5.4 switches used in the BAIS case. A Chi-square test was performed to test the differences between BIM and BAIS. Because 15 participants and 8 contact switches were employed in the recall test, there were 120 trials for each interaction style. There was a significant difference in the number of correct recalls between the interaction styles (Chi-square(1)=27.5, po0:001). This implies that BIM was easier to remember than BAIS. This can be anticipated from the results of the subjective evaluation of intuitiveness and body-task association test. 5.5. Participants’ profiles The differences of performance and preference across experience levels of the participants were analyzed. First, novices were found to spend more time (157,227 ms) completing the task than other user groups (135,569 and 133,723 ms for experts and intermediate users, respectively). On the other hand, the experience level did not affect the number of errors and preferences. We also analyzed whether the differences in the performance and preference were significant across the interaction styles in the novice, intermediate and expert groups or not, i.e. we were interested which, for example, interaction style was most preferred in the novice group. The data were divided into those for the novice, intermediate and expert groups. The mean data, obtained when the participants performed the MP3 player, were used in the analysis to compare all the four interaction styles. It was found that there were little differences across the experience level in subjective measures except for the reported fatigue level. Experts did not feel any differences among the interaction styles in terms of fatigue. However, novices complained that BIM was

1

1 14 (93%)

more difficult to work with ‘‘physically’’. This is in conflict with the general founding that BIM was easy to use. One possible interpretation is that BIM is ‘‘mentally’’ easy to use (e.g. the user knows what he has to do to accomplish the task), but it is difficult to carry it out physically (e.g. due to hand moving distances, the experimental set up and cumbersome devices). Among the novice and expert groups, the performances were found to differ in terms of both time and error. In the expert group, BAIS was in the slowest group of interaction styles, with much more errors. In the novice group, BIM and OM were in the slowest, but the number of errors was not significantly different across the four interaction styles. 5.6. Correlation between objective and subjective measures The correlation between objective performance and subjective preference measures were also analyzed (Nielsen and Levy, 1994) to examine whether user performance was the best with the most preferred interface. Mean values across interaction styles were used in this analysis. Task completion time was lowly correlated with all the subjective measures, i.e., intuitiveness, speed, ease of use, and fatigue. However, the number of errors was highly correlated with intuitiveness and ease of use. Their correlation coefficients were 0.93 and 0.98, respectively. Fewer errors occurred in the interaction styles that the participants thought intuitive and easy to use. For example, the participants thought that BAIS was the most difficult to use, and actually committed much more errors (see Fig. 10). Table 5 shows all the correlation coefficients between objective performance and subjective preference measures.

6. Discussion The task completion time in the second experiment session was lower than that in the first most probably because of the participants’ learning effect. Yet, the number of errors did not decrease. This conflicting result may be explained by the fact that the participants


272

Mean number of errors

0.08 BAIS 0.06 R2 = 0.9554 0.04 0.02

BIM

MM 0 50

60

70

OM 80

90

100

Ease of use Fig. 10. Correlation between ease of use and number of errors.

Table 5 Correlation coefficients between performance and preference measures Performance measures

Intuitiveness

Speed

Ease of use

Fatigue

Task completion time Number of errors

0.53 0.93

0.73 0.02

0.35 0.98

0.50 0.74

became familiar with the experimental conditions, but they were not briefed again about the mapping relationships in the second session. Meanwhile, the exposure of the participants to the different styles of body-based interfaces from the first session must have had an effect on the task completion time itself. This learning effect was apparent for BAIS (and MM that partially uses the same interface). For the case of OM, the participants seemed to become familiar with the optical mouse and the operation on the chest. However, the learning effect did not seem to affect BIM, because it must have felt already intuitive and understood well by the participants, plus the hand moving probably could not get any faster. BAIS exhibited similar task completion time to BIM in spite of the relatively shorter hand moving distance and to the strategic (easily seen and accessible) location of the interface. In terms of accuracy, because the position of the buttons for each task could not be remembered easily in BAIS, it produced a much higher error rate compared to the other three methods. However, the number of errors was reduced in MM drastically and almost as good as BIM. Thus, among the four, MM can be viewed as the best (with least errors and reasonable speed) body-based interaction method in terms of performance measures such as speed and accuracy. BAIS showed a very poor score in the subjective evaluation such as with intuitiveness and ease of use. This interesting result may be linked to the fact that it was also the most erroneous one. However, BIM made the participants feel more fatigued due to the relatively larger moving distance. MM were included in the most

intuitive and easiest to use group of the interaction styles. In short, MM turned out to be superior in most measures, taking advantages of the other two extreme methods as expected. It goes to show that as it is usually difficult to find a complete and intuitive set of metaphors, particularly when the number of tasks increases and thus the interactions become more complex, it is necessary to combine different interaction styles. Table 6 summarizes superior interaction styles for each measure. For BAIS, the experts completed the tasks more slowly and committed more errors than for the other interaction styles. Meanwhile, the novices completed the tasks more slowly in BIM, as expected. The relatively longer hand moving distances of BIM seem to have more effects on the novice users (e.g. higher fatigue level). Our guess is that this is simply a psychological effect of the novice user’s unfamiliarity with the interaction devices (e.g. headset, interactive shirt and the gloves). However, the correlation between the task completion time and perceived fatigue level was not significant in the analysis. The advantages and disadvantages of the two extreme methods, i.e., BIM and BAIS, can be concluded from this study as Table 7. These insights can help more usable body-based interfaces to be developed. Table 6 Superior interaction styles for each measure Measures

Superior interaction styles

Objective measures Task completion time Number of errors

MM BIM, MM, OM

Subjective measures Intuitiveness Speed Ease of use Fatigue

BIM, MM, OM Not different BIM, MM, OM BAIS

Table 7 Advantages and disadvantages of BIM and BAIS BIM Advantages Intuitive Easy to memorize the mapping

Disadvantages Long distance of hand movement Cause more fatigue (especially, in case of novices) Limited number of body parts mappable to many tasks

BAIS Short distance of hand movement Less fatigued Mappable to a lot of tasks

Not intuitive Difficult to memorize the mapping


7. Conclusion This study proposed several body-based interaction methods and examined their relative performances both objectively and subjectively. It was generally found that even though the BIM can cause longer hand moving distances and thus longer task completion times, it can still contribute to reducing the overall task completion time, because it produces much less errors as users find it natural and easy to use and remember. However, note that the excessive hand moving could cause muscle fatigue, and not all tasks can naturally be associated with body parts as well. The BAIS method had the contrasting character of higher error rates, but shorter task completion times. It was easy, with BAIS, to reach that part of the body that served as the point of interaction, however, difficult to remember the specific buttons to trigger particular functions. The fact that all the buttons/triggers were physically close together contributed to the high error rate as well. Overall, the MM was superior in terms of both the task completion time and error rate as it combined the merits of the above two methods. Thus, it is expected, for applications with many associated tasks, a judicious division of tasks among those that have natural semantic links to body parts and those that do not, is necessary to design the most performing body-based interface. In fact, even in the future years when wearable computing becomes more common and applied to wider areas of domain and tasks, we will probably not be able to afford a very large display like we have in our desktop systems. This means we need to rely more on other forms of (e.g. non-visual) interfaces. In this situation, the increased number of tasks and applications will make it more difficult for users to remember how to use these non-visual interfaces. BIMs seem to be one way to relieve this problem. In a very distant future, we might also speculate that various sensors will commonly be embedded in our everyday clothing (or even in our body) for wearable computing (Post and Orth, 1997). At first, these sensors might start to appear (simply) on sleeves of a shirt (and support BAIS), then later, spread to various strategic locations on our body (and support BIM). Our three applications were not a big enough pool to make generalizations about what type of body-based interaction schemes would be suitable for which application type. Only discrete choice tasks were examined in the experiment, while general interaction must include methods for input for continuous data (Lehikoinen, 2000; Fishkin et al., 2000). More experiments on a carefully classified and complete range of tasks might reveal certain guidelines on that topic. While this study concentrated on the comparison among different body-based interactions, it is also needed to compare body-based interactions, as part of future

273

work, to other conventional interaction methods (e.g. voice/gesture recognition, device-based, etc.). Acknowledgements This research was supported in part by the Korea Science and Engineering Foundation supported Virtual Reality Research Center, and the Korean Ministry of Education BK21 program. References Baber, C., Baumann, K., 2002. Embedded human computer interaction. Appl. Ergon. 33 (3), 273–287. Bowman, D., Kruijiff, E., Laviola, J., Poupyrev, I., 1999. IEEE VR Tutorial 2: The Art and Science of 3D Interaction. IEEE, New York. DDD, 2002. Direct digital depth homepage. http://www.ddd.com/ products. Fishkin, K., Gujar, A., Harrison, B., Moran, T., Want, R., 2000. Embodied user interfaces for really direct manipulation. Commun. ACM 43 (9), 75–80. Harrison, B., Fishkin, K., Gujar, A., Mochon, C., Want, R., 1998. Squeeze me, hold me, tilt me! An exploration of manipulative user interfaces. In: Proceedings of the ACM SIGCHI ’98. Los Angeles, 18–23 April, pp. 17–24. Hinckley, K., Tullio, J., Pausch, R., Profitt, D., Kassel, N., 1997. Usability analysis of 3D rotation techniques. In: Proceedings of the ACM UIST ’97, Canada, 14–17 October, pp. 1–10. IBM, 2002. A highly mobile processing communication device. Article on IBM Research Corporate Homepage, http://www.ibm.com/ products/gallery/wearablepc.shtml. Insko, B., 2001. Passive haptics significantly enhance virtual environments. Ph.D. Dissertation, Computer Science Department, University of North Carolina, USA. Keppel, G., 1991. Design and Analysis: a Researcher’s Handbook, 3rd Ed. Prentice-Hall, Englewood Cliffs, NJ. Kovacs, G.T.A., 1998. Micromachined Transducers Sourcebook. WCB, Boston. Lehikoinen, J., 2000. Virtual pockets. In: Proceedings of the Fourth International Symposium on Wearable Computers, Atlanta, 18–21 October, pp. 165–170. Lindeman, R., Sibert, J., Hahn, J., 1999. Hand held windows: towards effective 2D interaction in immersive virtual environments. In: Proceedings of the IEEE VR ’99, Houston, 13–17 March, pp. 205–212. Mine, M., Brooks, F., Sequin, C., 1997. Moving objects in space: exploiting proprioception in virtual environment interaction. In: Proceedings of the SIGGRAPH ’97, Los Angeles, 5–7 August, pp. 19–26. Miyake, K., 2002. HITACHI PC makes a fashion statement. Web Article, http://asia.cnn.com/2002/TECH/ptech/01/17/hitachi. fashion.idg/index.html. Montgomery, D.C., 2001. Design and Analysis of Experiments 5th Ed. Wiley, NY. Nichols, S., 1999. Physical ergonomics of virtual environment use. Appl. Ergon. 30 (1), 79–90. Nielsen, J., Levy, J., 1994. Measuring usability: preference vs. performance. Commun. ACM 37 (4), 66–75. Ockerman, J.J., Pritchett, A.R., 1998. Preliminary investigation of wearable computers for task guidance in aircraft inspection. In: Proceedings of the ISWC ’98, Pittsburgh, 19–20 October, pp. 33–40.


G.J. Kim et al. / Applied Ergonomics 35 (2004) 263–274

Post, E.R., Orth, M., 1997. Smart fabric, or washable computing. In: Proceedings of the First International Symposium on Wearable Computers, Cambridge, 13–14 October, pp. 167–178. Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., Carey, T., 1994. Human–Computer Interaction. Addison-Wesley, Reading, MA. Shankland, S., 2001. IBM clocks in with new LINUX watch. Web Article, http://zdnet.com.com/2100-1106-803567.html. Shneiderman, B., 1998. Designing the User Interface: Strategies for Effective Human–Computer Interaction, 3rd Ed. Addison-Wesley, Reading, MA.

Ullmer, B., Ishii, H., 2000. Emerging frameworks for tangible user interface. IBM Syst. J. 39 (3&4), 915–931. Wakefield, J., 2001. Suit yourself with a wearable computer. Web Article, http://zdnet.com.com/2100-11-530085.html. Wickens, C.D., Gordon, S.E., Liu, Y., 1997. An Introduction to Human Factors Engineering. Addison-Wesley, Reading, MA. Xybernaut, 2002. Xybernaut Corp. Homepage. http://www.xybernaut.com/newxybernaut/home.htm. Zhai, S., Milgram, P., 1993. Human performance evaluation of manipulation schemes in virtual environments. In: Proceedings of the VRAIS. Seattle, 18–22 September, pp. 155–161.