Capturing User Tests in a Multimodal, Multidevice Informal Prototyping Tool Anoop K. Sinha
James A. Landay
Group for User Interface Research UC Berkeley Berkeley, CA 94720-1776 +1 510 642-3437
Group for User Interface Research UC Berkeley Berkeley, CA 94720-1776 +1 (510) 643-3043
[email protected]
[email protected]
ABSTRACT Interaction designers are increasingly faced with the challenge of creating interfaces that incorporate multiple input modalities, such as pen and speech, and span multiple devices. Few early stage prototyping tools allow non-programmers to prototype these interfaces. Here we describe CrossWeaver, a tool for informally prototyping multimodal, multidevice user interfaces. This tool embodies the informal prototyping paradigm, leaving design representations in an informal, sketched form, and creates a working prototype from these sketches. CrossWeaver allows a user interface designer to sketch storyboard scenes on the computer, specifying simple multimodal command transitions between scenes. The tool also allows scenes to target different output devices. Prototypes can run across multiple standalone devices simultaneously, processing multimodal input from each one. Thus, a designer can visually create a multimodal prototype for a collaborative meeting or classroom application. CrossWeaver captures all of the user interaction when running a test of a prototype. This input log can quickly be viewed visually for the details of the users’ multimodal interaction or it can be replayed across all participating devices, giving the designer information to help him or her analyze and iterate on the interface design.
Categories and Subject Descriptors H.5.2 [User Interfaces]: –Prototyping, Interaction styles, Input devices and strategies. D.2.2 [Design Tools and Techniques] : – Evolutionary prototyping, User interfaces
General Terms
1. INTRODUCTION Our past study into multimodal user interface design practice [1] uncovered the lack of processes and tools for experimenting with multimodal interfaces. Non-programmer interaction designers are increasingly impacted by this problem, because many new interfaces are not for PC GUI’s or Web UI’s, but are instead target PDAs or physical devices. Furthermore, the set of tools for prototyping interfaces for heterogeneous devices is extremely small. Designers that we interviewed are also thinking about even more complex user interface scenarios, such as interfaces that span multiple devices simultaneously. One designer was planning a meeting room application with multiple participants sitting around a table, each seeing slides broadcast to their personal PDA. Traditional multimodal platforms have required programming expertise to deal with natural input recognition [2]. We have been working to remove programming as a requirement for multimodal interface design through the use of visual prototyping of simple multimodal interfaces [3]. This paper extends on our past experiments with multimodal storyboarding, introducing a storyboard style that also enables prototyping of multidevice interfaces (called multi-computer user interfaces by Rekimoto [4]). Designers can now create multimodal, multidevice interfaces, which can use unimodal natural input or multimodal input and can span multiple devices simultaneously. This allows interaction designers to conceptualize user interface scenarios that they were previously unable to create on the computer.
Keywords
Our new tool also formalizes the informal prototyping process, supporting distinct design, test, and analysis phases. This paper presents the designer’s process using our tool in each of those phases.
Informal prototyping, multimodal, multidevice, sketching, mobile interface design, pen and speech input
2. CrossWeaver
Design, Experimentation, Human Factors.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ICMI’03, November 5–7, 2003, Vancouver, British Columbia, Canada. Copyright 2003 ACM 1-58113-621-8/03/0011…$5.00.
CrossWeaver divides the prototype creation process into three distinct phases: design, test, and analysis. This methodology for informal prototyping was previously formalized in a speech interface prototyping tool called SUEDE [5]. In the design phase (see Section 2.1), the designer creates the storyboard, the artifact that describes the prototype, which includes sketches and input specifications. In the test phase (see Section 2.2), the designer can execute the prototype. Execution involves running the prototype with end-users. The analysis phase (see Section 2.3) occurs after a test. The analysis data
switching from one scene to another. An output target or output device is the label of the device onto which the scene will be shown, typically labeled by an identification number. In our tool, the storyboard is linear, though it also supports branching. The linear form encourages the designer to think in terms of short, stepby-step examples, which we have found to be quite common in the design process.
2.1.2 Drawing Area
Figure 1. The CrossWeaver Design Mode’s left pane contains the storyboard, which is made up of scenes and input transitions. The right pane contains the drawing area for the currently selected scene. contains a log of all of the user interaction – the scenes that were displayed across all participating devices and the user input that was received. CrossWeaver allows the designer to review the user test data and replay a user test later, across all of the participating devices. CrossWeaver uses sketches that are displayed in an unrecognized and un-beautified form. Rough sketches have been suggested as a better way to illicit feedback about interaction in the early stages of design rather than obtaining comments about fit-and-finish [6]. Comments about interaction and high level structure are what designers seek in the first stages of design.
2.1 Design Mode The CrossWeaver design mode is shown in Figure 1. The design being created is the Multimodal Toaster, an example given to us by one of the professional interaction designers who evaluated the tool. This designer said that he often uses 2-D representations of physical objects that he is designing. He wanted to add active behavior to his sketches but has previously not had a tool to allow him to do that.
2.1.1 Definitions A storyboard is a collection of scenes and transitions. A scene is a pane in the storyboard. An input transition, or simply, transition, is the input mode and parameter value that triggers
The right pane in Design Mode is the drawing area. In “Draw” mode, the designer can use a pen-based input device or a mouse to add strokes in the drawing area. The toolbar palette has buttons that can change the stroke color, fill color, and line width, represented in the fourth, fifth, and sixth buttons respectively. In “Gesture” mode, strokes can be selected, moved, and erased. Circling a stroke selects it and scribbling on top of a stroke deletes it.
The left pane contains the storyboard. The currently highlighted scene is shown in the drawing area. The transitions to the right of each scene show the possible inputs that go from scene to scene when testing the prototype (see Figure 2c).
2.1.3 Input Transitions Each transition can specify four different input modes. The top input mode is mouse click. Below that is keyboard press. The next after that is pen gesture. And the bottom input mode is speech input. Beneath each icon is an input area to specify the recognition parameter. For instance, the first transition in Figure 2c specifies ‘n’ as a pen gesture input. (For the purposes of our examples, we are using a letter recognizer and all gesture recognition parameters are letters.) The set of inputs specified in a group of vertical transitions represents a logical “or”. In Figure 2, this means a pen gesture ‘n’ or a keyboard press ‘n’ or a multimodal command of pen gesture ‘d’ and spoken input ‘down’ will lead to a transition from scene 2. The third transition is a multimodal command. It specifies that the gesture ‘d’ and the spoken command ‘down’ must both occur within one second of each other for the transition to proceed. None of the designers we interviewed prioritized the need to experiment with strategies for processing fused natural
d
a
b
c
Figure 2. A scene in the storyboard contains (a) a thumbnail of the drawing, (b) device targets and text-tospeech audio output, (c) input transitions showing the natural inputs necessary to move from scene to scene, and (d) a title. input commands. Thus, we have only incorporated a simple strategy for multimodal input fusion in the tool. A future version of the tool could add strategies for specifying fused input in different ways. The arrow and number at the top of the transition specify the next scene to show if the input is matched. For example, in Figure 3, each transitions will go to scene 3.
2.1.4 Processing Recognition We use an agent-based architecture for processing recognition input. In our examples, pen gesture recognition is done by a commercial letter recognizer or optionally by Wizard of Oz. A different recognizer could be used to return named gestures such as ‘up’, ‘down’, ‘copy’, or ‘paste.’ In our example, the speech command ‘down’ might be an individual speech command or it might be a keyword returned by a streaming speech recognition agent. We presently use a speech recognition agent written using Microsoft’s Speech Recognition Engine.
2.1.5 Output Devices The output devices for each scene are specified in the bottom panel underneath the thumbnail (see Figure 2b). The number next to the screen icon represents the PC screen identifiers that would show this scene (i.e., PC Device #0). The PDA icon is next to the PDA identifiers (i.e., PDA #0, PDA #1, PDA #2). The sound icon is next to the text-to-speech audio that is played when this scene is shown. Each of the devices specified needs to be running a standalone version of the test browser, which is further described in Test Mode (see Section 2.2). When the scene is shown, the audio is played via a text-to-speech agent. In Figure 2, this specific scene is broadcast to all four devices at the same time. With changes in the device identifiers, it is also possible to specify output to only a single device.
Figure 3. Here we specify an input region, a dashed area in the pane, in which linked gesture commands must happen to follow the transitions
2.1.6 Arrows The arrows in the storyboard show connections between scenes and transitions. In Figure 1, we see that the different scenes are specified linearly. Branching is fully supported by arrows that connect to non-adjacent scenes. There is also an optional comic strip view which displays the scenes and transitions in a grid.
2.1.7 Operations A designer can group together two scenes and create an Operation (see Figure 4). An operation is like a production rule for CrossWeaver. In the example shown in Figure 4, the tool calculates the difference between the two scenes; here the difference is the addition of an object, a slice of toast. During the test mode, the transitions in the operation can be used to add the object to an arbitrary scene. CrossWeaver interprets operations by looking at the difference between the two scenes and assigning meaning to that difference. In the current version, CrossWeaver understands adding an object, change of color, zooming in and out, moving objects, and deleting an object. These are the set of operations that are most useful in a map or drawing application. The specific parameters for each operation, e.g., location to move to, are inferred from the drawn scenes. This method of specifying operations focuses the designer on experimenting with different input commands that trigger the interaction. The designer can quickly try different input modes that trigger an operation. Operations were easily understood by the designers we spoke with and were called potentially useful. But in the first few trials, the designers were more concerned with using different storyboard scenes individually rather than parameterized operations. Thus, for designers, operations will be a more advanced feature that is learned over time.
2.1.8 Global Transition A transition that points back to its originating scene is a global transition, which can be activated from any other scene. This provides a method for a tester to jump to a specific scene from
Figure 5. A global transition points back to the scene from which it started. A gesture of ‘h’ for ‘home’ on any scene will take the system back to this scene, the starting scene of the storyboard. In the storyboard, each scene specifies target output devices. A standalone version of the multimodal browser, written in Java, can be started separately on the devices that are participating in the test (see the bottom of Figure 7). This standalone version works just like the built-in browser, displaying scenes and accepting mouse, keyboard, pen and speech input. The standalone browsers and the recognition engines are all joined together as agents using the Open Agent Architecture [7]. Figure 4. Grouping two scenes creates an Operation. Based on the difference between the scenes, this operation is the addition of a toast slice to a scene, triggered by any of the three input transitions joining the two scenes. any other place in the storyboard. In Figure 5, the gesture ‘h’ for home will transition to the first scene in the storyboard.
The state management of the test is controlled by CrossWeaver in Test Mode. CrossWeaver tracks the current scene being displayed on each device and also translates an input action into the next scenes to be shown. Any input is tagged by a specific device ID. Thus, CrossWeaver is managing a state machine based on the storyboard with multiple active states. Each active state corresponds to a device participating in the test.
2.1.9 Imported Images and Text Labels To allow the designer to quickly reuse past art work or previous designs, CrossWeaver allows the insertion of images into the scenes (see Figure 6). The designer can also insert typed text labels. These images and labels co-exist with drawn strokes. This combination of formal and informal representations is potentially quite powerful. It allows the designer to reuse elements that might already have been created in another tool, such as a background design or a template. The designer can also quickly change the elements that are more in flux using drawn strokes. Imported text labels can also be used for giving directions to test users, as shown in the example in Figure 6.
2.2 Test Mode Once the storyboard is complete, the designer can execute the application by clicking on the “Run Test…” button. A built in multimodal browser will be started, corresponding to PC device #0.
2.2.1 Architecture of Test Execution The browser accepts mouse button presses, keyboard presses, pen gestures, and speech input from a tester. Gesture recognition and speech recognition are performed in separate recognition agents participating with the test browser in the execution of the application.
Figure 6. A bitmap image of a toaster has been inserted into the scene. A typed text label has also been added. These images co-exist with strokes, providing combined formal and informal elements in the scene. Images can be imported from the designer’s past work or an image repository.
Figure 7. Clicking on the ‘Run Test…’ button in design mode brings up the Test Mode Browser, which accepts mouse, keyboard, and pen input. (Top) In the first sequence, the end user gestures ‘n’ on the scene and the scene moves to the appropriate scene in the storyboard. (Bottom) In the second sequence, the user accesses the ‘add slice’ operation, adding slices of bread to the scene. This is occurring in the standalone browser running on device PDA #0, as identified by the ID in the title bar of the window. Pen recognition and speech recognition results come into the browser from separate participating agents. Two or more standalone devices browsers can share the same device ID. This provides a shared scene configuration, where the device screens are kept in sync. A shared screen configuration would be used for a broadcast presentation tool, for example.
the test execution. The second transition from the left on the top row shows that the user on PDA #0 typed ‘t’ on the keyboard.
2.2.2 End-User Experience running a Test
This view allows the designer to examine the inputs attempted by the end participants even if they were not successful in triggering a transition. All input attempts are captured in this log. The designer can use the analysis display to see if the user switches input modes to attempt to trigger the next scene. These repair actions are especially important in multimodal applications.
Multiple users can participate in a test simultaneously, as in a CSCW application with one user per device. Each test device will run a standalone browser. The end-users running a test are not aware of the full storyboard that the designer has created. Each user is focused only on the scene in front of him and on his input. The designer can help the end-users discover commands by adding audio instructions or using text or handwritten prompts and instructions to each scene.
2.3 Analysis Mode After running a test, clicking on “Analyze” in Test Mode brings up a timeline log of the most recent test (see Figure 8). Each row in the display corresponds to a different participating device in the execution of the test. Any device that was identified in design mode will show up in the analysis view. In Figure 8, from left to right we see a time-stamped display of the scenes that were shown across each device and the corresponding input transition that led to a change in the state of
Some rows have a scene followed by a blank area, such as the second row corresponding to PDA #2. Blank grid spaces mean that no changes were made on that specific device.
2.3.1 Viewing Multimodal Input Figure 8 also shows the building of a multimodal input, as shown in the bordered vertical box in the middle of the screen. If an input can possibly match a multimodal command, the system waits for other inputs within a specified time period. If those other inputs happen, they are added to the display as a multimodal input transition, with multiple input mode slots filled in.
2.3.2 Replay From within the Analysis Mode display, a designer can replay the entire log of execution across all of the devices by pressing the play button in the toolbar. This replay allows the designer to see the user test again, in action, across all of the devices.
3.1.1 Informal Prototyping for Designers SILK [6, 8], a tool for sketching graphical interfaces, introduced the idea of sketching interfaces and leaving them in unrecognized form when testing. SILK also introduced a storyboarding paradigm for this type of design. CrossWeaver has extended SILK’s storyboard to multimodal commands. CrossWeaver introduces the new concept of multidevice prototyping as an extension to the work that SILK pioneered. DENIM [9] is an informal prototyping tool for web design. DENIM has sketched pages and transitions that are analogous to Figure 8. CrossWeaver’s Analysis Mode shows a running timeline of all of the scenes that were shown CrossWeaver’s scenes across all of the devices. It also displays the interaction that triggered changes in the state machine. and transitions. DENIM The outlined are in the middle of the screen represents the current state in the replay routine, which uses an infinite sheet steps through the timeline step-by-step replaying the scenes and inputs across devices. It is presently layout, arrows, and playing back a multimodal command, the pen gesture ‘d’ with speech input ‘down’. semantic zooming for web page layout and Replay is valuable for finding critical incidents or more subtle linking. CrossWeaver uses a linear storyboard to match the design flaws long after the user test is finished. multimodal designers’ preference for thinking in terms of short linear examples. In DENIM, transitions are based on mouse 2.3.3 Returning Full Circle to the Design events of different types (e.g., left or right click). CrossWeaver The analysis view purposely looks similar to the original design adds gesture transitions and speech transitions to promote view: it is linear for each device and has the same scenes and multimodal experimentation. Lastly, DENIM runs its designed transition layout. web pages as a single state machine in its integrated browser or Analysis files can be saved individually and loaded into separate in a standalone browser. CrossWeaver allows the designer to windows, allowing the designer to compare two or more conceptualize a multidevice application via multi-state different participants side-by-side. By having the analysis data execution of the storyboard. collected in the design tool itself, the designers can quickly The design, test, analysis paradigm for informal prototyping was review a number of user tests and refine the design. introduced in SUEDE, an informal prototyping tool for Wizard An analysis view does not have to correspond to the current of Oz design of speech interfaces [5]. SUEDE’s design mode scenes in the storyboard. For example, analysis views of creates a flow chart of the speech interface much like different user tests can be displayed even after the storyboard CrossWeaver creates a storyboard of the multimodal, itself has changed. Keeping all design, test, and analysis data multidevice interface design. In CrossWeaver test mode, the together in the same tool facilitates rapid iterative design. storyboard maintains a separate state per device and executes logic in the storyboard across multiple devices, whereas in SUEDE the state diagram maintains only one state. 3. RELATED WORK Furthermore, transitions in SUEDE are only speech commands CrossWeaver has been developed based on the specific focus on while in CrossWeaver transitions can be mouse clicks, keyboard the needs of the interaction designers that we studied [1]. input, pen gestures, and speech input. CrossWeaver has been significantly influenced by work in informal prototyping for designers and in multimodal interface CrossWeaver’s analysis mode is influenced by the Designer’s design. Outpost history mechanism [10] which captures the history of designers creating a web site architecture. CrossWeaver’s focus
is capturing the log of the test of an interactive multimodal, multidevice application, versus a history of commands in a design tool. DEMAIS [11] is an informal tool for multimedia authoring. It includes the concept of joined formal and informal representations. In DEMAIS, audio and video clips co-exist with sketched representations. It also includes the concept of rich transitions between scenes in its storyboard based on mouse events. As a multimedia tool, DEMAIS is focused on design of rich, diverse output. CrossWeaver’s focus is on natural input for an interactive application. The two tools are quite complementary.
3.1.2 Multimodal Prototyping The seminal platform for interactive multimodal application design is the QuickSet system for implementing multimodal applications built using the Adaptive Agent Architecture (AAA) [12]. QuickSet is a programming platform that has been used to create multimodal applications for map and military planning and mobile domains. It includes all of the capabilities for creating multimodal, multidevice applications, and the applications created with QuickSet could be considered target applications that could be designed in CrossWeaver. Prototyping by non-programmers has not been the target of QuickSet. STAMP [13], a multimodal logging tool accompanying QuickSet, has been used to captured detailed information about multimodal user input. These input logs can be analyzed to understand multimodal ordering, preferences, and statistics. CrossWeaver’s analysis mode shows much less information than the STAMP tool. The CrossWeaver analysis display is designed only to give the information that is of relevance to designers at the very first stage of multimodal, multidevice design: the attempted commands and subsequent scenes displayed. CrossWeaver also includes only simple multimodal input commands. Even simple multimodal commands have been previously unavailable for use by non-programmer designers.
three seconds” if he did it again. He also added that his HTML web page would not have gesture input. A second interaction designer stated that this was the first time he had designed any application that used gesture as an input. This was the first time that he had executed a sketched prototype on the computer, though he frequently used paper prototyping [14]. He said that incorporating gesture into the interface design was extremely simple with CrossWeaver. One of the interaction designers said that he sees “great potential for real world use of this application [CrossWeaver].” He immediately started talking through some of his current design tasks. He said that he often has to do 2-D mock-ups of physical devices to illustrate the behaviors of those devices. His idea was to use CrossWeaver to do mock-ups of the toaster, which is the example in this paper, and have different parts of the toaster be controlled by gesture or speech. He pointed out the need for a more flexible gesture recognizer that supports strokes rather than letters. However, he quickly saw how he could use CrossWeaver to mock-up the multimodal toaster and other appliances in the kitchen.
5. Future Work Various refinements can be made to CrossWeaver, including inclusion of additional recognition engines, drawing capabilities, and visual refinements that would make it more appealing to designers. Additionally, an aggregate view could be added to the Analysis Mode to help manage large numbers of user tests. Two other valuable directions for CrossWeaver to take are enhanced multimodal command design and enhanced sketch interpretation capabilities. For multimodal command design, CrossWeaver presently only includes a simple algorithm for multimodal fusion. CrossWeaver could incorporate a visual interface for adding multimodal parameters and disambiguating inputs. CrossWeaver’s analysis mode could also provide additional data for multimodal command design, making it more flexible for a CrossWeaver test to utilize streaming input sources.
An additional set of multimodal applications have been built using the Open Agent Architecture (OAA) from SRI [7]. In fact, CrossWeaver is built on top of OAA, using its distributed recognition agent capabilities and its ability to manage the standalone CrossWeaver browsers. Multimodal application design with OAA also has not previously been available to nonprogrammer designers.
For sketch interpretation, CrossWeaver’s concept of an Operation can be enhanced to include more capabilities in other domains based on sketch interpretation. The existing mechanism is extensible, and applications in different domains might have different requirements for baseline Operations. For an example, shared ink might be an Operation in a classroom presentation application.
4. Evaluation
6. Implementation Overview
We have run an evaluation of CrossWeaver with nine professional interaction designers from the San Francisco Bay Area. As testing tasks, we asked the participants to first create a multimodal web page, i.e., a web page that uses gestures and speech input to go from page to page. We next asked them to create a remote control for a PDA that can control the display of a large computer screen. All nine participants found the CrossWeaver design, test, and analyze methodology easy to understand. One of the interaction designers created the multimodal web page and then commented that he thought he could have created something similar in HTML in three minutes. Then he added that he sees how he could create the design in CrossWeaver “in
CrossWeaver is built as a Java application using JDK1.4.1. The core sketching capabilities in CrossWeaver and the scene models are extended from Diva, an application infrastructure for building sketch-based applications [15]. The standalone browser is built in Java as well, using only capabilities compatible with Java on mobile devices. CrossWeavers’s multimodal and multidevice capabilities are integrated through SRI’s Open Agent Architecture (OAA) and agents for handwriting recognition and speech recognition are available to communicate with the CrossWeaver system.
7. Conclusion CrossWeaver gives a non-programmer designer the ability to conceptualize, test, and analyze a multimodal, multidevice informal prototype. An interaction designer who is tasked with creating an application in this new domain can quickly create an prototype that incorporates pen gestures, speech input, and multimodal commands. They can also quickly create a user interface that spans multiple devices simultaneously, and test that interface with end-users. The design, test, and analysis methodology in CrossWeaver dovetails with existing design processes and provides the capabilities for rapid multimodal design in one tool. CrossWeaver’s analysis mode helps capture the spirit of each end-user tests, helping the designer decide on the multimodal vocabulary that they will use in their final application. Rapid design experimentation with CrossWeaver will help improve a design and help extend the proliferation of multimodal, multidevice interfaces. For more information on CrossWeaver, please see: http://guir.berkeley.edu/crossweaver/
8. ACKNOWLEDGMENTS Our thanks to the Diva team for use of their toolkit and to SRI for use of the Open Agent Architecture. This material is based upon work supported by the National Science Foundation under Grant No. 9985111.
9. REFERENCES [1] Sinha, A.K. and J.A. Landay. Embarking on Multimodal Interface Design. In Proceedings of International Conference on Multimodal Interfaces ICMI 2002. (Pittsburgh, PA, October 2002). pp. 355-360.
[2] Oviatt, S., P. Cohen, L. Wu, L. Duncan, B. Suhm, J. Bers, T. Holzman, T. Winograd, J. Landay, J. Larson, and D. Ferro, Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions. HumanComputer Interaction, 2000. 15(4): p. 263-322.
[3] Sinha, A.K. and J.A. Landay. Visually Prototyping Perceptual User Interfaces Through Multimodal Storyboarding. in Workshop on Perceptive User Interfaces PUI 2001 (Orlando, FL, November. 2001)..
[4] Rekimoto, J. Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments. in Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology. UIST 1997. (Banff, Alberta, Canada, 1997).
[5] Klemmer, S.R., A.K. Sinha, J. Chen, J.A. Landay, N. Aboobaker, and A. Wang, SUEDE: A Wizard of Oz Prototyping Tool for Speech User Interfaces. CHI Letters,
The 13th Annual ACM Symposium on User Interface Software and Technology: UIST 2000. November 2000. 2(2): p. 1-10.
[6] Landay, J.A. and B.A. Myers, Sketching Interfaces: Toward More Human Interface Design. IEEE Computer, 2001. 34(3): p. 56-64.
[7] Moran, L.B., A.J. Cheyer, L.E. Julia, D.L. Martin, and S. Park, Multimodal User Interfaces in the Open Agent Architecture. Knowledge-Based Systems, 1998. 10(5): p. 295-304.
[8] Landay, J.A. and B.A. Myers. Interactive Sketching for the Early Stages of User Interface Design. in Proceedings of Human Factors in Computing Systems: CHI ’95. (Denver, CO, 1995).
[9] Lin, J., M.W. Newman, J.I. Hong, and J.A. Landay, DENIM: Finding a Tighter Fit Between Tools and Practice for Web Site Design. CHI Letters, Human Factors in Computing Systems, CHI 2000, April 2000. 2(1): p. 510517.
[10] Klemmer, S.R., M. Thomsen, E. Phelps-Goodman, R. Lee, and J.A. Landay, Where Do Web Sites Come From? Capturing and Interacting with Design History. CHI Letters, Human Factors in Computing Systems: CHI 2002, 2002. 4(1): p. 1-10.
[11] Bailey, B.P., J.A. Konstan, and J.V. Carlis. DEMAIS: Designing Multimedia Applications with Interactive Storyboards. in Proceedings of the 9th ACM International Conference on Multimedia. (Ottawa, Ontario, Canada, 2001).
[12] Cohen, P.R., M. Johnston, D. McGee, S. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow, QuickSet: Multimodal Interaction for Distributed Applications, in Proceedings of ACM Multimedia 97. (Seattle, WA, 1997). p. 31-40.
[13] Clow, J. and S.L. Oviatt. STAMP: an Automated Tool for Analysis of Multimodal System Performance. in Proceedings of the International Conference on Spoken Language Processing. (Sydney, Australia , 1998).
[14] Rettig, M., Prototyping for Tiny Fingers. Communications of the ACM, 1994. 37(4): p. 21-27.
[15] Reekie, J., M. Shilman, H. Hse, and S. Neuendorffer, Diva, a Software Infrastructure for Visualizing and Interacting with Dynamic Information Spaces. http://www.gigascale.org/diva/, 1998.
