Ideas for Intelligent User Interface Design James R. Slagle, Zbigniew Wieckowski a.k.a. Bishak (slagle,
[email protected]) Department of Computer Science University of Minnesota 200 Union St. SE Minneapolis, MN 55455
ABSTRACT
A methodology for development of intelligent user interfaces is presented which allows for programming by correcting behavior, rather than writing programs or scripts. The importance of incremental knowledge accumulation with contributing individuals cooperating over the network is pointed out. The experience of the authors with the prototype of an interface builder and prelimnary results are outlined. The project aims at utilizing the flexibility, that Tcl/Tk offers, to allow for automatic creation of an interface on the basis of previous user interactions with applications. The possible role of the user as a tutor and cooperation of users is described.
Would you like to have your own intelligent assistant? Sure, but will such an assistant be smart enough? Can you rely on it? Well, no one knows whether it is feasible to design such an intelligent assistant, and there is little agreement on what it should resemble. This work presents one possible direction. For the purpose of this paper we will use the terms intelligent agent/assistant, and intelligent user interface as synonyms.
1. Inspirations No matter how simple the ideas are, they come into being either as a synthesis, variation, or antithesis (like Unix1, and, to some degree, Linux) of previous work. Expect [6] makes it possible to write a script for an agent that sits in between the user and an application, allowing automation of tasks with executions that vary, but there is a pattern to the interaction. An interesting application, written with the use of Expect, is kibitz [7], which allows two people on different machines to control the same program. Kibitz can also be used in case a cron job needs to contact the user and ask a question. ExpecTerm is an extension of Expect that allows more flexibility by adding terminal emulation, and also by allowing the user to specify rows and columns to be searched. TkEmacs is an emacs widget that is completely integrated with the Tk widget set. The Reactive Keyboard [3] is a program that tries to predict the next user’s command. Red Ryder is a Macintosh communication program with a “watchme” feature.
1. Unix is a trademark of USL, Inc., which employs Diego Baez - buyer of Bishak’s giant schnauzer puppy.
2. An Overview of the Intelligent User Interface Builder Like [5], we agree that an agent should be autonomous and intelligent. However, we disagree that it can not expect to ask for help from the user when problems arise. Our method makes such a feedback from the user a rule. In fact, no scripting language is even required. The essence of the method is capturing the ongoing user activity, so that later it could be reused. Of course, it is not possible in all cases to create a model of interaction that will work for all possible variants of user interaction. Generality is very desirable, however, and the user should guide the interaction in such a manner so as to maximize the chance of success when the interaction is reused later. The interaction consists of direct user commands and meta-activities, performed by the user, that allow the extraction of relevant information from the current environment, or the assertion that the activity makes progress. The interaction recorded by the user is an action. An action that does not contain meta-activities is a macro. We search for meta-activities that would allow effective heuristic classification [1] of situations encountered.Whenever metaactivity of an invoked action fails, the user is contacted in order to improve the interaction, so that the current case is taken care of. This can be regarded as knowledge acquisition, or case-based reasoning. User feedback will result in an improved action that will be augmented by a new conditional branch for that specific case. Unlike Red Ryder, our model does not assume focusing on a single domain. Instead we propose that useful actions be shared with others, e.g., through an anonymous ftp site. This would require each action to be appropriately commented, and its applicability context specified as a part of it. Also, credits should be
given to authors by inclusion of e-mail addresses. Assuming that a mechanism exists for an action to invoke other actions, this would allow data-driven accumulation of knowledge. This contrasts with expert-system development, where knowledge accumulation proceeds in a goal-driven fashion. Actions would have to be tested thoroughly, before being made available, due to security.
3. Current Limitations We have noticed that regular expressions help little in the task of extracting semantic information. Regular expressions might be good in a script. However, thinking of a regular expression quickly might be difficult.(Otherwise we would know of people using regular expressions for reading news under nn.) We need a simple, intuitive, but still robust, mechanism, and we have implemented searches for sequences of words, numbers, and single characters, but we are far from a satisfactory solution. The prototype we have developed shows that it is possible to program by giving an example [2], rather than writing scripts. For this approach to be a feasible way of accomplishing goals, however, there must be a mechanism for supplying additional knowledge, in cases that were not foreseen. Preferably, such a mechanism would be based on correcting the behavior, rather than editing accumulated knowledge. We think it would be also easier to do that within a context in which the undesirable behavior has occurred. Refinement of actions doesn’t require close cooperation between developers. Such cooperation would be solely based on closeness of interests of people who happened to use the same application. The output from an application may arrive in parts, instead of as a whole. There is a problem
with deciding when it is time to send the input to the application. We introduced delays, so as to decrease the chance of sending characters to the application while it still produces output (top is an exception). However, this solution is temporary. What is really needed is knowledge about the current application that would allow determination of the end of output. At each step of an interaction the action would know the current context, e.g., we could define a context for ftp, in which an action would wait for the prompt. Although the current prototype works with a character-based application on a terminal, it can cooperate with other window-based applications built using the Tk toolkit. In one respect it is easier that way to integrate applications into the interface, e.g., we did not have to modify the code of emacs, as in the case of TkEmacs, but we are still able to send character strings to it from other applications. It is interesting, though, to investigate what would be an equivalent of our interface for a windowbased environment. Would it be feasible for the agent to recognize patterns in a video being played and respond to them in real time? We are not currently pursuing these topics, but we would like to point out that the XTrap program is able to record X events and replay them at a later time.
4. Interaction among Tk windows Tk allows for dividing applications into several parts that communicate with each other. This results in a flexible interface paradigm, which prefers small interface components, as opposed to a monolithic application. An important aspect of an intelligent assistant is its ability to adjust easily when the user switches between different applications. Each such application may require a different user interface to support it. We claim that Tcl/Tk
provides a good support for enforcing semantic correctness by displaying only relevant part of the interface [8]. It is also possible to extend ideas of the Reactive Keyboard. Instead of predicting just the next command, the user could be presented with several most likely choices. The user may also prepare interface support for specific applications by writing Tk scripts or using XF [4] for interactive design.
5. Conclusions The need for intelligent agents is tremendous. In this paper we have presented an approach that allows the user to record semantic threads of interaction for later reuse. The advantages of our approach are the following: on-the-fly prototyping, no need for scripting language and writing scripts, incremental knowledge accumulation, and the possibility of sharing knowledge among users. The problems that we have encountered are: difficulties with coming up with effective general methods for extracting important information that could be used to guide the flow of an action. This results partly from the ubiquitous character of data dealt with, which is also a factor that allows the agent to operate on exactly the same data that the user is currently perusing. This is important, because users are currently the best source of knowledge for programs. The technique we propose should be regarded as a supplement to existing techniques, as high level language programming, shell or Tcl scripts, and Expect scripts. Our goal is to continually enhance the user interface by providing the user with a tool for creating high level interaction capabilities on the basis of existing applications [9].
6. References [1] William J. Clancey - “Heuristic Classification”, Artificial Intelligence 27 (1985).
[2] Cypher A.- “Programming Repetitive Tasks by Example”, Proceedings CHI ‘91. [3] Darragh, J.,J., Witten, I.H., and James, M.L. (1990) “The Reactive Keyboard: A Predictive typing aid”, IEEE Computer 23(11): 41-49, November. [4] Sven Delmas - “XF - Design and Implementation of a Programming Environment for Interactive Construction of Graphical User Interfaces” [5] Raymond W. Johnson - “Autonomous Knowledge Agents”, Tcl’93 Workshop. [6] Don Libes - “expect: Curing Those Uncontrollable Fits of Interaction”, Proceedings of the Summer 1990 USENIX Conference, Anaheim, California, June, 1990. [7] Don Libes - “Kibitz - Connecting Multiple Interactive Programs Together” - “Software - practice and Experience”, John Wiley & Sons, New York, Vol.23, No.5, May 93. [8] John K. Ousterhout - “Tcl and the Tk Toolkit”. [9] Henry Spencer - “How to steal code”, Proceedings of the Winter 1988 USENIX Conference. Short biography of Dr. James R. Slagle Dr. Slagle received his Ph.D. in Mathematics from MIT in 1961. He is a Distinguished Professor of Computer Science at the University of Minnesota, a position he has held since 1984. He is a member of ACM, fellow of the AAAI, AAAI, and IEEE. Dr. Slagle has received numerous honors throughout his academic and research career, including the Inventor’s Award from the Minnesota Intellectual Property Law Association. He has published over 130 papers as well as the book Artificial Intelligence : The Heuristic Programming Approach, McGraw-Hill, N.Y., 1971.
Zbigniew Wieckowski is a PhD candidate. His interests include Systems, Artificial Intelligence, Computer-Human Interaction, and Multimedia. He received his M.Sc. in Computer Science from Warsaw Technical University in 1989.
Fig.1. A simple application constructed with the use of the intelligent interface builder. Under XF it is easy to add more functionality provided that the user defines appropriate actions.
