11 interact 2005 - Semantic Scholar

11

WORKSHOP INTEGRATING SOFTWARE ENGINEERING AND USABILITY ENGINEERING

Morten Borup Harning, Rick Kazman, Daniel Kerkow, Kirstin Kohler & Jan Gulliksen

September 13 full day

In conjunction with: Tenth IFIP TC13 International Conference on Human-Computer Interaction 12-16 September 2005, Rome, Italy

INTERACT 2005

Table of Contents Modeling Dependencies for Cascading Selective Undo Aaron G. Cass and Chris S. T. Fernandes

Eliciting and Specifying Requirements for Highly Interactive Systems using Activity Theory Robert B. K. Brown, Peter Hyland, Ian C. Piper

Towards a Pattern Based Usability Inspection Method for Industrial Practitioners Martin Schmettow

The Role of Usability Testing in Software Quality Control: Finding Mismatches, not Defects Paul H. Vossen

Model-Based Software Development and Usability Testing Peter Forbrig, Gregor Buchholz, Anke Dittmar, Andreas Wolff, Daniel Reichart

Socio-Technical Support for Knowledge Collaboration in Software Development Tools Yunwen Ye

Principles for the Design of Web Applications Anthony I. Wasserman

Requirements for the Integration of UE Methods in SE Processes from the Perspective of Small and Medium-sized Enterprises (SMEs) Daniel Kerkow, Karin Schmidt, Frank Wiebelt

Modeling Dependencies for Cascading Selective Undo Aaron G. Cass and Chris S. T. Fernandes Union College, Schenectady, NY 12308, USA, {cassa|fernandc}@union.edu

Abstract. Linear and selective undo mechanisms have been studied extensively. However, relatively little attention has been given to the semantics of selective undo in situations in which the undone command has dependency relationships with actions later in the command history. We propose a new approach to selective undo using a process modeling language from the software process community to model dependencies between commands, and we outline semantics for undo in the face of these dependencies.

1

Introduction

In most applications today that implement an undo feature, a history list is kept of all user actions, and a linear undo model is followed. In this model, an undo command will only reverse the most recent action, and subsequent undo commands will iterate backwards through the history list. For example, given user actions A1 , ..., An , issuing an undo command will only undo action An , and one cannot undo any action Ai without also undoing actions Ai+1 , ..., An . Other undo paradigms exist, however, and many have studied non-linear undo models. One such model is the selective undo model, introduced in [1] and studied more formally in [2]. In this model, a user can undo action Ai without undoing other actions. The meaning of this kind of undo can differ, however. Most implementations of selective undo use the “script” paradigm [3], in which the result of undoing action Ai alone is equivalent to the result reached by executing user actions A1 , ..., Ai−1 , Ai+1 , ..., An in that order. That is, if the list of user actions is viewed as a script, undoing one action is equivalent to removing that action from the script, with no other changes. This can result in side effects the user may or may not have intended. If the given actions were the following word processor commands: 1. 2. 3. 4.

type “hello” italicize “hello” copy “hello” paste in position x

then selectively undoing the second action would result in the removal of italics from both the original and the pasted text. On the other hand, the semantics of

selective undo in [1] would result in the removal of italics from just the original text. This semantic ambiguity points to the disparity between the way undo is perceived by interface programmers compared to users [4, 5]. In this position paper, we explore the semantics of selective undo using ideas from the software process community. We believe this to be a good match since a primary cause of the ambiguity discussed above is the dependencies which can exist between user actions in any given “script”. In the above example, this dependency is seen when the italicized text is duplicated. Software engineering is a domain in which these dependencies frequently occur, and thus provides fertile ground for determining appropriate undo semantics. In addition, tools already exist in the software engineering community for capturing dependencies in complex process or workflow models. Conversely, the tools which are used for software development can greatly benefit from a feature allowing selective undo. One novel aspect to our approach is the way in which we handle recognized dependencies between user sub-tasks. In previous work, these dependencies were either not accounted for at all or else handled by disallowing the undo action if the result was not meaningful, such as in [1]. An example of a selective undo that creates a meaningless result is if a programmer first created a class “Car” and second created a method “fillTank” for “Car”. Selectively undoing the first action would cause the second action to be meaningless and under the semantics presented in [1] would be disallowed. We propose a different alternative – allowing an undone action to cause the undoing of other user actions until a meaningful state is reached (with appropriate user feedback and override controls). We believe this cascading selective undo offers more flexibility than previous approaches and that software engineering approaches can be used to help determine a meaningful cascade. Section 2 provides details about the software engineering concepts we will employ to capture the context of selective undo. Section 3 gives examples of the types of questions we will explore. Section 4 examines future work and conclusions.

2

Our approach

Cass has been developing software design environments for use in design guidance experiments [6] using programs written in a process-programming language called Little-JIL [7, 8]. Such process programs allow for a wide range of guidance to users of a design tool, from very flexible to very constrained sequencing of tasks. A process program specifies the allowable sequences of tasks and the data flow between them. The Little-JIL run-time environment works with this process model to make commands available to users on a hierarchical to-do list. As the user performs tasks, the system presents new tasks as controlled by the semantics of the Little-JIL program. The current implementation does not support undoing user actions. One key issue is that the process program represents a controlled walk through a design space and the run-time system as such maintains a representation of the

Fig. 1. Little-JIL Basics

state of this walk. Therefore, if we are to support undo, we need to do it in such a way that the run-time system can continue to maintain accurate process state information. Our approach is therefore to define undo semantics for the hierarchical process programs written in Little-JIL. This should give us robust semantics for cascading selective undo that can be extended to other dependency models. In this section, we will briefly introduce Little-JIL, how Little-JIL programs represent control and data dependencies between tasks, and how we can use that information to support cascading selective undo. 2.1

Introduction to Little-JIL

Little-JIL programs are hierarchies of steps which are instantiated at run-time to become items to be placed on the agendas of agents acting in the process. The step kind, represented by the sequencing badge on the step in the graphical representation of the program, specifies how the instantiated sub-steps are to be sequenced. The badges for the four step kinds are shown in the left-most legend of Figure 1. Children of sequential steps are performed left-to-right with one child available for execution only after the previous one is completed. Children of parallel steps are performed in any, possibly overlapping, order. Only one child of a choice step is executed – when one of the steps is started at the discretion of the agents involved in the process, the other children are retracted. Children of try steps are attempted in left-to-right order and as soon as one of them successfully completes, the try step is completed. Program hierarchies are arbitrarily deep, with leaf steps representing the primitive actions available to agents. Note that any undo mechanism for Little-JIL programs must allow a user to undo a primitive action that was not the most recently executed, for two reasons. First, Little-JIL programs can specify the coordination of actions of multiple agents and therefore multiple primitive actions can occur concurrently. At the minimum, an undo mechanism would have to allow an undo of one agent’s most recent action (a locally linear undo) in cases where this is not the globally most recent action. Second, the completion of a primitive action in Little-JIL can cause new actions to become available. For example, if the action just completed is the last of the children of a sequential step, the sequential step is completed

and possibly other actions are made available depending on the step kind of the parent of the sequential step. In order to maintain legal process state, undoing the user’s most recent primitive action will require undoing the completion of the sequential step and the posting of the new options. 2.2

Modeling Control and Data Dependencies

The Little-JIL program structure shows control dependencies between steps – steps that share a sequential ancestor are sequentially ordered themselves. Data dependencies are modeled by parameter passing along the edges between parent and child steps. Steps have parameters specified as in, out, in-out, or local, and edges can be annotated to indicate where these parameters get their values. If an in or in-out parameter is bound from a parameter in the parent, its value is copied from the parent to the child before the child starts. Conversely, if an out or in-out parameter is bound to a parameter in the parent, its value is copied from the child to the parent when the child completes successfully. Therefore, to indicate a data dependency between two sibling steps, we bind the out parameter of one to a parent parameter, which is then bound to the in parameter of the other sibling. Note that because data passing only occurs when steps start or complete, parameter passing can only be guaranteed between siblings if there also exists a control dependency between them. Siblings that are children of a parallel step, for example, can be started at the same time, so a parameter value copied from the parent might not reflect the value produced by a sibling. With this limitation, Little-JIL programmers indicate dependencies between tasks with control dependencies primarily. Note that in future versions of Little-JIL, a mechanism for sibling communication and synchronization will allow another way to specify task dependencies. 2.3

Cascading

Given a Little-JIL program from which a temporal sequence of primitive actions has been instantiated, and which defines the legal sequences of these same primitive actions, we can tell whether the resulting sequence of a selective undo would be meaningful according to the dependencies specified in the Little-JIL program. For example, if the leftmost child of the sequential step in Figure 1 were undone, sequential semantics would dictate the undoing of its siblings too. This can be done automatically and would avoid the problem with the “Car” class example given in Section 1 since the removal of the class would cause the removal of its methods as well.

3

Discussion

In this section we present important questions we plan to explore to determine undo semantics. First, is the Little-JIL language robust enough to capture all required semantics for undo? While we are currently working under the assumption

that control dependencies subsume all important dependencies, data dependencies may exist where there are no statically-determined control dependencies. Also, control dependencies can be rather complex because of Little-JIL’s additional semantics for cardinality and exception handling. Cardinality specifies the number of instantiations to create for a step: zero or more (*), one or more (+), zero or one (?), or somewhere between m and n (m-n) for specified non-negative integers m and n. Exception handling specifies how to handle exceptions thrown from a child step, and then how to continue execution afterward. Once the exception is handled by executing a handler substep (which can have children), execution continues in one of four ways, as shown in the right-most legend of Figure 1: continue continues with the siblings of the sub-step that threw the exception, complete completes the step, restart restarts the step, causing the sub-steps to be re-instantiated, and rethrow causes the step to throw the exception to its parent. With these rich semantics of control and data dependency, determining the semantics for cascading undo for arbitrary Little-JIL programs will be nontrivial. A second question asks which kinds of actions should the user be allowed to undo. In [9], a distinction is made between undo as a regular user action (which itself can be undone), and a meta-command (which cannot). Allowing the user to undo an undo may prove difficult to represent to the user as an option, especially under the cascade model where multiple sub-tasks are involved. This also relates to the issue of how much of the cascade should be presented to the user when it occurs. This question also pertains to the granularity of the task to be undone. Under the Little-JIL model, should selective undo be restricted to leaf steps or can internal nodes be undone? If the latter, this raises the issue of how much the user needs to know about Little-JIL semantics in order to effectively use the tool. A third question relates to the mechanics of reversing an action when an undo is called for. Is reversability always decided at run time? Clearly this is sometimes necessary. For example, if an online purchase of airline tickets were to be modeled, undoing the purchase may or may not be allowed depending on the current date or type of ticket purchased. However, run-time evaluation reduces performance and for any domain, there will be undo steps that work identically in every instance and can therefore be statically embedded. Can a running littleJIL process with instantiated steps be augmented, perhaps with annotations, to reflect where dynamic evaluation of undo is necessary? In a related vein, could annotations be embedded into instantiated steps to provide more information about data dependencies (such as the time of purchase for the above example)? These annotations could provide information that would otherwise have to be inferred from the original step diagram or else not be available at all.

4

Conclusions and Future Work

In this paper we have begun to explore issues of selective undo in the context of sub-tasks which may have dependencies with one another. By using formal process modeling techniques, we have provided a means to capture both control and data dependencies. Our goal is to facilitate cascading selective undo in a conservative way, i.e. in a way that will always leave the user in a meaningful state while providing more flexibility. It is important to note that context-dependent undo elicits other important issues besides the ones given in Section 3. For example, if a software designer renamed class X to Y, and then created a method “addYListener” and an interface “YListener” subsequently, then the undoing of the renaming step should cascade name changes in the method and interface too. The fact that these three constructs are related may originally only exist in the designer’s mind and capturing that external knowledge is an important issue. However, discovering this knowledge is an AI problem that, while interesting, we do not wish to pursue at the onset. For now, we wish to focus on a conservative model of cascading where the knowledge required for a successful cascade are statically determined before moving on to more complicated problems of context dependency. In time, we wish to tackle these other issues as well.

References 1. Berlage, T.: A selective undo mechanism for graphical user interfaces based on command objects. ACM Transactions on Computer-Human Interaction 1 (1994) 269–294 2. Myers, B.A., Kosbie, D.S.: Reusable hierarchical command objects. In: Proc. of the ACM Conf. on Human Factors in Computing (CHI 96), ACM Press (1996) 260–267 3. Archer, Jr., J.E., Conway, R., Schneider, F.B.: User recovery and reversal in interactive systems. ACM Transactions on Programming Languages and Systems 6 (1984) 1–19 4. Abowd, G.D., Dix, A.J.: Giving undo attention. Interacting with Computers 4 (1992) 317–342 5. Mancini, R., Dix, A.J., Levialdi, S.: Dealing with undo. In: Proc. of INTERACT’97, Sydney, Australia, Chapman and Hall (1997) 6. Cass, A.G., Osterweil, L.J.: Process support to help novices design software faster and better. Technical Report 2005-018, U. of Massachusetts, Dept. of Comp. Sci. (2005) 7. Wise, A.: Little-JIL 1.0 Language Report. Technical Report 98-24, U. of Massachusetts, Dept. of Comp. Sci. (1998) 8. Wise, A., Cass, A.G., Lerner, B.S., McCall, E.K., Osterweil, L.J., Sutton, Jr., S.M.: Using Little-JIL to coordinate agents in software engineering. In: Proc. of the Automated Software Engineering Conf., Grenoble, France. (2000) 9. Vitter, J.S.: US&R: A new framework for redoing (extended abstract). In: SDE 1: Proc. of the first ACM SIGSOFT/SIGPLAN software engineering symposium on Practical software development environments, New York, NY, USA, ACM Press (1984) 168–176

Eliciting and Specifying Requirements for Highly Interactive Systems using Activity Theory Robert B.K. Brown

Peter Hyland

Ian C. Piper

Centre for e-Business Applications Research (CeBAR) School of Information Technology and Computer Science, University of Wollongong c/o SITACS Bldg3, University of Wollongong, NSW 2522 Australia {bobbrown,phyland,ian}@uow.edu.au

Abstract The processes of eliciting user requirements and formalising these into specifications are critical for the success of highly interactive systems. These processes are still poorly understood, partly because current methods are usually ad hoc and lack any theoretical basis. A number of researchers have used Activity Theory (AT) to refine these processes and have met with some success. To date, this approach has been more useful explaining the processes post hoc. This positional paper proposes an AT method for requirement elicitation and specification definition. The method is sufficiently prescriptive and well formed that it does not require any detailed understanding of AT. Keywords Requirement Elicitation, Activity Theory, Requirement Specification, Interaction Quotient

1. Introduction Reportedly high abandonment rates and high cost overruns of computer based projects [14, 11] indicate probable misconceptualisation or misrepresentation of the purpose and requirements of the systems under development, requiring costly and time-consuming reengineering effort. It would seem that the products under development are too often not the product that stakeholders and users actually want. Formal methods of generating and specifying requirements have a chequered past when it comes to dealing with interface design. The Human Computer Interface (HCI) community have not adopted formal methods with open arms [12]. If the accurate determination of stakeholder requirements is a significant factor in determining software project success, then perhaps we can turn to a theory based in psychology and sociology to understand these requirements, We propose Activity Theory as a

basis for system design method. It is anticipated that such a method could bring significant benefits, especially to highly interactive systems.

2. The Interaction Quotient Increasing numbers of systems seemingly exist almost purely for the direct use of users and cease to function in any meaningful way without user interaction. A web-based e-commerce system would exemplify such a system. W adopt the term Interaction Quotient (IQ) [2], which reflects the proportion of a system’s functionality which directly interacts with the user. Much, if not most, of the functionality ‘resides’ in relatively close proximity to the user interface(s) of high-IQ systems . A greater number of high-IQ systems are being encountered over time as the World Wide Web becomes the medium for widespread access to systems; particularly those designed for use by a heterogeneous audience. A market for high-IQ systems exists in which we cannot assume any users’ prior computer skills. Ease of use is therefore a crucial factor. Defining the Requirement Specifications for high-IQ systems necessarily involves paying close attention to the so-called Non-Functional Requirements (NFR’s) often associated with the user interface. It is one claim of this proposal, that for highIQ systems, a number of aspects traditionally considered to be NFR’s are in fact Functional Requirements (FR’s). We suggest a simple alternative taxonomy of Internal and External requirements, indicating requirements for those elements visible to the User (external), and those which are not (internal).

3. Activity Theory (AT) Exploring the construction of human consciousness, Vygotsky [18] proposed that humans conceive actions upon an internalised plane of reality. The actions of the human actor (Subject) upon objective reality occur via various

mediating tools (both physical and psychological). The current conception of this is shown in Figure 1.

outcome which is unexpected, something of a side-effect. These outcomes may form components of neighbouring Activities. For example, one Activity may yield an outcome which contributes to, or even defines, the Rules or the Tools etc. of another. [16]

Figure 1: Vygotsky’s AT conception [18] Leont’ev [7] later focussed upon specific activities and proposed a hierarchic model shown in Figure 2, in which any given Activity has a Motive. Within that Motive are Goals oriented Actions. At the base level, atomic Operations are taken depending upon prevailing Conditions. Leont’ev’s conception wa s a powerful and dynamic vision which encompassed the notion of components ‘sliding’ to another level (typically upwards) as the Subject devotes more cognitive attention upon them in the face of some unforeseen complication. Downward ‘slides’ may arise as Subjects become more familiar with Actions and they devolve into near autonomic operations.

Figure 2: Leont’ev’s AT hierarchy [7] Interest turned more to the role of the people engaged in the Activity, and Yrjo Engström [4] developed a larger conceptual matrix, shown in Figure 3, which expands on Vygotsky’s earlier work. The Subject is the person or sub-group whose point of view is analysed. The relationships between these socio-cultural nodes are defined in the Division of Labour node and also in the Rules node which contains social norms, regulations, domain -specific procedures and other constraints. Engström observed that Activities are often interrelated and may impact upon one another. Kuutti [6] extended this observation and described a number of frictions between them. One of the bases of Activity Theory is the observation that an Activity may produce an

Figure 3: Engström’s AT matrix [4] Engstrom’s model incorporates Vygotsky and Leontev and is now the dominant model. The reader should not feel satisfied that this brief sketch has done justice to the domain. Engström’s matrix is a powerful conceptual system whose applications are widespread and much debated in a variety of fields. Bødker’s landmark PhD thesis and text “Through the Interface” [3] and later, Nardi’s text “Context and Consciousness: Activity Theory and Human-Computer Interaction” [9] laid out Activity theory as a useful tool and theoretical framework for Human-Computer Interaction (HCI) study. Several proposals have come to light, notably the checklist idea [5], however it has been stated that HCI has yet to benefit directly from AT [16]. McGrath and Uden [8] observed, as have the authors of this paper, that there is a near total lack of any prescriptive procedures for applying AT in software development. They found it difficult to apply Engström and Kuutti’s theoretical frameworks because prior AT case studies , were almost uniformly narrative in style and lacked a well defined and replicable process. An exception to this lack of well defined process is the work of Vrazalic [16] who proposed a method for evaluating the usability of a system after its completion. This method method is concerned with the broader social context in which the system is used. The user, her social environment, the system and all mediating technologies must be considered. Vrazalic adopts a broader distributed definition of usability in the manner of Spinuzzi [1 3] that incorporates assorted genres, practices, uses and

goals. Under this notion of distributed usability, Vrazalic considers the typical usability laboratory to be an artificial environment that has a number of shortcomings that can skew the results [17]. Vrazalic’s Distributed Usability Evaluation Method (DUEM) deploys a comprehensive series of tests based upon Activity Scenarios generated from intensive: observation of the user in their native work context, interviews with users and moderated focus group discussions. DUEM consists of three interacting phases: understanding user activities; evaluating the role of the system in user activities and analysing and interpreting the results. The first phase produces a shared understanding of user tasks and goals. The second phase produces rich qualitative descriptions of the users’ interaction with the system. The third phase concentrates on identifying points of breakdown, where the system and the activity map contradict. The problems are described via deeply contextual definitions which aid in reaching any negotiated solutions. DUEM uses the notion of distributed usability and AT principles to define contexts of a system’s use by humans. Evaluation is adjudged against criteria derived from these initial findings, based upon user activity rather than system specific requirements. Users are deeply involved in an iterative process through interviews, workshops and observations. [16]. It has been observed [2] that one drawback of DUEM is that evaluators must have an understanding of AT principles to inform their analysis and to help them guide users through the process. This precludes deployment by most software analysts and requirements engineers. It is also acknowledged that it is difficult to assemble a quorum of stakeholders and developers after finalisation to conduct the method. In any event, an evaluation of a product and identification of any areas of weakness after it has been constructed can not inform its design or construction. A chief motivation for the construction of the method briefly described in this paper is to investigate if migrating a DUEM styled AT analysis to the early-phase will address some of DUEM’s identified shortfalls.

4. Why use AT? We have chosen Activity Theory as the underlying theoretical basis for the proposed method for several reasons.

Firstly, AT has been acknowledged in the usability domain in the work of Bødker and Nardi and is recognised as a strong candidate for usability design methods. This stems from its being framed in terms of the Tool(s) used to mediate the creation of an Object by a Subject, as described above. Secondly, the proposed method described in this paper was partially motivated in response to the AT-flavoured DUEM usability evaluation method of Vrazalic. It was hoped that by migrating the AT analysis which underlies DUEM from the testing phase to the desig n phase, the several of the acknowledged shortcomings of DUEM could be addressed and improvements could be achieved in the development workflow. Thirdly, as the notion of modifying DUEM was being investigated, it became apparent that AT offered a strongly coherent and consistent terminology for describing the usability and functional requirements of the system under development. In comparison with the somewhat flexible deployment of terms under the UseCase and Scenario Planning methods, an AT informed method offers strong conceptual consistency, which meets one of the chief normative goals of the research – namely to improve on the stability and specificity of language in the process. By way of example, at least some commonly deployed uses of the term Scenario in Usecase planning are functionally equivalent to the term Activity in AT, suggesting that a more complete theoretical conception of human Activity has much to offer. In a further example of the suitability of AT to a design tool for highly interactive systems, we observe that Leontiev’s hierarchy gives the title Operation to the most basic, atomic task, performed under some Condition – such as depressing the brake pedal in a car when a child runs in front of you. Operational tasks and their informing Conditions therefore correlate strongly with the individual buttons, switches, dialogues and component widgets of computer system interface. Likewise, we find that Leontiev’s higher level, that of Goal driven Actions, is highly suggestive of how screen components may be composed meaningfully together into interface screens. The Motive driven Activity itself, Leontiev’s highest level, corresponds strongly with the set of interface screens used in a given role to facilitate a specific Activity. Fourthly, AT presents as a strong candidate for informing an end-to-end methodology. During early phase investigation it was hoped

that AT could drive the production of descriptive Requirements from user utterances. It has become apparent however that more prescriptive Specifications seem to fall out of the process with little extra work. When tied with a DUEM style cross -check in the end-phase, an end-to-end methodology seems to become an achievable goal. This paper presents early work towards that goal. Finally, we recognise that support for reusability would be an admirable trait in a system design method. It is apparent that at least some degree of reusability may be achieved if the conceptions of the system may be decomposed and recomposed in an efficient way to cater for the reallocation of tasks and actions among the users of a system (such as when an organisation undergoes a restructure of personnel). Considering this issue, we initially investigated some of the Actor Oriented Conceptual Modelling (AOCM) proposals such as i* [ref-Yu]. When considering that under organisational restructuring some tasks are reinstantiated and reconfigured under different coincidences of people in roles; perhaps it is better to consider the goal-driven tasks people are engaged in rather than those people who are assigned to those tasks under any single organisational structure. Accordingly, we lean towards an Activity Oriented rather than Actor Oriented conception of the system. Once again, AT presents as a strong candidate theoretical basis. The issue of Activity Oriented Conceptual Modelling and its contribution to reusability is a strong preliminary result, and one which will be explored in detail in later papers.

5. Towards Activity Theoretical Requirements Elicitation and Specification Although AT seems useful in requirements gathering, especially for interactive systems, a formalised and systematic method is required. This paper sets out early work in proposing such a method. While Vrazlic has demonstrated the usefulness of AT in the evaluation of interactive systems, any Usability Evaluation Method can, at best, indicate quality “after the fact”. What is needed is a simple, yet systematic and rigorous Activity Theory method for requirements elicitation and analysis. Such a method should be sufficiently defined that the user does not need to

understand AT to use this method. Moreover, the output of this method should be a set of requirement specifications that would fit seamlessly into more conventional systems development paradigms. Thus the proposed method provides a “grey-box” set of tools that can be plugged in to the requirements elicitation and analysis process by any competent analyst, and produces specifications that are useful and understandable to any competent programmer. It is our hope that an Activity Theoretical method for requirements elicitation and analysis should yield reliable and verifiable Requirement Specifications whilst taking account of elicited user activities. Using a common taxonomy of system-transparency, we aim to provide “grey box” specifications for the system components to be designed. We hope to present this description in the form of commonly encountered Requirements Specifications, familiar to most system builders; and we further hope that the method will be clearly prescribed as to allow analysts without AT experience to use it, most especially for high-IQ projects. The preliminary conception for the complete method comprises four components: (1) an extended set of elicitation questions, laid out in a systematic and hierarchic fashion; (2) a template for recording the higher-level goals and objectives onto an Engström matrix; (3) a template for recording the lower-level goals and objectives, using a graphical conceptual matrix for the Action level, and (4) a set of ordered steps for processing the output and producing workable, reliable and verifiable Requirements Specification, of a form familiar to system developers. We present some preliminary ideas for steps (1) through (3) below.

5.1. Initial Analysis A nested sequence of elicitation questions will be put to stakeholders, clients and users. The objective of the questions is to identify: ? Subjects (including sub-groups) – being those persons and groups who perform an Activity o Relevant communities they belong to o The Divisions of Labour between the community members o Rules of membership and responsibility

? ? ? ?

Identification of all Community members, which may identify Subjects of other Activities. The high level Object and Motives of these Subjects Lower level Goals within these Objects, which indicate Actions The Conditions under which Actions occur, which identifies Operations and indicate functions and controls which may need to appear as user interface components. Note: All the actors who share an identical point of view on the Object comprise the Subject of an Activity. Conversely, associated actors who share the Object but may perform slightly different roles may appear in the Community node.

Observe that each person (who is later fitted into a Subject group) may use different terms to describe their Activity, complicating the capture of such details [15]. The Activity Theoretic approach could and should generate an Engström matrix for each identified Subject, however, a layer of processing will be required to synthesize the varied Subject generated descriptions into a maximally consistent subset of Activities. The most basic Activity Theoretical principle to inform this process is that Subject-specific matrices with a common Object should merge to a single Activity, whilst those with identifiably different Objects should form distinctly separate though neighbouring Activities. A fully developed process of this kind will elicit sufficient data to produce a set of annotated Engström matrices, awaiting processing. We anticipate that this stage of the process could benefit from a semi-automated aid, which would assist analysts who lacked significant experience in Activity Theory.

5.2 Processing the Output From the stakeholder descriptions, a complex set of interrelated Activities – an Activity Network - will be identified. From this Activity Network, a subset of Activities will be identified as being supportable by computerised Tools, which together form our interactive system. The rema ining steps in our proposed method will produce requirement specifications for these tools and for the system as a whole.

This is the first analytical scratch-board used, the System Space Build. Each Activity in the network will have a set of Actions, elicited during the initial analysis. These Actions will be collected into a Combined Action Table (CAT), the second analytical scratch-board. Analysis of the CAT will reveal connective flows of data between the actors and their Activities. Other connective relations may be observed, such as the granting of access, a request for action, the imposing of an obligation and so on. A resulting analytical scratch-board is the Patch Panel, which sets out all the connective relations identified from the CAT. The termini of these connective relations indicate where some interaction between users or between a user and the system occurs. These strongly correlate with interface components (Switches), which are also informed directly from Operations, as identified from Activities in the initial analysis. The final analytical scratch-board is actually a model of the overall system. It is a hierarchic representation referred to as the “5-S” model. This model is not one of the processing steps as such, but is constructed as a part of the output. Ultimately the method will produce as output a series of requirement specifications. Currently, we envision these having a naturallanguage format of a sort familia r to most system designers. Rather than following the common taxonomy of Functional, Non-Functional, however, we observe that under this analysis many of the interface related specifications usually considered NFR’s may now be seen to be functional (especially for high-IQ systems). We therefore propose a replacement taxonomy: External Requirements, being those visible to the user and comprised largely from the classifications and composition of connective relations and their Switches; and Internal Requirements, being those not visible to the user and comprised largely from the linkages between Activities.

5.2.1 The “5-S” Model Using our AT approach to requirements elicitation and analysis, it appears that five distinct layers of specification components are supportable and useful. As shown in Figure 5, these are:

1. System. This is a broad descriptive specification of the common computerised system of tool(s) that best mediate and facilitate the activity(s) under examination. Care should be taken not to simply replicate the functionality of any pre -existing tools, especially computer-based ones. Consider also, that it is most unlikely that all Actions within the Activity will require the mediation of the system; telephones, paper, physical aids and face to face conversation among many other tools, are likely never to be fully subsumed into computerised systems. Initially we envision the System as a space whose external boundary only is known. The specifications produced ultimately by the method should provide meaningful definition to the boundaries of this space as well as a number of insights into its internal structures by way of describing data flows and other connective relations. In this manner, we hope to produce grey-box specifications of the system. 2. Station. The Activity Network comprises a number of Activities. Each Activity has a Subject and an Object and a comprehensive analysis of the Activity Network will identify all the discrete Activities i.e. those which do not have the same Subject and Object as any other Activity. . Activities which share common Subjects may be considered to belong to a set of Roles, referred to as a Station, played by any given individual or group of individuals, which in AT terms constitutes the Subject. This is a reflection of the stakeholders and the structure of their activities which are under investigation. Note that similar Activity Networks may exist with differently composed Stations. For example, two different client organisations may have seemingly identical objectives, high-level goals and motives and thus conduct seemingly identical Activities; however, they may assign these roles differently among their members, and such a different composition of Actions and Operations would require differently composed Stations. This is a key finding and permits the reuse of the hierarchically modular components identified by our method. 3. ScreenSet These are associated one to one with each Activity of the Activity Network. These are sets of user-interfaces required for any one Activity. Stations may contain several ScreenSets, one for each Activity it contains. 4. Screen. Each Screen should correspond closely with one or more identified Actions of the Activity which

informs its parent ScreenSet.. Identification of the Screen emerges from careful analysis of the previously observed connective relations. Any ScreenSet may exhibit a functional aggregation of such relations, which will suggest the formulation of a single interface Screen. In this manner, our method provides some insight into a workable composition of interface components to accord with the Activities of the stakeholder(s). 5. Switch. AT defines an Operation as a simple, even autonomic response to changing conditions. In our method, each Operation may correspond to a Switch i.e. some component on the screen. These Switches may take the form of a menu, a single menu item, dial, button, dialogue, slider or any other mechanism for low-level human-computer interaction.

Figure 5: The “5-S” Model Note: As described above, according to Leont’ev, elements of the Activity residing at one level, may occur at a higher level, often under error-related circumstances. To some degree this is subsumed in the human use of the system; when a user encounters some difficulty with an Action, they may seek help or deploy other tools or rules to aid them, and thus elevate an Action to the status of an Activity. At a more concrete level, this hierarchic upwards slippage is accommodated by the notion that a given Operations level component (Switch) may open up an Action level component (Screen) to resolve an issue or allow for configuration etc. A Screen level issue may

give rise to the need to step into, or summon, another Role level component (ScreenSet). Hierarchic ‘slippage’ downwards, typically familiarity-driven, arises when for example, an Action becomes autonomic. A well-designed system should identify such potential down-slips and provide the user with the means to assign an entire Action to a Switch, in the manner of a macro or customised button. Such notions are already well known in HCI and need no further explication here. For further detail we recommend the work of Jacob Nielsen. [10] We observe that in building these five layers of specification, the System, ScreenSets and Switches should be more-or-less directly derivable from the Activity analysis. Deriving Stations and Screens however requires making informed choices as to the clients’ organisational structure of roles and the composition of connective relations and Switches from the Patch Panel.

5.2.2 Patch Panel Defining the layers of the specification is not sufficient in itself to provide workable speculations. At this stage we have merely demonstrated the shape of the System and sketched the facets with which users interact (Stations, Screens etc). To yield a true grey-box specification, it is necessary to identify some of the inner workings behind the interface. [1] In this early proposal, these take the form of connective relations primarily between both Screens and Stations. In a sense, these may be thought of as the ‘wiring’ behind and between the ‘control panels’ of the grey box. Indeed, we choose to view the ScreenSets, Screens, Switches and connective relations as analogous to a patch-board. The CAT will indicate the locations of the ‘wiring’ junctions, but it is the composition of these ‘wires’ which will define individual Screens. There are three principle classes of connective relations we need to identify: 1. Intra-Screen These join switches to switches within one interface, and yield External requirements of the interface layout and design. 2. Intra-Screenset These join between interface elements from Screens within the one ScreenSet, and thus indicate the need to move from one interface to another. Our intuition is that the composition of

Screens should attempt to minimise the number of such moves. 3.

Inter-ScreenSet a. Station-to-Station These represent linkages between Actors and directly reflect the passing of data, control, responsibility etc between users. b. Intra-Station These join screen elements from screens within the one ScreenSet, and thus indicate situations where an actor needs to change ‘mode’ as they adopt one of their other ‘roles’. Within these classes of connective relations, we hope to be able to identify and classify ‘joins’ of data flow, of access, of control, of responsibility, issues of compatibility and other aspects relevant to the framing of Requirements Specifications for the System. We anticipate greater formality may be seen if connective relations are expressed in terms of Deontic and Temporal logics.

6. Sample Activity Network To illustrate the proposed method, we consider an activity familiar to many academics; that of administering an undergraduate assessment task. Here an Academic (A) devises an assignment task and a marking-scheme. A Tutor (T) employed under (A) deals most directly with the Student (S). The student receives the assignment from the Academic, performs the requested task, and submits a paper back to the Tutor, who marks it, according to the marking-scheme and forwards the results to the Academic for submission into a permanent record. Upon examination of this Activity, reflection on our own experiences and casual interviews with our peers, tutors and students, we derived tables of Actions. We carefully avoided recording the functions of any of the extant computer-based systems that are currently used at our test site. There are numerous Activities in this network, one for each Objective in accordance with Activity Theory. These may readily be clustered within three Stations (Roles), those of Academic (A), Tutor (T) and Students (S). This paper proposes the approach only, so for the sake of simplicity and brevity, this example subsumes ScreenSets within their parent Stations. Likewise, we do not here present the individual

Activity Tables, the System Space Build or the Patch Panel. A partial CAT is here given, with brief indications of some components that derive from it, simply to offer the flavour of the method. A more complete working of this example will be given in future publications. From the partial CAT below, Entry 3 indicates a data flow from (A) to both (S) and (T). This implies the presence of ‘send’ and ‘get’ Switches. Entry 4 however requires secure transfer, implying an internal requirement of encryption and/or access control and external requirements for appropriate Switches. Entry 5 implies a conversational exchange which may benefit from properties of simultaneity and/or persistence. A simple email module may not suffice. Entry 16 implies a secure long-term storage facility to which (A) has write access and (S) has limited read only access. Deeper analysis of the connective relations at (A) may suggest that the Entry 16 Switches reside in a different Screen (or ScreenSet) to, say, the Entry 3 Switches. No Screen

Sets

Actions …

(A) Send assignment > (S),(T) 3 (A)(S)(T) (T) Get assignment < (A) (S) Get assignment > (A) 4 (A)(T) (A) Send mark-scheme > (T) (T) Get mark-scheme < (A) 5 (A)(T) (T) Query < > (A ) (A) Field query < > (T) (S) Query assignment < > (T) 6 (T)(S) (T) Field query < > (S) …

11

(T)(A)

12 13

(A)(T) (A)

Comment

(T) Submit mark > (A) (A) Get mark < (T) (A) Check mark < > (T) (A) Adjust mark

document transfer document transfer SECURE exchange SECURE exchange document transfer SECURE create local record

…

16

(A)(S)

(A) Declare mark > (S) (S) Get mark < (S)

write-read secure

…

Table 1: Extract of a Combined Action Table (CAT) for the Activity Network, ‘administer undergraduate assignment’ 7. Future Work Several foreshadowed papers will set out each of the method components in greater detail within a fully worked case study; these being the Initial Analysis, the System Space Build, the

Combined Action Table, the Patch Panel and the 5-S hierarchic model. We see potential for the Initia l Analysis to be facilitated by use of a semi automated tool. The nature of such a tool will be informed by the full development and explanation of these steps. We anticipate that a useful degree of formalisation may be introduced to the formulation of Requirements by deploying elements of Deontic and Temporal logic notations to the analysis of the Patch Panel. As observed in the brief description above, the Station level of the model can be seen to reflects organisational structure(s) and their analysis may permit the re-use of the hierarchically modular components identified by our method. This is, in itself, a significant result and will form the substance of a later paper.

8. Conclusions Our method shows potential to be a systematic and prescribed process with a solid theoretical base. We believe it will elicit useful Requirements from statements elicted from stakeholders without requiring the analyst to have a deep knowledge of Activity Theory. For high-IQ systems at least, we note that the NFR/FR taxonomy commonly applied to issues of usability and interface design, has less meaning under our representation. We therefore propose the simple External/Internal requirement taxonomy, tied directly to the analysis of a given Systems’ Patch Panel. We believe our system has the potential to address an ongoing fundamental problem in system analysis and design, that of the coincidence of roles in users. Many systems are designed with a role-based interface division unmapped to organisational division of labour, and incapable of ready translation or adaptation in the face of any organisational re-distribution of roles.

9. References [1] Arpaci-Dusseau, A.C. & R.H., “Information and Control in Gray-Box Systems”, proceedings of the 18 th ACM Symposium on Operating System Principle Vol35 Issue 5, 2001 [2] Brown, R.B.K., “Design-Side Considerations: A Reaction to DUEM”, proceedings of the Australian Conference on Computer Human Interaction, (OZCHI’04), Wollongong, 2004

[3] Bødker, S., Through the Interface: A human activity approach to user interface design. Lawrence Erlbaum Associates, Hillsdale, NJ. 1991 [4] Engström, Y., Learning by Expanding: an activitytheoretical approach to developmental research, Orienta-Konsultit Oy, Helsinki, Finland, 1987 [5] Kaptelinin, V., Nardi, B., & Macauley, C., The Activity Checklist: A tool for respecting the “space”of context, interactions, Vol ^ Issue 4, pp.27-39 1999 [6] Kuutti,K., Activity Theory and its applications to information systems research and development, in Nissen, H.E., Klein, H.K., & Hirsheims, R. (Eds) Information Systems Research: Contemporary Approaches and Emergent Traditions Elsevier Science, Amsterdam, pp.529-549, 1991 [7] Leont’ev, A.N., Activity, Consciousness, and Personality, Prentice Hall, 1978 [8] McGrath, G.M. & Uden, L., “Beg, Borrow or Steal: OK, but it’s not all One-Way Traffic!”, 2000, http://citeseer.ist.psu.edu/668644.html [9] Nardi, B. A., Context and Consciousness: Activity Theory and Human-Computer Interaction. MIT Press, Cambridge, MA, 1996 [10] Nielsen, J., Usability Engineering, Academic Press, Boston, 1993 [11] OASIC Study, “Why do IT projects so often fail”, OR Newsletter, 1996, 309, pp12-16. [12] Paterno, F., & Palanque, P., “Formal Methods in Computer Human Interaction: Comparisons, benefits, Open Questions”, workshop session, CHI’96, 1996 [13] Spinuzzi , C., “Grappling with distributed usability: A cultural-historical examination of documentation genres over four decades”. Proceedings of the 17th annual international conference on Computer Documentation, 12-14 Sept. 1999, New Orleans, USA. [14] Standish Group, The CHAOS Report, 1994, http://www.standishgroup.com/sample_research /chaos_1994_1.php [15] Sommerville. I., P. Sawyer, and S. Viller, Viewpoints for requirements elicitation: a practical approach, Proceedings of the 3rd IEEE international conference on requirements engineering (ICRE’98), Colorado Springs, USA, 6-10 Apr. 1998, pp.74 – 81. [16] Vrazalic, L., “Towards Holistic HumanComputer Interaction Evaluation”, PhD Thesis, University of Wollongong, 2004 [17] Vrazilic, L., Gould, E., “Towards and Activitybased Usability Evaluation Methodology”. … 2003 [18] Vygotsky, L.S, Mind in Society Harvard University Press, Cambridge, MA, 1978

Towards a Pattern Based Usability Inspection Method for Industrial Practitioners Martin Schmettow Fraunhofer IESE, Fraunhofer-Platz 1, D-67663 Kaiserslautern [email protected]

Abstract. Usability Inspections are essential for early detection of defects in UI design, but they require sound usability knowledge. Usability Patterns are the state-of-the-art format for describing usability knowledge. Thus, it seems obvious to use them as a means for evaluating the design of user interfaces. In this paper a usability inspection method based on usability patterns is introduced, which may be especially well suited for practitioners in software development, who are in general not usability experts.

1

Introduction on Usability Inspections

Inspections in Software Engineering are techniques to find defects in software products. As they can – opposed to testing – be performed on static artifacts (meaning not runable modules), they are excellent in finding defects very early, which knowingly saves time and money [1]. Usability inspections in general share these goals and are in the underlying principles and processes quite in line with software inspections. But they seem to not be as established in software development, which regularly leads to late and costly defect elimination or, in the worst case, unusable products. The main problem in current software development practice is, that construction as well as evaluation of user interfaces (UI) are most often done by developers, who are in general not dedicated usability experts [2]. Thus usability evaluation methods are needed, which are well suited for non usability experts. To be well suited for non usability experts a method must at least provide comprehensible usability guidelines (or usability knowledge in general) and must support the evaluator to take the perspective of the common user. Of additional benefit is, if a method facilitates explicit recommendations for design improvement instead of mere defect identification. Indeed, the known methods provide usability knowledge via differing forms of ergonomic guidelines, like heuristics [3], psychological theories [4] or interface standards. But these guidelines have some shortcomings which make them cumbersome or simply unusable for software developers. And, in fact, several studies show, that less skilled evaluators are less efficient in identifying usability defects and that there are considerable differences in efficiency even for high skilled evaluators [3, 5]. In the following we propose the construction of a usability inspection method, which uses usability patterns as a source of usability knowledge. We are optimistic, that this

method would be well suited for software developers, because it makes use of detailed and problem-oriented usability knowledge and has a procedure to strongly encourage taking the perspective of the user.

2

Usability Patterns as a Source for Usability Inspections

Usability patterns are on the way to become the most prominent format to collect, describe and structure usability knowledge [6]. They describe well established solutions together with preconditions and rationales for ergonomic design of user interfaces. Usually they are collected and classified in pattern collections or languages for general application domains or more specific purposes [7-9]. Having a further look into the nature of usability patterns, several advantages and also a few problems arise for basing a usability inspection method on patterns. Compared to usability heuristics and interface standards, patterns have several advantages: x Patterns state preconditions and rationales for the proposed solution, which helps to identify the correct pattern for a specific situation, while the decision of when to apply a specific heuristic is left to the evaluator. x The descriptions of problems and solutions are far more verbose and problemoriented than with heuristics, which makes them easier to understand and correctly apply for evaluators.

x

Opposed to heuristics patterns describe concrete solutions, which makes them a natural means for design recommendations in an evaluation. Additionally, the idea of describing best practices as patterns is well established in other fields of software engineering for years [8, 10, 11] and thus is widely known and accepted by software developers. On the other hand there are several problems in using patterns as a source for usability inspection: x Since pattern collections use to be very large (but still not as large as design standards), a powerful information retrieval mechanism has to be provided to find applicable patterns for each step of an inspection. x Established pattern collections use to refer to quite general application domains (WIMP interfaces, Web sites), which might be applicable but not sufficient for applications in specific domains. However, there already exist some collections for special purposes [12, 13]. x The fact that a pattern proposes a good solution for a specific problem does not deny, that the designer might have found an innovative or at least alternative good solution for the problem. However, using a standardized solution for similar design problems leads to a high consistency inside and across applications, which is desirable for e.g. better learnability. x The completeness of defect identification is highly dependent on the existence of applicable patterns. But this is an inherent problem of any guideline or the mere expertise of an evaluator, which might always be limited, inadequate or outdated. With the pattern-based method this problem will degrade

with the appearance of more complete and also domain-specific pattern collections. These problems have to be solved by the construction of the inspection method or at least have to be kept in mind when using it. In the following a proposal for the construction of a usability pattern based inspection method (UPI) is outlined.

3

Description of the UPI method

Process of the method The overall process of the UPI method is very similar to other inspection methods in UE and SE: In the preparation phase evaluators will be selected and advised and the artifacts and evaluation environment is prepared. Especially the compilation of an adequate pattern collection is crucial here. For many applications the general pattern collections available [14, 15] should be sufficient, but these may be augmented with domain specific patterns for special purposes (examples of how to build domainspecific pattern collections can be found in [12, 16]). Then several single evaluation sessions are performed before a final review meeting is held which consolidates the results and prepares them for presentation. The innovative core of the method is the application of usability patterns as evaluation criteria in the single evaluation sessions – therefore this will be the focus of the further description. Principles of the method In a first overview the method can be described in seven principles which form the basis of the single evaluation procedure: 1. Guidance for the inspection is provided by detailed descriptions of user tasks, which are to be performed by the evaluator. 2. The evaluator is advised to always be aware of what user activity she is performing, what is called self monitoring further on. The user activity is selected from a predefined set. 3. Checkpoints for the inspection are defined by changing dialogues and shifting user activity. 4. Matter of inspection are dialogues, screens and single screen elements. 5. Criterion for evaluation is a match or mismatch of a dialogue or screen element to a specific pattern. 6. Selection of applicable patterns is done by the current user activity and – of course – the patterns described preconditions. 7. Recording is done immediately for every single step of the evaluation for positive as well as negative results.

Single Evaluation Procedure The procedure of the method as well as the recording format is quite well-defined. Since the inspection path is guided by a detailed task description, the first step for the evaluator is to start doing a task or a subtask. While he is doing the task he is held to monitor his own flow of activities. These activities stem from a predefined set of 15 general user activities (some examples are shown in Figure 1) and every pattern in the used pattern collection is classified to one or more of them. This self monitoring of activities is one central idea in the method and acts in a threefold way: 1. It is the pacemaker in the procedure: Whenever the current user activity changes, the current dialogue is compared to usability patterns. 2. It is the main information retrieval mechanism for preselecting patterns for a given situation. 3. It enables the evaluator to become better aware of the user perspective. This works, because the self monitoring hinders the evaluator to use the application in an intuitive or highly automated manner - especially when he is an application or domain expert. For example, when a new dialogue appears, the evaluator could ask: Would the user immediately comprehend the dialogue and start entering data or would he first orient himself or search for help? The procedure of selecting and applying patterns to each dialogues, as outlined in Figure 1, is as follows: When the activity changes and/or a new dialogue appears, the evaluator will select all patterns, which are classified to this activity, and check their applicability in two steps: First he compares the current situation to the preconditions (problem, context and forces) of each pattern. Then he tries to imagine roughly, if and how each pattern could be instantiated into the current dialogue. If both checks preconditions and imagination – pass, the pattern is considered as applicable. The set of applicable patterns is then compared to the current dialogue to decide, if they match fully, partly or not at all. In the latter two cases a mismatch is recorded and a recommendation according to the pattern’s solution is given.

Self Monitoring Identify

User Activities: x x x x x

Orientation Data Entry Search data Find function Recover from error

Select Pattern Collection

Apply Dialogue

Figure 1 Procedure of selecting and applying patterns to the dialogue in question

4

Evaluation

The method was successfully applied in one industrial and two governmental projects as well as in several minor examples, where it in fact revealed a significant number of usability problems. In one of the projects, it was also shown, that the method is quite applicable to early design artifacts (static UI prototypes). In all three projects the results were presented to the developers and project sponsors and were in general easily understood and accepted by those. The retrieval of applicable patterns through the selection by user activities works quite well. For example the collection of patterns for windows-like GUIs has about 75 entries. Far most user activities reduce the number of preselected applicable patterns to below ten results (which is below the number of heuristics proposed by Nielsen [3]). At the moment a comprehensive validation study is being prepared, which will provide statistical measures for reliability, validity and efficiency compared to either heuristic evaluation or usability testing. Since the participants will mostly have few experiences with usability only, this experiment will also stress our hypothesis, that the Usability Pattern Inspection method is outstanding well-suited for non-experts.

5

Conclusion

In this paper a usability inspection method is presented which makes use of usability patterns as a source of usability knowledge. It is believed that the method is well suited for software developers, who are in general not experts in usability evaluation. This is mainly due to two facts: (1) patterns describe usability knowledge in detailed and well structured way, stating preconditions and rationales for concrete and well established solutions (2) the principle of self monitoring forces the evaluator to take the perspective of the common user. The main problem in using patterns is the mere size of pattern collections. This was met with the procedure of selection by user activity, which basically results in a context-adaptive checklist. When the method is empirically proven to be reliable, valid and efficient, in the future we will focus on the following improvements: 1. The collection of patterns will be enhanced for special application domains (e.g. business process modeling tools [16]). 2. Tool support will be enhanced for session recording and report generation. 3. Comprehensive training material for industrial practitioners will be developed. 4. When the detailed characteristics of the method are explored, it should be integrated into a well defined UE/SE process framework.

6 [1]

References B. Boehm and V. R. Basili, Software Defect Reduction Top 10 List. In: Foundations of Empirical Software Engineering, B. Boehm, H. D. Rom-

[2]

[3] [4]

[5]

[6] [7] [8] [9]

[10] [11]

[12]

[13] [14] [15]

[16]

bach, and M. V. Zelkowitz, Eds. Heidelberg, Germany: Springer, 2005, pp. 426-431. J. McKirdy, An Empirical Study of the Relationships Between User Interface Development Tools. University of Glasgow, Department of Computing Science, Technical Report TR-1998-06, 1998. http://www.cs.unb.ca/profs/jlumsden/publications/tech_report_tr-199806.pdf J. Nielsen, Heuristic Evaluation. In: Usability Inspection Methods, J. Nielsen and R. L. Mack, Eds.: John Wiley & Sons, 1994, pp. 25-61. C. Wharton, J. Rieman, C. Lewis, and P. Polson, The Cognitive Walkthrough Method: A Practitioner's Guide. In: Usability Inspection Methods, J. Nielsen and R. L. Mack, Eds.: John Wiley & Sons, 1994, pp. 105-139. H. W. Desurvire, Faster, Cheaper!! Are Usability Methods as Effective as Empirical Testing? In: Usability Inspection Methods, J. Nielsen and R. L. Mack, Eds.: John Wiley & Sons, 1994, pp. 105-140. J. Borchers, A Pattern Approach to Interaction Design. Chichester, England: John Wiley & Sons, Ltd, 2001. M. Fowler, Analysis Patterns : Reusable Object Models. Boston: Addison Wesley, 1996. M. Fowler, Patterns of Enterprise Application Architecture. Boston: Addison-Wesley, 2002. E. Gamma, R. Helm, R. Johnson, and J. Vlissides, Design Patterns - Elements of Reusable Object-Oriented Software. Boston: Addison Wesley, 1994. D. M. Dikel, D. Kane, and J. R. Wilson, Software Architecture: Organizational Principles and Patterns: Prentice Hall PTR, 2000. L. Hagge and K. Lappe, Patterns for the RE Process. In: 12th IEEE International Requirements Engineering Conference. Kyoto, Japan: IEEE, 2004, pp. 90 - 99. M. Leacock, E. Malone, and C. Wheeler, Implementing a Pattern Library in the Real World: A Yahoo! Case Study. Presented at Information Architecture Summit, 2005. R. Gibbert, Eine Design Pattern-Sprache für mobile Applikationen mit dem Schwerpunkt Navigationssysteme. diploma thesis 2003. M. v. Welie, Patterns in Interaction Design. http://www.welie.com/patterns/gui/index.html (last accessed: 2005) J. Tidwell, COMMON GROUND: A Pattern Language for HumanComputer Interface Design. http://www.mit.edu/~jtidwell/interaction_patterns.html (last accessed: 2005) K. Kohler, D. Kerkow, S. Hess, and K. Schmid, Best Practices und Usability Pattern für Geschäftsprozess-Modellierungswerkzeuge. Fraunhofer IESE, Kaiserslautern, Germany,IESE.Report, 060.05/E, July 2005. http://www.iese.fhg.de/Publications/Iese_reports/

The Role of Usability Testing in Software Quality Control: Finding Mismatches, not Defects Paul Hubert Vossen, {voice & visual interaction design}, Kanalstraße 25, D-70771 Leinfelden-Echterdingen, Germany, [email protected]

Abstract. Suppose, you have thoroughly tested some application system. You have found some software bugs, but all critical errors and even most cosmetic errors have been repaired, as confirmed by regression tests. Furthermore, due to the use of advanced testing specification techniques, the expected coverage of potential failures was very high. Why should you still want to do Usability Testing? Some answers are given below. Most importantly, the notion of a mismatch between human beings and information appliances is introduced, which differs on several points from the classical notion of a software defect. This has farreaching implications for testing management and practice, so at the end of this paper we will introduce a general reference model combining Software Testing with Usability Testing.

1 Introduction Usability testing is necessary, when the software or product developed will actually be used by human beings, i.e. when we are talking about interactive systems as opposed to automated or autonomous systems. In interactive systems, the computer is dependent upon input from a human being for its ongoing operation and vice versa: a human being who is in charge of the computer needs specific information from the computer in order to check the progress and state of the ongoing job and to decide what to do next. In this context, it is irrelevant, whether we are talking about experts or novices, about regular or casual use, about work-related or leisure-related activity. What counts, is that human beings will interact and communicate with a responsive device and thus become part of an overall human-computer system, sometimes even without an (easy) way out. The crucial issue with interactive systems is to draw the right boundary between system and environment, or context. This means, taking the human being on board of the system, and to regard him or her in the role of the user of this system as an essential part of the system, albeit one that cannot as easily be modeled or changed as the rest of the man-made system. This view is not at all new, being the essence of a genuine systems approach to information engineering, including disciplines such as human-computer-interaction (HCI), industrial or engineering psychology and information or cognitive ergonomics.

Trivial as it may seem, this is all too often forgotten, neglected or downplayed in the turmoil of software and hardware development and testing, which has severe and deep implications for the so-called User Interface, because there is a strictly asymmetric relationship here: you have to adapt the computer system to the human capabilities and limitations in order for the entire system to work all right. Or, in the words of Raskin [1, p. xix]: “If a system’s one-to-one interaction with its human user is not pleasant and facile, the resulting deficiency will poison the performance of the entire system, however fine that system might be in its other aspects.” Failure to do this will result in software that may work fine from a purely technical point of view, i.e. according to design specifications based on technical (mainly functional) requirements, but that does not match actual human, organizational or social requirements –and thus will be useless from a pragmatic point of view –a fate shared by so many software products of the past.. This paper will address these issues in more detail through the following topics.

2 Mismatches in the user interface and interaction Mismatches between human beings and information systems can occur at all levels of Human-Computer-Interaction (HCI): the lexical, syntactical, semantic and pragmatic level of user dialogs (cf. the famous Seeheim model, defined in analogy to the OSI model). Also, mismatches may pertain to human perception of system output (e.g., reading a screen of text, viewing a picture, hearing a voice or signal) or to system input of human behavior (e.g., keyboard typing, moving or clicking the mouse, selecting an area on a touch-sensitive screen). Plentiful examples may be found e.g. in the books and columns of computer-risk-guru Peter Neumann about cases where assumed safe technology has (deeply) failed by so-called human error [2], or in books and papers of HCI-converted computer specialists like Alan Cooper [3]. Even a technically perfect system, i.e. all hardware and software bugs have been detected and removed 1, may contain all kinds of user interaction mismatches. The basic reason and explanation is, that mismatches can only be observed and identified, if and when the interactive system is put to test with actual users, not just by software testers alone. Of course, software testers may act as stubs for the users, but they will only be able to detect those mismatches which happen to correlate to deviations from usability specifications, i.e. a type of specification rarely seen in practice, although there have been several initiatives since the mid 80's to integrate them in the software design and development cycle. Detecting, analyzing and resolving those human-computer mismatches is indeed a real challenge, but a deeply rewarding activity too, as it tells us a lot about our own 1

The assumption, that this is at all possible, is indeed not supported by mainstream testing experts for several reasons, one of which is that it is still very uncommon to find fully up-todate and worked out hardware and software specification documents, so that looking for and finding bugs is very often left to the tacit knowledge and vivid imagination of testers, who should at the same time be knowledgeable in whatever application domain is being tested.

3

“make-up” [1]. Also users – even expert users – aren’t always aware of the mismatches they are exposed to, e.g. because they have learned to take shortcomings in software and other interactive products for granted, as evident by the market success of a range of inferior software products. But then: market success is not the same as product quality. Human-computer interaction experts have also learned to deal with that and to offer advice on how to get the right balance between (minimal) ergonomic requirements2 and (maximal) economic profit of applications under development.

3 Tracking down usability problems Great software: few bugs left, delivered just in time within budget, customer completely satisfied and willing to pay. But wait a moment, there’s a tiny problem left: users can’t work with it, they even refuse to do so, because the application is difficult to learn, to use, to adapt to their working habits. Unfortunately, this happens more often than not, and it is documented often enough, e.g. [4]. Why does it happen? We suppose, that it is not simply that usability requirements engineering is not well mastered at this moment and not well integrated into design and development activities, despite some very useful texts aimed at software engineers [5] and despite noteworthy initiatives to get standards in place (e.g., ISO 9241 and ISO 13407). In fact, the source of most (severe) usability problems in software is more hidden and complex: many violations of usability requirements have little to do with or can’t be easily tracked down to a single piece of code –and even if they could, their rectification often involves and affects neighboring or dependent elements, modules and other components in the user interface/interaction and beyond. That makes usability engineering (in particular, testing) so hard to understand and to integrate with standard software design and development (testing) approaches. Presumably, it calls for a kind of expertise, i.e. knowledge and skills, that is not yet part of the mainstream professional training of software architects and engineers, as will be explained in the next topic, and less still to be observed in current practices of software design and development.

4 The profession of usability testing Usability testing is a well-developed, well-established profession. As such, it is a subdiscipline of usability engineering3 , which itself is the “hard”core of the interdisciplinary field of Human-Computer-Interaction (HCI) –with deep roots in biology, er-

2

3

See e.g. ISO 9241 (the 17 part international standard on hardware-software ergonomics) and many related international as well as national ergonomic standards and initiatives for testing, accreditation and certification procedures. See e.g. Nielsen [6] for an easy introduction to the field. More up-to-date views and topics can be found in e.g. Chen [7], especially the first three chapters.

gonomics, psychology, sociology, and some other human (behavioral as well as cognitive) sciences. As a testing discipline, usability testing has much in common with conventional software testing approaches, but it also clearly differs on a couple of methodical and technical points. However, it is by no means new, or underdeveloped. On the contrary: all you might ever want to know about usability in general and usability testing in particular can be found on the web4 or by more conventional means, including professional5 and commercial organizations like the UPA. So why this recent hype about usability? Let's face the current situation: a lot of young IT managers and professionals are new to the field of Human-ComputerInteraction or don’t even know about its existence, still they are forced –in particular by the internet market – to take usability testing into account, hoping to find “somewhere” ready-made answers to all of those "awkward" (while subjective) usability questions (it can’t be so difficult, after all, it’s just usability ...), e.g., by adopting selfmade checklists or questionnaires, and are surprised to find out, that it doesn’t work out as expected. Indeed, all caveats and myths about software testing have their counterparts in usability testing: the challenge of test case coverage, abundance of testing methods and techniques, lack of money and/or time for thorough testing, bad integration within the design and development cycle, and so on. Even more to the point: you will need special training in models, methods, techniques and tools which you even didn’t know exist when you are new to the HCI field [8]. One of the challenges of usability testing in particular is that for obvious reasons the user is not and will never be fully specified beforehand, not even during use. This makes it hard for software architects or designers to take user requirements into account and eventually makes them reluctant to adopt a user-oriented perspective. The way out of this dilemma is evident –but not always simple to implement: (1) always take representatives of users on-board of your development team, (2) hire a professional usability tester.

5 Usability testing and software quality control Software testing without coordinated usability testing is like manufacturing a hammer and checking its suitability without ever giving it in the hands of a real craftsman and observing under different conditions, if he can use it right away, with minimal instruction and whether he still likes to use it after having used it for different purposes. If you have designed some software merely based on usability requirements and user-centric design specifications, you are completely at the mercy of those requirement specifications as far as overall system and software quality is concerned. And how often will you find out that the specs are missing, incomplete, ambiguous, outdated? And even if you have got a nice requirements document, this does not neces4

5

A good starting point might be "http://www.hcibib.org", a professional bibliographic database for the entire field of human-computer-interaction with an excellent reputation. Particularly noteworthy to mention in this context is the Usability Professionals Association with world-wide subchapters and its own series of yearly conferences.

5

sarily guarantee that the intended users will ultimately understand, learn, use and enjoy working with this “virtual”machine. A lot of human factors, capabilities as well as limitations, regarding the use of information appliances are difficult to codify in requirements or – worse – have been left open (“for the moment … ”) in the design specifications. Thus usability testing has to be integrated and coordinated – within a generalized approach to Software Quality Assurance and Control –with software testing, following slightly different routes as in the following overall reference model:

Software Developer

Model of the system (specs & docs)

Usability Tester

End User

Software Tester

SYSTEM (h&s) (information appliance)

Fig. 1: Reference model for the relationship between usability testing and software testing in an overall model of software quality control

Clearly, such an approach has practical consequences for management and control of ICT development projects, in particular its testing subprojects, e.g. there must a place for and there must be time to involve the end user in the testing activities! The good message however is, that most if not all ingredients for such a symbiosis already exist: what are needed are more awareness and more commitment on part of software quality managers to put the necessary structures and processes in place. And the reward will be evident: end users will thank you for more usable [9] and enjoyable [10] interactive systems!

References 1. Raskin J., The Humane Interface: New Directions for Designing Interactive Systems, ACM Press, 2000 2. Neumann, P.G., Computer related risks, ACM Press, 1995 3. Cooper, A., The inmates are running the asylum: why high-tech products drive us crazy and how to restore the sanity, SAMS P.C., 1995 4. Cooper, A., About Face: The Essentials of User Interface Design,, IDG Books, 1995

5. Constantine, L.L., Lockwood, L.A.D., Software for Use: A Practical Guide to the Models and Methods of Usage Centered Design, Addison-Wesley Professional, 1999 6. Nielsen, J., Usability Engineering, Academic Press, 1993 7. Chen Q., Human Computer Interaction: Issues and Challenges, Idea Group Publishing, 2001 8. Helander, M.G., Landauer, T.K., Prabhu, P.V., Handbook of Human-Computer Interaction, North-Holland, 1997 9. Norman, D.A., Things that make us smart: defending human attributes in the age of the machine, Addison-Wesley, 1993 10. Norman, D.A., Emotional Design: why we love (or hate) everyday things, Basic Books, 2004

Model-Based Software Development and Usability Testing Peter Forbrig, Gregor Buchholz, Anke Dittmar, Andreas Wolff, Daniel Reichart University of Rostock, Department of Computer Science, Albert-Einstein-Str. 21 18051 Rostock, Germany [email protected] http://wwwswt.informatik.uni-rostock.de/

Abstract. In this paper we discuss a user-centered approach of developing software, which is based on models and allows bridging the gap between software engineering and usability engineering. The development process consists of a sequence of interactive model transformations. It is demonstrated how first prototypes of interactive systems, which are animated models, can help to capture requirements and how the models evolve to the final interactive system. This model-based approach provides excellent opportunities for testing the usability of software based on the tasks users have to perform. A tool is presented, which visualizes the activities of a test person based on the models. The tool supports remote testing, which can be performed even on mobile devices.

1 Introduction It is commonly accepted that software development has to start with an analysis of the problem domain users work in. There are some discussions whether one has to start to analyze objects, tasks or interactions first but at the end there is a common understanding of the importance of all aspects of the problem domain. It is also more and more accepted that the point of view of the user is most important for the software under development. Indeed, a user-centered development process supports this idea perfectly. Model-based development of software systems becomes more and more popular. Even if it is up to now not used very extensively it is an attractive process with proven record of success, especially in the context of developing multiple user interfaces. There are approaches focusing on object models first like the model-driven architecture of UML [24]. However, we follow task-based approaches like ADEPT [26], CTTE [4] or Cameleon [1]. Typically, such systems are used to model existing or envisioned tasks. They help to understand the tasks a user has to perform in more detail by allowing simulations. Additionally, systems like TERESA [23] support the development of user interfaces. Our System DiaTask [21] follows a similar approach. Based on task-, object-, user- and environment-models interactive systems are developed. Chapter 2 will

describe this approach in a little bit more details. Afterwards opportunities for remote usability testing will be sketched and at the end we will discuss achieved and further goals.

2 Model-Based Development of Interactive Systems We strongly believe that software engineers and user interface designers have to base their work on the same models. Fig. 1 visualizes this fact by presenting these models on the left hand side. Task-, business-object-, user- and device models are very much interrelated. The device model is a representative of a general environment model here. These models are as well the basis for the development of the software developed by software engineers as those for the software developed by user-interface experts. Furthermore, we consider software development as a sequence of transformations of models that is not performed in a fully automated way but by humans using interactive tools.

Fig. 1. Model-based development process model Even if our work is especially focused on methods and tools supporting transformations by patterns (The transformation of class diagrams by patterns using the case tool Rational Rose is described in [11]. In [22], the idea of supporting the development of task models by patterns is presented.) we want to focus this paper on the models, which are within the development process.

In the following, we demonstrate the application of our ideas to a very small example of developing a mail management system. Fig. 2 is the result of the interviews of forthcoming users. It demonstrates how task models look like.

Fig. 2. Task model for the mail management system According to the task model of Fig. 2, a user may either read his mail, or write a new one. To read his mails, he has to select a specific mail from a list that is generated and presented to him by the application. Once he has selected a mail, he gets its content displayed. “Select” and “display” are consecutive subtasks of an iterative task that can be cancelled at any time. Writing mails is modeled in a similar manner. After a user decides to write a mail he has to enter the iterative task produce mail, where he is requested to compose a new mail and, after finishing this, the application will send it away. This sub-task may also be cancelled at any time. Additionally to the classical temporal operations like >> - enabling, ||| - in parallel, [] – alternative a new operation symbol # is introduced. It represents the “instance iteration” operation. In contrast to the classical iteration * it allows to start a new iteration before the old one is finished, a specification feature, which is very helpful in a lot of applications. With one of our tools called DiaTask [21] we are able to develop dialog graphs that represent the navigation structure of the interactive systems. Such a graph is based on the previously specified task model. A dialog graph consists of a set of nodes, which are called views and a set of transitions. There are 5 types of views: single, multi, modal, complex, and end views. A single view is an abstraction of a single sub-dialog of the user interface that has to be described. A multi view serves to specify a set of similar sub-dialogs. A modal view specifies a sub-dialog, which has to be finished in order to continue other sub-dialogs of the system. Complex views allow a hierarchical description of a user interface model. End views are final points in (sub-) dialogs. Each view is characterized by a set of (navigational) elements. A transition is a directed relation between an element of a view and another view. Transitions reflect navigational aspects of user interfaces. It is distinguished between sequential and concurrent transitions. A sequential

transition from view v1 to view v2 closes the sub-dialog described by v1 and activates the sub-dialog, which corresponds to v2. In contrast, v1 remains open while v2 is activated if v1 and v2 are connected by a concurrent transition. Fig. 3 shows the graphical notation for the different types of views and transitions. Unlike TERESA [23] the dialog graph is the result of a design process and not the result of automatic transformations. DiaTask allows assigning several different dialog graphs to one task model. Fig. 3 demonstrates the example of our mailing system with single views (main window, write mail), multiple views (read mail) and the end view (end). The screenshot is produced using the eclipse [9] plug-in for DiaTask.

Fig. 3. Dialog graph for the management system of mails There are concurrent transitions from “mail client” to “read mail” and “write mail”. This means that “main window” stays open and can be activated by a mouse click. Multiple views are able to instantiate several instances. In the example of a mail system this means that users can read several mails in parallel. Multiple views go together with concurrent transitions. Fig. 3 gives an impression of the abstract prototype generated from the model of Fig. 2. It captures a situation where one mail is read and another one is written. The “main window” is active and as well pressing the corresponding button can activate “end”, “read mail” or “write mail”. Fig. 4 demonstrates an abstract user interface.

Fig. 4. Canonical abstract prototype of Fig. 3 in animated mode The abstract user interface of Fig. 4 is automatically generated from the specification of the dialog graph and behaves according to the temporal relations of the task model. It is already very good to improve the communication with users during the requirements analysis phase. Unfortunately it has the drawback of a very abstract interface. It was our intention to improve this situation. It was our idea to generate the abstract user interface (e.g. Fig. 4) in a language, which can be picked up by a GUI-editor for improvements. Our decision was to use XUL [11] for this purpose. This user-interface description language was introduced with the Mozilla project and is part of its success story. Based on an existing GUI plug-in for eclipse an editor for XUL was developed. This editor is able to replace existing elements by other ones. In this way the abstract user interface can be improved to a more useful one while keeping up the reference from the GUI elements to tasks. We are especially working on the problem how patterns can be used for this purpose. Possible tool support is discussed in [22]. The interpreter of the models, which controls the animation, recognizes the existence of improved windows and includes them into the animation process. In this way, the user is able to have a look at a user interface, which is already a candidate for the final interactive system.

Fig. 5. Designed GUI for Select & Read

Details of the GUI-editor can be found in [27] and [28]. During simulation our system offers a view on the prototype of the user interface and a view on the animated task model. Fig. 6 demonstrates how the animated task model is visualized. Basic tasks (represented by squares) with green circles can be executed. Red crosses represent a status of the task that allows no execution because of restrictions (e.g. temporal relations between tasks). A blue tick signals the successful execution of a task.

Fig. 6. Visualization of an animated task model

3 Remote Usability Testing Usability testing is often the first activity, where software engineers come into contact with usability engineers. It is our idea that they start their co-operation already by commonly designing models. Nevertheless, testing the usability of software is a time consuming process. Test scenarios have to be developed, test persons have to be hired and experienced usability experts have to observe the behavior of the test person. Sometimes it is very difficult to have test person and the usability expert as test supervisor at the same time at the same place. This is especially true for mobile applications. Sometimes the presence of the test supervisors influences the test persons by executing their tasks. These problems can be reduced by remote usability tests. This kind of test allows the test person and the test supervisor to work at different places. This is not new but it opens new opportunities based on our model-based development approach. 3.1 Software Architecture It is possible to run a model-based system on client-server architectures. In this way models are interpreted on severs and the results are delivered to the clients. Fig. 7 gives an impression how this architecture looks like.

Server • Task model • Dialog graph • UI specification • Interpreter

Fig. 7. Software architecture The architecture of Fig. 7 opens new opportunities for remote usability testing because in addition to videos and the capturing of screens it is possible to observe the status of the interpreted models. Some features of our TMSClient and TMSServer (TMS = Task Model System) will be discussed within the following paragraph.

3.2 Tool Support for Remote Usability Tests Based on our model-based approach a TMSServer was implemented, which allows first remote usability tests. Up to now, the system’s input is a generated Javaapplication (see Fig. 8) that was built using the transformations described in the previous paragraph.

Fig. 8. User interface for the test person Model related events (creations, state changes) are captured on client side and transmitted to the server where the model is reconstructed and sent to the supervisor’s client(s). The usability experts get an impression of the test by looking at the visualization of the actual state of the execution of the task-model instance, which is presented in Fig. 9. We do not want to comment to all the information presented but would like to draw the attention of the reader to the left side of the screen shot, where the activated task model can be seen.

Fig. 9. User interface (animated task model) for usability experts

This user interface of Fig. 9 was technically produced based on a Java implementation, where parameterized cascaded observers were used to observe the states of the models on the server. At the moment the usability expert has to watch the changes of the task model instance. He has to observe whether the test person behaves as expected or whether tasks are activated that have nothing to do with the actual test scenario. In this last case something is wrong with the system and a usability problem was found. In the future there will be further tool support such tests like automatic extraction of critical actions. One can imagine that predefined scenarios are stored and as long as the test person acts according to these scenarios nothing happens. Otherwise the usability expert is informed that something happened that was not expected. It is his decision what to do next. This kind of usability testing offers a new type of information, which is not strongly connected to specific UI-elements like buttons etc but to structural information that was transformed from models to the final software. This allows a much more detailed analysis of the software’s logical structure than other test methods, and different versions of the software generated from different versions of the task model can be compared and optimized. Of course, it is not the intention to replace all other kinds of observations by this method but it is intended to use this idea additionally.

6 Summaries and Outlook Within this paper a model-based development (Fig. 1) process was proposed. This process model postulates the idea of having the same models as basis of the work of software engineers and usability engineers. This fact is very important for us. Even if not every detail of the models is important for both groups of the different developers there have to be common models as basis of the development process. This is one big first step towards bridging the gap between software engineering and usability engineering. The approach has the following advantages: •

•

Software engineers are focused on the tasks a user has to perform, on the object he has to manipulate and on the work situation (context of work). In this way it is guaranteed that the developed interactive systems are user-centered. Usability experts are involved in the specification of models in very early development stages. In this way they can influence the implementation process of the software engineers and integrate usability test results to ensure the task model and the modeled temporal relations to fulfill the user’s needs and support their work in an efficient manner.

•

Usability evaluation is supported during all stages of the development process based on the existing models. Especially remote usability testing can be supported. We have been using our approach in different small projects. However, up to now we did not use models for projects of large scale. Currently we are working in a project for a mobile maintenance system together with 8 partners from university and 9 partners from industry, which follows our model-based approach. The project started 2004 and will be finished 2006. We are sure that the demonstrated approach will lead to a successful result of the project and that we will gain further knowledge to improve our tool for remote usability testing in a mobile environment.

References 1. Cameleon: http://giove.cnuce.cnr.it/cameleon.html. 2. Clerxkx, T.; Luyten K.; Conix, K.: The Mapping Problem Back and Forth: Customizing Dynamic Models while preserving Consitency, Proc. TAMODIA 2004, P. 33-42. 3. Constantine L.L: Canonical Abstract Prototypes for Abstract Visual and Interaction Design, in Jorge J. A. et. al (Eds): Proceedings DSV-IS 2003, LNCS 2844, Springer Verlag, Berlin, 2003, P. 1-15. 4. CTTE: The ConcurTaskTree Environment. http://giove.cnuce.cnr.it/ctte.html. 5. Deakin, N.: XUL Tutorial. XUL Planet. 2000. 6. Dittmar, A., Forbrig, P.: The Influence of Improved Task Models on Dialogues. Proc. of CADUI 2004, Madeira, 2004. 7. Dittmar, A., Forbrig, P., Heftberger, S., Stary, C.: Tool Support for Task Modelling – A Constructive Exploration. Proc. EHCI-DSVIS’04, 2004. 8. Dittmar, A., Forbrig, P., Reichart, D.: Model-based Development of Nomadic Applications. In Proc. of 4th International Workshop on Mobile Computing, Rostock, Germany, 2003. 9. Eclipse: http://www.eclipse.org. 10. Elwert, T., Schlungbaum, E.: Dialogue Graphs – A Formal and Visual Specification Technique for Dialogue Modelling. In Siddiqi, J.I., Roast, C.R. (ed.) Formal Aspects of the Human Computer Interface, Springer Verlag, 1996. 11. Forbrig, P.; Lämmel, R.; Mannhaupt, D.: Patterns-oriented develpment with Rational Rose, Rational Edge, Vol. 1, No. 1, 2001. 12. Hilbert, D.M., Redmiles, D.F.: Extracting usability information from user interface events, in: ACM Compter Surveys, Vol. 32, Issue 4, ACM Press New York. 13. Limbourg, Q., Vanderdonckt, J.: Addressing the Mapping Problem in User Interface Design with USIXML, Proc TAMODIA 2004, Prague, P. 155-164.

14. López-Jaquero, V.; Montero, F. ; Molina, J.,P.; González, P.: A Seamless Development Process of Adaptive User Interfaces Explicitly Based on Usability Properties, Proc. EHCI-DSVIS’04, p. 372-389, 2004. 15. Luyten, K., Clerckx, T., Coninx, K., Vanderdonckt, J.: Derivation of a dialog model from a task model by activity chain extraction. In Jorge, J., Nunes, N.J., e Cunha, J.F. (ed.), Proc. of DSV-IS 2003, LNCS 2844, Springer, 2003. 16. Mozilla.org: XUL Programmer’s Reference 2001. 17. Nielsen, J.: Usability Engineering, Academic Press, Boston, 1993. 18. Paterno, F.; Mancini, C.; Meniconi, S: ConcurTaskTrees: A Diagrammatic Notatiob for Specifying Task Models, Proc. Interact 97, Sydney, Chapman & Hall, p362-369, 1997. 19. Paterno, F., Santoro, C.: One Model, Many Interfaces. In Proc. of the Fourth International Conference on Computer-Aided Design of User Interfaces, p. 143154. Kluwer Academics Publishers, 2002. 20. Puerta, A.R. and Eisenstein, J. Towards a General Computational Framework for Model-Based Interface Development Systems. Proc. of the 4th ACM Conf. On Intelligent User Interfaces IUI’99 (Los Angeles, 5-8 January 1999). ACM Press, New York (1999), 171–178 21. Reichart, D.; Forbrig, P.; Dittmar, A.: Task Models as Basis for Requirements Engineering and Software Execution, Proc. of. Tamodia, Prague, 2004, p. 51-58 22. Sinnig, D., Gaffar, A., Reichart, D., Forbrig, P., Seffah, A.: Patterns in ModelBased Engineering, Proc. of CADUI 2004, Madeira, 2004. 23. TERESA: http://giove.cnuce.cnr.it/teresa.html 24. UML: http://www.uml.org/ 25. UsabilityNet: http://www.usabilitynet.org/tools/r_international.htm 26. Wilson, S.; Johnson, P.: Bridging the generation gap: From work tasks to user interface design, In Vanderdonckt, J. (Ed.), Proc. of CADUI 96, Presses Universitaires de Namur, 199, p. 77-94. 27. Wolff, Andreas, Ein Konzept zur Integration von Aufgabenmodellen in das GUIDesign , Master Thesis, University of Rostock, 2004. 28. Wolff, A.; Forbrig, P.; Dittmar, A.: Reichart, D.: Linking GUI Elements to Tasks – Supporting an Evolutionary Design Process, accepted for TAMODIA 2005, Gdanks. 29. Wolff, A.; Forbrig, P.; Dittmar, A.: Reichart, D.: Development of Interactive Systems Based on Patterns, accepted for the workshop “Development of Interactive Systems Based on Patterns” at INTERACT 2005, Rome. 30. XUL: http://www.xul.org/

Socio-Technical Support for Knowledge Collaboration in Software Development Tools Yunwen Ye1,2 1

SRA Key Technology Laboratory, 3-12 Yotsuya, Shinjuku Tokyo 160-0004, Japan 2 Department of Computer Science, University of Colorado Boulder, CO80309, USA [email protected]

Abstract. Software development is inherently a knowledge-intensive and distributed cognitive activity. From this perspective, one of the major issues in user-centered software development tools is to provide support for knowledge collaboration between the developer and the external knowledge resources, which include both information in repositories and knowledgeable peers. This paper describes two prototype systems CodeBroker and STeP_IN that we are developing. The two systems combine to provide continuous support for the acquisition and application of external knowledge resources, designed to follow a design principle we call layered information-on-demand.

1.

Introduction

Software development is inherently a knowledge-intensive and distributed cognitive activity [16]. It requires knowledge from several different domains. Few developers, if any, have all the knowledge needed in their own heads. The knowledge is distributed between the developer and the external world [9], and the development of a software system requires learning from and integrating the knowledge of various external knowledge sources in the world. Knowledge in the world comes from cognitive tools that support programming and knowledgeable collaborating peers. The development of software is therefore no longer confined to an individual software developer but has to rely on distributed cognition by reaching into a complex networked world of information and computer mediated collaboration. Most existing software development tools have not paid enough attention to the above viewpoints: they are either product-oriented or process-oriented. Earlier CASE tools have mainly focused on the management of products generated in the process of software development, and those later process-oriented tools have mainly focused on the management of the process that dictates how software development should be conducted. Both groups of tools are methodology-oriented because those tools are developed based on an adopted software development methodology to support and manage activities of software development sanctioned by the methodology. Consequentially, such tools view methodology as the most important thing and thus often

require that programmers adapt their practice to incorporate the new methodology. Needs and concerns of tool users (software developers) are not fully considered. In contrast, the user-centered perspective--in this case, the developer-centered perspective, focuses on the behavior of software developers and aims at providing support for the cognitive process of software development [10]. Since software development is a process of progressive crystallization of software developers’ knowledge into a system, lack of needed knowledge is one of the major reasons that cause poor quality and productivity [8]. To create convivial tools that are consistent with the cognitive process of software developers requires the provision of means for software developers to acquire the needed external knowledge as easily as possible. Usability of software development tools in this vein goes beyond the careful layout and design of interfaces, but also include considerations of the way when the external information is presented, how the information is accessed and presented. Another major shortcoming of existing software development tools is that most of them are designed to support individual software developer, although the design process now is inherently distributed. Although there are tools that support the coordination and cooperation of developing software systems, such as concurrent version control systems and groupware systems, but those tools mostly considers coordination and cooperation through the mediation of managing products or processes. They are not meant for the collaborative knowledge construction among software developers, through which knowledge gets transferred from one developer to another developer during the production of software systems and new knowledge gets created. Most existing software development tools do not have integrated support for collaborative knowledge construction. Software developers have resort to other tools and communication media for knowledge collaboration. This paper describes our efforts in creating developer-centered software development tools from the perspective of framing software development as knowledgeintensive and distributed cognitive activities. We use reusable software components as the example domain to illustrate how external knowledge acquisition can be seamlessly integrated with current software development tools. With the advent of objected-oriented technology, reusable software components has now become an indispensable part of the knowledge required for software development. Supporting easy access to needed external information, reusable components in particular, to complement the insufficient knowledge of software developers is thus critical to the improvement of the quality and productivity of software development [5].

2.

Problems with Knowledge Acquisition in Software Development

We use the distributed cognition as the theoretical foundation to support our claim that easy acquisition of external knowledge, either from an information repository or a knowledgeable peer is a critical issue in creating usable and convivial software development tools. In particular, we want to develop software development tools that support the two kinds of distribution of cognitive process identified by Hollan et. al. [9]:

cognitive processes that involve the coordination between internal and external structure, and cognitive processes that are distributed across the members of a social group. Our previous research has revealed the following cognitive challenges in utilizing external information and knowledge [22]: software developers may not know the existence of useful external information and create suboptimal solution; software developers may not be able to find the right external information; software developers may not be able to understand and apply the new information; and different software developers have different knowledge needs. When perusing the external information itself is not enough for learning, software developers often turn to knowledgeable peers for help, as one empirical study has found that one important factor that differentiates expert developers from novices is that expert developers are able to “use other experts faster” to help them solve problems collaboratively because they often know the expertise of other experts and have a more reciprocal relationship with one another [2]. Therefore, utilizing external knowledge sources also poses the following social challenges: software developers may not know to whom they can turn for help on this particular problem [1]; and experts who are able to help may not be willing to due to the interruption to their own work and other various reasons [4]. Using one of the typical software development activity—software reuse—as an example, the following sections briefly introduce two systems that we have been developing: CodeBroker and STeP_IN (standing for Socio-Technical Platform for in situ Networking) that combines to provide a continuous support for timely access to external information and expertise of knowledgeable peers. The CodeBroker system addresses the cognitive challenges in software reuse: being aware of the existence of reusable components as well as being able to locate the needed component from a large repository; and the STeP_IN system helps a software developer find peer developers who have the expertise on the reusable component of interest and who are likely willing to offer timely help. The STeP_IN system is based on a new conceptual framework Dynamic Community that we have proposed to support situated knowledge collaboration.

3.

The CodeBroker System

Successful reuse requires that a software developer know the reusable component, when to reuse it, and how to reuse it in his or her task. If the developer has learned about the reusable component in advance and the reusable component is a part of his

or her working knowledge, reuse is a fairly straightforward activity. However, the vast amount of modern reuse repository, like the API libraries of Java that consists of thousands of reusable components (both classes and methods), makes it hardly possible for software developers to know everything about the reuse repository. The notknown-yet part of the reuse repository thus becomes the external resources that software developers have to utilize for developing software systems efficiently. If software developers realize that some reusable components they do not know yet can be reused in their current task, they might start browsing or searching the reuse repository. However, in many situations, they may not even be aware of the existence of applicable reusable components, therefore they would not make an attempt of locating the reusable components at all and reuse fails in the first place. CodeBroker aims to make software developers aware of unknown reusable components with the information delivery mechanism. It supports Java developers by delivering task-relevant and personalized components—components that can potentially be reused in the current development task and that are not yet known to the developers (see [22] for details). CodeBroker is seamlessly integrated with the programming environment Emacs. It consists of an interface agent and a backend search engine. Running continuously as a background process in Emacs, the interface agent infers and extracts reuse queries by analyzing the partially written program in the normal editing space of Emacs (Fig. 1(a)). Inferred queries are passed to the search engine, which retrieves matching components from the repository. The list of retrieved components is then delivered by the interface agent back into the programming environment (Fig. 1(b)) after it removes those components that the developer has known already by using user models for filtering. An adaptive and adaptable user model is used for each developer so that each software developer will get a different set of delivered components based on what they have known already [6].

Fig. 1. The CodeBroker System in Use

CodeBroker makes it possible for software developers to reuse those reusable components that they do not even know exist by creating an awareness of those unknown components that are potentially reusable in the current task. If the developer finds one of the delivered components interesting, a click will take him or her to the full document of the reusable components in the external documentation system for Java libraries (Fig. 1(d)).

4.

The STeP_IN System

When reading the document is not enough for software developers to understand how to use the reusable components, software developers need further help. The STeP_IN system that we are currently developing provides the social support for understanding the reusable component of interest. It creates a social platform for knowledge collaboration by superimposing the current Java documentation system with three kinds of support, finding example programs, browsing the archives of past discussion, and sending questions to selected experts. In the new documentation system, in addition to the conventional document created by the Javadoc program, each method in the reuse Java repository is enhanced with three clickable buttons: Examples, Discussion Archive, and Ask Experts (Fig. 2).

Fig 2: Enhanced Java Documentation

Clicking the Examples button of a method brings example programs that use the method. Those example programs are written by other software developers who are the participants of the knowledge collaboration space of the STeP_IN system. Clicking the Discussion Archive button of a method brings the archived previous discussions about the usage of the method. The archive stores messages that participants of STeP_IN have exchanged in the past using the following Ask Experts support. Clicking the Ask Experts button of a method sends an email requesting for help on the method to a small group of experts on the method who are selected from the STeP_IN community. The small group of experts is chosen based on the principles of the new conceptual framework Dynamic Community that we have proposed to support situated knowledge collaboration (see the next section for a brief introduction, for more details see [4]). When the selected experts exchange messages with the developer who is seeking for help, those messages are stored in the discussion archive associated with the method, so that other developers who have similar problems later can benefit by clicking the Discussion Archive button.

4.1

Dynamic Community

A dynamic community, or DynC, is dynamically formed in a knowledge work space. A knowledge work place consists of a group of knowledge workers, and the knowledge that the workers hold. The group can either be a formal organization such as a company or an informal online community. A DynC is a subgroup of knowledge workers that forms ad hoc in support of a particular user and a particular task, and dissembles as the task is finished. Unlike a static community that forms around a particular domain and exists for a long time [20], a dynamic community forms for a particular task and a particular user, and exists only for a short period. The members of a DynC are selected using the following two criteria: they have expertise on the particular task, and they already have social contacts with the particular user. The first criterion is grounded in the observation that in today’s highly specialized world expertise is no more an absolute attribute of person but a relative function of a person and a task. In other words, omniscient experts for many domains do not exist any more, and experts can be identified only after the task is known. The second criterion is grounded in the assumption that existing social contacts between the helpers and the helpee could provide extra motivation to engagement in knowledge collaboration based on the social norm of generalized reciprocity [7]. In summary, a DynC is task-specific: for different tasks, different DynCs will form even for the same user because different groups of experts will be identified; a DynC is user-specific: for different users, different DynCs will form even for the same task because users has different social contacts; and a DynC is in-situ and ephemeral: it is created dynamically when a need for collaboration arose in a real task and dissembles when the task is solved. 4.2

Creating Dynamic Communities in STeP_IN

The STeP_IN system can be used to support either an online community or an organization, and the system can be set up to support any Java reuse library. For simplicity of discussion, let’s say for the standard Java API library. The system maintains, in its central server, a database that stores three kinds of relation: (1) the relation from a library component to example programs, (2) the relation from a library component to experts, and (3) the relation from a member developer to other members. Upon joining the collaboration space supported by STeP_IN, a member has to perform the following task to initialize his or her profile in the database. First, he or she has to use one of the STeP_IN’s subsystems, in his or her own computer, to analyze all the Java programs that he or she has written in the past. The analysis examines what reusable components of the repository have been used by the member to identify his or her expertise on those reusable components.

Second, he or she is required to provide a set of Java programs that he or she agrees to share with other members. Those programs will be uploaded into the STeP_IN’s central server and will e used as example programs. Third, he or she has to use another STeP_IN’s subsystem to analyze his or her emails. The analysis mines the emails to create an initial social network for him or her based on the email exchanges. To protect privacy, the analysis is conducted in the member’s own computer and only the resulted relation (nothing about the contents) will be uploaded into the central database after he or she has confirmed and edited with a provided editing interface. The social network is updated whenever knowledge collaboration activities take place in the STeP_IN space. With the database that has all three kinds of relations, STeP_IN is able to locate appropriate examples when the Examples button is clicked. At first, when multiple examples exist, the system will list them in the order of their length based on the rather simplistic assumption that shorter programs are easier to understand. Each example will be associated with a voting mechanism for members to vote on its helpfulness for understanding the reused component. As votes accumulate, examples with more votes will advance to the head of the list of the examples and the number of votes is also shown to assist selection of an appropriate example. When a member (i.e. a helpee) clicks the Asking Experts button, according to the principles of DynC, STeP_IN goes through two steps—expert identification and expert selection—to create the list of experts who should receive the help request. The expert identification process examines the relation from the component to the experts in the database to find all the members who have used the component before in their programs, and create a list of Candidate Helpers. The expert selection process examines the social relation from the helpee to other members in the database to select a small group of people from the list of Candidate Helpers. Principles for selection are discussed further in the next subsection. The selected people become the DynC Members. An email is then sent to DynC Members to ask for their help. The DynC Members who receive the email requesting for help can send back their help through emails and the emails are captured by the system and stored in the database in association with the component. Such messages are then displayed for later members who click the Discussion Archive button. 4.3

Social Factors Considered in STeP_IN

To initiate and sustain successful knowledge collaboration requires that experts be motivated to engage in sharing their valuable time and knowledge. In the design of the STeP_IN system, we have taken into consideration of the following social factors as a way of increasing motivation for knowledge collaboration during the process of selecting experts. Economic use of expert attention. Experts have their own work, and sharing their knowledge with others costs extra time and attention. Sending a request to all possible experts demands the time and attention of all of them and creates a huge waste when some of them are not experts on this particular task. To better utilize experts’ time, attention and good will, we use the expert identification and selection process to limit

the number of experts to be contacted to a much smaller one and thus reducing the collective waste of expert attention and time. Utilizing existing social networks. Collaboration happens more spontaneously and smoothly when strong ties exist between participating members [18]. To motivate software developers to participate in knowledge collaboration in STeP_IN, we seek to utilize existing personal social networks of each member to seek those experts that have strong social ties with the helpee. The social network is initially mined from their emails, and gets updated when collaboration takes place. The personal social network is also editable and updatable by its owner who can specifically specify whom he or she would like to help and also whom he or she does not want to help. If a person X specifies he or she would not like to help Y, then any request from Y would not go to X even if X is identified as an expert for a component that Y wants to learn about. Promoting reciprocity. Generalized reciprocity is the foundation for long-term collaboration [12]. To promote reciprocity, in the expert selection process, we first seek to enlist the help from those experts who have been helped by the helpee in previous occasions. Avoiding forced collaboration. When a helpee requests help from experts he or she knows well, the social pressure is often shift on the experts because they might fear potential social backlash if they do not provide help and violate the rule of reciprocity. In such cases, the experts are often forced into collaboration even if they might not in a condition or a mood for collaboration at this particular moment. This would be detrimental to the long-term sustainability of the collaboration platform and some experts might simply withdraw from it. We believe avoiding such forced collaboration is essential for creating a sustainable voluntary collaboration platform that relies on the good will of benefactors. In STeP_IN, when the request for help is sent out to the DynC members, the helpee does not know to whom the request is sent, so that even if a DynC member does not offer help due to various reasons, the helpee would not notice. However, any help message from a DynC member to the DynC shows its origin so that the DynC member’s contribution to the group is publicly recognized and properly acknowledged.

5.

Individualized and Layered Presentation of Information

Locating, comprehending, and using information involves at least three different phases: information discernment, detailed evaluation, and information application [3]. In the information discernment phase, users quickly determine whether a piece of information is relevant to their current task. The detailed evaluation phase involves a relatively thorough study of the information. In the information application phase, users need not only to understand the information but also to apply it in their current situation. Each phase requires different levels of details of the information. For the information discernment phase, an overview of the information is enough. This phase also requires the simultaneous presentation of several candidates for users to compare and find the information that is most relevant. For the detailed evaluation phase, users

need to focus on the details of a chosen piece of information. Examples that use the information are very effective in helping users understand, adapt, and integrate the information by providing context and enabling users to draw an analogy between their task and examples [13, 15]. The details required for information comprehension and use depend greatly on the knowledge of users. A continuum of demands for different levels of details exists. On one extreme, if the user already knows the presented information vaguely, an overview of the information may be enough. On the other extreme, if the user has never encountered the information before, he or she may need to go through all the three phases, and that iteratively because he or she may not be able to make the right choice at first. Because of the tacit nature of knowledge [14], information repository cannot capture all the knowledge. In such cases, finding an expert on the topic becomes necessary for the user to understand and apply the information [11, 19]. In the design of CodeBroker and STeP_IN that combine to become an integrated socio-technical platform in support of knowledge collaboration in software development, we have followed the principle of layered information-on-demand to guide the design of interaction and coordination between users and external knowledge resources. The design principle of layered information-on-demand aims to provide a natural link from the task at hand to a variety of external information and knowledge resources, which are presented to users at different levels of abstraction in response to their different needs. Depending on the working context and background knowledge of a user, a unique network of information and knowledge is dynamically constructed and presented to provide a seamless transition from users’ interaction with tools to interaction with external information and other knowledgeable peers. Structuring information into different layers and presenting each layer on demand to accommodate the different needs of users has its root in the concepts of progressive disclosure and details on demand [17]. Progressive disclosure is a design principle to reduce the complexity of user interfaces by presenting the most common choices to users while hiding more complex choices or additional information, which is displayed when needed. Details-on-demand is widely applied in information visualization systems that allow users to obtain details about a selected element from the visualized overview of an information set. Layered information-on-demand supports a new interaction style between information systems and users that focuses on the economical utilization of human attention by presenting information in different levels of details and on demand from overviews to details to examples and to experts. This design principle accommodate the differing needs of each individual user by putting control at the hand of users, who can stop at any level whenever they deem the presented information enough. For experts, the system remains brief and simple, without taking too much attention from their real task; for novices, the system has a complete socio-technical support, using human experts as the social infrastructure to back up the inadequacy that is inherent in knowledge repository systems due to the tacit nature of knowledge. The following subsections describe the layers of information that are provided in the two systems.

5.1 Autonomous Delivery of Task-Relevant and Personalized Information Overviews In CodeBroker, software developers do not need to explicitly input queries for external information. As software developers enter document comments and method signatures in the editing space (Fig. 1(a)), the CodeBroker system extracts the comments and signatures and uses them as a query. A list of task-relevant components that match the query is autonomously retrieved from the Java API library. Different users have different knowledge and needs different set of external information even for the same task. CodeBroker, therefore, uses user models to personalize the retrieval results before they are delivered. The retrieved task-relevant components are compared against the user’s user model that contains the components he or she already knows. The known components are removed because the developer would be able to reuse those known components by himself or herself if they are reusable in the current situation. The personalized task-relevant components are delivered into the delivery buffer (Fig. 1(b)). Each delivered component has the rank of the relevance between the component and the current task, the relevance value, the name, and the synopsis of its functionality. This is a context-aware list of reusable components for the developer to browse and serves as the first layer of information presented on demand. Not all the components are useful for the current task, but many of them are. In fact, traditional recall and precision measurement shows that for achieving 80% recall rate, the precision rate is 33.71%. In other words, about 1/3 components returned could be used to implement the task in one way or another [21]. 5.2

Mouse Movement-Triggered Information

Another important piece of information for a Java API method is its signature. We decide not show it in the delivery buffer because signatures are usually long and takes too much of the scarce screen resource. Furthermore, signature becomes relevant only when the developer finds the reusable component has the potential to be reused. In this sense, the signature of a component is secondary information needed to determine the reusability of the component. Therefore, only when the developer moves the mouse pointer over the component names in the delivery buffer (Fig. 1(b)), which triggers the display of the component’s full signature in the mini-buffer (Fig. 1(c)), the second layer of information is presented on demand to assist the developer in determining further the relevance of the component. The full signature shows the package, the class, and the types of input and output data. To note that all the interaction with CodeBroker by far was conducted within the programming environment. This makes the location of reusable components a natural extension to the developer’s current programming practice because he or she does not need to make a conscientious decision to shift from programming mode to reusing mode.

5.3

Jumping To The Documentation

When the developer finds one promising reusable component and wants to know more about the component, a click on the component name brings up a web browser that shows its full documentation (Fig. 1(d)), and with the integration of the STeP_IN system that are still under development, the developer will see Fig. 2, with three new buttons added to the standard Java documentation. 5.4

Finding Examples

If the developer still does not know how to use the component, an example of its use might be helpful. He or she can then press the Examples button (Fig. 2), which has been added to the standard Java documentation system in STeP_IN, to get an example program that illustrates the use of the component. 5.5

Reading the Discussion Archive

When an example is still not enough for understanding or using the component, the developer can, by pressing the Discussion Archive button, to the archive of previous discussions that are saved from previous information exchange about the component between helpers and helpees. 5.6

Finding Experts

If the developer’s question does not have an existing answer in the discussion archive, he or she can press the Ask Experts button to formulate a dynamic community for knowledge collaboration. As we have mentioned in Sections 4.1 and 4.2, this dynamic community is user-specific, it is different for different users because each user has different inter-personal relationship network. The discussion that takes place afterwards is saved in the Discussion Archive for the benefit of later developers.

6.

Summary

To improve the usability of software development tools, we have to focus on the needs and concerns of software developers. Since software development is inherently a knowledge intensive and distributed cognitive activity, in addition to methodological support, user-centered software development tools need to provide support for knowledge collaboration with external knowledge sources that include both information repository and knowledgeable peers. Successful knowledge collaboration requires the awareness of available external information, the identification of knowledgeable peers, and the willingness of the peers to participate in collaboration, as well as the obtaining of such information and expertise in a timely manner. This paper

described our two research prototypes CodeBroker and STeP_IN that combine to provide continuous support for knowledge collaboration in software development. The former creates the awareness of and the access to unknown reusable components, and the latter creates and facilitates collaboration among software developers as a social backup to complement the insufficiency of reuse repositories in support of software reuse. The two systems are designed under the guidance of the principle of layered information-on-demand that can satisfy the wide variety of individual needs for external knowledge.

Acknowledgments. This research is supported by the Ministry of Education, Culture, Sports, Science and Technology of Japan, Grant 15103 of the Open Competition for the Development of Innovative Technology program.

References 1.

2.

3.

4.

5.

6.

7.

8. 9.

Becks, A., Reichling, T., and Wulf, V.: Expertise finding: Approaches to foster social capital, in Huysman, M. and Wulf, V., (eds.): Social capital and information technology. 333-354, The MIT Press, Cambridge, MA, 2004. Berlin, L.M.: Beyond program understanding: A look at programming expertise in industry, in Cook, C.R., Scholtz, J.C., and Spohrer, J.C., (eds.): Empirical studies of programmers: Fifth workshop. 6-25, Ablex Publishing Corporation, Palo Alto, CA, 1993. Carey, T. and Rusli, M.: Usage representations for reuse of design insights: A case study of access to on-line books, in Carroll, J.M., (ed.) Scenariobased design: Envisioning work and technology in system development. 165182, Wiley, New York, 1995. Cross, R. and Borgatti, S.P.: The ties that share: Relational characteristics that facilitate information seeking, in Huysman, M. and Wulf, V., (eds.): Social capital and information technology. 137-161, The MIT Press, Cambridge, MA, 2004. Detienne, F.: Design strategies and knowledge in object-oriented programming: Effects of expertise. Human-Computer Interaction, 10(2/3):129-169, 1995. Fischer, G.: User modeling: The long and winding road, in Kay, J., (ed.) Proceedings of user modelling 1999. 349-355, Springer Verlag, Banff, Canada, 1999. Fischer, G., Scharff, E., and Ye, Y.: Fostering social creativity by increasing social capital, in Huysman, M. and Wulf, V., (eds.): Social capital. (in press), 2004. Green, T.R.G.: The nature of programming, in Hoc, J.-M., et al., (eds.): Psychology of programming. 21-43, Academic Press, New York, 1990. Hollan, J., Hutchins, E., and Kirsch, D.: Distributed cognition: Toward a new foundation for human-computer interaction research, in Carroll, J.M., (ed.)

10. 11.

12. 13. 14. 15. 16. 17.

18. 19.

20. 21.

22.

Human-computer interaction in the new millennium. 75-94, ACM Press, New York, 2001. Jarzabek, S. and Huang, R.: The case for user-centered case tools. Communications of the ACM, 41(8):93-99, 1998. Mockus, A. and Herbsleb, J.: Expertise browser: A quantitative approach to identifying expertise., in 2002 International Conference on Software Engineering. 503-512, Orlando, Florida, 2002. Nahapiet, J. and Ghoshal, S.: Social capital, intellectual capital, and the organizational advantage. Academy of Management Review, 23:242-266, 1998. Neal, L.R.: A system for example-based programming, in Human factors in computing systems (chi'89). 63-68, ACM Press, Austin, TX, 1989. Polanyi, M.: The tacit dimension, Doubleday, Garden City, NY, 1966. Repenning, A. and Perrone, C.: Programming by analogous examples. Communications of the ACM, 43(3):90-97, 2000. Robillard, P.N.: The role of knowledge in software development. Communications of the ACM, 42(1):87-92, 1999. Shneiderman, B.: Designing the user interface: Strategies for effective human-computer-interaction, 3rd edition, Addison-Wesley, Reading, MA, 1998. Taylor, J., Sumner, T., and Law, A.: Talking about multimedia: A layered design framework. Journal of Educational Media, 23(2/3):215-241, 1997. Vivacqua, A.: Agents for expertise location, in Proceedings of 1999 AAAI Spring Symposium on Intelligent Agents in Cyberspace. 9-13, Stanford, CA, 1999. Wenger, E.: Communities of practice — learning, meaning, and identity, Cambridge University Press, Cambridge, England, 1998. Ye, Y.: Supporting component-based software development with active component repository systems. Department of Computer Science, University of Colorado, Boulder, CO, 2001. Ye, Y. and Fischer, G.: Supporting reuse by delivering task-relevant and personalized information, in Proceedings of 2002 International Conference on Software Engineering (ICSE'02). 513-523, Orlando, FL., 2002.

Principles for the Design of Web Applications Anthony I. Wasserman Center for Open Source Investigation (COSI) Carnegie Mellon West Moffett Field, CA 94035 USA [email protected]

Abstract. The development of high-volume web applications draws on many principles and guidelines drawn from software engineering, human-computer interaction, and other aspects of system design. This paper identifies those principles and guidelines, with a focus on abstraction and modularity principles, using a content management system as an illustration of using high-level abstractions to create web applications.

1 Building Web Applications The development of a successful web application (website) involves many different kinds of design, including functional design, software architecture, business process or workflow design, user interface design, and database design. More than any other type of application, web applications require effective integrated use of software engineering practices and interaction design principles. In the decade since the first web applications were created, mechanisms and tools for the design and development of such websites have evolved very substantially. While early websites might have had a handful of server-side actions, today’s sites routinely include JavaScript (ECMAscript) for client-side interactivity and checking, dynamically generated web pages, and a complex “back-end” that often includes extensive processing logic written in Java or a scripting language (Perl, PHP, or Python), distributed processing functionality in a J2EE container, a database management system, and specialized components for security, e-commerce, and management of web services. Applications developed for high-volume use on the World Wide Web are highly complex, and illustrate the practical application of principles and guidelines for human-computer interaction. The most heavily used websites are characterized by high reliability, high availability, high security, and rapid interactive response. For web applications such as Google, Amazon.com, eBay, and Yahoo, these qualities are essential, and are a key reason why they are trusted by millions of users. Other web applications, such as those used for major sporting events and viedo streaming, are designed to support spikes in traffic.

2

Anthony I. Wasserman

One result of the vast experience in website development is that many aspects of this development have become systematized, allowing the development team of a new site to follow a well-understood set of principles. Many of today’s tools for web application design incorporate some of these principles, effectively encoding the knowledge that has come from the collective experience of building so many sites. For example, it’s extremely easy to create a robust e-commerce site without writing any code at all. The builder of such a site chooses a template, customizes some graphical elements and menu items, and defines the inventory with pricing. After a few additional steps, the application is up and running on the server site. (See, for example, CityMax, at http://www.CityMax.com) One could make the case that the “coding free” version of a web application is very constraining, both in terms of the styles of user interaction and the site content. While that’s true, it also should be noted that the ability to enable the “instant” creation of a site that meets the needs of at least 90% of all users represents a formidable achievement in software engineering and human-computer interaction. The design knowledge for building e-commerce web sites has been captured and embedded in the application builder and its server environment. This knowledge includes site and page templates for presentation of the site content, the functionality needed by an ecommerce site, and the infrastructure of the application as needed to assure acceptable performance, reliability, and security. One might say that the creators of such an application builder have “boiled it down to a science,” incorporating principles of interaction design (forms, navigation mechanisms, page layout), site architecture (server side processing, data management), and key application functionality (e.g., shopping carts, integration with payment mechanisms, security certificates, site registrations for express checkout). These developers also know the traffic limits of sites created in this way. Dantzig’s history of IBM’s experience in building high volume web sites [1] describes their growing sophistication in site design over a five year period. Yen, Hu, and Wang have described a framework for effective web site design [2]. Menascé and Almeida [3] have defined a four level reference model that links together e-business requirements with capacity planning. Killilea [4] has focused on server configurations in his coverage of performance issues. Barish [5] has focused on scalability and performance issues for the Java Enterprise platform. (Other good sources of information on performance aspects of web application design are the annual meetings of the International Workshop on Web Site Evolution and the International Conference on Web Site Engineering.) There is also a substantial body of literature of the human factors aspects and visual design of web applications. Van Duyne and colleagues [6] have identified important guidelines for site design. Nielsen has written a widely followed book on usability [7]. By contrast, though, there has been relatively little written about the software engineering design principles and scientific foundations that underlie this kind of software. Rosenfeld and Morville [8] have addressed both the aesthetics and the mechanics of web site design, but there are few other works that do so.

Principles for the Design of Web Applications

3

In summary, there is a significant challenge to unify design principles from the various disciplines that influence the development of web applications and their subsequent usefulness. This work cuts across all aspects of software development from requirements analysis through implementation (coding and deployment architecture) and testing, as well as across numerous aspects of human-computer interaction, including task flow analysis, user interface design, and web site responsiveness. If one views software engineering as the application of underlying scientific formulas, laws, and principles, then the longer term goal of this work is to create this scientific foundation that provides a solid basis for the systematic development of usable web applications, as well as expanded automated support for the engineering processes. The remainder of this paper identifies some of the unifying themes across software engineering and human-computer interaction as they apply to web application development as a starting point for creating an integrated design framework.

2 Software Design Principles There have been many previous efforts, including that of the author [9, for example], to identify the essential qualities of software design. More recently, the US National Science Foundation has sponsored a workshop and funded research projects on the theme of a Science of Design. The summary of the workshop, held in November, 2003, cited the need for a science of design, and enumerated goals for such an effort, including: “To enable the design of software-intensive systems as dependable and fit to purpose as mechanical, civil, and other engineering systems typically are today; …. to develop rich behaviors at complex interfaces between software-intensive systems and their human and physical environments; to enable automated tools based on sound science to replace costly and error-prone human activities in software design; to leverage knowledge of design from other fields; to develop design methods for emerging technology platforms; to understand how to design software-intensive systems as social agents; to understand the characteristics of successful designs and how they are achieved; to learn how to represent designs at a much higher-level than code but nevertheless rigorously embody the key constraints; to discover how multidisciplinary communities of designers and users can communicate to jointly advance design processes” [10] Many of these goals depend on integration of design principles across domains of software engineering. Because web application development is software-intensive

4

Anthony I. Wasserman

and touches many different domains, it makes an excellent area for the exploration of these principles. We now identify some basic design principles, drawing in part from [8], and showing how they fit in the context of web application development. We illustrate the application of some of these principles with examples from an open source content management system (CMS), Drupal (http://www.drupal.org), described by its creators as a “dynamic web site platform which allows an individual or community of users to publish, manage and organize a variety of content.” Drupal provides a site development environment with minimal coding, including weblogs, collaborative tools and discussion-based community software. The citizen journalist site, NowPublic (http://www.nowpublic.com), is an example of a site managed by Drupal. 2.1 Abstraction Abstraction is a fundamental technique for problem understanding and solving. The psychological notion of abstraction permits one to concentrate on a problem at some level of generalization without regard to irrelevant low-level details. Abstraction is a common intellectual technique for managing the understanding of complex items, and is pervasive in software engineering. In the requirements stage, methods and notations such as use case diagrams, scenarios, work flow models, and conceptual data models allow analysts to understand the problem independently of its implementation. At the design stage, user interface designers can think in terms of images, menus, buttons, text fields, and other visual design elements. Software designers can think in terms of major system functions and interfaces among them. Principles of information hiding, as described by Parnas [11], are a central notion of abstraction. A CMS, such as Drupal, provides its users with an abstracted view of the basic functions of a web site, allowing the site administrator(s) to select components for use in a site managed by that CMS. Among the components provided in Drupal are user registration and login, static page creation, content aggregation (RSS), discussion forums, weblog (blog) creation, and site searching. From a user interface perspective, the CMS provides visual design themes: templates that control the layout and format of the content. The site designer is presented with a collection of themes, which define the layout for a set of “blocks” on the page. The choice of theme determines the basic look-and-feel of the site, and can be used “as is”, or by adding some HTML code to add logos and other graphical elements within the constraints of the theme. One such theme is employed in the Drupal-based Spread Firefox site (http://www.spreadfirefox.com). The important concept here, from the standpoint of design principles, is that the developer of the web application is able to work at a much higher level of abstraction than was previously possible. There is no need to write code to build a user

Principles for the Design of Web Applications

5

registration form, populate the registration database, or manage the subsequent user login process. All of these functions are bundled and automatically handled by the user registration and login component. Much the same is true for the other high level functions provided by the CMS. In the absence of a user interface designer, the developer can choose from a selection of site design layout themes, virtually all of which are superior to a design that could be created by someone without user interface design knowledge and experience.

2.2 Modularity and Architecture Software architectures play a major role in determining the quality and maintainability of a system. The overall architecture of a software system has long been recognized as important to its quality (or lack thereof). An architecture’s building blocks include units that carry out the system’s behavior, connections that show how these units transmit or share information, and connections that show how the various program operations are activated, either sequentially or concurrently. Web applications have a well-known and widely followed n-tier site architecture. Traditionally, the client side is a web browser that uses a network connection to a web server that directly processes requests for static HTML pages and passes along any requests that require additional server-side processing. The server side requests are often managed by an application server, which provides additional services for clustering, fail-over, and load balancing, in addition to managing connections to database management systems. Application servers are a central component of high volume content management systems, intranets, portals, and custom applications. With the appropriate architecture, these servers, as well as supporting database management systems, can be replicated and clustered to provide the desired levels of reliability and scalability. In its simplest form, the site structure is 3-tiered, with the presentation layer on the client side, the business logic embodied in code processed by the application server, and the data managed by the database management system. This three-tier model is supported directly by Struts (http://struts.apache.org), which uses a variant on the model-view-controller (MVC) architectural design pattern. Design patterns [12, 13] are yet another example of allowing the designer to work at a higher level of abstraction than was previously the case. With Struts, for example, the developer writes a configuration file (in XML) to organize the different actions of the architecture into user inputs (Action Forms), system activities (Action Mappings), and output page selection (Action Forwards). Design patterns apply not only to functional and structural components, but also to elements of the user interface. Martijn van Welie has compiled a collection of patterns in interaction design, covering many common patterns of web site and GUI design, including navigation mechanisms, site structures, searching, menu structures,

6

Anthony I. Wasserman

and basic page types (http://www.welie.com). He has also written about mechanisms for organizing pattern languages [14]. Plug-ins are another important architectural contribution. Plug-ins were introduced as components of professional design and publishing tools, such as Adobe Photoshop and QuarkXPress (XTensions). The developers of these tools created extensible architectures that allow third parties to extend the functionality of the basic tool by creating new components that work within the tool’s architectural framework. Plugin architectures are now found in a broad variety of software components, including the Eclipse development platform and application frameworks (http://www.eclipse.org) and the Firefox browser (http://www.mozilla.org). In summary, there is now a significant body of knowledge about the ideal structure of software systems and web applications, building upon first principles of abstraction and modularity. Knowledge and use of these successful structures is a key principle for creating new web applications.

2.3 Logging for Analysis and Testing Traditional software packages can be instrumented internally or externally to trace their execution flow, invocation of other software components, and other performance measures. There are numerous tools for performance tuning, load testing, GUI testing, real-time analysis, and related measures. Many such tools are valuable in the development and testing of web applications. The typical application server, e.g., WebLogic Server or JBoss, produces detailed log files that are valuable for studying system performance, identifying errors, and determining general patterns of use. Beyond that, though, a web application can be extensively tested and evaluated from the perspective of its users. An HTTP server, such as Apache, produces the Common Logfile Format, a standard log that shows details of every request made from a web browser accessing the application (site). Basic analysis tools, such as Analog (http://www.analog.cx) and Webalizer (http://www.mrunix.net/webalizer), provide information on traffic by browser, time of day, and originating domain, as well as counts of page viewings, errors, and searches. While this information is useful for finding errors and scaling the hosting resources, it is possible and often more valuable to collect higher level information. For example, it is useful to understand such items as the amount of time that a user spends on the site, which pages have been viewed, the sequence of pages navigated by a user, the time taken to complete a task, login and registration information, and many more related items. Operators of e-commerce sites not only want to know which products were viewed, but also which items were placed into shopping carts and ultimately purchased; at the same time, they also want to know when shopping carts were abandoned. Advanced log analysis tools, such as 3DStats (http://www.3dstats.com) and WebTrends (http://www.webtrends.com) can provide this data.

Principles for the Design of Web Applications

7

Such information provides web application developers with rapid, even realtime, feedback on usage patterns for their site(s). For example, traffic data and user behavior can validate the success of a marketing campaign or the popularity of a new product on an e-commerce site. Amazon.com, for example, is able to use this data to maintain a dynamically updated list of their most popular products in various categories. Problems can be found and fixed. The underlying principle is that multiple levels of logging data is essential to continuous improvement of the web application, from both the standpoint of application reliability, security, and efficiency, as well as the perspective of the user experience. Mechanisms for collecting and analyzing this data should be built into the ongoing operation of the site.

2.4 Drupal and the Design Principles The Drupal CMS adheres quite well to these principles. Modularity and abstraction are well supported, since additional functionality and content can be added to a Drupal-based site without access to or knowledge of the underpinnings of the system (an HTTP server, PHP, and either the MySQL or PostgreSQL DBMS). The underlying site infrastructure is hidden, and the site administrator(s) can enable various functional components above and beyond those provided by default. In other words, the CMS has functional modules that implement abstractions such as content aggregation, blogging, and site searching. Drupal’s open architecture allows third-party development of add-on modules. Among the more than 100 such modules (see http://drupal.org/project/Modules) are those for display of ad banners and for e-commerce. These modules can be freely downloaded and integrated with other modules, sharing the same look-and-feel theme. The site administrator has access to logs that show various system events, including user sessions and security violations. These logs afford a task-oriented view of user behavior at a higher level of abstraction than that provided by the logs created by the HTTP server and the DBMS. While Drupal remains a work in progress, the functionality currently available through the base CMS and the add-on modules (akin to plug-ins) is such that one can build significant web applications with minimal coding. Drupal is an excellent example of a software system that is built on well-proven underpinnings and that follows good design principles on its own, therefore simplifying the task of creating new web applications.

8

Anthony I. Wasserman

Beyond Drupal, numerous other web applications, such as SugarCRM (http://www.sugarcrm.com), are following the same principles, creating an open source software application that builds on proven components, such as the Apache HTTP server and the MySQL RDBMS. The installation of these applications invokes a script that creates the database schema needed by the application, populating it as needed. Both the Drupal CMS and the SugarCRM application are written in a scripting language, PHP (http://www.php.net) [15, 16] in these examples, but there are numerous other languages that could alternatively be used. Web pages containing PHP script have a php extension instead of an htm or html extension. Two lines are added to the configuration file for the Apache HTTP server to indicate that the PHP interpreter should be invoked for those pages with the php extension. (Similarly, Java Server Pages are denoted with a jsp extension.) Another approach to web application development may be found with Ruby on Rails [17], a new framework that builds on the Ruby programming language [18]. The Rails framework builds upon a specialized web server and overcomes a major inefficiency of traditional web applications. The Basecamp project management system (http://www.basecamphq.com) is built on this framework. An important aspect of all of these tools is that they make effective reuse of major software components and also enable future tools to be built upon the foundation that they have established. The net result of reusing these mature components will be growing quality of the applications, less need to write new code, and reduced time for creating both prototypes and production versions of web applications.

3 Conclusion Abstraction and modularity are key software engineering principles that are central to the design of web applications. Patterns for site architectures, software components, and interaction design provide a powerful foundation for the effective creation of usable and efficient web applications. Logging provides valuable information on both application performance and user behavior. When new web applications are built upon proven components that follow these principles, the development time for new applications is drastically reduced, while the quality of the application is increased. The domain of web application design shows numerous areas where user interface design interacts with these software engineering principles, making it an excellent area for further study of integrated methods and processes involving these domains, as well as a basis for a science of design.

Principles for the Design of Web Applications

9

References 1.

2.

3. 4. 5. 6.

7. 8. 9. 10.

11. 12. 13. 14.

15. 16. 17. 18.

Dantzig, P.: Architecture and Design of High Volume Web Sites. In: The Fourteenth International Conference on Software Engineering and Knowledge Engineering, Ischia, Italy, July, 2002, pp. 17-24. Yen, B., P. Hu, and M. Wang: Towards Effective Web Site Designs: A Framework for Modeling, Design Evaluation and Enhancement. Proceedings. IEEE International Conference on e-Technology, e-Commerce and e-Service, 2005, pp. 716-721. Menascé, D.A. and V.A.F Almeida: Scaling for E-Business: Technologies, Models, Performance, and Capacity Planning. Prentice Hall, 2000. Killelea, P.: Web Performance Tuning, 2e. O’Reilly, 2002 Barish, G.: Building Scalable and High-Performance Java Web Applications using J2EE Technology. Addison-Wesley, 2002. Van Duyne, D.K., J. Landay, J.Hong: The Design of Sites: Patterns, Principles, and Processes for Crafting a Customer-Centered Web Experience. Addison-Wesley, 2002. Nielsen, J. Designing Web Usability: the Practice of Simplicity. New Riders, 1999. Rosenfeld, L. and P. Morville: Information Architecture for the World Wide Web: Designing Large Scale Web Sites, 2e. O’Reilly, 2002. Wasserman, A.I.: Toward a Discipline of Software Engineering, IEEE Software, November, 1996, pp. 23-31 Sullivan, K.J.: Science of Design: Software-Intensive Systems -- A National Science Foundation Workshop. Airlie, Virginia, November, 2003. http://www.cs.virginia.edu/~sullivan/sdsis/workshop%202003.htm Parnas, D L.: On the Criteria to be used in Decomposing Systems into Modules, Communications of the ACM, 15, 12 (December, 1972), pp. 1053-1058. Gamma, E., R. Helm, R. Johnson, and J. Vlissides. Design Patterns: Elements of Object-Oriented Software Architecture. Addison-Wesley, 1995. Alur, D., D. Malks, and J. Crupi. Core J2EE Patterns: Best Practices and Design Strategies, 2e. Prentice-Hall, 2003. Van Welie, M. and G. van de Veer: Pattern Languages in Interaction Design: Structure and Organization. In: Human-Computer Interaction – INTERACT’03, Zürich, September , 2003, pp. 527-534. Lerdorf, R. and K. Tatroe. Programming PHP. O’Reilly, 2002. Williams, H.E. and D. Lane. Web Database Applications with PHP and MySQL, 2e. O’Reilly, 2004. Thomas, D. and D. Heinemeier Hanson. Agile Web Development with Rails: a Pragmatic Guide. Pragmatic Bookshelf, 2005. Thomas, D., C. Fowler, and A. Hunt. Programming Ruby: the Pragmatic Programmer’s Guide, 2e. Pragmatic Bookshelf, 2004.

Requirements for the Integration of UE Methods in SE Processes from the Perspective of Small and Medium-sized Enterprises (SMEs) Daniel Kerkow1, Karin Schmidt2, Frank Wiebelt3 1

Fraunhofer Institute Experimental Software Engineering Fraunhofer-Platz 1 67663, Kaiserslautern [email protected] 2

DaimlerChrysler Research Software Engineering Department Software Processes and Quality (REI/SP) Postfach 2360 Wilhelm-Runge-Str. 11 89013 Ulm, Germany [email protected] 3

Institute for Technology and Work (ITA) Kaiserslautern Kurt-Schumacher-Strasse 74a 67663 Kaiserslautern, Germany [email protected]

Abstract. This article describes the collection and derivation of requirements for the integration of usability engineering (UE) methods in software engineering (SE) processes from the perspective of small and medium-sized enterprises (SMEs). An exploratory, not generalisable survey of 11 typical SMEs was carried out in order to register their special needs.

1 Introduction The gap between established software engineering (SE) methods and methods for the determination, assessment, and increase of usability of software systems has not been closed. In both areas, established methods and tools are employed (Law, 2003). The integration of well-known usability engineering methods (UE) into current working practices of software design companies has not been successful from the scientific perspective (Baumann, 2004; Harrison, 2004; Kohler, Leidermann & Birk, 2003; Seffah, 2004). In systems development, the designer, scientists, and engineers have to enter into a dialogue. Only in close collaboration and with interdisciplinary methods can the transfer of principles, knowledge, and objectives be assured in development projects.

The following exploratory survey had the objective to ascertain the nature of an interdisciplinary approach in German small and medium-sized enterprises and to determine their requirements with regard to integrated UE/SE methods and processes. This survey was carried out within the framework of the BMBF (Bundesministerium für Bildung und Forschung – Federal Ministry of Education and Research) project USEKIT. USEKIT is intended to provide, for SMEs in particular, methods for useroriented software development. The main focus lies on interlocking the methods of software engineering and usability engineering. In section 2.1 we will present the methodology used for the survey, in section 2.2 the characteristics of the SMEs and the derived requirements for technologies that integrate SE and UE are described.

2 Method requirements from the SME perspective In a two-stage interview we questioned eleven german enterprises from various industrial domains. The objective of the first stage was the collection and evaluation of the current development processes in selected enterprises. Boundary conditions regarding the software engineering process were also researched. The purpose of the second stage of the interviews was the theoretical evaluation of individual UE methods. The enterprises were selected in such a way to represent the widest and varied markets and software products. The statistical analysis of the evaluation was not the main goal.

2.1 Survey methodology The software development process applied in the enterprises was surveyed free from hypotheses. As survey instrument we chose a combination of semiformal interview and scenario-based evaluation. The three stages of the survey are described below (see figure 1): 1.

Initial interview: recording of the current SE processes, UE processes, and specific boundary conditions in the individual enterprises

2.

Generation of scenarios: targeting the optimisation of product quality and risk reduction through the use of UE methods

3.

Interview 2: carrying out a plausibility check of the scenarios selected

Process Model Interview 1

plan Actions

Interview Guide

Action Prototyp

Interview 2

Interview2 Feedback

Requirements

Prerequisites

Figure 1: Three stages of the survey While this method does not generate a generalisable picture, it succeeds in indicating the diversity in the enterprises.

2.1.1 Interview and interview guideline In the first level of the survey, the characteristics of the development processes in the enterprises were explored free from significant assumptions. The questioning was carried out using a self-developed guideline and was equivalent to a semi-structured interview. The objective of the initial interview was to achieve a sufficiently detailed view of the current engineering processes in the organisation, to recognize potentials and deficits in the use of UE methods, and to describe concrete suitable actions. The interview guideline was structured into the following categories: 1.

General information about the organisation

2.

General information about software development

3.

Boundary conditions of software development

4.

Software development processes

5.

General usability questions

The interview focused mainly on those parts of the processes interview, concerned with requirements and tests. The results of the first part of the interview are presented in chapters 2.2.1 and 2.2.2.

2.1.2 Development of scenarios Working hypotheses were generated from the results of this survey. The hypothesis included determining the UE methods and processes that could be used and the way the enterprises could employ them in order to be able to utilise improvement potentials. The hypotheses were based on evidence and explicit statements generated in the first interviews. The purpose was to present not only new but also modified UE methods that could lead to possible improvement or to a risk minimisation in the development process of the respondent enterprises. In preparation for the second interview, concrete scenarios were developed. They included one or more actions such as:

x x x

Description of indicators gained from the interview Description of the identified potential improvements and their risks for the enterprise Concrete actions for the enterprise to obtain improvements through one or more UE methods

The actions were selected in such a way that they could be integrated into current development processes without major modifications. This seems to be an important prerequisite, as the cost of redesigning current processes may be an obstructive aspect for many SMEs. If the integration of UE methods is kept simple, there is less bias against UE integration in SME-specific processes. For the respondents, it was important that a realistic assessment of the feasibility was made. Therefore the necessary steps for the implementation of the suggested actions were described. An estimate of the costs to be incurred in the introduction of methods and the implementation during current operations was made. In the second interview, the actions developed were presented to the enterprises in the form of scenarios. In the following interview, the respondents were asked to evaluate whether the scenarios generally reflected the correct conclusions drawn from the results of the initial interview, whether they appeared generally coherent with regard to the cost estimation and, whether, in principal, the documented form of the proposed actions for the current working process was suitable. Special attention was paid to the boundary conditions in the enterprise or to any other miscellaneous influences that could hinder or prevent an implementation. Where appropriate, the scenario and/or the documented activities were adjusted. In order to obtain generalised results, the respondents were requested to ignore enterprise specific conditions in their evaluation (e.g. “My superior is principally not interested in usability.”)

2.2 Results The followings section summarize the results of the survey, characterising the participating companies, their SD- processes and usability maturity, as well as their reaction to the actions and methods proposed by the scenarios.

2.2.1 Characterisation of the Enterprises The number of employees varied greatly in the enterprises surveyed. However many of the enterprises had less than 50 employees. Thus the required personnel resources became an important constraint on the methods. The staffs of the enterprises surveyed were mostly organised in interdisciplinary teams, their background was predominantly technological.

All these enterprises develop their software products in projects. They vary greatly in the time and cost expended. Typically, the developed systems were built for the business-to-business (B2B) sector. In most cases, the target group for the software products was well-known and distinguishable. It was possible to identify a small number of users. Only two enterprises developed business-to-customer products (B2C). In this case, it was difficult to identify the typical user. The results of the survey showed that there was rarely direct contact to the end user. It was striking that enterprises specialised in individual domains. Often standard solutions were developed and customised. Table 1 shows an overview of the characteristics of the enterprises surveyed.

1

SME

Project, only used in DCAG branches

Product or Project

Web-based customer information system

Type of Product

30

32

Number of Staff

700

70000

Number of Users

Computer scientists, economists, MBAs

MBAs, more seldom computer scientists

Qualifications of Staff

Major phase approx. six months, with approx. 5-6 PJ and approx. 20 persons

Nine months to one year

Development Cycle

Standard development. Every 4-5 years (90-100 PJ);

Definition und tool-aided support of processes in research

Release every 6 months

Project for internal use

Teachers and trainers, psychologists, computer scientists, industrial engineers, MBAs

Initial development 3.5 years, afterwards annual advancement

2

3000

Designers, media designers, and engineers

Six months or longer

40

>10000

Engineers (Polytechnic, University) and career changers

Three to six months

400

Web-based workflow system for the support of updating management

40

May 00

Computer scientists, business economists, IT apprenticed professions, commercial training

Military systems for BWB; conventional WIMP and media projectors

Product manufacturing within the project

Internet presence incl. content management system and Web applications

Ca. 50

1000-2000

Prouct/Programme/P roject

4

Product manufacturing within the project

Media software for Free TV support

30

3

5

Project

Web-based system supporting strategic purchasing

Computer scientists, communication engineers, MBAs, economists, soldiers

6

Both

10

9

8

Both

Both

Both

Project

ERP

ERP

Warehouse Management Systems (WMS)

Public administration

153

26

80

Ca. 100

10-100

150-300

3-300

Oct. 50

Computer scientists, mathematicians, or engineers

Computer scientists, engineers

Computer scientists, technicians, physicists

Career changers with university degree, computer scientists

Standard development. every 2 years; customer project days to months

Irregular

Irregular; code from project development transferred into general code basis

From two months up to two years

25

7

11

Table 1: Characteristics of the respondent SMEs

2.2.2 Typical UE and SE processes For the small and medium-sized enterprises interviewed, a broad variation of SE processes could be determined. A closer look at this variability, however, yields three main directions: 1.

The current software process does not follow a procedure found in the literature, nor is there an underlying process model. It has grown through the course of the years from own experience, ideas, and solutions. Normally the process does not employ SE terminology.

2.

The development process is derived from a procedural model: frequently, these are derivatives of the Rational Unified Process (RUP), the V –Model, or the Waterfall Model. In part, the software development process is not completely process oriented.

3.

The process concept is given. It is typically a classical V Model process or a process that has been specially developed and implemented for these enterprises (in most cases, a version of the V Models).

What most of the SMEs' clients and suppliers regarded as paramount during the SE process was the collection and implementation of functional requirements, error tolerance, and the performance and stability of the system. However, the clients and suppliers had little or no understanding of UE activities as the following section indicates.

2.2.3 Usability activities in SMEs Generally, systematic usability engineering is not required by the clients of SMEs: it is therefore not practiced. In summary, three basic directions can be identified in companies from the usability perspective: 1.

Usability is neither an explicit objective nor a specific activity. Usability is neither required by the clients in the systems requirement documentation nor offered by the supplier. Technical feasibility stands in the foreground.

2.

Systematic UE is not practised. The necessity is known however both from the follow-up costs for corrective actions required for previous projects that led to, in some cases, high start-up costs made up of the user costs for the product and from user feedback. Experienced developers have learned basic ergonomic principles that they intuitively employ in new projects. That these requirements are not systematically determined, supplemented, and examined is a problem. The knowledge that is stored only in the heads of the individual staff members becomes outdated or is lost through staff fluctuation.

3.

The necessity and importance of usability efforts is well known. Usability is performed and even required by the customer. Some variations can be inter-

preted: ¾

In some cases, the user interface is created in cooperation with the customer. Because of this approach, products that comply largely with the perceptions of the user group are developed. The significance of the functionality is overshadowed by the nature and manner of the access.

¾

Some enterprises have integrated UE methods in their software development process and have their own UE procedure model. One is orientated by the implementation of control criteria (for example, conformity to expectations, functional adequacy, error tolerance, and selfdescriptiveness). (ISO 9241/11, 1997) Different UE methods are used (for example, development of user profiles, execution of a task analysis, expert evaluation, and iterative walkthroughs with and without user, usability tests).

¾

Yet it is necessary to note that UE and SE are not interlocked. A transfer of UE process results takes place only through an intensive communication among all the key stakeholders.

The under-representation of UE activities results in part from the lack of knowledge with regard to usability (caused in part by missing qualifications) or lack of understanding for the utility (for example, saving of time through efficient handling). As a result, only the assumed additional costs or the higher time expenditure are assessed. Deficits regarding the use of UE in SE processes are apparent on several levels: ¾ ¾ ¾

Organisational deficits: the organisational sector is ruled by a lack of understanding of the importance of usability. Methodical deficits: SE methods lack the extensions for the integration of UE aspects (Identification of Boundary Objects) (Star, 1990; Walenstein, 2003) Boundary conditions make the application of conventional methods impossible. Methods do not include adaptability information.

Basically the awareness of the need of UE integration in SE processes in SMEs is established. However, often the necessary know-how for handling the new (that is, for the enterprise) methods is missing. It is not enough merely to create or describe methods isolated from possible SE processes. Simple and easy-to-implement connections have to be emphasised or created in both the UE methods and the SE processes. This lack of interlocking of UE and SE processes has led to many levels of improvement potential being identified. These are mainly essential organisational improvement potentials (understanding of the importance of usability) and methodical improvement potentials (integration of UE methods; see section 2.2.4).

2.2.4 Requirements of UE methods Different concern-specific scenarios were derived from the surveyed company characteristics. These scenarios presented the introduction of individual or multiple methods, the associated costs, and the expected benefits. The following constraints for the methods could be derived from the feedback given by the respondents. Requirement 1: Closeness to typical working style The results of the interviews showed that people in companies had a certain working style, a culture. Most enterprises had a more technical background, where UE principles are not known. The presentation of different methods showed that those requiring an intense communication with the user tended to result in higher bias. The reason for this might lie more in the organisational and less in the (programmer-specific) task range. Usability methods are often accompanied by completely new activities that are very different from the typical programmer formats (e.g., ManPages). The transfer of usability principles into the development processes would be more successful if known methods from the SE process could be “enriched” by UE aspects. Methods concerning client involvement were often rejected: it is difficult to persuade the client to write specific requirements for the SME so that lower consolidation costs accrue. Regarding objectivity and reliability, methods should be optimised such that they can be employed without help of experts. Requirement 2: Support for scalability / adaptability Before the method is applied, an estimation of the effort is necessary. This effort has to be given relative to an existing environment with pre-existing artefacts, processes, resources and methods. This also means that a micro-invasive integration (not altering the entire SE process) into the pre-existing context may be done. The advantages of using UE methods are not of a binary nature: some of them are expected to be scalable, for example, performing a usability test with only one person yields a certain understanding of the perspective of users even though the method would favour a larger empirical foundation. This means that UE methods would be much more widely accepted in the SE world, if information about their scalability were at hand. Requirement 3: Estimation of risk “No-Go-Criteria“ (e.g. because of time pressure) are desired. The risk to apply an unknown method is too high for a company, if the responsible person can not estimate the consequences, the benefits, the goodness and the probability to be successful. Requirement 4: Support for introduction Guidelines on how to initiate the process to intertwine methods with preexisting processes are necessary. These could be: initial efforts and best practices.

Requirement 5: Estimation of goodness Methods should be optimized with respect to their objectivity and reliability, in order to be applicable by non- experts. Requirement 6: Support for introduction Guidelines on how to initiate the process in order to intertwine methods with preexisting processes are necessary. These guidelines can embrace initial efforts and best practices. Requirement 7: Expected value and contribution Methods shall be able to solve short term and actual problems. The “incubation1” time might be a relevant factor for the acceptance of a method. Companies on a rather low maturity level hesitate to invest in methods that do not solve actual project specific problems and whose benefit and value is not proved. Methods that have the goal to grow and mature are rather accepted in larger companies with resources for growth. The value and the benefit shall be recognizable and empirically proved, also in relation to efforts to be spent. The value and benefit is not only a matter of financial metrics, but also includes the rather process oriented aspect of impact, effect or contribution: The expected results shall be known and anticipated. Other requirements “Good“ methods increase the motivation of the developer (satisfaction with the result) and act as an “enabler” of creativity. All in all, such methods are more likely to be accepted, if they serve as communication instruments, reduce development costs, and improve aesthetic aspects of the product.

These requirements should be considered when we talk about the development of methods and their integration into preexisting processes. In summary they show the necessity of a flexible and value centered approach for the selection of usability methods within software engineering.

3 Conclusion The development of methods is not completed with instructions and guidelines. Rather, methods must include the intentions, cultural principles, and manageability of the (that is, justification for the) recommended scaling. In a critical situation, the risk 1

Pharmaceutical term that refers to the time from the moment of exposure to the development of symptoms of a particular infectious disease. In our context the time between method application and the first contributions to value.

of employing unknown UE methods in SMEs is too high if management is unable to balance the consequences regarding the quality and possibility of success. Through continued method research (and method development) and standardisation of characteristic method indices, a professional handling and risk-minimised implementation of methods for an assured use in SMEs should no longer be a handicap. Moreover, not only is client bias lowered, clients also learn to value the employment of UE methods. This demand provides a foundation for methods and their characteristics for the method developed within the framework of USEKIT.

4 References 1. 2. 3.

4. 5. 6. 7.

8.

Baumann, K. (2004). Position paper for the CHI 2004 Workshop #2 on Identifying and Bridging Gaps between Disciplines. Paper presented at the CHI 2004. Harrison. (2004). Bridging between disciplines in the development of dependable interactive systems. Paper presented at the CHI 2004. Kohler, K., Leidermann, F., & Birk, A. (2003). Der Weg zu einer stärkeren Verzahnung von Usability Engineering und Software Engineering. Paper presented at the First Workshop des German Chapters der Usabiltiy Professionals Association, Stuttgart. Law, E. L.-C. (2003). Bridging the HCI-SE Gap: Historical and Epistemological Perspectives. Paper presented at the Interact 2003. Seffah, A. (2004). Bridging the Educational Gap between SE and HCI: What Software Engineers Should Know? Paper presented at the ICSE 04. ISO 9241/11, (1997). Ergonomic requirements for office work with visual display terminals (VDTs) - Part 11: Guidance on usability. Genf, NR: ISO/FDIS 9241-11:1997(E). Star, S. L. (1990). The structure of ill-structured solutions: Boundary objects and heterogenous distributed problem solving. Distributed artificial intelligence vol. 2, Morgan Kaufmann Publishers Inc. San Francisco, CA, USA. 2: 37-54. Walenstein, A. (2003). Finding Boundary Objects in SE and HCI: An Approach Through Engineering-oriented Design Theories. ICSE 03.

Acknowledgement This work is funded by the German Federal Ministry for Education and Research (BMBF) under Contract No. 01ISC23D.