REMOTE - Mathematical and Computer Sciences

1 downloads 0 Views 2MB Size Report
of the commando brigade by the incoming mechanical brigade; for condition 2 ..... 45. Communicating over electronic media was not as effective as face to face.
REMOTE – Desk-top VR for future command and control? R.S.Aylett1, C.Delgado1, J.H.Serna1, R.Stockdale2, H.Clarke2, M.Estebanez2, P.Goillau2, T.Lynam2. Centre for Virtual Environments, University of Salford, Business House, University Road, Salford, M5 4WT, Manchester, UK 1 QinetiQ, St. Andrews Road, Malvern, Worcestershire, WR2 3PS. 2 Email: [email protected], [email protected]

Abstract The paper discusses a study in supporting collaborative military planning in which groupware, videoconferencing and a desktop Collaborative Virtual Environment (CVE) were used. It discusses the design and implementation of the CVE and the set-up and execution of the study using questionnaires and observation. The results of the study questionnaires showed that the CVE was not seen by users as the best of the ways offered to support collaborative planning; these results are discussed and their implication for the design of such a CVE are assessed.

Keywords Virtual environment, collaboration, groupware, avatar, net-VE. 1. Introduction This paper reports work carried out by QinetiQ and the University of Salford, both in the UK, to compare the use of a desktop collaborative virtual environment with two other modalities as support for a distributed military command team. The system produced, REMOTE (Reach-back Enabling Multiple Objective Team-working Environment), was included in an initial study evaluating potential novel command technologies for distributed team working, and more detail can be found in the resulting QinetiQ technical report [Clarke et al., 2003]. The customer for this work was the UK Ministry of Defence (MoD), who are, like other national defence departments (for example the United States Department of Defense), looking at ‘command posts of the future’. Command posts are

responsible for decision-making, the co-ordination of manoeuvres, and issuing orders to troops, and are made up of commanders with an often substantial support staff. Such large command posts are difficult to deploy, especially in proximity to a battlefield - not the safest place to be - and yet there are many advantages in doing so for rapid and efficient decision-making. As a result the idea of using technology to combine a small mobile group with all the distributed resources they need – especially experienced people who should not be risked by putting them close to combat - has become an attractive one. This produces a geographically distributed team. Much research has been carried out into distributed team working in the much broader context of the globalisation of organisations. The issues and problems are comparatively well known as a result, and mostly arise from the absence of spatial co-location. Colocation has many advantages, especially for tightly coupled synchronous interaction [Olson et al92, Covi et al 98]. A shared local context, including shared time frame and many outside events, supports easy socialising, building trust and mutual understanding [Bradner and Mark 02] and supporting informal interaction before and after more formal episodes. Personal identities are known and can be taken into account – including cultural and language differences - and peripheral interactions can be seen in passing and interpreted. Mutual awareness is known to be central to collaborative working [Steed and Tromp 97]. Within interactions themselves, many channels other than the overt semantic content are available – for example gesture, gaze, posture, facial expression and prosody. It has been estimated that more than 65% of the information exchanged during a face-to-face interaction is expressed through these largely non-verbal mechanisms [Argyle 88], which

are mainly analogue, continuous, and support subtle distinctions and nuances. Flowing conversation depends on many such channels for ‘back-channel’ regulation and successful turn taking [Cassell et al 01]. Moreover, they play important roles in establishing a common frame of reference both for concepts and objects, for example through deictic gestures such as pointing. This in turn relies on the spatiality of such references – people and objects are located in a common space. In multi-person meetings, such channels can also be used to express or judge the reaction of other participants to the current speaker. Finally, co-location minimises communication delays. This analysis has produced a demand for ‘virtual co-location’ (VCL) – a loosely defined concept which can be taken to mean capturing some or all of the characteristics of colocation for participants not actually co-located, using a variety of information and communication technologies: audio, video and computer-based. The basis for the study reported here was the hypothesis that some of the characteristics of co-location that are difficult to support through mechanisms such as email, electronic chat, videoconferencing and groupware, could be supported through a collaborative virtual environment (CVE) [Churchill and Snowden 98]. Here, a multi-user 3D graphicallyrealised shared virtual space would play some of the role of shared physical space through the sense of physical presence – the feeling of being physically located in the virtual space – which with multiple users might then produce a feeling of physical colocation. 2. Study methodology There are known methodological problems in evaluating CVEs [Steed and Tromp 97] related to the prototypical nature of most applications and the need to deal with complex

social behaviour both inside and outside the CVE. The geographic distribution and number of users makes conducting proper controlled experiments very difficult, while the prototypical nature of applications usually makes it infeasibly expensive in time and other resources to create different experimental conditions. These constraints are particularly acute where the task involved is unpredictably structured and has an extended duration, as in military operations (note that Steed and Tromp considered video-conferencing and booking holidays as their applications, both much less problematic than military operations). In addition, studies involving military teams are inevitably hard to set up. Not only is it inconceivable that wholly novel approaches would be tried in a live situation, even their use in official exercises would require a degree of maturity in the technology and acceptance by the prospective users still a long way from being achieved. Given that this work involved an initial study, or ‘proof of concept’, participant numbers were bound to be too small to adopt a rigorous quantitative approach, forcing a qualitative and exploratory stance. Nevertheless, qualitative approaches are well established in HCI work where they are seen as fundamental to user-centred design, drawing on cognitive and social psychology and ethnography [Hughes et al 95]. Qualitative methods are also widely used in the broader field of Information Systems [Myers and Avison 02]. The qualitative technique adopted was that of the exploratory case study. This refers to the collection and presentation of detailed information about a particular participant or small group, frequently including the accounts of subjects themselves [Sechrest et al 96]. The case study technique gives special attention to completeness in observation,

reconstruction, and analysis of the cases under study and deals with a situation in which there are many more variables of interest than there are data points. This contrasts with controlled experiments in which there are typically several orders of magnitude more data points than variables. A case study is more concerned with establishing variables and questions for future research than with final, generalisable conclusions, but it can also be seen as an individual iteration in a larger process of user-centred design, with particular relevance to use-in-context and interface and interaction issues [Shneiderman 92]. This is particularly true where a case study can legitimately be categorized as one involving ‘typical’ users carrying out a ‘typical’ task. Two teams of four participants made up of ex-service personnel were made available. These were experienced personnel who had carried out such operations in their military careers and could therefore be regarded as typical of the user group being targeted. However these participants were only available for the week-long study in March 2003 itself, and not for prior development of the user requirements. Here, experience of past studies in other military domains and some comment from a military link officer were the only input. The case study approach is familiar to these users and is widely-used within the military environment since it is a basic training method and the framework for military exercises. One can therefore have some confidence in the typicality of the scenarios used and their relevance to the evaluation of the technology. As we have commented, it has often been found infeasible in evaluating CVEs to create different experimental conditions. However it was possible to do this for the study being discussed given that the participants were available for a week, rather than the shorter times of some other studies [Steed and Tromp 97]. An initial view was taken that since

groupware offers the most mature technology for distributed teams, this should form the base-line condition for the study. The package GROOVE Workspace [Groove-Networks 02] was the groupware used. Two further experimental conditions were then added, as enhancements to, not replacements for, groupware: the addition of video-conferencing, using CUSeeMe [CUSeeMee-Networks 01], and the addition of a CVE. It was hoped in this way to capture differential benefits of the second and third experimental conditions. Thus it was a fundamental requirement for the CVE that it interface with the chosen groupware platforms and support the invocation of groupware components from within it. Finally, it is important to remember that the target users are very unlikely to be sophisticated users of IT and are used to coordinating activity at a distance by voice only, using radio. They are however also used to working in a disciplined style controlled by Standard Operating Procedures (SOPs). SOPs are detailed written instructions to be routinely followed so as to achieve uniformity in the performance of a specific function. They are used in many large organizations, especially where the consequences of procedural errors can be very serious, as for example in military operations, and experienced personnel, such as those in the current study, are familiar with the relevant SOPs and will apply them appropriately under case-study conditions. 3. The construction of REMOTE The experimental constraints were the starting point for the requirements of the REMOTE software. Objects spatially located within the CVE were to be used as a means of invoking external applications such as email, web browser, and groupware components. A degree of fidelity to the physical environment was required – the virtual world should look something like a command post and include objects such as a bird

table (i.e. large table for the display of maps etc) that would be the central feature in a real command post. It was decided that a central room with other rooms leading off would be included, with each room containing a particular piece of functionality. There were also requirements on the user’s representation, or avatar, that will be discussed below. 3.1 Choice of development framework An initial stage of the work was to evaluate available CVE platforms. It was considered desirable to use commercial software since this has generally higher levels of support and fewer bugs than research platforms and is consistent with the policy of the end-user to use COTS (Commercial Off-The-Shelf software) whenever possible. For this reason as well as issues of availability and learning curve, the two large research platforms DIVE [Frécon 98] and MASSIVE-3 [Greenhalgh et. al 00] were not adopted. Three commercial packages were evaluated. These were: Adobe Atmosphere [Adobe 02], a multi-user web virtual world package, then available in beta-release; the game engine distributed with the game Unreal Tournament [Epic-Games 99]; and the WildTangent software environment [Wildtangent 02], originally a games engine but now marketed as a builder of interactive web media environments and a rapid prototyping toolkit. All of these were capable of supporting a client-server architecture, of handling a 3D graphical world, and of importing 3D models from widely available modelling packages such as 3ds max [DISCREET]. Atmosphere was however felt to have a rather low frame-rate and a very limited API for adding new behaviours, only offering JavaScript. It was not clear that it would be feasible to call other applications from within it, such as GROOVE, as required.

The standard API for the Unreal Tournament engine was limited to its proprietary Unreal Script, and while the Gamebots [Kaminka et al. 02] extensions make it possible to run external code on a socket interface this seemed cumbersome. However WildTangent has a very open API supporting Visual C, Visual Basic or Java (though it requires the Microsoft JVM, not updated past Java 1.1.4), and is easy to integrate into web browsers, unlike Unreal. While Unreal uses OpenGL as its underlying library, and is thus multiplatform, WildTangent sits on DirectX and is Windows-specific, but this was not seen as an issue for a proof-of-concept system that was intended to run under Windows. In any case, this close relationship with DirectX makes WildTangent fast, and as it handles all of the graphics, there is little speed penalty in using Java as a framework around it with the accompanying benefits of a modern web-oriented language. Thus WildTangent was chosen as the development platform and Java was used to access its facilities. 3.2 Embodiment and identity In considering what user representation to support within REMOTE, certain basic considerations were important. User representations – avatars1 – are added to a CVE to meet six generic requirements, as proposed byPandzic et al 97: (i) Perception allows a user to register the presence of others, and (ii) localization to see where in the environment they are. (iii)Identification is then the ability to know who they are. These three capabilities are basic but can be met by fairly crude graphical representations. A further three capabilities require more sophisticated representations. (iv)Visualization of interest focus usually requires the representation of gaze direction, most obviously through a user representation with eyes, whether realistic, as in a humanoid figure for

1

We use the term avatar here in its original VR sense [Stephenson 93] to mean any representation of the user in the graphical world, not in the more recent sense of any humanoid virtual agent, user-driven or not

example or iconic, such as a personal emblem or object. This is also achievable using a sensor beam as in virtual robots. (v) Visualization of actions can only occur if the avatar has the ability to carry out actions, using some form of end-effector, in robot terminology. This could be a robot gripper, or arms and legs, but again, could also be represented very simply by extending a line to a manipulated object, or through a 3D cursor or virtual hand grasping the object. (vi) Avatar Communication poses the heaviest requirement - in real life, as already argued above, it is supported by gesture, posture, facial expression and vocal characteristics. Creating an expressive avatar requires a relatively sophisticated representation, which may then create a problem for the user who has to manipulate this expressive repertoire while also carrying out whatever collaborative task is at issue. Perception, localisation and identification of course include self-knowledge too, and this depends on whether a first-person or third-person view is adopted. In the former case, the user does not see their own avatar since their position is as if they were looking through the avatar’s eyes; in the latter case, the user is positioned a little above and behind the avatar. Atmosphere and Unreal Tournament both take a first-person view (with the weapon being carried visible in Unreal) but a third-person view was selected in REMOTE. In the real world we always have some awareness of what we look like and where our body is in relation to objects around us, and this effect of the third-person view was felt to outweigh the inconvenience of having the avatar body partially block the user’s field of vision. A humanoid form for an avatar is not always essential, but in terms of expressiveness it has many advantages [Benford et al 97]. It seemed especially relevant to REMOTE’s target users given their lack of familiarity with graphical environments. It also provided

an intuitive method for displaying the service, rank and role of a particular user, significant identifiers in military organizations and especially in control rooms where commanders of all services may collaborate along with civilian or government specialists. A natural way of identifying individuals would be to texture their face onto the face of their avatar, technically quite straightforward. The conditions of the study made this impossible, as the participants were unknown until shortly before it took place; instead the name, role and rank of the user were attached above the head of the avatar, using a billboard so that it always faced towards a viewing user, the data being gathered at logon. However this was removed from the user’s own avatar (they know who they are) as it blocked too much of the field of view. Uniform was used to indicate service membership – camouflage for army, white shirt and dark blue trousers for navy, and light blue shirt and blue trousers for air force. Both male and female avatars were provided, the gender user-selected at logon, and figures of somewhat heroic physical proportions were produced (see Figure 1). Had timescales allowed it, different character designs would have been developed and evaluated with end-users: as it was, only feedback from the military liaison officer was available and his response to this design was positive.

Figure 1 Avatars in REMOTE

As already suggested, there is a trade-off between the communicative advantage of giving avatars more expressive behaviour and the disadvantage of producing a cognitive overload for users in trying to invoke such behaviour appropriately. A small set of animations was therefore provided as seen in Figure 2. It was seen as particularly important to indicate when the avatar was inactive because the user driving it was absent – the ‘chill-out’ animation. The rotate and walk animations were invoked when the user navigated their avatar within the environment. However it was also considered important to a feeling of physical presence and realism that avatars could not walk through walls or furniture, so that obstacle avoidance was required. The method of separating axes was used to determine whether or not two bounding boxes intersected. Using the data for the centre, axes and extents of the avatar’s box and for the bounding box of the object to be tested, classical algorithms to test if two boxes have an intersection as described in Eberly 01 can be used.

If they do collide then the velocity of the avatar is reduced to 0 and the position is updated accordingly. Latency is always an issue in a

start

Chill-out ( not Present)

CVE, and if the position of an lean

avatar is updated only on coordinates from the server,

Still

latency effects can produce a Salute

very jerky animation. Dead-

Rotate Walk

Figure 2 State Diagram of animations.

Run

reckoning can be used to improve this situation by

making a local prediction in the client about where an avatar is likely to move, moving towards that point and updating with the incoming coordinates from the server. Thus dead reckoning consists of two parts prediction and convergence [Singhal and Zyda 99]. Prediction is the way the entity's current state is computed based on previously received update packets while convergence defines how the object’s position and trajectory is corrected as in the following equation: Pred.pos after t secs = pos + velocity x t

The difference between the real position and the predicted position is next computed and a new path is calculated, which is used in the convergence algorithm. Finally a point in the future is selected on the new predicted path, and the navigation is updated accordingly. Several techniques have been proposed [Singhal and Zydah 1999], although a linear convergence (using velocity but not acceleration) was selected for REMOTE,

Figure 3: REMOTE CVE and launched applications.

because latency was not thought to be a big problem in the setup of this study given that it ran on a LAN in two neighbouring rooms. 3.3 Interacting with other applications Object Room

Action

Board

Main room

Launch Newsgroup

Table with map

Bird table room

Groove sketchpad tool

Cabinets

Files room

Groove files tool

Board

Pictures Room

Groove Pictures tool

Presentation dais with stairs

Meeting room

Nothing

Flying doc

Documents room

Groove notepad tool

Other Avatars

All

Netmeeting call to Avatar's IP (was not used in actual test)

Table 1 Objects and locations in REMOTE

As discussed above, the ability of the CVE to interact with other applications was a fundamental requirement. Applications were associated with particular objects in the environment, mostly located in specific rooms, each labelled with the function. Table 1

shows the objects, associated applications, and the room in which they were located. A bounding box was implemented around the launch objects and the associated application for a user launched when their avatar intersected the bounding box. To make this visible to the user, the bounding box was rendered as partially opaque on intersection – as in the top left component of Figure 3, in which the Board object in the Pictures room has been intersected, bringing up the GROOVE Picture seen bottom right in Figure 3. When the avatar leaves the bounding box the application is closed, and a delay factor was incorporated to prevent rapid zigzags between opening and closing. Interactions with another user were initiated by approaching that user’s avatar, and such interactions were limited to one-to-one. The main purpose here was to invoke Netalk to allow voice interaction between two users. The bottom-left component of Figure 3 contains the logging file listing specified events in the VLE, though the text is not readable in this diagram, and the top-right component contains the login facility. 3.4 Overall architecture Figure 4 shows the client-server architecture of the overall system. Client-side servers handle the invocation of GROOVE and other applications, a login application records significant actions of the avatar and the login functionality handles the identification of users and their attachment to an avatar. The architecture is divided into two logical components: the client and the server side. The server side handles the data received from the clients, that is positions and actions of the avatar and it replicates the data to the clients so that the all clients have the data regarding the other users' avatars. To handle communication between the server and the

Figure 4 Overall architecture diagram.

clients, the Observer-Subject Pattern was used, updating information when a message is received by the listener (as seen in figure 4). This runs in a separate thread. On the client side, (Figure 4) several subcomponents were developed to handle the required tasks. Object Orientation and Design Patterns [Gamma et al, 1996] are heavily used in the architecture offering support for modularity and the use of existing abstract solutions to common problems like the use of controllers (the singleton pattern was used to have just one instance of a class of controllers in the code). These controllers are used to handle the animation of the avatars, the network communications and the data relating to all the objects displayed in the virtual environment. Also important was the use of the command pattern to link an object in the virtual environment to an action, for example to open a GROOVE subcomponent. GROOVE is itself a set of subcomponents, and as Table 1 shows, four of these – sketchpad, files, notepad and picture – were invoked by an avatar approach to the relevant object. This was handled by the GROOVE Server: the object has an associated message, which is sent to the server, creating a link between object and GROOVE subcomponent. A similar mechanism was used to open other Windows applications using the application server,

and an editable XML file was used to specify the repertoire of connections making the system easily extensible. Another external program is a Logger Server, used to store copies of messages so that a history of the interactions between avatars and objects can be maintained. This was also configurable via XML. The login is used to enter the profile of the user and allows a user to choose their sex (male, female), their service, their role and their rank. The selected profile is used to generate an avatar in the appropriate uniform and the label above the avatar's head. 4. Study and results As already mentioned, the study involving the use of REMOTE ran in March 2003 and involved two teams of four ex-military personnel, each forming a HQ. A team contained a Chief of Staff (COS), an Intelligence coordinator (J2), Operations controller (J3) and Logistics controller (J4). These are standard and well-understood HQ roles. Each team was located in a separate room but shared the same LAN. No attempt was made to simulate latency, bandwidth constraints or other WAN conditions. 4.1 Use of scenarios The scenarios were all set within the context of a UN peace-keeping operation in the fictitious country of New Albion, an island archipelago in the North Atlantic approximately 600Km to the west of UK. New Albion comprises three ethnically divided provinces: Wessex (Angle), Siluria (Celtic) and Devonia (Celtic). After 10 years of bitter civil war the UN is invited to intervene to monitor a lasting ceasefire, however UN intervention fails through lack of a clear mandate and other factors. A new ceasefire is brokered and NATO agrees to intervene with New Albion consent and a revised UN

mandate. The US, UK and France each agree to send Divisional sized forces (NAFOR). Despite the ‘ceasefire’ there is an operationally urgent need to get new forces on the ground as quickly as possible. The UK contributes a framework Multi-National Divisional HQ, some Divisional troops, a Mechanised Brigade and an embarked Commando Brigade. The background information remained the same throughout all three conditions, minimising the additional read-in time each day and allowing more time for planning. Each of the three experimental conditions described above in section 2 was introduced by a new task set within the overall scenario. For condition 1, this was to organise the relief of the commando brigade by the incoming mechanical brigade; for condition 2 to deploy forces to secure a town in which ‘ethnic cleansing’ was threatening; for condition 3 to evacuate threatened settlers from the town back to their ethnic home area and protect those staying on. The output from each scenario was a plan briefing in which a remotely situated representative Commander was presented with potential courses of action (COA). Scenarios were such that they deliberately forced collaboration between HQs for example, sources of information were placed at different levels within each HQ. A control cell ran the simulation, generating military and political events. The maintenance of Situational Awareness (SA) is usually ranked among the most important tasks for all command post team members [Taylor 1990] and experienced personnel such as those used in the study perceive it as a key task. A widely (though not universally) accepted definition is that of [Endsley 95] in which SA is the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future: informally ‘what is

going on’. Military planning in command posts is naturally hierarchical (for example strategic, tactical or operational), and in the scenario used, planning invariably commenced with one HQ having better higher-level SA while the other would have better lower-level SA. This was achieved by having one HQ ‘on the ground’ operating at the tactical level and the other working at the operational level out of the area. The study used a within groups (related, same-subject) design [Clarke et al 03] so that all subjects performed all conditions: GROOVE alone, GROOVE + CUSeeMe and the GROOVE + REMOTE. The constraints on the availability of participants, and the resources available, made this the most feasible approach. There are known difficulties with this design: for example, there can be a learning effect from one condition to the next and a potential order effect where performance either improves as a result of practice and familiarity or degrades because of fatigue or boredom. Changing the scenario and the role of the team members between the conditions can counterbalance some of these effects. However, this needs to be taken into account when analysing the results and is discussed in section 4.4 below. In the five-day programme, each 4-person team spent the first day separately carrying out non-technology-based team-building exercises, to avoid an increasing team familiarity and cohesion among the co-located participants impacting the study. During the second day, familiarisation with GROOVE was carried out. Then one condition was run on each day, with the last part of days 3 and 4 spent on training in CUSeeMe and REMOTE respectively. The last part of day 5 was used to complete the questionnaires discussed below and for a debrief.

4.2 Data collection and analysis Data was collected by observations made during each experimental condition and through questionnaires completed after each experimental condition, see Appendix 1. Two observers were present in each team room and used an observational ‘crib sheet’ as a guide – see Appendix 3. Observations covered: the human factors issues associated with working in a distributed command environment; how the tools supported the VCL concept though team and task work; the effects of the different tools on local and distributed team interactions; the effect of the tools on the military task; difficulties in using the tools. A set of categories was developed to allow observations to be grouped and this can be seen in Appendix 4. The debrief was carried out using the prompt sheet in Appendix 2. Three questionnaires were used, containing a 7 point Likert rating scale and completed electronically. The three questionnaires covered: aspects of distributed and collaborative team working effectiveness; the degree to which the collaborative technologies meet VCL (Virtual Co-Location) requirements; and aspects of the collaborative technologies including general comments / strengths / weaknesses / missing features, and tool design aspects and the effect of the collaborative technologies on co-located team working. Subjects were invited to add any comment they wished after any question. The debrief followed the questionnaires and was structured around five ‘open’ questions and associated prompts as seen in Appendix 2. 4.3 Results summary The averaged responses to the VCL requirements questionnaire, which was composed of 41 statements each with a 7-point Disagree-Agree response scale, are shown as graphical profiles in Figure 5. There was considerable variation in responses across individual

questions, of which all but two – questions 2 and 9 - were posed positively (i.e. higher score = better). Overall, the GROOVE + CUSeeMe condition achieved higher ‘agreement’ ratings than did the GROOVE alone condition, which in turn was rated higher than the GROOVE + REMOTE condition. Overall means across all conditions and participants were 4.87 (GROOVE+CUSeeMe), 4.12 (GROOVE alone) and 3.38 (GROOVE+REMOTE). Separating these profiles by team showed that Team 1 rated GROOVE alone and GROOVE+CUSeeMe similarly to team 2. However, GROOVE+REMOTE achieved higher agreement ratings from Team 2 (overall mean 3.86) than Team 1 (overall mean 2.9). This difference appeared to relate to the very different ways of working of the two teams: observation logs showed that Team 2 spent far more time in own-team meetings to discuss plan options, details of how they were going to go about the task, the information that they had available to them, initial ideas, and the tasks each of the team members would carry out. They would most often turn away from their workstations to hold such meetings and as a result message alerts and calls from team 1 would be missed. Team 2 members also stood up to leave their workstations and temporarily gather around other members’ workstations to discuss information far more than team 1 members. A consequence of this was that Team 1 members were frequently unable to contact Team 2 members and expressed a great deal of frustration. REMOTE had no facility to signal when such own-team meetings were taking place and thus contributed to rather than relieving Team 1’s frustration. REMOTE’s best scores related to Q11 (‘The software enabled me to rapidly exchange key information with another individual on a one-to-one basis.”) and to Q21 (“The

software enabled all members of the team to undertake their roles and responsibilities”) strongly suggesting that the overall critical response was not due to functional incapability or fragile implementation, but rather to the basic design decisions made. Its worst scores relate to Q18 (“The software effectively supported social interaction (e.g. chatting or interacting separate to the work task) with another individual on a one-to-one basis.”) and Q19 (“The software effectively supported social interaction with several other team members at the same time.”), reflecting the constraint of one-to-one communication only. In terms of meeting distributed team members as easily as co-located members, REMOTE was overall considered better at supporting this than GROOVE, but was considered worse than CUSeeMe. This was due to an exceptionally high rating by Team 2, which judged REMOTE superior to both GROOVE and CUSeeMe in this regard. Team 2 rated REMOTE on a par with CUSeeMe, however, in its ability to support a sense of shared meeting space. The ranking between the software of the three conditions shown in these subjective scales is reflected in the participants’ written questionnaire comments, and observers’ observations. The two other questionnaires alternated positive and negative questions, so that the result profiles are less easy to visualise than the one shown, but they also gave a compatible ranking of the study conditions. We return here to the issue raised in 4.1 above: whether there was a learning effect from one condition to the next and a potential order effect either improving performance as a result of practise and familiarity or degrading it because of fatigue or boredom which might itself explain the ranking of the three conditions. First we would restate that the

first day included a scenario based team task without any of the software under examination in order to minimise the effect of improving social cohesion among subjects who were new to each other. Secondly, GROOVE was in use for all three days of the experimental condition as well as in training on the second day. A positive learning effect for it as between condition 1 and condition 2 was noted in observation. However had this effect been dominant, then one would not expect the second experimental condition to be ranked higher than the third one, as it was. If a boredom factor for GROOVE was involved, one would have expected the observation process to detect it, and it did not. The area in which boredom and frustration may have impacted the study lay in the difference between Team 1 and Team 2’s work styles, and the frustration suffered by Team 1 as a result discussed above. This may have been a further factor in the more critical assessment of the third experimental condition by team 1. However both teams ranked the three experimental conditions the same, suggesting that the order effect, if present, was not decisive. In general, the observation regime acted as a control on both factors since it produced a detailed picture of action and interaction. 4.4 User -identified problems The overall questionnaire results above show that users were critical of REMOTE compared to GROOVE and CUSeeMe. The underlying problems prompting these results are brought out in user comments which were collected by the observers, made during the debrief session, and included in the questionnaires. Table 2 lists the most significant problems, illustrated with specific comments, which are naturally there taken out of the context in which they occur. Nevertheless, the typicality of the users and of the scenarios

7

6

Disagree - Agree 1-7

5

4

3

2

1

0 1 2 3 4 5 6 7

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

FIGURE 5: VCL requirements results

VCL requirements

T2 Groove

T2 CUseeMe

T2 REMOTE

asserted in section 2, gives us some confidence that our identification of the generic problems is correct. Discussion follows in the next section. Positive comment related to creating a feeling of presence and good awareness of other team members and what they are doing, for example: “The main strength is that it creates a feeling of the team being present especially the distributed team.”; “The avatars provide good awareness of other team members and what they are doing”; “ CVE provides a good sense of virtual presence”.

Problem

Comment

Navigation slow and

J2: "Where do I find the notepad?", COS: "In the documents room", J3: "see you there in 10

laborious

minutes!" "I just can't see the logic of it…it's just an arcade game!". Moving people around the environment is laborious. I can't see the point of moving mannequins around a set of corridors". COS: "OK, Let's all go to the bird table", J3: "Could take a while".

Difficulty in finding

"Where is he? He was in the files room, he was there a minute ago". (At the central table)

colleague’s avatars

"OK, he's landed".

Only one app at a time

"Can I put a map in there/ I want to be able to drag and drop. Need to import the map, but

can be used

can't import the map via sketchpad. It's hopeless. If I can't load a map into sketchpad, can only start from scratch." "If I want to work on a file, I have to come out of notepad". ”Difficult to know whether individuals in different teams are looking at the same map. Quite difficult to confirm, as you can't look at the map and chat at the same time” “The REMOTE environment forces you to only take in one piece of information at a time. Splits your attention.”

Avatar jams by app

"Queuing up at filing cabinets with 4 big guys in the way".

launchers – see Fig 6 Wrong information

"To look at a file, we need to leave the meeting room, bump into walls and go to the files

access metaphor

room…. I think that's ridiculous". "You're constantly thinking where is everything… and how do I get there?" "You shouldn't have to go into another room to get something" "I just think the analogy is wrong with rooms etc. You put together your folder and carry it around with you".

Only 1-1 voice

"This is even worse, we're just one-to-one chatting now". "I'm going to leave this now,

interaction

because there's no point"

Table 2. User-identified problems

5. Discussion and Further Work The results of the study show that the REMOTE CVE was not seen as the best way of supporting collaborative team working by its users. However one must take into account here the inherent difficulties in supporting distributed teams with technology. The following comment is instructive, arising from a study into the use of non-CVE collaborative tools to support distributed design teams in manufacturing engineering: “Findings from this study suggest that collaborative tools must be clearly superior to existing practices to merit the effort of deployment, adoption and subsequent use, since the burden of learning and mastering a new tool in a demanding and fast-paced engineering design environment may not outweigh the perceived benefits.” [Wierba et al 01 p1]. In a similar collaborative design environment, albeit with physically co-located teams, the technology that was most clearly valued was a printing wallboard that saved participants the time previously spent copying informal diagrams to take away [Olson and Olson 00]. Here, ‘just one thing’ was added to existing and familiar technologies. Work in the field suggests that involvement of users in the design of collaborative environments – difficult to set up in military domains – plays a very important role. For example, a successful ‘collaboratory’ of space physicists [Olson and Olson 00] has involved 10 major redesigns over 7 years, all involving the users. In this collaboratory, scientists organise themselves into virtual rooms of objects and remote partners to run experiments using real-time data from geographically distributed instruments, suggesting that the basic metaphor of the virtual room may play a positive role.

Figure 6 Avatars queuing to access the File resource. The key question is whether a CVE is inherently incapable of delivering clearly superior results both to existing practice and other technological solutions such as the two first experimental conditions, or whether a different set of design decisions, together with some maturing of underlying technologies, might produce a more successful CVE. This question is related to an ongoing ‘ space-versus-place’ debate within the CSCW community as summarised in [Benford et al 2001]. The arguments for the ‘space’ metaphor already outlined suggest that it is independent movement within a shared coordinate system, combined with the representation of others' positions through avatars that allows a CVE to support social interaction. Those supporting the ‘place’ argument suggest that other important aspects of an environment beyond the provision of a shared coordinate system are responsible for supporting social interaction. We are more convinced by the potential benefits of a shared space, and here put forward lessons learned from the study on the design of such a space for distributed military command teams.

REMOTE applied a relatively naturalistic design to its virtual world, while interfacing it to applications such as groupware with a desktop metaphor. These two metaphors did not appear to marry together well for the users, and the conflict between them was sharpened by locating each application launcher in a different virtual room. Unlike the collaboratory cited above, in which an experiment is a specific activity and thus fits conceptually into a separate room, military planning requires a strong integration of different information sources from different locations. Thus the bird table plays a very central role in real command posts and this was not successfully replicated in REMOTE. One user commented: “The key to digitisation is a version of the electronic bird table. Showing the same picture to all team members, and can plan concurrently”. REMOTE applied a naturalistic metaphor for physical space, this was not true of its information metaphor: “Compared to the analogue system, you have a map and notepad which you carry with you anywhere. So you shouldn't have to go to other rooms - not realistic". Allowing avatars to carry information sources would also make it natural to have many open simultaneously and support transfer between them – the restriction of having one tool only open at a time was felt acutely, especially .for maps, often used with other documents, and a natural focus for voice interaction. It would also have avoided queues around central resources such as in the Files Room (Figure 6). The consequence of the decision to distribute information sources was a high requirement for navigation. The effort this produced for users underlines a basic difference between real and virtual spaces. In the real world, most spatial navigation in office environments is carried out at a sub-cognitive level – we walk and think, or talk, at the same time. This is not true of navigating an avatar in a desktop CVE and it is hard to conceive of a

navigational mechanism that would be as low in effort. But this does not mean that a 3D space is in itself a bad thing, merely that one should not try to ‘walk’ avatars about a large space with separate rooms as in REMOTE. A further consequence of the design was that users found it hard to locate the avatars of users with whom they wanted to interact. A ‘plan view’ was suggested as a result, and this together with some colour-coding would support easier location. A single-room design would fundamentally reduce the problem, but might then meet over-crowding problems in a command post with many more participants. Centering interaction on avatars rather than information sources might produce a single-room design with ‘team areas’ or stations for particular roles to interact – an intelligence or a logistics desk for example. The key issue for these users is visualising the state of the task; one user commented “the environment doesn't change, it's the task that changes”. This is not entirely true of course – the changing element in the environment is however the avatars themselves, so that their identity and activity is what the environment really ought to communicate. However supporting the visualisation of maps, the key information source for military planning, would be a better use of 3D for these users: “Really good (virtual) 3D mapping would be excellent. Especially if you are able to work with and annotate the maps.” The identification mechanism adopted of labels over avatars’ heads was not completely successful. Such legends may be hard to read – as can be seen in Figures 1, 3 and 6 - and at a sensible size take up too much space. Texturing avatar faces with those of the users themselves ought to be tried –not possible in this study because the identity of the individuals participating was known only a short time before the study took place.

However individualised faces along with further clothing cues might have been attempted. As with navigation, it is hard to think of a currently viable mechanism to allow a user to invoke complex expressive features in an avatar without imposing a high degree of effort. The positive response of the users to web cam data suggests that compositing a video head-and shoulders onto the avatar’s face is worth considering in spite of the technical difficulties, at least when voice interaction between users takes place. User responses underlined the importance of voice interaction in collaboration, and the importance of multi-voice conferencing. In spite of the use of a LAN there were still problems with audio quality in the experiment, and this has been confirmed by a number of earlier studies [Tang et al 94; Olson and Teasley 96; Finholt and Olson 97; Covi et al 98]. The voice facilities of GROOVE were criticized, and though NetMeeting was seen as better in quality, participants would have preferred to use the phone. Voice-over-IP is still less reliable and of a much lower quality than telephony. Attempts to integrate NetMeeting into REMOTE along with the other applications were not successful, so that it had in fact to be run in parallel outside the environment instead of being triggered when two avatars met. Even then, the restriction to one-to-one conversation was far too constricting. 6. Conclusions Running studies such as this one and learning from the results seems fundamental to the development of CVEs into a successful and useful technology. Collaborative activity is a particularly difficult area to support with technology, and this is all the more so for tightly-coupled tasks with a high degree of inter-dependence such as in military planning [Olson and Olson 00]. A deep understanding of the user requirements and how these can

be supported with the technology as it now is cannot be achieved without such experimental work. In this case, a desktop VE was poorly received, but as we have tried to indicate in this final section, this seems more a case of iterating the design concepts than abandoning the approach. There is still a need to assess more explicitly the impact of these technologies on task performance and conduct to determine whether they provide the ability to enable the task to be done more quickly and more efficiently. References Adobe (2002). Adobe Atmosphere, http://www.adobe.com/products/atmosphere/main.html. Argyle, M. (1988) Bodily Communication , New York: Methuen and Co., 1988. Benford, S. D., Bowers, J. M., Fahlen, L. E., Greenhalgh, C. M., and Snowdon, D. N. (1997) Embodiments, Avatars, Clones and Agents for Multi-user, Multi-sensory Virtual Worlds, Multimedia Systems, 5, 2, 1997, 93-104. Benford, S., Greenhalgh, C., Rodden, T. and Pycock, J. (2001) Collaborative Virtual Environments, Communications of the ACM, Volume 44 , Issue 7 pp79 – 85, July 2001 Bradner, E. and Mark, G. (2002). Why Distance Matters: Effects on Cooperation, Persuasion and Deception. In proceedings of Computer Supported Cooperative Work (CSCW 2002). November 16-20, New Orleans, Louisiana Cassell, J., Nakano, Y., Bickmore, T., Sidner, C., and Rich, C. (2001). “Non-Verbal Cues for Discourse Structure.” Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics, pp. 106-115. July 17-19, Toulouse, France Churchill, E and Snowdon, D (1998) Collaborative Virtual Environments: an introductory review of issues and systems, in Virtual Reality: Research, Development and Application, 1998 Clarke, H; Goillau, P and Stockdale, R. (2003) Distributed Teamworking Experiment, QinetiQ May 2003, Malvern UK

Covi, Lisa M., Olson, Judith S., Rocco, Elena, Miller, William J., and Allie, Paul. (1998) A Room of Your Own: What do we learn about support of teamwork from assessing teams in dedicated project rooms. In Cooperative Buildings: Integrating Information, Organization and Architecture: First International Workshop, Co'Build '98. Darmstadt, Germany. CUSeeMe-Networks (2001). CUSeeMe, http://ic2.cuseeme.com/. DISCREET : http://www.discreet.com/support/max/ Eberly, D. H. (2001). 3D Game Engine Design, San Francisco: Morgan Kauffman Publishers. Endsley. M.R. (1995) Toward a theory of situation awareness in dynamic systems. Human Factors ,37 (1), 32-64. Epic-Games (1999). Unreal Tournament, Infogrames. Frécon, E. and Stenius, M. (1998). DIVE: A scaleable network architecture for distributed virtual environments, Distributed Systems Engineering Journal (DSEJ), 5, pp 91-100, Special Issue on Distributed Virtual Environments. Finholt, T. A., and Olson, G. M. (1997) From laboratories to collaboratories: A new organizational form for scientific collaboration. Psychological Science. 8(1), 28-35 Gamma, E; Helm, R., Johnson, R and Vlissides, J. (1996) Design Patterns, Addison-Wesley Longman, Reading Massachusetts. Greenhalgh, C., Pubrick, J., and Snowdon, D. (2000). Inside MASSIVE-3: Flexible support for data consistency and world structuring. In Proceedings of the 3rd International Conference on Collaborative Virtual Environments, (CVE ‘2000), San Francisco, California, USA, ACM Press. Groove-Networks (2002). Groove Workspace, http://www.groove.net. Hughes,J; King,V; Rodden, T and Anderson,H. (1995) The Role of Ethnography in Interactive Systems Design, ACM interactions , v.2 n.2, p.56-65, April 1995

Kaminka, G. A.; Veloso, M.; Schaffer, S.; Sollitto, C.; Adobbati, R.; Marshal, Andrew N.; Scholer, Andrew, S.; and Tejada, S. (2002). GameBots: the ever-challenging multi-agent research test-bed, In Communications of the ACM, January issue. Myers, M.D. and Avison, D.E. (eds.). (2002) Qualitative Research in Information Systems: A Reader, Sage Publications, London, 2002. Olson, J. S., and Teasley, S. (1996) Groupware in the wild: Lessons learned from a year of virtual collocation. Proceedings of the Conference on Computer Supported Cooperative Work. New York: ACM. pp-419-427 Olson, G.M., and Olson, J.S. (2000). Distance matters. Human Computer Interaction, 15, 139-179. Pandzic, I; Lee, E; Magnenat Thalmann, N; Capin, T and Thalmann, D. (1997) A flexible architecture for Virtual Humans in Networked Collaborative Virtual Environments Computer Graphics Forum Vol 16, Issue 3 Conference Issue (September 1997) Sechrest, L., Stewart, M., Stickle, T., and Sidani, S. (1996). Effective and Persuasive Case Studies. Cambridge, MA: Human Services Research Institute. Shneiderman, B. (1992). Designing the user interface: Strategies for effective human-computer interaction, 2nd edn, Reading, MA: Addison-Wesley Singhal, S., and Zyda, M. (1999) Networked Virtual Environments, New York:ACM Press, Siggraph Series. Steed, A and Tromp, J. (1997) "Experiences with the Evaluation of CVE Applications", in Proceedings of Collaborative Virtual Environments 1998, 17-19th June 1998, Manchester, UK Stephenson, N.(1993) Snowcrash. Spectra Books , 1993. Tang, J. C., Isaacs, E. A., and Rua, M. (1994) Supporting distributed groups with a Montage of lightweight interactions. Proceeding of the ACM Conference on Computer Supported Cooperative Work (CSCW94), Pp. 23-34

Taylor, R. M. (1990) Situational awareness rating technique (SART): The development of a tool for aircrew systems design. In Proceedings of the Situational Awareness in Aerospace Operations, AGARD CP-478, Paper 3. Wierba, E.E., Finholt, T.A., and Steves, M. (2001). "What have you done for me lately?" A case study of barriers to collaborative tool adoption in a manufacturing engineering setting (PDF file). At: http://www.crew.umich.edu/technical_reports.htm Wildtangent (2002). Wildtangent Web Driver, http://www.wildtangent.com

Appendix 1 – The Questionnaires A1.1 the team-working questionnaire QUESTION NUMBER 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Questions In the co-located team, it was possible for team members to have a similar understanding of the situation. In the distributed team, it was possible for team members to have a similar understanding of the situation. When the team was co-located, technology hindered the team members’ understanding of the situation. When the team was physically located together, it was possible to have an understanding of the overarching team goal. In the distributed team, it was difficult to maintain an understanding of where the team fitted into the bigger picture. It was easier to have an awareness of other team member’s roles and responsibilities in the co-located, rather than the distributed, team In the co-located team, it was easy to monitor and assess the progress of fellow team members. In the co-located team, it was easy to ask for guidance from the Chief of Staff. In the co-located team, it was difficult to notice and prevent team problems. It was difficult to notice and prevent team problems, when working in a distributed team. It was more difficult to resolve conflicts when team members were distributed. It was difficult to know when to provide constructive feedback to team members who were not in the same room. It was easy to identify when co-located team members required assistance or support. It was easy to ask for support when the team was co-located. Feedback was more discrete and less formal when given face to face. In the distributed team, it was difficult to know when other team members were having task-focused problems. It was easier to develop trust between members in the same room than team members that were distributed. When separated from the rest of the team, it was possible to experience feelings of isolation.

19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

39 40 41 42 43 44 45 46

of isolation. It was easy to develop and maintain interpersonal relationships between colocated members of the team. The technology hindered the development and maintenance of interpersonal relationships. Unfamiliarity within the distributed team, prevented team members from asking for support. It was easy to build and maintain a good team atmosphere in the co-located team. In the co-located team there was a high level of satisfaction. The technology hindered the overall level of satisfaction in the co-located team. In the distributed team there was a high level of satisfaction. The technology hindered the overall level of satisfaction in the distributed team. In the co-located team there was a high level of motivation. The technology hindered the overall level of motivation in the co-located team. In the distributed team there was a high level of motivation. The technology hindered the overall level of motivation in the distributed team. In the co-located team, there was high level of cohesion. The technology hindered the development and maintenance of cohesion in the co-located team. In the distributed team, there was a high level of cohesion. The technology hindered the development and maintenance of cohesion in the distributed team. It was difficult to interact and communicate with co-located team members. When team members were not co-located, additional co-ordination and communication demands arose. It was difficult to interact simultaneously with co-located and distributed team members. It was difficult to conduct brainstorming sessions with co-located team members whilst maintaining awareness of the activities of distributed team members. It was difficult to maintain eye contact with co-located team members. In the co-located team, it was difficult to manage team members’ tasks and coordinate a unified effort. When a message was not communicated verbally, face-to-face, it was difficult to understand whether a team member had understood the message. Requesting information was easier when face-to-face. In the distributed team, it was difficult to know how and when to update others with information. Lack of face to face communication increased the number of misunderstandings that occurred. Communicating over electronic media was not as effective as face to face communication. The physical layout of the work environment inhibited non-verbal communication between co-located team members.

A1.2 The Requirements Questionnaire QUESTION Questions NUMBER 1 The software did not constrain team working between members of the co-located team.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Co-located team members felt forced to stay at their workstations to undertake their tasks. The software enabled me to communicate with other team members whenever I needed to. It was useful having a choice of ways to contact other team members (e.g. by using text or voice). The software made it easy to determine whether team members were currently available for communication. The software enabled me to communicate effectively with others whilst maintaining awareness of current information. The software made it easy to ensure that all team members had a similar understanding of the situation. The software enabled communications to be brief and concise. A high number of misunderstandings and communication breakdowns occurred when using the software. The application enabled me to ‘Reachback’ (i.e. to locate staff functions outside of the traditional area of operations) to the distributed team location. The software enabled me to rapidly exchange key information with another individual on a one-to-one basis. The software enabled me to effectively exchange information with several other team members at the same time. The software enabled information to be produced and circulated, using common formats with visual or graphic representations where appropriate. The software made it easy to check that information was received and understood as intended. The software enabled all team members to interact with one another as required. The means of electronically meeting other team members and interacting with them was appropriate, and natural Ad-hoc, spontaneous interactions with other team members could take place electronically. The software effectively supported social interaction (e.g. chatting or interacting separate to the work task) with another individual on a one-to-one basis. The software effectively supported social interaction with several other team members at the same time. The software was better at helping team members to exchange information and facts relevant to the task than in helping them get to know each other as colleagues. The software enabled all members of the team to undertake their roles and responsibilities. The software helped the team to synchronise their activities. The software made it easy to keep track of each other’s activities. It was easy to switch between individual tasks and team activities using the software. The software made it easy to manage the team’s workload effectively. It was easy to maintain awareness of all distributed team members through the use of the software. The software made it easy to monitor distributed team members actions. The software made it easy to know when to provide feedback to team members who were not in the same room. The software made it easy to identify when distributed members of the team required assistance and support. The software made it easy to identify and deal with problems within the distributed team.

31

The application allowed me to ‘Virtually Co-locate’ with team members in the other room i.e. to interact with distributed team members as if physically co-located with them.

32

I could as easily electronically ‘meet’ the team members located in the other room as I could meet my co-located colleagues in the same room. Overall, I experienced a sense of ‘shared meeting space’ with other team members when using the software. Overall, I felt that the software supported me in feeling present in the distributed team. The software enabled me to feel a member of the distributed team. The software enabled me to feel familiar with my distributed team members.

33 34 35 36 37 38 39 40 41

The software aided the creation of a continuing sense of common team purpose. The software made it easy to build and maintain a good team atmosphere. The software enabled the distributed team to create and maintain an atmosphere of trust. The software is a useful ‘proof of concept’. The software should be developed further for military use.

A1.3 The Technology Questionnaire QUESTION Questions NUMBER 1 The training I received on Groove was adequate to conduct the experiment. (Condition 1 only) 2 Overall, I found the software easy to use. 3 I liked the software and enjoyed using it. 4 The software helped me to do teamworking in an efficient and timely manner. 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

It was straightforward to learn and become familiar with the software. I had to frequently ask the advice of technical staff on how to use the system. The system behaved in a consistent way enabling me to understand how it worked. The system enabled easy access to common documents. The system enabled remote briefings to be conducted. System functions were well integrated. Few actions were required to perform functions. The interface enabled me to efficiently access information. Contacting individual users was quick and simple. I used the voice chat facility frequently The voice chat facility was easy to use. I felt that the voice chat facility was effective. I used the text chat facility frequently. The text chat facility was easy to use. I felt that the text chat facility was effective. I used the discussion tool frequently. The discussion tool was easy to use. I felt that the discussion tool was effective. I used the sketchpad tool frequently. The sketchpad tool was easy to use. I felt that the sketchpad tool was effective. I used the instant messaging tool frequently.

27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

The instant messaging tool was easy to use. I felt that the instant messaging tool was effective. I used the electronic filing system frequently. The electronic filing system was easy to use. I felt that the electronic filing system was effective. I used the contacts tool frequently. The contacts tool was easy to use. I felt that the contacts tool was effective. I used the notepad tool frequently. The notepad tool was easy to use. I felt that the notepad tool was effective. The training I received on CUSeeMe was adequate to conduct the experiment. (Condition 2 only). I used the video conferencing facility frequently (Condition 2 only). The video conferencing facility was easy to use (Condition 2 only). I felt that the video conferencing facility was effective Condition 2 only). The training I received on REMOTE was adequate to conduct the experiment. (Condition 3 only). The virtual environment was natural and intuitive. (Condition 3 only). The avatars were effective for interacting with distributed members of the team. Condition 3 only). The avatars were effective for interacting with objects in the virtual environment (e.g. filing cabinets and post box). (Condition 3 only).

Appendix 2: The debrief prompt sheet

EXP3 - VCL debrief sheet

(PJG 21_2_03)

Debrief sessions will occur after electronically completing the questionnaires for each experimental condition. This sheet is intended as an ‘aide memoire’ to structure the main themes to be covered in the debriefs. Five ‘open’ style interview questions are given. Questions with brief ‘Yes / No’ answers are to be avoided. These are not intended to be prescriptive. Priority should be given to asking about the issues and problems that will have surfaced during the experimental condition. Finally, a number of non-directive questioning and ‘prompt’ techniques have been included as suggested ways to keep the debrief discussions flowing.

1. Debrief Topics for exploration ALL CONDITIONS (1, 2 and 3) Q1. Were there any particular issues, problems or successes that emerged during the last condition? (e.g. concerning the scenario, SOP’s, EXCON, software / technologies, teamworking issues – local co-located team vs. distributed team?) (Can you say more? What did you particularly like / dislike / feel was missing during the condition?)

Q2. Did the software help you in carrying out your task activities? (In what way ? / Can you say more? / Can you think of specific examples?) Q3. Did the software help you in interacting with and getting to know other team members? (In what way? / Can you say more? / Can you think of specific examples?) Suggestion - probe specific software applications and features: • Groove (voice chat, text chat, document tool, sketch pad tool, instant messaging facility, electronic filing system, contacts tool, notepad tool) CONDITION 2 ONLY (CU-SEE ME VIDEO CONFERENCING) Q4a. Did the CU-SeeMe video conferencing enable you to accomplish objectives that you couldn’t achieve using Groove alone? (What were they? / Can you think of specific examples?) Suggestion - probe specific software applications and features: • CUseeMe desktop video conferencing (1:1, 1: many) CONDITION 3 ONLY (REMOTE VIRTUAL ENVIRONMENT) Q4b. Did the REMOTE virtual environment enable you to accomplish objectives that you couldn’t achieve using Groove alone? (What were they? / Can you think of specific examples?) Q5. How did REMOTE with its virtual environment and avatars compare with the previous CU-SeeMe video conferencing condition? (What were the relative strengths and weaknesses / advantages and disadvantages?) Suggestion -probe specific software applications and features: • REMOTE (overall virtual environment, avatars – presence / location / state, avatar-avatar trigger of chat, avatar-object trigger of Groove tools) ALL CONDITIONS (1, 2 and 3) Q6. Is there anything else we forgot to ask, that you’d like to mention? (Catch all – last chance to elicit any remaining issues).

2. Example ‘open’ style guidance questions for ‘probing’ topics 1. Concerning X, what were your thoughts / overall impressions? 2. What issues became evident with X? 3. What did you think of the specific feature / aspect of X? 4. What were the strengths / what worked well / what did you like? 5. What were the weaknesses / what didn’t work well / what did you dislike? 6. What was missing / what additions were needed / what would you like to see added in the future? 7. Other specifics (what / when / who / where / how? ) 3. Prompts – to keep the discussion going I see. That is interesting. Can you say more? Did that always occur? Can you give me an example, a ‘for instance’, when that occurred? In what way was that a

problem for you? Broadening to everyone in the team, did you all think that was an issue for the team? Etc etc.

Appendix 3 – The Human factors ‘crib sheet’ using during observation

Appendix 4: Observer data analysis coding table Abbrev usr_str

Definition User identified strengths

usr_weak

User identified weaknesses

usr_ftr

User identified areas for future development

usr_cmnt

User comments

app/funct

Functionality of the underlying applications

co-ord

Team Co-ordination and Interaction.

Tm_clim

Team Climate

ad_be

Adaptive Behaviours

mon/fed

Performance Monitoring and Feedback

under

Understanding

comm

Communication

proc/mow

Planning process issues or methods of working

lead

Leadership

pos

Productivity of Staff

info_shr

Information Sharing / Management

hci

Human Computer Interface Design Issues

pres

Sense of Shared Presence and Awareness of Others.

train

Training Issues

ergo/phys

Ergonomics or Physical Issues

tech

Other Techical Issues

ease_use

Ease of Use

scenario

Scenario Issues

crash

System Crashes

z_question

Observers’ Questions.