Web designers and web users: Influence of the ... - Semantic Scholar

ARTICLE IN PRESS

Int. J. Human-Computer Studies 64 (2006) 1031–1048 www.elsevier.com/locate/ijhcs

Web designers and web users: Influence of the ergonomic quality of the web site on the information search Aline Chevalier, Maud Kicka Cognitive Processes and Interactive Conducts Laboratory (EA 1588), Department of Cognitive Psychology, University of Paris X, 200 avenue de la Re´publique, 92001 Nanterre cedex, France Received 11 February 2005; received in revised form 29 April 2006; accepted 1 June 2006 Communicated by S. Wiedenbeck Available online 28 July 2006

Abstract Despite rapid growth in the number of web sites, there is still a significant number of ergonomic problems which hinder web users. Many studies focus on analysing cognitive processes and difficulties experienced by web users, but very few are interested in web designers’ difficulties or in comparing their respective activities. Towards this end, the two experimental studies presented in this article compare the strategies developed both by professional web designers and (novice vs. experienced) web users while searching for information on web sites of varying ergonomic quality. More precisely, we investigated whether web designers can effectively use their own strategies as web users when designing web sites. We presented a comparison of novice web users, experienced web users and professional web designers searching behavior and cognitive load when using ergonomic and non-ergonomic web sites. In addition, we asked web designers to predict the strategy used by novice web users. Based on the results obtained in the two experiments, we conclude that web designers are not able to predict strategies of novice users and do not behave like novice users. Consequently, ways for supporting web designers in developing a user-centered activity are necessary, and certain ways are suggested at the end of this article. r 2006 Elsevier Ltd. All rights reserved. Keywords: Web designers; Web users; Information search; Cognitive load; Ergonomics; Experience

1. Introduction Koenemann and Belkin (1996) have signaled a dramatic increase in the number of information sources that have become available to an exponentially growing number of users with very little training in searching for information in such systems. Pirolli and Card (1999) have underlined the fact that the quantity of web pages doubles every year. In more recent years, this trend has continued to increase, and e-commerce sites are contributing to in this rapid growth (Forrester, 2002; Richard and Chandra, 2005; Wang and Emurian, 2005). The facility with which a large public can be reached in a short time partially explains this interest in web sites. Nevertheless, they are still considered as difficult to use and to access (Lee, 1999; Mc Crickard, Corresponding author. Tel.: +33 1 40 97 75 28; fax: +33140977158.

E-mail addresses: [email protected] (A. Chevalier), [email protected] (M. Kicka). 1071-5819/$ - see front matter r 2006 Elsevier Ltd. All rights reserved. doi:10.1016/j.ijhcs.2006.06.002

2001; Teo et al., 2003; Bhatt, 2004; Ling and van Schaik, 2006; Stronge et al., in press). Given this rapid growth in number, web sites are constantly in competition. Consequently, they have to be precise, concise and rapid, since web users, having more choice than ever, will not waste time on web sites which are confusing, too slow and not well adapted to users’ needs (Helander and Khalid, 2000; Wang and Emurian, 2005). Therefore, the implementation of an organized and intuitive navigational system is critical to user success, especially in web navigation, because users see individual sites as subsets within their overall Internet experience, and users frequently are unaware they have left a specific site (Spool et al., 1999; Nielsen, 2000; McEneaney, 2001). In order to determine precisely the problems and cognitive difficulties that web users experience, more and more researchers study the cognitive processes involved and difficulties experienced by users when navigating the Web (see, e.g. Rouet and Tricot, 1998; Navarro-Prieto

ARTICLE IN PRESS 1032

A. Chevalier, M. Kicka / Int. J. Human-Computer Studies 64 (2006) 1031–1048

et al., 1999; Pearson and van Schaik, 2003; Rouet, 2003; Pratt et al., 2004). Findings about users’ cognitive functioning have ergonomic implications for designers (see, e.g., Hong et al., 2004; Oulasvirta, 2004) and for the creation of guidelines and checklists (see, e.g., Bastien et al., 1999; Spool et al., 1999; van Duyne et al., 2002; Ozok and Salvendy, 2004). Nevertheless, we also assert that understanding designers’ activities and identifying the difficulties that designers experience are essential to improving the ergonomic quality of web sites. More precisely, research on users’ and designers’ cognitive activities must be conducted jointly to help designers concretely and efficiently consider future users’ needs when designing web sites. In this paper, we assert that determining differences between designers and users while navigating will clarify the reasons why designers cannot sufficiently take into account users’ needs when creating web sites. The question of designers’ and users’ differences have been largely ignored, except one recent article by Park et al. (2004) has examined users’ esthetical preferences. In order to address the research needs in this area, we present in this article two experiments conducted with novice web users, experienced web users and professional web designers involved in information search tasks. These experiments aim to compare the search strategies and cognitive resources that (novice vs. experienced) users and professional designers employ while searching for information according to the ergonomic quality of the site (a site with ergonomic problems vs. a site incorporating ergonomic recommendations). The following section provides an overview of information search activity and cognitive load. Sections 3 and 4 present the two experimental studies. The results obtained are discussed in Section 5. Suggestions to help web designers’ activities are presented in Section 6. 2. Searching for information in e-documents: processes and cognitive load The first models of information searching described this activity as cyclical, i.e. the individual defines a (cognitive) goal, selects an information category, extracts information and integrates it into previous extracted information; the individual begins this cycle over and over again until s/he reaches her/his search goal (Guthrie and Mosenthal, 1987; Guthrie, 1988; Dreher, 1992; Armbruster and Armstrong, 1993). These models do not explain why users fail when searching for information. Rouet and Tricot (1996, 1998) defined a more complete cognitive model that includes different factors involved in this cognitive activity, such as the degree of precision of the user’s objective (vague vs. precise), the extraction of unique or various sources of information and the experience of users. This model is close to those used for searching in electronic information systems proposed by Marchionini et al. (1993) and Shneiderman et al. (1998). There is one major difference however: The latter models did not consider, for instance,

the specific differences between the World Wide Web and bibliographical database systems. The model developed by Rouet and Tricot proposes an information search activity which is both cyclical (like Guthrie’s model, 1988), and similar to text comprehension, problem-solving and decision-making activities. More precisely, their model involved three phases: (1) The evaluation phase consists in determining whether, at an instance in time, available information is relevant regarding the goal to be reached. Three situations are possible: information perfectly fits the individual’s goal representation, so the search activity is over. The information is partially relevant to the search task; the individual has to evaluate the gap between her/his goal and selected information, and, if required, has to begin a new cycle by defining new criteria. Third, the information is not quite relevant to the goal representation; the individual has to revise or reevaluate the search information task and to begin a new cycle. Consequently, if necessary, the individual changes her/ his goal representation in order to make it more accurate with precise information to be found. (2) In the selection phase, the individual has both to maintain the goal representation, previously selected information, as well as information currently in working memory. This phase allows the individual to evaluate an information set which is considered to be the most accurate of all of the available information. (3) During the processing phase, the individual tries to understand the information set by taking into account its particularities (such as verbal and visual information, animations or sounds). Furthermore, according to Rouet and Tricot (1996, 1998), three processes supervise and manage information search activity: planning, control and regulation. During the planning process, the individual determines a strategy to reach relevant information. The elaborated plan may be coherent and stable or ineffective, according to whether the individual has relevant knowledge for finding information or not. The control process allows the individual to evaluate the relevance of the information found during the selection phase. Finally, the regulation process allows the individual to make decisions during the information search, and if the result is not satisfactory, to modify her/his goal representation or different strategies in order to improve her/his information search activity. Consequently, we can assert that searching for information is a particularly complex cognitive activity that involves many cognitive resources. The cognitive load of individuals may vary according to the experience and the presentation of information, as described below. Cognitive load is usually evaluated according to the quantity of information to be memorized and the amount of processes involved to perform the task (Hoc, 1988; Paas and van Merrie¨nboer, 1994). Two aspects are

ARTICLE IN PRESS A. Chevalier, M. Kicka / Int. J. Human-Computer Studies 64 (2006) 1031–1048

preponderant: the first aspect refers to information from the task to be performed, and the second is related to individual characteristics (e.g., experience, motivation, etc.). These two aspects are also important for the cognitive load theory defined by Sweller (1988). This theory initially is concerned with the development of instructional methods for learning systems. Cognitive load theory is also used as a theoretical framework to explain difficulties that people may experience while searching for information in e-documents (Rouet and Tricot, 1996, 1998; Tricot, 2003). This theory underscores the importance of the limited cognitive capacities of the working memory (according to Baddeley’s memory model, 1986; see Sweller, 1988, 1999; Kalyuga et al., 2003; van Merrie¨nboer and Sweller, 2005). When the individual has to understand information (text, image, animation, sound), this information is first processed in working memory before being stored in long-term memory. More precisely, Sweller (1988, 1999) distinguished: 1. Intrinsic cognitive load which refers to the task at hand, i.e. it depends on the difficulty of the content to be learned and on the amount of information that the individual has to simultaneously process in working memory. Intrinsic cognitive load decreases as knowledge in long-term memory increases. Consequently, a high intrinsic cognitive load corresponds either to a highly complex material or to an individual’s low expertise. 2. Extraneous cognitive load which concerns instructional materials used to present information. For instance, teaching materials addressing a concept (e.g. continental drift) will be more effective if they present appropriate graphics instead of more text presentation. More precisely, extraneous cognitive load is divided into two components: Irrelevant extraneous cognitive load corresponds to the cognitive resources involved in processing information but not essential to learning or understanding. It is often linked to a poorly adapted information presentation. Relevant extraneous cognitive load corresponds to the fact that an available processing space in working memory is used for the enrichment of mental schemata or for their automation. Relevant extrinsic cognitive load is dependent on information presentation, which encourages the individual to focus her/his attention on the relevant information regarding the task to be performed. Intrinsic and extraneous cognitive loads are additive to create a total cognitive load (van Merrie¨nboer and Sweller, 2005). Consequently, cognitive overload is influenced by users’ experience in navigation (see, e.g., Yantis, 1993; NavarroPrieto et al., 1999; Ihadjadene et al., 2003; Pratt et al., 2004; Tabatabai and Shore, 2005) and, according to Sweller’s

1033

theory, we hypothesize that the ergonomic quality of presentation of information in web sites will also influence the search for information and increase the cognitive load of individuals. In order to test this hypothesis we conducted two experimental studies. 3. Experiment 1 3.1. Research problem and objectives Studies show that designers have tried to understand and consider users’ needs, but that they have experienced important difficulties (Card et al., 1983; Olson and Olson, 2003). In web site design, these results may seem surprising, as one particularity of web site design is that designers are also web users, which is not the case in most design situations (e.g., design of aerospace products). Nevertheless, previous studies have shown that web sites do not fit the users’ needs (Chevalier and Ivory, 2003; Chevalier, 2004). So, how can we explain why designers, who are also web users, develop sites which are difficult to use? Among the hypotheses suggested, we argue that even if designers are also web users, they are not novices; on the contrary, with practice, they have become expert users. Thus, designers have acquired procedures related to navigation on the Web that they automatically apply without requiring many cognitive resources; this is not the case for users, particularly for novice users who occasionally navigate the Web. Consequently, do professional designers experience the same difficulties as users (in particular novice users) when searching for information? Do designers develop an information search activity in the same way as novice or experienced users? We hypothesize that professional designers will experience few difficulties and will have a lower cognitive load than users (particularly novice users), especially with the non-ergonomic web site. To provide answers to these questions and to test these hypotheses, two experimental studies are presented in this paper. The first experiment aims at determining whether ergonomic problems identified by designers and users in an exploration task actually hinder their information search activities and thus lead them to develop different search strategies. Towards this end, two versions of the same web site were created: one site had ergonomic problems whereas the other site was in line with ergonomic recommendations. More precisely, the first experiment aims at examining the following aspects according to the web site version and the participant’s experience (novice and experienced users vs. professional designers):

The time necessary to find information. The number of steps (i.e. the number of hyperlinks visited by the participants) required to find information. The amount of cognitive resources involved in finding information. The participants’ usability satisfaction with regard to the visited site.



3.2. Hypotheses Hypotheses were grouped according to the two experimental factors: experience of the participants and the ergonomic quality of the web site. Hypothesis 1. Effect of the experience. Experts are different from novices because they automatically activate accurate knowledge stored in their longterm memory (or mental schemata, see Barlett, 1932; Rumelhart, 1978; Sweller, 1988) to perform the task at hand. Consequently, the time necessary to find information, the number of visited hyperlinks and the amount of cognitive load should decrease as participants’ experience increases. In addition, because novice users do not navigate the web as regularly as experienced users and professional designers, novice users should be less critical than others. Thus, usability level of satisfaction should decrease as experience increases. Hypothesis 2. Effect of the ergonomic quality of the site. According to Rouet and Tricot (1996, 1998), the individual has to maintain an initial goal in working memory, along with search criteria and information acquired during the search activity. When information presentation increasingly adapted to the users’ cognitive capacities, the extraneous cognitive load is reduced (van Merrie¨nboer and Sweller, 2005). Therefore, the time required and the number of visited hyperlinks to find information as well as the amount of cognitive load should be lower when participants perform the information search task on the ergonomic site than on the non-ergonomic site. Recall that the ergonomic site was created to fit users’ cognitive needs. Thus, participants navigating the ergonomic site should be more satisfied than participants navigating the non-ergonomic site. 3.3. Study participants Forty web users (students at the University of Paris X) and sixteen web designers participated in this study:

Twenty experienced web users: Following Ho¨lscher and Strube (2000), a user was considered as experienced when s/he used the Web at least for 1 h/day (this criterion could be compensated if s/he indicated using the Web several times a day for at least a quarter of an hour per connection). All of the experienced users had from 3 to 4 years of intensive experience with the Internet and used all of the functionalities (mailbox, various search tools, forums, etc.). Experienced users navigated the Internet with a high-speed link. Average age: 24-year old. Twenty novice web users, who used the Web occasionally to search for information (about once a month) and checked their e-mails. Most of them only had an Internet connection at their university. Average age: 23-year old.

Sixteen professional web designers, who had created web sites within companies for about 6 years. These designers were specialized in designing e-commerce sites. They used the Internet every day for work and for pleasure. Average age: 32-year old.

The participants used either the ergonomic web site or the non-ergonomic site (see Table 1). 3.4. Experimental design 3.4.1. Cognitive load measurements In the domain of e-documents, various methods are used to measure cognitive load (for a review, see Paas et al., 2003): such methods include physiological techniques (e.g. measuring of electro-cortical activity), rating scale techniques (e.g. questionnaires), performance and the dual task technique (used since 1967 by Kalsbeek and Sykes). For the two experiments presented here, we chose the last method, i.e. the dual task technique. This technique is an on-line method used to measure cognitive resources which are still available in the individual’s working memory to perform a task at an instant in time. The idea underlying the dual task paradigm is that residual capacity (i.e. capacity not used to perform the main task) can be used to perform a secondary task. While the individual is performing a task, s/he has to react as fast as possible to visual or auditory signals (mostly auditory signals), often by clicking with a mouse. The time necessary to answer auditory signals reflects the amount of cognitive resources allocated to the main task. This technique has been used by many authors in various domains such as text writing or design, and it is used more and more today (see, e.g., Kellogg, 1987, 1994; Levy and Ransdell, 1995; Marcus et al., 1996; Sweller, 1999; van Gerven et al., 2002; Bonnardel and Piolat, 2003; Bru¨nken et al., 2003) because it is a reliable technique, and because it takes into account cognitive load fluctuations. Nevertheless, methodological concerns have been expressed about the dual task technique (see Fisk et al., 1986–1987); the main reservations center on the fact that auditory signals may hinder the individual’s activity. In text writing, writers admit that they are hindered by auditory interruptions (especially if the probes are frequent); however, the quality of their writing is not affected. Table 1 Participant assignment to the two site versions Participants

Site version

Total

Ergonomic web site

Non-ergonomic web site

Novice users Experienced users Professional designers

10 10 8

10 10 8

20 20 16

Total

28

28

56


This is true only if the writer must consider auditory signals as a secondary task (see Levy and Ransdell, 1995; Piolat et al., 2001). Writers succeed in performing a cognitive tradeoff in cognitive resources management and produce a text whose quality is preserved. On the other hand, if the dual task is considered as the main task, the individual’s activity is hindered (Kalsbeek and Sykes, 1967; Leplat, 1997; Richard, 1997; Spe´randio, 1988). Accordingly, the use of the dual task technique brings to light two important methodological points (applied in our two experiments): (1) The choice of the accurate probe. For the two experiments, a pilot study allowed us to identify the optimal probe interval to measure the majority of the participants’ actions (for our two studies, probes varied from 3 to 15 s). (2) It is important to inform participants of these signals and insist on the fact that the main task remains information search despite auditory signals. 3.4.2. Experimental task and data analysis The experiment was divided into the three following stages: Stage 1: Before starting the information search task, the participants were trained to respond to thirty auditory signals from Tholos software to determine their baseline reaction times (the description of Tholos is presented in Appendix A). Stage 2: The participants had to search individually for information in order to answer three questions, presented successively, by counterbalancing the order of the question presentation (see Appendix C for instructions and questions). These questions had only one correct answer. For each question, the participants had to start from the homepage. All of the participants used the same personal computer. The navigation activities of the participants (visited pages, changing pages, etc.) were recorded using Find-outs (Applied Logic Corporation). The time required to find the targeted information was calculated from the moment the participant saw the homepage to the moment s/he said s/he had found the information. While searching for the information, the participant had to react to auditory signals (from Tholos) by pressing a pedal with her/his foot (her/his hands remained free to use the computer). This made it possible to determine an average reaction time (in ms). The participant’s baseline reaction times measured during the training phase (first stage) was subtracted from the reaction times measured during the experimental task (information search task), thus providing ‘‘reaction time interference scores’’. As already noted, such scores allowed us to measure the participants’ cognitive resources: the greater the reaction time, the more cognitive resources were involved. The instructions and the experimental procedure are detailed in Appendix C.

1035

As indicated, two versions of the same web site were created: (1) The ergonomic site was composed of eighteen pages, which were hyperlinked in such a manner that participants could navigate the site. Eight graduate students in cognitive psychology and ergonomics at the University of Paris X evaluated it. These students used two methods: the cognitive walkthrough method (Blackmon et al., 2002) and an evaluation of conformity to the ergonomic recommendations as defined by Bastien et al. (1999) and Nielsen (2000). (2) The non-ergonomic site included the main ergonomic problems identified by designers and users in a previous study (see Chevalier, 2005). This site was also composed of 18 pages. The web pages were hyperlinked in such way that participants could navigate the site. Each page included the same number of ergonomic problems, i.e. 14 problems per page (see Figs. 1–3 for examples of the web pages, and Appendix B for the distribution of the ergonomic problems in the nonergonomic site). These two web sites presented the same e-shop selling music products (CDs, show tickets, etc.). We chose an e-commerce site selling music since those products are bought by many people on-line and do not require specific knowledge linked to the site content. To respect the ergonomic recommendations and to compare the search time between the two sites, two steps (or two hyperlinks) were necessary to find the three correct answers. These three questions allowed the participants using the non-ergonomic site to be confronted with all of the ergonomic errors at least once if they used the optimal path for finding answers. Stage 3: After searching for information, the participants had to freely navigate the site and then answer a usability satisfaction questionnaire without time limits (this questionnaire was inspired by the Web Site Analysis and MeasureMent Inventory developed by Kirakowski et al., 1998). Our questionnaire included seventeen affirmations (see Appendix D) about which the participants had to indicate the degree to which they agreed on a 5-point-scale; the more the participant evaluated the site as satisfactory, the closer to 5 the grade was. 3.5. Results Results are presented in the following order: search time and number of steps (Section 3.5.1.), cognitive load (Section 3.5.2) and usability satisfaction (Section 3.5.2). 3.5.1. Time required and number of steps for answering questions 3.5.1.1. Time for answering questions. All of the participants succeeded in finding the three correct answers.



Fig. 1–3. 1. The ergonomic site homepage. 2. The non-ergonomic site homepage. 3. Page dedicated to the catalog of Cds (the non-ergonomic site).

There was a significant effect of participants’ experience on the time needed to find answers (F(2,50) ¼ 13.756; po.0001; see Table 2): experienced users required less time than novice users (po.0001) and professional designers (po.0001). But, surprisingly, there was no significant difference between novice users and professional designers. These results are not exactly in accordance with Hypothesis 1. In accordance with Hypothesis 2, all of the participants who used the non-ergonomic site spent more time searching for information than the participants who used the ergonomic site (F(1,53) ¼ 37.684; po.0001; see Table 2). 3.5.1.2. Steps for answering questions. Recall that the optimal number of steps was the same for the two sites. In accordance with Hypothesis 1, the number of visited hyperlinks decreased as expertise increased (F(2,50) ¼ 24.385; po.0001): novice users made significantly more

steps than experienced users (po.005) and than designers (po.001). There was no significant difference between experienced users and professional designers (p4.1). The non-ergonomic site required a higher number of steps compared to the ergonomic site (F(1,50) ¼ 33.562; po.001; see Table 2), in accordance with Hypothesis 2. More precisely, there was a significant difference between novice users and experienced users depending on the site version (po.001 and po.01), but no significant difference appeared for professional designers.

3.5.2. Cognitive load Cognitive resources were measured using Tholos software (detailed in Appendix A). Hypothesis 1 was not verified since we noticed no significant difference in the cognitive resources used by the participants (F(2,50) ¼ .295; p4.1; see Table 3). No


1037

Table 2 Mean time (and standard deviation), in seconds, necessary and mean number (and standard deviation) of steps to answer each of the three questions according to the web site version and to participants’ experience Ergonomic site

Time to find information (s) Number of steps

Non-ergonomic site

Novice users

Experienced users

Professional designers

Novice users

Experienced users


9.15 (3.76)

4.09 (2.09)

12.5 (6.65)

22.86 (6.43)

14.8 (4.66)

21.15 (9.2)

3.9 (.46)

3.4 (1.1)

2.5 (.37)

6.6 (2.03)

5.2 (1.34)

3 (.86)

Table 3 Mean reaction time (and standard deviation), in milliseconds, necessary to answer each of the three questions according to the web site version and to participants’ experience Participants

Site version

Table 4 Mean notes (and standard deviation) on 5 points to the satisfaction questionnaire according to the site version and the participants’ experience Participants

Mean

Ergonomic web site

Non-ergonomic web site


78.53 (45.39) 87.33 (59.2) 54.48 (22.87)

45.77 (21.8) 52.07 (24.67) 66.92 (23.89)

Mean

74.78 (46.66)

54.06 (24.19)

62.15 (38.5) 69.7 (47.7) 60.67 (23.5)

significant difference between participants due to their levels of experience appeared. Contrary to Hypothesis 2, the ergonomic site required more cognitive resources than the non-ergonomic site, but only for the users (F(1,36) ¼ 6.09; po.05; see Table 3); no significant difference appeared for professional designers. 3.5.3. Usability satisfaction for the two web sites The questionnaire presented seventeen affirmations. Participants had to indicate for these affirmations the extent to which they agreed on a 5-point scale, ranging from (1) ‘‘I completely disagree’’ to (5) ‘‘I completely agree’’. The more satisfied a participant was, the closer to 5 the evaluation was (for the questions, see Appendix D). In accordance with Hypothesis 1, usability satisfaction decreased as expertise increased (F(2,50) ¼ 4.097; po.05; see Table 4): designers were less satisfied than experienced users (po.02) and than novice users (po.02), but there was no significant difference between novice and experienced users (p4.1). In accordance with Hypothesis 2, the ergonomic site obtained a significantly higher score than the nonergonomic site for all of the participants (F(1,50) ¼ 8.832; po.005; see Table 4). 3.6. Discussion Here, we discuss the findings that led us to carry out experiment 2. All of the main results will be discussed in the general discussion (Section 5).

Site version

Mean

Ergonomic web site

Nonergonomic web site


3.27 (.11) 3.26 (.18) 3.1 (.32)

3.1 (.3) 3.08 (.26) 2.79 (.43)

Mean

3.22 (.22)

3 (.34)

3.18 (.23) 3.17 (.24) 2.94 (.39)

First, a surprising and interesting result appeared for professional designers; they did not find the answers significantly faster than novice users, although designers visited fewer hyperlinks than novice users. To explain this result, we hypothesize that the professional designers will not use the web interface in the same way as novice users. In their professional activities, before developing a site, designers analyse existing web sites from the same field with a critical eye. In this experiment, professional designers may have thoroughly explored the opened web pages in addition to their information search activities. On the other hand, because of their lack of understanding, the novice users may have rapidly compromised themselves by using hyperlinks and opening new pages without examining opened web pages. This hypothesis is tested in experiment 2. Second, even though all of the users using the ergonomic site found information faster than users using the nonergonomic site (by clicking on fewer hyperlinks and by evaluating it as globally more satisfying), they used more cognitive resources while navigating the ergonomic site than the non-ergonomic site. This result is particularly surprising, since on the non-ergonomic site irrelevant elements were introduced which may have distracted users, and, for the less experienced among them, may have overloaded working memories (Sweller, 1988, 1999). In this study, users may not have been able to ignore and inhibit irrelevant elements present on the web pages of the non-ergonomic site. So, web users who used the non-ergonomic site, especially the novices, may have



experienced difficulties in successfully and quickly selecting relevant elements to answer questions. This may have generated difficulties both for planning their future actions, evaluating what hyperlink to open and more generally anticipating their activities (according to the Rouet and Tricot model). Thus, they may have clicked on more hyperlinks, spent more time searching and tried to find information through trial and error (strategy traditionally observed with novices in problem-solving, see Matlin, 1998; Chen et al., 2006). In the same way, distracting elements in web pages may have slowed down the information search (see Anderson et al., 1998); so users confronted with the non-ergonomic site may have experienced more difficulties focusing their attention on the task to be performed, and they may have used fewer cognitive resources than users experimenting the ergonomic site. This hypothesis is related to the Sweller model (1988, 1999), since the adapted presentation would allow and/or encourage the participant to focus attention on relevant information. This point is clarified in experiment 2 by determining more precisely whether irrelevant elements are actually difficult to inhibit for users, particularly for novice users. In experiment 2, we used part of Oulasvirta’s experimental procedure (2004) to evaluate the implicit memory of users and professional designers in order to determine if they try to understand all elements present on the site even if they are not relevant to their tasks. Third, professional designers did not grade the nonergonomic site as satisfactorily as users did; thus, it would seem that they were aware of the web site’s usability difficulties. Although they identified that the non-ergonomic web site was not easy to use, could they anticipate novice users’ activities and difficulties that novices might experience? Experiment 2 also aims at answering this question.

elements introduced into the non-ergonomic site, and more generally if participants build a coherent mental model of the site, even if they use the non-ergonomic site (for more details on this experimental procedure see Grey, 1990; Otter and Johnson, 2000; Oulasvirta, 2004). Moreover, we asked designers to anticipate the path that they thought novice users would follow to answer the same questions to which they found answers. This would enable us to determine the representation they have of novice users’ activities and to explain the difficulties they experience in creating web sites that fit users’ needs. Furthermore, we hypothesized that professional designers and novice users would not use the site in the same way: designers would develop two activities, analysing and searching for information, whereas novice users would experience difficulties making decisions to go on with their activities. In order to test this hypothesis, we analysed in this second experiment the parts of the web interface used by participants, though the movements of the mouse. More precisely, this second experiment aimed at determining the effects of the experience level of the participants and the ergonomic quality of the web site on:

4. Experiment 2

Hypothesis 1. Recalled elements. In experiment 1, designers were aware of difficulties related to poor ergonomic navigation (results of the usability satisfaction questionnaire). Nevertheless, when they have to create web sites they introduce ergonomic errors (see Chevalier and Ivory, 2003; Chevalier, 2004). In order to explain these results, after searching for information, all of the participants had to remember elements seen on visited web pages. Given their experience of web navigation, professional designers and experienced users should chunk elements together (elements sharing similar properties or linked semantically, see Miller, 1956; Chase and Simon, 1973); so they should recall more elements from the web pages than novice users. Moreover, if irrelevant elements are actually difficult to inhibit and keep users from focusing attention on relevant elements for their search activities, the participants visiting the nonergonomic site should remember more irrelevant elements than relevant ones compared to participants visiting the ergonomic site. In particular, novice users should experience more difficulties distinguishing relevant elements from

4.1. Objectives and hypotheses Based on the results obtained in experiment 1, three main questions were asked: how can it be explained that the non-ergonomic site required fewer cognitive resources than the ergonomic site? Why did professional designers significantly not take less time than novice users to find the correct answers, whereas professional designers opened up fewer hyperlinks than novice users? How can we explain that professional designers were less satisfied with the web sites (especially the non-ergonomic site) than users, whereas designers made ergonomic errors in their own web sites? To answer these questions, a second experiment was carried out wherein we tested the mental model of the visited web site built by professional designers and users. Towards this end, we instructed participants to remember page titles, content and hyperlinks seen in the visited web site. This procedure makes it possible to determine if users experience difficulties ignoring and inhibiting the irrelevant

The time, the number of steps required to find information and the cognitive resources involved to find information (as in experiment 1). The number, the nature and the relevance of the recalled elements seen on web site to find information. The movements of the participants’ mouse on the screen to find information. The anticipation by professional designers of the novice users’ activities in the search for information.

Based on these objectives, we formulated the three following hypotheses:


irrelevant elements than professional designers and experienced users. Hypothesis 2. Movements of the mouse. Professional designers would not use the web interface as novice users use it. Indeed, in their professional activities, before developing a site, designers analyse existing web sites from the same field. So, designers would develop two activities, i.e. analysing and searching, whereas novice users would rapidly compromise themselves by using hyperlinks and opening new pages. Thereby, designers should scan the whole web page for potential targets in order to build a coherent mental model of the site before clicking on a hyperlink; on the other hand, novice users should click on more hyperlinks without scanning the whole web page, and stop their activities to linearly read specific points (particularly texts) more often than professional designers and experienced users. Hypothesis 3. Anticipation of novice users’ paths by professional designers. With training, professional designers are become expert web users, so ergonomic problems introduced into the site would not be problematic for them, and they would not identify them on web sites. Consequently, they should experience difficulties anticipating and then describing the paths of novice users searching for information on the site. Moreover, designers should not be able to detach themselves from their own points of view, so they should describe the paths of novice users as being similar to their own (what van Duyne et al. called the ‘ego bias’—2002). 4.2. Study participants and procedure Twenty web users (students at the University of Paris X) and eight web designers participated in this study. The same criteria as in experiment 1 (see Section 3.2) were used to chose participants:

Ten experienced web users: Average age: 25-year old. Ten novice web users: Average age: 22-year old. Eight professional web designers: Average age: 29-year old.

Half of the participants were either confronted with the same ergonomic web site or with the same non-ergonomic site, as in experiment 1, and they used the same personal computer. The procedure was the same as in experiment 1 without free navigation. The second experiment utilized the incidental learning paradigm (see Oulasvirta, 2004): participants carried out navigational and content-related tasks not knowing that their memories were later tested. In this experiment, after answering the same three questions, participants had to answer a questionnaire about their level of experience in using the Internet (the same as in experiment 1) in order to distract them before

1039

recalling elements seen in the visited web pages. Then, the experimenter instructed participants to recall elements seen on previous web pages by writing them on paper. At the end, the professional designers orally had to indicate the path they thought that novice web users would follow to answer questions (procedure inspired by Park et al.’s work regarding users’ esthetic preferences—2004). The questions, the instructions and the experimental procedure are detailed in Appendix E. All of the mouse movements of the participants were recorded with a software of video capture and a camera in order to analyse them. In addition to the analyses carried out in experiment 1, we analysed movements of the mouse in web pages for all of the participants. This technique is often used to understand human movement, which is central to improving input devices and interaction techniques (Hwang et al., 2005). This technique provides accurate data about which region is being focused (see also Jansen et al., 2003). Moreover, Chen et al. (2003) showed a strong correlation between the movements of the mouse and the eye movements of participants navigating web sites. 4.3. Results Search time, number of steps and cognitive load are presented in Section 4.3.1. In Section 4.3.2., we present the elements recalled by participants (nature and number). Then, the movements of the mouse of the participants and the anticipation of the user’s path by designers are presented (Sections 4.3.3 and 4.3.4.). 4.3.1. Time, number of steps to find information and cognitive load All of the participants succeeded in finding the correct answers (as in experiment 1). The statistical analyses showed results which were very similar to those obtained in experiment 1. 4.3.1.1. Time for answering questions. As in experiment 1 and surprisingly, the time was different according to the level of experience (F(2,22) ¼ 5.431; po.02; see Table 5): novice users needed more time than experienced users (po.04) and professional designers (po.001). There was no significant difference between novice users and professional designers (p4.1). The ergonomic site required less time to find information than the non-ergonomic site for all participants (F(1,22) ¼ 37.5; po.0001; see Table 5). 4.3.1.2. Steps for answering questions. The novice users required more steps than experienced users (po.3) and designers (po.01), but no significant difference appeared between designers and experienced users (p4.1). The nonergonomic site required, on average, a higher number of steps than the ergonomic site (F(1,22) ¼ 18.555; po.0003; see Table 5). More precisely, the experienced users opened more hyperlinks while navigating the non-ergonomic site than the ergonomic site (po.04), but there was no


1040

Table 5 Mean (and standard deviation) of times, of steps and cognitive load (reaction times) necessary to answer each of the three questions according to the site version and the participants’ experience Ergonomic site

Time to find information (s) Number of steps Cognitive load (ms)

Non-ergonomic site

Novice users

Experienced users


Novice users

Experienced users


9.02 (3.25)

6.31 (1.22)

9.45 (2.28)

17.1 (5.35)

12.16 (3.51)

18.25 (1.39)

3.49 (.66) 98.45 (41.85)

2.8 (.84) 86.67 (11.66)

2.25 (.17) 77.05 (8.11)

5.24 (.69) 76.18 (22.58)

3.58 (.99) 55.87 (14.49)

3.08 (1.19) 64.29 (10.74)

Table 6 Means (and standard deviations) of recalled elements according to the site version and the participants’ experience Ergonomic site

Non-ergonomic site

Novice users

Experienced users

3.2 (1.3) 4.6 (2.88)

3.2 (.84) 2.8 (2.28)

3.75 (.96) 6.5 (7.89) 2.5 (2.89)

Recalled page titles Recalled web page content Recalled hyperlinks

—

.6 (.89)

Total of recalled elements

7.8 (2.68)

6 (3.65)


12.75 (8.77)

significant difference for novice users, or for professional designers. 4.3.1.3. Cognitive load. Recall that cognitive resources were measured using the dual task technique: as in experiment 1, when the participant is searching for information, s/he has to react as fast as possible to auditory signals (distributed by Tholos software, presented in Appendix A). The time (in ms) required to answer signals reflects the amount of cognitive resources allocated to the information search. Results did not show any significant difference linked to experience level (F(2,22) ¼ 1.716; p4.1). In contrast, we noted that the ergonomic site required, again for the users, a higher cognitive load than the non-ergonomic site (F(1,22) ¼ 6.683; po.02). 4.3.2. Elements recalled by participants The recalled elements were grouped into three categories (following Oulasvirta, 2004): recall of web page titles, recall of web page content (elements such as images, color, etc.) and recall of hyperlinks. 4.3.2.1. Total of recalled elements. In accordance with Hypothesis 1, the level of experience had a significant effect on the mean number of recalled elements (F(2,22) ¼ 6.704; po.006; see Table 6): novice users recalled fewer elements than designers (po.002) and experienced users (po.05), but no significant difference appeared between designers and experienced users (po.1). No significant difference appeared for the site (F(1,22) ¼ 2.2; p4.1), contrary to

Novice users

Experienced users


1 (0) 4 (2.74)

1.4 (.89) 9.4 (.89)

3.75 (1.5) 8.75 (3.3)

—

3.4 (.89)

3 (3)

5 (2.74)

14.2 (2.28)

15.5 (5)

Hypothesis 1. An interaction effect appeared between the level of experience and the site version (F(2,22) ¼ 3.396; p ¼ .051): experienced users recalled more elements while navigating the non-ergonomic site than the ergonomic site (po.005). No significant difference appeared for designers or for novice users. Moreover, a significant difference appeared between participants navigating the non-ergonomic site (F(2,11) ¼ 13.599; po.002): designers and experienced users recalled more elements than novices (po.0007 and po.002). These recalled elements were divided into the three categories, as just described. 4.3.2.2. Recalled page titles. There was a significant effect of experience level (F(2,22) ¼ 6.871; po.005; see Table 6): designers recalled more titles than experienced users (po.007) and than novices (po.003). There was no significant difference between novice users and experienced users (p4.1). The participants who used the ergonomic site recalled more titles than the others (F(1,22) ¼ 12.142; po.003; see Table 6). In particular, experienced users and novice users recalled more titles while navigating the ergonomic site than the non-ergonomic site (po.02 and po.006). There was no significant difference for professional designers. 4.3.2.3. Recall of web page content. No significant difference appeared for the level of experience (F(2,22) ¼ 1.785; p4.1) or for the site (F(1,22) ¼ 1.752; p4.1). Especially, experienced users recalled more elements while navigating the non-ergonomic site than the ergonomic site (po.0004),


1041

Table 7 Percentage of relevant vs. irrelevant recalled elements according to the site version and the participants’ experience Ergonomic site

Non-ergonomic site

Novice users

Experienced users


Novice users

Experienced users


Relevant elements

63.67

88.73

74.08

38.58

58.5

77.99

Irrelevant elements

36.33

11.27

25.92

61.42

41.5

22.01

but no significant difference appeared for the novice users (p4.1) or for the professional designers (p4.1). 4.3.2.4. Recalled hyperlinks (images and/or texts). The level of experience had a significant effect on the total number of recalled hyperlinks (F(2,22) ¼ 9.505; po.002; see Table 6): designers and experienced users recalled more links than novices (po.0004 and po.005), but there was no significant difference between experienced users and designers (p4.1). The participants who used the nonergonomic site recalled more hyperlinks than the others (F(1,22) ¼ 4.246; p ¼ .051; see Table 6). There was a significant difference for experienced users only (po.002); novice users did not recall any hyperlink. 4.3.3. Relevance of recalled elements We grouped recalled elements according to their relevance. Relevant elements concerned the elements that participants needed in order to answer three questions, such as hyperlinks, title pages. Irrelevant elements were elements that were considered as additional to necessary ones, such as advertisements, information not linked to the questions, animations, etc. We noted significant differences in the proportion of relevant elements recalled by participants according to their level of experience (F(2,22) ¼ 5.426; po.02; see Table 7): designers and experienced users recalled more relevant elements than novices (po.01 and .02), but there was no significant difference between professional designers and experienced users (p4.1). In accordance with Hypothesis 1, participants using the ergonomic site recalled more relevant information than participants using the non-ergonomic site (F(1,22) ¼ 2.198; po.03). More precisely, experienced users as well as professional designers recalled more relevant elements than irrelevant elements regardless of the site version (po.0004 and po.004). On the other hand, novice users recalled more relevant elements while navigating the ergonomic site and more irrelevant elements while navigating the nonergonomic site (po.05). 4.3.4. Movements of the mouse These analyses allowed us to identify the actions of the users and designers on the web site required to answer the

three questions. We grouped movements of the mouse according to the four following activities:

Looking at or looking for the navigational framework (i.e. the menu) at the top of the page: the ergonomic site had a navigational framework on every page whereas the non-ergonomic site did not. Consequently, the participant may have looked for this navigational framework on the non-ergonomic site. Linear reading: the participant linearly read part of a web page (without analysing the page entirely) as well as s/he would read a paper document. While reading, s/he displaced the mouse onto texts. Scanning of the whole page: the participant scanned the page when s/he went rapidly through the entire page before stopping to focus her/his attention on a specific point to read it. Here, the participant explored the web page before making her/his choice by clicking on a hyperlink.

We counted the total number of occurrences for these three activities for each participant. For instance, if a participant did some linear reading, then went to the navigational framework and went back to linear reading, three actions were taken into account. 4.3.4.1. Looking at/for the navigational framework. We noticed a significant effect of participants’ experience (F(2, 22) ¼ 4.582; po.03; see Table 8): designers used or looked for a navigational framework more often than novice users (po.007), but no significant difference appeared between designers and experienced users, or between novice and experienced users (p4.1). There was no significant difference linked to the web site (F(1,22) ¼ 2.907; p4.1). A significant interaction appeared between experience and site version (F(2,22) ¼ 8.007; po.0002): experienced users as well as designers used or looked for a navigational framework more often while navigating the non-ergonomic site than the ergonomic site (po.006 and po.05), whereas novice users used or looked for it more often while navigating the ergonomic site than the non-ergonomic site (po.008). 4.3.4.2. Linear reading. The experience of the participants had a significant effect (F(2,22) ¼ 3.554; po.05): in



Table 8 Means (and standard deviations) of the movements of the mouse into web pages according to the site version and the participants’ experience Ergonomic site

Menu/navigational bar Linear reading Scanning of the whole page

Non-ergonomic site

Novice users

Experienced users


Novice users

Experienced users


2.2 (.84) 1.6 (.54) —

1.2 (.45) .4 (.54) 2 (.71)

1.75 (.5) .5 (.57) 1.75 (.5)

.6 (.55) 1.6 (.54) .4 (.89)

2.8 (.84) 2 (.71) 1.4 (.55)

3.25 (1.26) 1.25 (.5) 1.5 (.58)

accordance with Hypothesis 2, novice users read more linearly than designers (po.05), but no significant difference appeared between designers and experienced users or between novice and experienced users (p4.1). Participants navigating the non-ergonomic site read more often linearly than participants navigating the ergonomic site (F(1,22) ¼ 12.715; po.002). An interaction effect appeared between the experience of the participants and the site version (F(2,22) ¼ 4.794; po.02): experienced users read more linearly in the non-ergonomic site than in the ergonomic site (po.004), whereas no significant difference appeared for novice users or for designers (p4.1).

Consequently, and in accordance with Hypothesis 3, professional designers found it difficult to detach themselves from the their own viewpoints and to consider the user’s viewpoint.

4.3.4.3. Scanning of the whole web page. The experience of the participants had a significant effect (F(2,22) ¼ 18.698; po.0001): designers and experienced users scanned the web pages more often than novice users (po.0001 and .0001), but no significant difference appeared between designers and experienced users (p4.1). No significant difference appeared for the site version (F(1,22) ¼ .42; p4.1).

Results of these two experiments showed that the experienced users and professional designers developed closely related search activities (except for some results such as the time required to find information or the answers to the grades obtained in the questionnaire on usability satisfaction), whereas professional designers and novice users developed two different and unrelated search activities. In the two experiments, search times for novice users were longer than for experienced users, regardless of the ergonomic quality of the site. These results were corroborated by the fact that novice users visited more hyperlinks, opened up more web pages and more often read web pages linearly than experienced users and professional designers. Moreover, novice users recalled fewer elements seen on visited web pages and more irrelevant ones than experienced users and professional designers. These results reflect understanding difficulties for novice users of the search task to be performed: they spent more time developing and using cognitive processes adapted to their information search activities, as shown by Ho¨lscher and Strube (2000) and Sweller (1998, 1999). In addition, novice users, while navigating the non-ergonomic site, recalled more irrelevant elements than relevant elements. Novice users’ activities were guided by the structure and elements on the web pages, and so they were more affected by the ill-structured web site than experienced users (and designers), as shown by Hofman and van Oostendord (1999), and Potelle and Rouet (2003). Consequently, novice users experienced difficulties building up a global organization of the web site (i.e. a coherent mental model) and chunking elements, unlike experienced users and professional designers. Furthermore, and contrary to our hypothesis, professional designers did not spend significantly less time than novice users finding correct answers (in the two

4.3.5. Anticipation of novice user’s activity After answering the three questions, the professional designers had to describe from the visited web site and explain the path they thought that users would follow to answer the same three questions to which they found answers. Analyses showed two main important results:

Seven designers out of the eight (87.5%) indicated that novice users, before making the decision to click on a hyperlink, would click on all of the provided hyperlinks on the interface one after the other. Once all of the hyperlinks were visited, they would click on the one they thought was the most appropriate with regard to their search task. Consequently, they would choose the correct hyperlink by trial and error in the order presented on the web interface. In this experiment, only two out of ten novice users (20%) actually adopted this strategy while navigating the non-ergonomic site. Once the accurate hyperlink was selected by novice users on the web page, designers indicated that novice users would follow the same path as designers. However, this was not the case, since novice users recorded more steps than designers, and so novice users did not use the web site as professional designers.

5. General discussion We discuss the results of the two experiments on both the role of the level of experience (Section 5.1) and the ergonomic quality of the site (Section 5.2). 5.1. The role of the level of experience


experiments). However, designers visited fewer hyperlinks than the other participants. On the non-ergonomic site, designers looked for a possible navigational framework at the top of the page, whereas novice users spent little time looking for it. Therefore, professional designers automatically applied procedures acquired with experience (looking for a menu that usually supports their navigation activities in web sites), while novice users had not yet acquired this knowledge. In addition, professional designers scanned the web pages more often than novice users, while novice users read the web pages linearly more often than professional designers. Thus, designers and novice users did not use the web site in the same way, although their search times were not significantly different (in the two experiments). Indeed, novice users, because of their lack of understanding, rapidly compromised themselves by opening hyperlinks and visiting new web pages, whereas all of the professional designers opened web pages before clicking on a new hyperlink. Results from the usability satisfaction questionnaire (experiment 1) show that the professional designers were aware of the difficulties generated by the non-ergonomic site, but that they experienced difficulties anticipating the users’ navigation activities: the designers thought that the novice users would choose the correct hyperlink by trial and error. In reality, only two out of the ten novice users adopted this strategy while navigating the non-ergonomic site. Once the accurate hyperlink was selected by novice users on the web page, designers indicated that the novice users would follow the same path as the designers. In these experiments, we noticed that this was not the case, since the novice users recorded more steps than the designers and read linearly more often than the designers. These results reflect the fact that designers actually experienced difficulties detaching themselves from their own viewpoints to consider the user’s viewpoint, in part because they have developed automatic procedures in searching for information. They thought that the novice users would adopt the same behavior or strategy as them. This corresponds to what van Duyne et al. (2002) call the ‘ego bias’, i.e. designers believe that all individuals use web sites like them. 5.2. The effect of the ergonomic quality of the web site On the non-ergonomic site, many irrelevant elements were introduced (advertising banners, flashing items, etc.). According to Sweller (1988, 1999), irrelevant elements distract users and, for the less experienced, may overload working memories. In our two studies, the novice users experienced difficulties ignoring and inhibiting irrelevant elements presented in the non-ergonomic site, since they recalled more irrelevant elements that relevant ones, and novices recalled fewer relevant elements than experienced users and professional designers. This generated difficulties in planning their future actions, anticipating their activities and so selecting relevant elements to answer questions. Thus, novice web users experienced difficulties in develop-

1043

ing evaluation activity, which consists in determining whether available information is relevant to the objective (goal representation maintained in working memory—see Rouet and Tricot, 1996, 1998). These results also corroborate previous findings which showed that distracting elements slowed down the information search, and that certain visual characteristics, such as colors, shapes, size, structure of web pages, guided users’ attention (Anderson et al., 1998; Wickens and Hollands, 2000; Hong et al., 2004). This can explain the increase in the search time, the number of visited hyperlinks for novice users and the few recalled elements (with a significant proportion of irrelevant ones). Another very interesting result appeared. Contrary to our hypothesis, the ergonomic site required significantly more cognitive resources for the users than the nonergonomic site. These results, not in accordance with Sweller’s (1988, 1999) in which a non-adapted presentation of information leads to a higher extraneous cognitive load, can be explained by different ways of managing of cognitive resources: resource commitment leads to the protection of cognitive resources, i.e. through reducing cognitive load, whereas task commitment leads to the monopolization of resources involved in the task (see Cegarra and Hoc, 2006). Indeed, users visiting the ergonomic site could dedicate more cognitive resources to their information search activities than users visiting the non-ergonomic site, since the users of the ergonomic site were not distracted by irrelevant elements which could disturb their attention. They could focus their attention on the task to be performed. We can also explain these results by the fact that users navigating the non-ergonomic site did not try to assimilate all of the elements presented on the web pages (they recalled fewer elements on the nonergonomic site than on the ergonomic site), contrary to learning systems used by Sweller, so their working memory were not overloaded. Therefore, the cognitive theory of Sweller should be adapted to information search by decreasing the importance allocated to the process and the understanding of all of elements present in the site, at least when experience in navigation increases. Professional designers navigating the non-ergonomic site did not use significantly more cognitive resources than designers navigating the ergonomic site. Using the nonergonomic site, professional designers looked for the navigational framework (missing on the web pages), while very few novice users looked for it. Professional designers scanned pages more often than novices, whereas novices linearly read the web pages more often than designers, and professional designers recalled more page titles than novice users. Consequently, designers’ schemata got activated almost automatically while navigating the web sites, and added extraneous load did not disturb their activities. Their level of experience allowed them to overcome ergonomic problems, to deal with them without requiring many cognitive resources, and thus the construction of a coherent mental model of the visited web site.



6. Implications for helping web designers These two studies sought to determine the different strategies involved in information searching, depending on the ergonomic quality of the web sites (a site with ergonomic problems vs. a site incorporating ergonomic criteria and recommendations) and the level of experience of the participants (professional designers, novice users and experienced users). These two studies were the first to attempt at a better understanding of what distinguishes professional designers from novice and experienced web users while searching for information on web sites. Further studies are required to better understand the strategies of information searching developed by designers and users, especially when they have to find information not only on a precise site (as in this article), but on the whole of the Internet (by using search engines, for instance). Our first two studies show that professional designers have automated procedures in navigation, which allow them to overcome difficulties generated by usability problems present on certain web sites. These procedures are effective for search activities, but they prevent designers from efficiently anticipating and considering users’ needs when they design web sites. Based on these results and previous ones (see Chevalier and Ivory, 2003; Chevalier, 2004), two aspects of professional designers’ activities should be supported to help them develop a user-centered activity:

Evaluation process: it seems important to encourage professional designers to consider users’ needs while evaluating web sites, before creating their own site(s). For instance, ergonomic guides could be developed to aid designers (during the evaluation process) to develop an activity similar to the cognitive walkthrough method, in order to help designers identify the usability problems in the evaluated web sites which could slow down and disturb users’ activities. This would help them to identify the ergonomic problems present in existing web sites in order to avoid them in their own productions. Design process: it is also very important to train professional designers to efficiently take into account the novice users’ needs while creating web sites. Towards this end, we conducted research about the usefulness of MetroWeb, a computer support tool developed to help designers create web sites easier to use. MetroWeb structures usability knowledge according to a conceptual framework which is composed of the usability knowledge itself and everything that is considered useful for developing a user-centered design based on usability guidelines (Mariage and Vanderdonckt, 2004). Our first results showed that professional designers who used MetroWeb took more users’ needs into account than designers without MetroWeb, and their web sites were easier to use (Mariage et al., 2005). In addition to the use of MetroWeb, designers could use an ergonomic guide (as suggested in the previous paragraph to support the

evaluation process) to evaluate the ergonomic quality of their work during different design stages. If designers identify ergonomic problems in their web sites, MetroWeb could help them to concretely apply ergonomic recommendations and to rectify the ergonomic problems that might still be present in their productions. Acknowledgments We are very grateful to Andre´ Tricot for his precious advice, to James Mc Cabe for proofreading this article, to the anonymous reviewers and to Jean Scholtz, associate editor, for their relevant comments and interest in this paper. We also thank C. Cabello, A. Delavenne, N. Fouquereau, C. Vale´rian and the participants for their contributions. Appendix A. Tholos experimental software for calculating cognitive resources

First, we indicate the participant’s name and the scenario (see Part 1). Then, the participant listens to an auditory signal by clicking on a button ‘‘Test the beep’’ (see Part 3); if necessary, the sound can be adjusted. In part 2, a space is allocated for parameters capture during the training task. During this phase, the mean baseline reaction time is determined. The researcher notes, in corresponding cells, the number of signals to be presented during the training task (here: 30 beeps), the interval schedule of signals—randomly determined by Tholos—and the number of signals not to be considered while calculating the mean baseline reaction time (here: the first five signals). The researcher presses the ‘‘Run the training task’’ button to launch the training. The part 3 is allocated to capturing the parameters which are specific to the main task (in our study, information search). Then, the researcher presses the ‘‘Run the dual task’’ button to launch the dual task.


Once the task is over, all of the data about participants are recorded in a Microsoft Excels document. Tholos has been developed for an MS Windowss environment (see Chevalier et al., 2004; Cegarra and Chevalier, 2006). Tholos allows the use of three techniques to measure cognitive resources (pupil dilation, questionnaire and dual task). For our experiments, we only used the dual task technique. For the dual task, Tholos presents

1045

functionalities close to Scripkell, software for Macintosh (for more details on Scripkell, see Piolat et al., 1999). The dual task technique has been used to determine the cognitive load of writers, but also of graphic designers (Bonnardel and Piolat, 2003), web designers (Chevalier and Bonnardel, 2000), and individuals searching for information in an e-encyclopedia (Piolat et al., 2002).

Appendix B. Distribution and description of the ergonomic errors introduced in the non-ergonomic web site (see illustrations with Figs. 1–3)

Ergonomic criteria

Description of the introduced usability problems

Guidance. Prompting.

1. The menu is not on each page. For instance, the participant must return to the homepage to find the menu (i.e. the navigational framework). 2. No distinguishing items by location on the web page.

Guidance. Grouping and distinguishing items by location. Guidance. Grouping and distinguishing items by format.

Guidance. Legibility.

Workload. Brevity. Minimal Actions.

Workload. Information Density. Consistency.

Significance of Codes.

3. In the menu, no markers appear regarding the item corresponding to the page read by participant. 4. The hyperlink in the page that the participant is reading is still active in the menu. 5. Some texts are underlined but they are not hyperlinks. 6. The font is not usual (e.g., Comic or Mistral). 7. The contrast text/background is not strong enough. (e.g., yellow and green). 8. The texts are too long (over 50 lines). 9. Some pages are very long (with texts and pictures). 10. From each page, we cannot systematically return to the homepage (via a hyperlink). 11. Irrelevant elements may distract the participant (e.g., advertising banners). 12. The pages do not have the same structure (e.g., the backgrounds have different colors). 13. The hyperlink title of the homepage is not always the same. 14. The hyperlinks do not systematically and clearly reflect the semantic content of their destination pages.

Appendix C. Experiment 1 session Once participant filled in the questionnaire to evaluate her/his level of experience regarding Internet use, s/he started the information search with the same instructions: The site which you are going to navigate contains an eshop, named Espace Culture, which sells music products, CDs, tickets for shows. This shop is located in Aix-enProvence (France). If you do not know this town, it does not matter. Espace Culture invites clients to become members, which allows them to get discounts on the products they purchase.

You have to answer three questions, and the answers are contained on the site. Your task is to try to find the answers as fast as possible. While you are searching for information, you will hear auditory signals like this one (the experimenter makes the participants listen to an auditory signal). You will have to respond to the signal by pressing a pedal located on the ground in front of you, with the same foot during the experiment and as quickly as possible. This task is not as the important as the information search, so I will ask you to really concentrate your attention on the questions and not on the auditory signals. You will also try to always place your foot at about the same distance from the pedal.


1046

Now, to get used to responding to these auditory signals, we are going to proceed to a training phase in which you will hear 30 signals to which you will have to respond. You have to focus your attention on these auditory signals. Before starting, you have to choose the foot you are going to use, in order to position it properly. After these instructions, the participant went though a training phase, which also served to measure her/his baseline reaction times to auditory signals. After the training phase, the experimenter provided these instructions: Now, we are going to start the main task. You will have to answer three questions by finding the information within the site. To get started, let’s begin with the homepage of the site. When you find an answer, and before writing it on your answer sheet, you must press this red button. The participants answered these three questions, one after the other (the presentation order was counterbalanced): Question A: What is the main difference between an analogical recording and a numeric recording? Question B: You wish to go to the Espace Culture shop. You are a student at the University of Aix-en-Provence and you have no car. Which buses do you have to take? Question C: Next week is your grandmother’s birthday. She particularly likes classical music. You decide to buy her a Hector Berlioz CD (featuring Charles Munch and Suzanne Danco Boston). How much do the shop members and non-members have to pay for it on the Espace Culture site? Once the participant has answered the three questions, s/he is are asked to freely navigate the site: Now, you are going to freely navigate the site. As soon as you think you have visualized all of the web pages, close the site by pressing on the red cross above on the right. Once the free navigation task is over, the participant had to answer a satisfaction questionnaire (the usability satisfaction questionnaire, see Appendix D).

(3) This web site’s organization seems logical. (4) It might be useful to give more details on the homepage. (5) Web pages are fun and interesting to explore. (6) Navigating this web site is easy. (7) I find the amount of information I need on this web site. (8) The directions for using this web site are simple and clear. (9) I like to use this web site. (10) This web site provides opportunities for me to communicate with its authors. (11) I feel efficient when I use this web site. (12) At all times, I can control the information I wish to see on this web site. (13) The initial use of this web site is easy. (14) Some elements contained in this web site are annoying. (15) While navigating this web site, I can easily figure out where I am. (16) All buttons and other navigation mechanisms on this Web site function as they should. (17) On this web site, I can easily understand everything. Appendix E. Experiment 2 session From the experiment presentation to the end of the main task (i.e. after answering the three questions), instructions were the same as in experiment 1, except that the participant had to answer the questionnaire to evaluate her/his expertise and in order to distract them. Once the main task was over, the participant had to recall on paper the elements seen in previous visited web pages: Now, you have to try to recall as many elements as possible that you have seen on the site. These elements may be titles, pictures, links, etc. You can note them on this paper. Then, for professional designers only, we added: To finish, you have to imagine the path that a novice user would follow to answer the three questions to which you have just answered.

Appendix D. Usability satisfaction questionnaire References Participant has to indicate for these 17 affirmations her/ her degree of agreement on a 5-point scale, ranging from (1) ‘‘I completely disagree’’ to (5) ‘‘I completely agree’’, as |______|______|______|______|______|

follows:

1

2

3

4

5

I completely disagree ...................... I completely agree

(1) I can control how fast I move through this web site at all times. (2) In this web site, I can rapidly find what I am looking for.

Anderson, J.R., Matessa, M., Lebiere, C., 1998. The visual interface. In: Anderson, J.R., Lebiere, C. (Eds.), The Atomic Components of Thought. Lawrence Erlbaum, Mahwah, NJ, pp. 143–168. Armbruster, B.B., Armstrong, J.O., 1993. Locating information in text: a focus on children in the elementary grades. Contemporary Educational Psychology 18, 139–161. Baddeley, A., 1986. Working Memory. Clarendon Press, Oxford. Barlett, F.C., 1932. Remembering: A Study in Experimental and Social Psychology. Cambridge University Press, London. Bastien, C., Scapin, D., Leulier, C., 1999. The ergonomic criteria and the ISO/DIS 9241-10 dialogue principles: a pilot comparison in an evaluation task. Interacting with Computers 11, 299–322.

ARTICLE IN PRESS A. Chevalier, M. Kicka / Int. J. Human-Computer Studies 64 (2006) 1031–1048 Bhatt, G., 2004. Bringing virtual reality for commercial Web sites. International Journal of Human-Computer Studies 60 (1), 1–15. Blackmon, M.H., Polson, P.G., Kitajima, M., Lewis, C., 2002. Cognitive walkthrough for the Web. CHI Letters 4, 463–470. Bonnardel, N., Piolat, A., 2003. Design activities: how to analyze cognitive effort associated to cognitive treatments? The International Journal of Cognitive Technology 8 (1), 6–15. Bru¨nken, R., Plass, J.L., Leutner, D., 2003. Direct measurement of cognitive load in multimedia learning. Educational Psychologist 38, 53–61. Card, S.K., Moran, T.P., Newell, A., 1983. The Psychology of Human–Computer Interaction. Lawrence Erlbaum, Hillsdale, NJ. Cegarra, J., Chevalier, A., 2006. Tholos software: a tool for combining measures of cognitive load. Unpublished manuscript. Cegarra, J., Hoc, J.-M., 2006. Cognitive styles as an explanation of experts’ individual differences: a case study in computer-assisted troubleshooting diagnosis. International Journal of Human–Computer Studies 64, 123–136. Chase, W.G., Simon, H.A., 1973. The mind’s eye in chess. In: Chase, W.G. (Ed.), Visual Information Processing. Academic Press, New York, pp. 215–281. Chen, M.S., Anderson, J.R., Sohn, M.H., 2003. Eye-hand coordination during Web browsing. In: Ratner, J. (Ed.), Human Factors and Web Development, second ed. Lawrence Erlbaum Associates Publishers, Mahwah, NJ, pp. 207–224. Chen, S.Y., Fan, J.P., Macredie, R.D., 2006. Navigation in hypermedia learning systems: experts vs. novices. Computers in Human Behavior 22 (2), 251–266. Chevalier, A., 2004. L’orientation du cahier des charges constitue-t-elle une aide a` la conception de sites web ergonomiques? Revue d’Intelligence Artificielle 3, 337–366. Chevalier, A., 2005. E´valuer un site Web: les concepteurs et utilisateurs parviennent-ils a` identifier les proble`mes d’utilisabilite´? Revue d’Intelligence Artificielle 19, 319–338. Chevalier, A., Bonnardel, N., 2000. The role of constraints in creativity: a study on the design of Web sites. Communication presented at the XXVII International Congress of Psychology, Stockholm, Sweden, July. Chevalier, A., Ivory, M.Y., 2003. Web site designs: influence of designer’s experience and design constraints. International Journal of Human Computer Studies 58 (1), 57–87. Chevalier, A., Kicka, M., Cegarra, J., 2004. Quels sont les effets de la qualite´ ergonomique d’un site web et de l’expe´rience des utilisateurs sur la charge cognitive et le temps de navigation? Communication pre´senteé a` la 10e`me JETCSIC, Geneva, Switzerland, June. Dreher, M.J., 1992. Searching for information in textbooks. Journal of Reading 35, 364–371. van Duyne, D.K., Landay, J.A., Hong, J., 2002. The Design of Web Sites. Principles, Processes, and Patterns for Crafting a Customer-Centered Web Experience. Addision-Wesley, Reading, MA. Fisk, A.D., Derrick, W.L., Schneider, W., 1986–1987. A methodological assessment and evaluation of dual-task paradigms. Current Psychological Research and Reviews 5, 315–327. Forrester, 2002. Q2 2002 online sales: Stable and on track. Available at http://www.forrester.com/ER/Research/Brief/Excerpt/0,1317,15462,00. html at July, 2001. Guthrie, J.T., 1988. Locating information in documents: examination of a cognitive model. Reading Research Quarterly 23, 178–199. Guthrie, J.T., Mosenthal, P., 1987. Literacy as multidimensional: locating information and reading comprehension. Educational Psychologist 22, 279–297. Grey, S.H., 1990. Using protocol analyses and drawings to study mental model construction during hypertext navigation. International Journal of Human–Computer Interaction 2, 359–377. Helander, M.G., Khalid, H.M., 2000. Modeling the customer in electronic commerce. Applied Ergonomics 31, 609–619. Hoc, J.M., 1988. Cognitive Psychology of Planning (C. Greenbaum, Trans.). Academic Press, London.

1047

Hofman, R., van Oostendord, H., 1999. Cognitive effects of a structural overview in a hypertext. British Journal of Educational Technology 30 (2), 129–140. Hong, W., Thong, J.Y.L., Tam, K.Y., 2004. Designing product listing pages on e-commerce websites: an examination of presentation mode and information format. International Journal of Human–Computer Studies 61 (4), 481–503. Ho¨lscher, C., Strube, G., 2000. Web search behavior of Internet experts and newbies. Computer Networks 33, 337–346. Hwang, F., Keates, S., Langdon, P., 2005. A sub movement analysis of cursor trajectories. Behaviour & Information Technology 24 (3), 205–217. Ihadjadene, M., Chaudiron, S., Martins, D., 2003. The effect of individual differences on searching the web. In: Proceedings of the Annual Meeting Humanizing Information Technology: From Ideas to Bits and Back, Westin Long Beach, CA, October. Jansen, A.R., Blackwell, A.F., Marriott, A.F., 2003. A tool for tracking visual attention: the restricted focus viewer. Behavior, Research Methods, Instruments & Computers 35 (1), 57–69. Kalsbeek, J.W., Sykes, R.N., 1967. Objective measurement of mental load. Acta Psychologica 27, 253–261. Kalyuga, S., Ayres, P., Chandler, P., Sweller, J., 2003. The expertise reversal effect. Educational Psychologist 38 (1), 23–31. Kellogg, R.T., 1987. Effects of topic knowledge on the allocation of processing time and cognitive effort to writing processes. Memory & Cognition 15, 256–266. Kellogg, R.T., 1994. The Psychology of Writing. Oxford University Press, New York. Kirakowski, J, Claridge, N., Whitehand, R., 1998. Human Centered Measures of Success in Web Site Design. In: Proceedings of the Human Factors and the Web Workshop, Basking Ridge, NJ. Koenemann, J., Belkin, N.J., 1996. A case for interaction: a study of interactive information retrieval behavior and effectiveness. In: Proceedings of CHI 1996, Vancouver, Canada, April. Lee, A., 1999. Web usability. A review of the research. SIGCHI Bulletin 31, 38–40. Leplat, J., 1997. Regards sur l’activite´ en Situation de Travail: Contribution a` la Psychologie Ergonomique. PUF, Paris. Levy, C.M., Ransdell, S.E., 1995. Is writing as difficult as it seems? Memory & Cognition 23 (6), 767–779. Ling, J., van Schaik, P., 2006. The influence of font type and line length on visual search and information retrieval in web pages. International Journal of Human–Computer Studies 64 (5), 395–404. Marchionini, G., Dwiggins, S., Katz, A., Lin, X., 1993. Information seeking in full-text end-user-oriented search systems: the roles of domain and search expertise. Library & Information Science Research 15, 35–69. Marcus, N., Cooper, M., Sweller, J., 1996. Understanding instructions. Journal of Educational Psychology 88, 49–63. Mariage, C., Vanderdonckt, J., 2004. Creating Contextualised Usability Guides for Web Sites Design and Evaluation. In: Proceedings of the Fifth International Conference on Computer-Aided Design of User Interfaces CADUI’2004, Funchal, Portugal, January. Mariage, C., Vanderdonckt, J., Chevalier, A., 2005. Using the MetroWeb tool to improve usability quality of Web sites. In: Baeza-Yates, R., Olsina, L. (Eds.), La-Web. IEEE Computer Society, Los Alamos, CA, pp. 213–222. Matlin, M.W., 1998. Cognition, fourth ed. Harcourt Brace, New York. Mc Crickard, D.S., 2001. The effect of changes in information access times on hypertext choices. In: Smith, M.J., Salvendy, G., Harris, D., Koubek, R.J. (Eds.), Usability Evaluation and Interface Design. Lawrence Erlbaum Associates, Mahwah, NJ, pp. 1086–1090. McEneaney, J.E., 2001. Graphic and numerical methods to assess navigation in hypertext. International Journal of Human–Computer Studies 55 (5), 761–786. Miller, G.A., 1956. The magical number seven plus or minus two: some limits in our capacity for processing information. Psychological Review 63, 81–97.



van Merrie¨nboer, J.J.G., Sweller, J., 2005. Cognitive load theory and complex learning: recent developments and future directions. Educational Psychology Review 17 (2), 147–177. Navarro-Prieto, R., Scaife, M., Rogers, Y., 1999. Cognitive strategies in web searching. In: Proceedings of the Human Factors and the Web, Gaithersburg, Maryland, June. Nielsen, J., 2000. Designing Web Usability. New Riders Publishing, Indianapolis. Olson, G.M., Olson, J.S., 2003. Human–computer interaction: psychological aspects of the human use of computing. Annual Review of Psychology 54, 491–516. Oulasvirta, A., 2004. Task demands and memory in web interaction: a levels of processing approach. Interacting with Computers 16 (2), 217–241. Otter, M., Johnson, H., 2000. Lost in hyperspace: metrics and mental models. Interacting with Computers 13 (1), 1–40. Ozok, A.A., Salvendy, G., 2004. Twenty guidelines for the design of Webbased interfaces with consistent language. Computers in Human Behavior 20 (2), 149–161. Paas, F., van Merrie¨nboer, J.J.G., 1994. Instructional control of cognitive load in the training of complex cognitive tasks. Educational Psychology Review 6, 51–71. Paas, F., Tuovinen, J.E., Tabbers, H.K., Van Gerven, P.W.M., 2003. Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist 38, 63–71. Park, S., Choi, D., Kim, J., 2004. Critical factors for the aesthetic fidelity of web pages: empirical studies with professional web designers and users. Interacting with Computers 16 (2), 351–376. Pearson, R., van Schaik, P., 2003. The effect of spatial layout of and link colour in web pages on performance in a visual search task and an interactive search task. International Journal of Human–Computer Studies 59 (3), 327–353. Piolat, A., Olive, T., Roussey, J.-Y., Thunin, O., Ziegler, J.C., 1999. Scriptkell: a computer assisted tool for measuring the relative distribution of time and cognitive effort in writing and other tasks. Behavior Research Methods, Instruments, & Computers 31 (1), 113–121. Piolat, A., Kellogg, R.T., Farioli, F., 2001. The triple task technique for studying writing processes: on which task is attention focused? Current psychology letters. Brain, Behavior and Cognition 4, 67–83. Piolat, A., Roussey, J.Y., Ge´rouit, C., 2002. Prise de notes par des e´le`ves de 10–12 ans plus ou moins bons lecteurs et re´dacteurs. Repe`res 26, 277–291. Pirolli, P., Card, S., 1999. Information foraging. Psychological Review 106 (4), 643–675. Potelle, H., Rouet, J.-F., 2003. Effects of content representation and readers’ prior knowledge on the comprehension of hypertext. International Journal of Human–Computer Studies 58 (3), 327–345. Pratt, J.A., Mills, R., Kim, Y., 2004. The effects of navigational orientation and user experience on user task efficiency and frustration levels. Journal of Computer Information Systems 44 (4), 93–100.

Richard, M.-O., Chandra, R., 2005. A model of consumer web navigational behavior: conceptual development and application. Journal of Business Research 58 (8), 1019–1029. Richard, J.F., 1997. Attention, controˆle et gestion des ressources. In: Mellier, D., Vom Hofe, A. (Eds.), Attention et controˆle cognitif: mećanismes, de´veloppement des habilite´s, pathologies. PUR, Rouen, pp. 7–16. Rouet, J.F., 2003. What was I looking for? The influence of task specificity and prior knowledge on students’ search strategies in hypertext. Interacting with Computers 15 (3), 409–428. Rouet, J.F., Tricot, A., 1996. Task and activity models in hypertext usage. In: van Oostendorp, H., de Mul, S. (Eds.), Cognitive Aspects of Electronic Text Processing. Ablex, Norwood, NJ, pp. 239–264. Rouet, J.F., Tricot, A., 1998. Chercher de l’information dans un hypertexte: vers un mode`le des processus cognitifs. Hypertextes et Hyperme´dias, hors se´rie, pp. 57–74. Rumelhart, D.E., 1978. Schemata: The Building Blocks of Cognition (Report #79). University of California, San Diego. Shneiderman, B., Byrd, D., Croft, B., 1998. Sorting out searching: a userinterface framework for text searches. Communications of the ACM 41 (4), 95–98. Spe´randio, J.C., 1988. L’ergonomie Cognitive du Travail Mental. Masson, Paris. Spool, J.M., Scanlon, T., Schroeder, W., Snyder, C., DeAngelo, T., 1999. Web Site Usability: A Designer’s Guide. Morgan Kaufmann, San Francisco, CA. Stronge, A.J., Rogers, W.A., Fisk, A.D., in press. Web-based information search and retrieval: effects of strategy use and age on search success. Human Factors. Sweller, J., 1988. Cognitive load during problem solving: effects on learning. Cognitive Science 12, 257–285. Sweller, J., 1999. Instructional Design in Technical Areas. ACER Press, Melbourne. Tabatabai, D., Shore, B.M., 2005. How experts and novices search the Web. Library & Information Science Research 27, 222–248. Teo, H.H., Oa, L.B., Liu, C., Wei, K.K., 2003. An empirical study of the effects of interactivity on Web user attitude. International Journal of Human–Computer Studies 58 (3), 281–305. Tricot, A., 2003. Apprentissage et recherche d’information avec des documents e´lectroniques. Me´moire pour l’habilitation a` diriger des recherche, Universite´ de Toulouse le Mirail, France. Van Gerven, P.W.M., Paas, F., Van Merrie¨nboer, J.J.G., Schmidt, H.G., 2002. Cognitive load theory and aging: effects of worked examples on training efficiency. Learning & Instruction 12, 87–105. Wang, Y.D., Emurian, H.H., 2005. An overview of online trust: concepts, elements, and implications. Computers in Human Behavior 21 (1), 105–125. Wickens, C.D., Hollands, J.G., 2000. Engineering Psychology and Human Performance, third ed. Prentice-Hall, Upper Saddle River, NJ. Yantis, S., 1993. Stimulus-driven attention capture and attentional control settings. Journal of Experimental Psychology: Human Perception & Performance 19 (3), 676–681.