elab.vanderbilt.edu
Measuring the Flow Experience Among Web Users Thomas P. Novak (
[email protected]) Donna L. Hoffman (
[email protected]) Project 2000, Vanderbilt University http://www2000.ogsm.vanderbilt.edu/
July 1997
Paper Presented at Interval Research Corporation, July 31, 1997
Draft 1.0
Measuring the Flow Experience Among Web Users Thomas P. Novak and Donna L. Hoffman Project 2000, Vanderbilt University http://www2000.ogsm.vanderbilt.edu/ Abstract The flow construct has recently been proposed as essential to understanding consumer navigation behavior in online environments. We review definitions and models of flow, and describe an empirical study which measures flow in terms of respondents’ skills and challenges for using the World Wide Web. Skills and challenges are shown to correlate in anticipated ways with scales measuring constructs of flow, control, arousal, and anxiety that underlie previous models of flow. By taking the sum and difference of skills and challenges as axes of a two dimensional space, we derive a simple conceptualization of flow. The sum and difference of skills and challenges for using the Web relates in hypothesized ways to measures of consumer search and purchase behavior in online and traditional media.
1
1) Introduction Objectives The flow construct (Csikszentmihalyi 1977) has been recently proposed by Hoffman and Novak (1996) as essential to understanding consumer navigation behavior in online environments such as the World Wide Web. Previous researchers (e.g. Csikszentmihalyi 1990; Ghani, Supnick and Rooney 1991; Trevino and Webster 1992; Webster, Trevino and Ryan 1993) have noted that flow is a useful construct for describing more general human-computer interactions. Hoffman and Novak define flow as "the state occurring during network navigation which is: 1) characterized by a seamless sequence of responses facilitated by machine interactivity, 2) intrinsically enjoyable, 3) accompanied by a loss of self-consciousness, and 4) selfreinforcing." To experience flow while engaged in an activity, consumers must perceive a balance between their skills and the challenges of the activity, and both their skills and challenges must be above a critical threshold. Hoffman and Novak (1996) propose that flow has a number of positive consequences from a marketing perspective, including increased consumer learning, exploratory behavior, and positive subjective experience. Over the past 20 years, numerous researchers have attempted to measure flow for a wide variety of activities, including composing music, sports, work activities, hobbies, and computer usage. We will summarize the results of many of their efforts, and the models of flow that have been proposed. Our interest is in measuring and modeling flow specifically for consumer behavior on the Web. Our own measurement effort builds upon previous research by operationalizing flow in terms of the combination of a Web user’s perception of their skills at using the Web and how challenging they find using the Web to be. Specifically, we seek to: 1) Review previous definitions and models of flow. 2) Describe our empirical study which measures a series of constructs related to flow. 3) Develop and evaluate the reliability of a series of Likert Scales for components of the eight channel flow model proposed by Massimini & Carli (1988): skill, challenge, and bipolar constructs of flow/apathy, control/worry, arousal/relaxation and anxiety/boredom. 4) Evaluate the convergent and discriminant validity of the above, in terms of the relationships among these constructs as predicted by the eight channel flow model. 5) Develop a simple two-dimensional model of flow, where the dimensions underlie the four eight channel flow models, and are defined by a) the sum of skills and challenges, and b) the difference of skills and challenges. We will investigate demographic and usage differences on these two dimensions. 6) Specify and test a series of hypotheses regarding the relationship of consumer search and purchase behavior on the Web and in traditional media to the sum and difference of skills and challenges using the Web. We begin with a literature review of the flow construct, and models of flow, and then consider the topics listed above.
The Flow Construct What is flow? Table 1 provides definitions of flow from 16 different studies. As one reads through this list, the phrases listed here seem to make intuitive sense - flow is "a holistic sensation where one acts with 2
total involvement, with a narrowing of focus of attention," and so forth. However, the exercise of reading through these phrases in an attempt to define flow is deceptive. One is not left with a central definition of flow, but rather a wide variety of constructs which tend to be experienced when one experiences flow. Some of these constructs define or cause flow, and some of these are experienced as a result of being in the flow state. Hoffman and Novak (1996) propose, for example, that centering of attention is a necessary condition for achieving flow, as are congruent skills and challenges that are above a critical level. Consider two definitions that have been proposed by Trevino & Webster (1992) and by Csikszentmihalyi & Csikszentmihalyi (1988). Trevino and Webster (1992) operationally define flow as the linear combination of four characteristics: control, attention, curiosity, and intrinsic interest. But, it is not clear why these four characteristics should be used. Do these define flow, or are they better thought of as antecedents or consequences of flow? The other definition, from Csikszentmihalyi & Csikszentmihalyi (1988) is quite different, focusing upon the congruence of a person's skills in a given activity, and their perceptions of the challenges of the activity. The person's evaluation of the skills and challenges is typically thought of relative to other activities the person performs, rather than on an absolute scale. That is, how challenging is surfing the Web, compared to other ways the person has available of spending their time? The definition also states that there is a critical value that skills and challengers must be above. Thus, it is not simply the fact that skills and challenges are congruent, they must also be high. This is different than many early definitions of flow in terms of skills and challenges, which considered low skill and low challenge activities (take chewing bubble gum as an extreme example) to also produce flow (Csikszentmihalyi 1977). In Hoffman and Novak (1996), we define flow in terms of the experience of flow (intrinsic enjoyment, loss of self-consciousness), structural properties of the flow activity (seamless sequence of responses facilitated by interactivity with the computer and self-reinforcement), and antecedents of flow (skill/challenge balance, focused attention, and telepresence). In this research, we focus upon high skills and challenges for using the Web as a requirement for the experience of flow while using the Web. In addition, we use the construct of computer playfulness, as operationalized by Webster and Martocchio (1992) as an indicator of the experience of flow while using the Web. The role of skills and challenges in defining flow is a central portion of the majority of the definitions in Table 1. We do not address the larger question in this paper of the exact configuration of antecedents and consequences of flow, focusing instead upon the more limited research question of the role of skills and challenges. Research in progress (Novak, Hoffman and Yung 1997) addresses the larger issue of the structure of antecedents and consequences of flow. Table 1 - Definitions of Flow
Reference:
Conceptual or Operational Definition:
Csikszentmihalyi (1977)
"the holistic sensation that people feel when they act with total involvement" (p36) when in the flow state "players shift into a common mode of experience when they become absorbed in their activity. This mode is characterized by a narrowing of the focus of awareness, so that irrelevant perceptions and thoughts are filtered out; by loss of self-consciousness; by a responsiveness to clear goals and unambiguous feedback; and by a sense of control over the environment...it is this common flow experience that people adduce as the main reason for performing the activity" (p72)
Privette and Bundrick (1987)
"Flow..., defined as an intrinsically enjoyable experience, is similar to both peak experience and peak performance, as it shares the enjoyment of valuing of peak experience and the behavior of peak performance. Flow per se does not imply optimal joy or performance but may include either or both." [p316]
Csikszentmihalyi and
"The flow experience begins only when challenges and skills are above a certain level,
3
Csikszentmihalyi (1988)
and are in balance." [p260]
Mannell, Zuzanek, and Larson (1988)
"Csikszentmihalyi (1975) describes the flow experience as 'one of complete involvement of the actor with his activity' (p. 36), and he has identified a number of elements that are indicators of its occurrence and intensity. These indicators include: the perception that personal skills and the challenges provided by an activity are imbalance, centering of attention, loss of self-consciousness, unambiguous feedback to a person's actions, feelings of control over actions and environment, and momentary loss of anxiety and constraint, and enjoyment or pleasure." [p291] "Flow was operationalized by measuring the affect, potency, concentration, and the perception of a skill/challenge balance ." [p292]¨
Massimini and Carli (1988)
congruent skills and challenges that are above each subject's average weekly levels
LeFevre (1988)
"a balanced ratio of challenges to skills above average weekly levels" (p307)
Csikszentmihalyi and LeFevre (1989)
"When both challenges and skills are high, the person is not only enjoying the moment, but is also stretching his or her capabilities with the likelihood of learning new skills and increasing self-esteem and personal complexity. This process of optimal experience has been called flow."
Csikszentmihalyi (1990)
we feel "in control of our actions, masters of our own fate...we feel a sense of exhilaration, a deep sense of enjoyment" (p3) "the state in which people are so intensely involved in an activity that nothing else seems to matter; the experience itself is so enjoyable that people will do it even at great cost, for the sheer sake of doing it."
Ghani, Supnick and Rooney (1991)
"two key characteristics of flow: the total concentration in an activity and the enjoyment which one derives from an activity...the precondition for flow is a balance between the challenges perceived in a given situation and skills a person brings to it" (p230) "a related factor is the sense of control over one's environment" (p231)
Trevino and Webster (1992)
"flow characterizes the perceived interaction with CMC technologies as more or less playful and exploratory"..Flow theory suggests that involvement in a playful, exploratory experience - the flow state - is self-motivating because it is pleasurable and encourages repetition. Flow is a continuous variable ranging from none to intense." [p540] "Flow represents the extent to which (a) the user perceives a sense of control over the computer interaction, (b) the user perceives that his or her attention is focused on the interaction, (c) the user's curiosity is aroused during the interaction, and (d) the user finds the interaction intrinsically interesting.." [p542]
Webster, Trevino and Ryan (1993)
"the flow state is characterized by four dimensions...(a) the user perceives a sense of control over the computer interaction, (b) the user perceives that his or her attention is focused on the interaction, (c) the user's curiosity is aroused during the interaction, and (d) the user finds the interaction intrinsically interesting. [p413]
Clarke and Haworth (1994)
"the subjective experience that accompanies performance in a situation where the challenges are matched by the person's skills. Descriptions of the feeling of 'flow' indicate an experience that is totally satisfying beyond a sense of having fun." [p511]
Ellis, Voelkl and Morris (1994)
"..an optimal experience that stems from peoples' perceptions of challenges and skills in given situations. Situations in which challenges and skills are perceived to be equivalent are thought to facilitate the emergence of such indicators of flow as positive affect and high levels of arousal, intrinsic motivation, and perceived freedom" [p337]
Ghani and Deshpande (1994)
"The two key characteristics of flow are (a) total concentration in an activity and (b) the enjoyment which one derives from an activity...There is an optimum level of challenge relative to a certain skill level. ...A second factor affecting the experience of flow is a sense of control over one's environment." [p383]
4
Lutz and Guiry 1994
"Psychologists use the term 'flow' to describe a state of mind sometimes experienced by people who are deeply involved in some event, object or activity...they are completely and totally immersed in it...Indeed, time may seem to stand still and nothing else seems to matter while engaged in the consump†ion event." [from respondent instructions]
Hoffman and Novak (1996)
"the state occurring during network navigation which is 1) characterized by a seamless sequence of responses facilitated by machine interactivity, 2) intrinsically enjoyable, 3) accompanied by a loss of self-consciousness, and 4) self-reinforcing"
Flow Measurement Methodology Table 2 summarizes the 16 articles shown in Table 1 according to the approach taken to measuring flow. There are three major approaches to measuring flow that have been taken in empirical research: 1) Respondent provides a narrative description of a flow experience and then evaluates the experience using a survey instrument (Narrative/Survey). 2) Respondents who have participated in a selected activity are asked to retroactively evaluate their experience using a survey instrument (Activity/Survey). 3) In the Experience Sampling Method (ESM), respondents are paged throughout the day for a one week period, and evaluated their activity at the time of being paged using a survey instrument. Let us consider a few examples of these three approaches. First consider the Narrative/Survey method. This method was used by Privette and Bundrick (1987) to measure six "construct events" - events characterizing constructs from the theoretical and research literature. Subjects are given short descriptions of peak performance, peak experience, flow, average events, misery and failure, and asked to write a short narrative description of one of each of these experiences. From the point of view of flow measurement, a major weakness of Privette's work is that flow is always operationalized as "describe the last time you played a sport or game." This is a very weak definition of flow. Each of the six experiences is then evaluated on a 47 item "Experience Questionnaire". The objective is to understand the nature of the differences among these six construct events. Thus, the analysis of flow is at a very general level. This methodology is not used to identify the extent to which flow occurs in different people for different types of events. The second method we call the Activity/Survey method. Webster, Trevino and Ryan (1993) describe two studies which provide examples of this method. In the first, 133 subjects attended a one-day Lotus 1-2-3 course, and in the second 43 subjects who were regular users of electronic mail were selected. In both studies, a 12-item scale measured flow as a combination of items for control, attention focus, curiosity and intrinsic interest. Table 2 - Methods for Measuring Flow
Method Narrative + Survey
Reference
Description
Privette & Bundrick 1987
Subjects provide narrative descriptions of six "construct events”: peak performance (one incident in your life characterized by functioning at your best); peak experience (one incident in your life characterized by highest happiness); flow (the last time you played a sport or game); an average event (something you did between 3 pm and 6 pm yesterday); misery one incident in your life characterized by deepest misery; failure (one incident in your life characterized by total failure).
5
These construct events were each evaluated on the "Experience Questionnaire" consisting of 47 rating scales. Lutz & Guiry 1994
Subjects provided narrative descriptions of four categories of consumption experiences: peak experiences, peak performance, flow, ordinary consumption These consumption experiences were rated on Privette's Experience Questionnaire consisting of 42 ratings scales, plus single item measures of the challenges and skills of the event.
Activity + Survey
Trevino & Webster 1992
154 respondents at a midwestern health care firm completed surveys. Employees were users of email (average of 8.5 months) and voicemail (average of 7 months). A voluntary one-day sample of email and voicemail was obtained and coded for each respondent. Survey data was collected on flow (four items measuring control, attention focus, curiosity, and intrinsic interest) and a large number of antecedents and consequences.
Webster, Trevino & Ryan 1993
Study 1: Subjects (133 MBA students) attended a one-day Lotus 1-2-3 course. Study 2: Subjects (43 employees in the accounting department of a large firm) who were regular users of electronic mail. Survey: Both studies used a 12-item scale which measured flow as a combination of 1) control, 2) attention focus, 3) curiosity, and 4) intrinsic interest. Correlates of flow were measured with scales for flexibility, modifiability, experimentation, expected voluntary use, and communication effectiveness.
Ghani, Supnick & Rooney 1991
59 undergraduate business students were randomly assigned to three-person groups and completed three group activities: 1) an orientation to computer-mediated (CM) conferencing, 2) a CM group exercise, and 3) a face-to-face group exercise. Respondents completed questionnaires measuring flow (operationalized as four items for enjoyment and four for concentration), challenges, skills, and control.
Ghani & Deshpande 1994
Sample was 62 managers from a variety of manufacturing, service, and government organizations who used computers as part of their day-to-day work. Respondents completed questionnaires measuring flow (operationalized as four items for enjoyment and four for concentration), challenge, control, exploratory use, and extent of use
Experience Sampling Method
Csikszentmihalyi & Csikszentmihalyi 1988
Objective is to measure flow and other states of consciousness occurring in activities encountered in everyday life. For each respondent in the study, day-to-day activities are sampled by paging the respondent 8 times a day for a week, for a total of 56 times. For example, Csiksentmihalyi & LeFevre 1989 paged respondents randomly within 2-hour periods from 7:30 am to 10:30 pm. The Experience Sampling Form, completed at each paging, measures challenges and skills, respondent's mood, motivation for doing the activity, and many other correlates of flow.
Mannell, Zuzanek & Larson 1988
Sample was 92 retired adults. Participants carried pagers for one week. Signals were sent between 8 am and 10 pm, with one signal occurring at a random time within every 2-hour block. ESF was filled out for each signal. Participants responded to 76% of signals, providing 3,412 self-reports.
Massimini & Carli 1988
Sample was 47 students between 16 and 19 years old living in Milan, Italy. An average of 7 signals a day were sent between 8 and 10 pm for one week, and ESF filled out. Activities were coded into categories.
Carli, Delle Fave &
Replicated Massimini & Carli with a sample of 75 U.S. adolescents from a diversified
6
Massimini 1988
community high school.
LeFevre 1988
Sample was 107 workers recruited from five large companies in the Chicago area that agreed to cooperate in a study of work satisfaction. Participants carried pagers for a week, with signals sent between 7:30 am and 10:30 pm, with one signal occurring randomly within 2-hour periods. Participants responded to 85% of all signals, providing 4,800 self reports.
Csikszentmihalyi & LeFevre 1989
(The data are from LeFevre 1988)
Clarke & Haworth 1994
Sample was 35 students aged between 16 and 18 years old from a college in the UK. Signals were sent between 11:30 am and 11:30 pm. Eight signals were issued at set, but randomly determined times. A total of 1745 self reports were obtained.
Ellis, Voelkl & Morris 1994
Study 1 - sample was 12 older adults residing in nursing homes between ages 73 to 95. Signals were sent six times a day between 7 am and 7 pm for one week. Participants responded to 74% of all signals, providing 307 responses. Study 2 - sample was 59 student volunteers. Signals were sent 5 times a day between 8 am and 10 pm for one week. Participants responded to 51% of all signals, providing 1057 responses.
The Activity/Survey method can also be used in laboratory experiments. Ghani, Supnick & Rooney randomly assigned undergraduate business students into face-to-face or computer-mediated groups, and had respondents complete questionnaires measuring flow after their group experience. The Activity/Survey method is, in principle, useful for either concurrently or retrospectively determining the experience of flow for specific events. One question is whether respondents can reliably evaluate flow after rather than during an activity. It may be that surveys completed immediately after completion of an activity have greater validity than surveys describing an activity which the person has engaged in for a long period of time. The most commonly used method of measuring flow is Csikszentmihalyi's "Experience Sampling Method." The ESM is uniquely suited to measuring flow and other states of consciousness occurring in activities occurring in everyday life. From the distinction between flow, peak experience and peak performance (Privette and Bundrick 1987), flow experiences are the sorts of experiences that occur on a regular, ongoing basis, rather than being unusual or atypical events of peak experience or peak performance. In the ESM, respondents are randomly paged 8 times a day for a period of a week, for a total of 56 times. When paged, respondents complete a two-page Experience Sampling Form, which measures the challenges and skills of the activity engaged in at the time of paging, plus a variety of measure of respondent's mood and motivation. The skills and challenges of each respondent are standardized after data collection, so they represent, for each respondent, whether the skills or challenges were above or below the average level of all activities rated during the one week period.
Models of Flow Now, let us consider approaches to constructing models of flow. These models relate to both the definitions we have discussed and also to the methodologies for measuring flow. Models of flow go beyond a definition of flow by describing causes, effects, and correlates of flow. We identify three broad approaches to constructing models of flow: (1) (2)
conceptual models causal models
7
(3)
flow channel segmentation models
We discuss each in turn. Conceptual Models. Our general conceptual model of flow in computer-mediated is described in detail in Hoffman and Novak (1996). Key features of this model are that flow is determined by high skills and challenges and focused attention, and is enhanced by interactivity and telepresence. A simplified version of Hoffman and Novak’s conceptual model is shown in Figure 1. Time per day use Web
S K IL L When first used Web
P O S ITIV E A F F E C T
FLOW CHALLENGE PLAY
EXPLORATORY B E H A V IO R
TELEPRESENCE
IN T E R A C T IV IT Y
F O C U S E D A T T E N T IO N
Figure 1 - Simplified Version of Hoffman & Novak’s (1996) Conceptual Model
Causal Models. Causal models provide a second approach. Flow research which has used the Activity /Survey methodology has typically used causal models to analyze the results of the survey data. Causal models look much like conceptual models, but provide an empirical test of the magnitude and directionality of hypothesized relationships. We briefly consider three causal models presented in the literature. Consider the causal model fit by Ghani, Supnick & Rooney (1991). Control and challenges were found to predict flow, which was operationalized as four items for enjoyment and four for concentration). Control and flow predicted exploratory use, which in turn predicted extent of use. In a later study, Ghani and Deshpande (1994) included skill as well as challenge. The resulting causal model is simple, but quite interesting, in that skill leads to control which leads to flow. Skill also directly affects flow, as does perceived challenge. This model provides empirical support for definitions of flow that specify that flow occurs when challenges and flow are both high, since skill and challenges independently contribute to flow. A final causal model was fit by Trevino and Webster (1992). A different operational definition of flow is used in this research, consisting of four items measuring control, attention focus, curiosity, and intrinsic interest). Skill was measured, but not challenges. Ease of use was identified as an intermediate variable between skill and flow. One difficulty with the above research is the operational definition of flow. Constructs of enjoyment, concentration, control, attention focus, curiosity and intrinsic interest are used to define flow, rather than being considered as potential antecedents or consequences of flow. Our intention in this current paper is not to fit structural equation models to constructs relating to flow, although we have work in progress (Novak, Hoffman & Yung 1997) which aims to construct comprehensive structural equation models which test and refine the model shown in Figure 1. In this current paper, we focus upon the relationship of skills and challenges with flow. This relationship is an integral component of Hoffman and Novak’s (1996) conceptual model, has been tested in the causal models discussed above, and is the key defining characteristic of the flow channel segmentation models we 8
discuss next.
Flow Channel Segmentation Models. The third and final category of models of flow are "flow channel segmentation models." While causal models have been used in studies employing the Activity/Survey method, channel segmentation models have been exclusively used in studies employing the Experience Sampling Methodology. This distinction is arbitrary, however, in that it stems from different methodologies chosen by different sets of researchers. In this paper, we will collect data using the Activity/Survey method, but analyze it from the framework of flow channel segmentation models.
High
Flow channel segmentation models are based upon Csikszentmihalyi's definition of flow in terms of skills and challenges. However, the segmentation models attempt to account for all possible combinations (channels) of high/low skills and challenges. Here are two simple models. The early three channel model shown in Figure 2 identified flow as congruent skills and challenges, both high and low. Anxiety is identified as high challenges and low skills, and boredom as high skills and low challenges.
CHALLENGE
A N X IE T Y
FLOW
Low
BOREDOM
Low
S K IL L
High
Figure 2 - Three Channel Flow Model Greater empirical support has been found for the reformulated four channel model shown in Figure 3, where flow is defined as high skills and high challenges, and apathy as low skills and low challenges. Numerous researchers (e.g., Ellis, Voelkl & Morris 1994; LeFevre 1988; Nakamura 1988; and Wells 1988) have found clear patterns of differences among the four “flow segments,” where the differences can be characterized as: 1) Flow is distinctly different from other states. Respondents who rated the skills and challenges of an activity as higher than average for the week rated those activities as higher on virtually all of the positive indicator variable collected in these studies: enjoyment, positive affect, activation, concentration, creativity, self-esteem, etc. 2) Flow is in complete opposition to apathy. The empirical evidence suggests that low challenge/low skill activities are opposite to flow activities, and the four channel model is more appropriate than the three channel model.
9
High
FLOW
APATHY
BOREDOM
Low
CHALLENGE
A N X IE T Y
Low
S K IL L
High
Figure 3 - Four Channel Flow Model A natural extension of the four channel model is the eight channel model (Massimini & Carli 1988; Ellis, Voelkl & Morris 1994), which also allows for intermediate (moderate) levels of skills and challenges, and identifies four additional channels: arousal, control, relaxation, and worry. Figure 4 shows our conceptualization of the eight-channel model, superimposing dimensions of Skills and Challenges. Figure 4 represents a 45 degree rotation of Figure 3, so that the horizontal direction in Figure 4 now corresponds to the sum of skills plus challenges (apathy vs. flow), while the vertical direction corresponds to the difference of skills minus challenges (boredom vs. anxiety). The southwest to northeast direction corresponds to skills (control vs. worry), while the northwest to southeast direction corresponds to challenges (arousal vs. relaxation). The four channel model thus proposes four bipolar constructs which lie in a two dimensional space, and where the space is spanned by orthogonal vectors for skills and challenges.
10
S kills - Challenges
S
ki
lls
BOREDOM
CONTROL
RELAXATION
APATHY
S kills + Challenges
FLOW
WORRY
AROUSAL
C A N X IE T Y
ha
ll
en
ge
s
Figure 4 - Eight Channel Flow Model
Massimini and Carli (1988) reported mean ratings on 23 scales, by the eight segments shown in Figure 4, for a sample of 47 Italian students between 16 and 19 years of age. By reanalyzing Massimini & Carli's data, we can demonstrate clear support for the 8 channel model. A principal components analysis of the means of the 23 items on the 8 flow channels identifies two factors which explain 89% of the variance. In Figure 5 we plot means of the 8 flow channels on the two factor scores, and represent (most of) the 23 items in the plot according to their correlations with the factors.
Figure 5 - Reanalysis of Massimini & Carli (1988) The plot is extremely interesting. First, we notice that the pattern of means of the eight flow channels is
11
virtually identical to the conceptual diagram of the eight flow channel model presented in Figure 4. Second, we can visually see that all items point to the right (flow) side of the map. There is also a clear differentiation between items that point toward the top right (control) and the bottom right (arousal).
2) Description of Empirical Research Questionnaire Development and Data Collection Our objective in our empirical research was to develop a comprehensive instrument for measuring the constructs discussed in the conceptual model from Hoffman and Novak (1996). The ultimate objective of this instrument was twofold; first, to provide data on skills and challenges of Web users that could be used to validate the eight-channel flow channel model and relate flow to consumer behavior on the Web (our current paper), and second, to empirically test the conceptual model in Hoffman and Novak (1996) (subsequent research). The process of questionnaire development involved a) construction of a preliminary questionnaire based upon a literature review; b) a series of four pretests; and c) the final field survey. We describe each in turn. Preliminary Questionnaire. Based upon the 16 studies in Tables 1 and 2, a comprehensive list of 100 items were compiled that have been used to measure the following constructs: flow, skill, challenge, control, attention focus, concentration, telepresence, curiosity, intrinsic interest, extrinsic interest, play/fun, affect, activation, ease of use, self-reinforcement, interactivity, self esteem, outer structure of task, and performance. Pretesting. A series of pretests were conducted over a four month period, which drew from and refined the items from the preliminary questionnaire. The major objective of the pretests was to refine items used to measure skills and challenges for using the Web, plus constructs measuring flow, play, and the eight channel flow model. The four pretests are briefly described below: Pretest 1.
(n=49, in-person 22 item written questionnaire administered in person to MBA students at a top 25 business school, December 1996). Subjects rated seven-point semantic differential scales describing how they usually feel when using the Web for “in flow vs. apathetic,” “anxiety vs. bored,” “in control vs. worried,” and “excited vs. relaxed.” Subject ratings for skills and challenges for using the Web on 9 point rating scales were also obtained, both from an absolute perspective (none to very high) and from a relative perspective (how much more or less compared with other activities?). From Figure 4, we can predict that skill should positively correlate with flow and control, and negatively correlate with anxiety, while challenge should positive correlated with flow, arousal, and anxiety. The standardized regression coefficients summarized in Table 3 show that the predictions were empirically supported for skills, but not for challenges.
12
Table 3 - Pretest One Results
Flow Channel: Flow Control Arousal Anxiety
Hypothesis: Skill + + -
Challenge + + +
Empirical (Absolute): Skill Challenge
Empirical (Relative): Skill Challenge
.22 .44* -.02 -.20
.42* .43* .07 -.01
-.17 .01 .33 -.05
-.04 -.23 .18 .12
*p value for standardized beta < .05
To explain the poor performance of the challenge scales, we examined two open ended questions: “What makes you skilled at using the Web?” and “What makes the Web challenging.” We found that the skill item was being interpreted as we desired (frequency of use, understanding search engines, experiences, curiosity, computer at home, frequency, experimentation, trial and error, practice, prior knowledge of computers, etc.). However, the challenge item was interpreted in a very negative manner. Rather than being interpreted as a stimulating cognitive task, the challenge of using the Web was described by phrases such as: loading time too slow, browser crashes, too much information, Web is saturated, wait time for connection, computer freezes, dial up busy signals, knowing the correct key words, etc. In subsequent pretests, we were careful to develop items for challenge that specifically captured the construct we were seeking to measure. It was clear from the first pretest, that unlike typical flow studies employing the ESM that have measured challenge with a simple single item rating scale asking for a direct assessment of the challenge of the activity, this would not be possible when measuring the challenge of the Web. Pretest 2.
(n=108, 59 item Web fillout form1 administered to Project 2000 Pretest Panel, January 1997). Based upon the results from Pretest 1, 15 items dealing with skills and 15 items dealing with challenges of using the Web were constructed, plus additional items for barriers to using the Web, dimensions underlying the eight channel flow model, and correlates of flow.
Pretest 3.
(n=86, in-person 59 item written questionnaire administered in person to MBA students at a top 25 business school, February 1997). Based upon item analyses of Pretest 2, skill and challenge items were further modified, and a list of 14 each were included in the questionnaire. In addition, items were included for playfulness, dimensions underlying the eight channel flow model, and correlates of flow.
Pretest 4.
(n=146, 78 item Web fillout form2 Project 2000 Pretest Panel, March 1997). Based upon item analyses of Pretest 3, skill and challenge items were further modified, and reduced sets of 7 items for each were included in the questionnaire. In addition, items were included for playfulness, dimensions underlying the eight channel flow model, and correlates of flow.
1 2
http://www2000.ogsm.vanderbilt.edu/cgi-bin/SurveyArchive/pretest.jan97.pl http://www2000.ogsm.vanderbilt.edu/cgi-bin/SurveyArchive/pretest.march97.pl 13
Final Survey. The final survey consisted of 77 items, and was administered as a Web fillout form3 which was posted from April 10 to May 10, 1997 in conjunction with the 7th WWW User Survey fielded by the Graphic, Visualization, and Usability Center (GVU) at the Georgia Institute of Technology4. Respondents who registered to participate in the 7th WWW User Survey were given a unique identifying code, and were presented with an online list of 13 different surveys, including our survey, which they could potentially fill out. With the exception of our survey, which was posted on Project 2000 Web server (www2000.ogsm.vanderbilt.edu), all WWW User Surveys were posted on the GVU server. However, a respondent’s unique identifying code was passed back and forth between the GVU and Project 2000 servers, as a hidden field in the fillout form. Thus, depending upon the overlap in respondents who filled out each of the 13 surveys, the Project 2000 survey can be linked to the 12 other surveys fielded by GVU. In this paper, we combine our data with items from one of these 12 surveys, the “Information Gathering and Purchasing Questionnaire5” developed by Sunil Gupta. The GVU WWW User Survey employs non-probabilistic sampling and self-selection (GVU 1997), and is not representative of the general population of Web users. Comparison with population projectable surveys of Web Usage (e.g. Hoffman, Kalsbeek and Novak 1997) shows the GVU User Survey sample to contain more long-term, sophisticated Web users than the general population. Participants were solicited by announcements placed on Internet-related newsgroups, banner ads placed on specific pages on high exposure sites (e.g. Yahoo, Netscape, etc.), banner ads randomly rotated through high exposure sites (e.g. Webcrawler, etc.), announcements made to the www-surveying mailing list maintained by GVU, and announcements made in the popular media. 19,970 respondents filled out at least one of the 13 surveys that comprised the 7th WWW User Survey, and 4,550 filled out the Project 2000 survey. Of the 4,550 respondents, we retained 4,232 for analysis purposes, eliminating 318 respondents who: • • • •
had more than 5 of 77 items missing (135 respondents) had constant responses throughout (i.e. all 1's or all 7's, only 2 respondents) did not have a valid GVU identifying code (17 respondents) had strong evidence of responding randomly to survey items by either a) responding with all 5's to one of two sets of semantic differential items, or b) by having large variability (top 5%) on summed rating skills for flow, play, and challenge (164 respondents).
3
http://www2000.ogsm.vanderbilt.edu/gvusurvey/project2000.gvu.html http://www.gvu.gatech.edu/user_surveys/survey-1997-04/ 5 http://www.gvu.gatech.edu/user_surveys/survey-1997-04/questions/purchase.html 4
14
3) Scale Construction and Reliability Tables 4, 5, and 6 provide coefficient alphas and item-total correlations for scales for skill, challenge and play. All items in these Tables were measured on 9 point scales, and items whose scale values were reversed for the purpose of combining summed rating scales are noted. The items for the skill and challenge scales were developed from the series of pretests, while the play items are from Webster and Martocchio’s (1992) scale of computer playfulness. Coefficient alphas are above .8 for all scales reported in these Tables. For skill and challenge, coefficient alpha was improved by reducing the seven item scales from 7 items to 5 and 4 items, respectively. Although coefficient alpha was reduced slightly by eliminating two items from the 7 item play scale, a principal components analysis of the 7 play items produced two factors with eigenvalues greater than one. Playful and spontaneous were the two items which correlated the most with the second rotated factor, and the least with the first rotated factor. When these two were eliminated, only one eigenvalue was greater than one for the remaining five items. For all full 7-item and reduced-item skill and challenge scales in Tables 4 and 5, only one eigenvalue greater than one was found. In subsequent analyses reported in this paper, we use five item scales for skill and play, and a four item scale for challenge.
Table 4: Scale construction for Skill
Skill item: I am very skilled at using the Web. I consider myself knowledgeable about good search techniques on the Web. I know less about using the Web than most users. (reversed) I find the Web easy to use. I know how to find what I want with a search engine. Downloading software is easy for me to do on the Web. It is hard for me to find information on the Web. (reversed) coefficient alpha
item-total correlations: 7 item 6 item 5 item scale scale scale .708 .742 .736 .786 .794 .802 .668 .694 .686 .680 .650 .633 .680 .645 .664 .567 .587 .428 .865
.875
.875
Table 5: Scale construction for Challenge
Challenge item: Using the Web challenges me. Using the Web challenges me to perform to the best of my ability. Using the Web provides a good test of my skills. I find that using the Web stretches my capabilities to the limits. The Web provides many opportunities for action Using the Web makes me think. The Web provides many possible things for me to do. coefficient alpha
15
item-total correlations: 7 item 6 item 5 item scale scale scale .745 .760 .759 .728 .726 .708 .742 .758 .768 .659 .672 .692 .564 .528 .506 .546 .519 .514 .868
.865
.865
4 item scale .772 .682 .789 .695
.876
Table 6: Scale construction for Play item-total correlations: 7 item 6 item 5 item scale scale scale .549 .585 .611 .572 .550 .507 .686 .652 .602 .622 .645 .660 .604 .640 .670 .548 .495 .454
Play item: I feel unimaginative when I use the Web. (reversed) I feel flexible when I use the Web. I feel creative when I use the Web. I feel unoriginal when I use the Web. (reversed) I feel uninventive when I use the Web. (reversed) I feel spontaneous when I use the Web. I feel playful when I use the Web. coefficient alpha
.829
.825
.818
Table 7 provides item-total correlations and coefficient alphas for the four constructs underlying the eightchannel flow model in Figure 4. These items were measured as nine-point semantic differential scales, and were developed over the course of the four pretests. There are four scales, one for each of the following bipolar constructs: flow/apathy; control/worry, arousal/relaxation, and anxiety/boredom. One item in each of these pairs is simply a rephrasing of these adjective pairs (excited was used instead of “aroused” because of the possible sexual connotation of “aroused,” given the media publicity about the extent of pornography on the Internet (Hoffman and Novak, 1995)). With the exception of the arousal scale, and the three-item anxiety scale, coefficient alphas are above .6 and acceptable for all scales. In subsequent analyses, we use the two-item anxiety scale. For simplicity, in the following discussion, we refer to bipolar scales (e.g. “Flow/Apathy”) by the first name of the pair (e.g. “Flow”).
16
Table 7: Scale construction for Flow, Control, Arousal, and Anxiety
Flow/Apathy items (alpha=.722): “in flow” vs. apathetic active vs. passive the Web challenges my capabilities to their limits vs. I don’t use the Web much and don’t care to
item-total correlations: .606 .524 .523
Control/Worry items (alpha=.664): worried vs. in control (reversed) clearly know the right things to do vs. feel confused about what to do frustrated vs. not frustrated (reversed)
.482 .442 .527
Arousal/Relaxation items (alpha=.541): calm vs. excited (reversed) stimulated vs. relaxed alert vs. soothed
.316 .422 .307
Anxiety/Boredom items (alpha=.564): using the Web is boring vs. using the Web makes me anxious (reversed) I’ve lost interest in using the Web lately because I’m too skilled vs. I’d enjoy using the Web more if I were more skilled at it (reversed) when I encounter a problem using the Web, I get stuck because I don’t know what to do next, vs. the Web isn’t as challenging to me as it used to be Anxiety/Boredom items, reduced scale (alpha = .689) I’ve lost interest in using the Web lately because I’m too skilled vs. I’d enjoy using the Web more if I were more skilled at it (reversed) when I encounter a problem using the Web, I get stuck because I don’t know what to do next, vs. the Web isn’t as challenging to me as it used to be
.174 .546 .472
.533 .533
4) Congruence of Constructs With Eight-Channel Flow Structure Figure 4 identifies the sum and difference of skills and challenges as central to the distinction between flow/apathy and the orthogonal construct of anxiety/boredom. Table 8 presents correlations of constructs underlying the eight channel model in Figure 4. Sample sizes for the correlations range from 4088 to 4164, depending upon the pattern of missing data. Due to the large sample sizes, all correlations in Figure 4 are significantly different from zero at p $50 other Web services
1.303*** 1.368*** 1.268*** 1.344*** 1.202***
Group B: home electronics < $50 home electronics > $50 videos/movies CDs/tapes/albums books/magazines concerts/plays Group C: legal services food/condiments/beverages investment choices travel arrangements apparel/clothing/shoes sunglasses/personal items real estate insurance services automobiles/motorcycles jewelry/watches household appliances banking/financial services
Startuse
Online Purchase: Skill Chall.
Startuse
1.095** 1.051 1.095** 1.085* 1.180***
1.368*** 1.607*** 1.283*** 1.417*** 1.072
1.347*** 1.289*** 1.230*** 1.251*** 1.203***
1.075** 1.073* 1.065* 1.077* 1.181***
1.406*** 1.495*** 1.260*** 1.374*** 1.108*
1.141** 1.173*** 1.151*** 1.221*** 1.279*** 1.183***
1.090** 1.078** 1.085** 1.058* 1.090** 1.079**
1.291*** 1.356*** 1.184*** 1.182*** 1.154* 1.187**
1.259* 1.244* 1.145* 1.268*** 1.198*** 1.189*
1.167** 1.220*** 1.219*** 1.096** 1.107*** 1.093*
1.428** 1.454*** 1.257* 1.205** 1.301*** 1.157
1.085 1.102* 1.104* 1.117** 1.069 1.144* 1.084 1.064 1.112* 1.085 1.003 1.139**
1.169*** 1.131*** 1.088** 1.110*** 1.147*** 1.218*** 1.115*** 1.157*** 1.049* 1.211*** 1.119*** 1.084**
1.088 .977 1.264*** 1.228*** 1.145* 1.103 1.143* 1.209* 1.135* 1.125 1.365*** 1.237***
.975 .977 1.066 1.115* 1.196* .903
1.256* 1.228** 1.076* 1.113*** 1.096* 1.212*
1.052 1.331* 1.353*** 1.339*** 1.043 1.311*
*** p $50 videos/movies CDs/tapes/albums books/magazines concerts/plays
1.024 1.041 1.041 1.013 .989 1.010
1.062* 1.002 1.108 1.106 1.021 .975
.987 1.046 .967 1.005 1.035 1.058
1.092 1.112* .970 1.046 .933 1.035
1.127*** 1.065* 1.133*** 1.106*** 1.126*** 1.081
1.128* 1.111 1.017 .940 .990 .978
Group C: legal services food/condiments/beverages investment choices travel arrangements apparel/clothing/shoes sunglasses/personal items real estate insurance services automobiles/motorcycles jewelry/watches household appliances banking/financial services
1.058 .935* 1.015 .960 .895* .923* .936* .935 .959 .944 .906* .968
1.099** 1.087** 1.016 1.045* 1.105*** 1.148*** 1.093*** 1.133*** 1.030 1.158*** 1.095*** 1.071*
.977 .956 1.101* 1.047 .995 1.001 1.080 1.066 1.012 .914 1.022 1.123*
.974 .988 1.012 .905 .937 .938
1.348** 1.143** 1.202* 1.251*** 1.110*** 1.202***
.778 .967 1.115 1.059 .994 .985
*** p $50 videos/movies CDs/tapes/albums books/magazines concerts/plays
.980 .995 .965 1.004 .930 .964
1.095** 1.069* 1.159*** 1.154*** 1.164*** 1.119*
.954 .971 .885* .908* .989 .992
1.067 1.062 1.033 1.167* 1.149* 1.068
1.030 1.055* 1.003 .939* .955 .956*
1.096* 1.032 .967 .971 1.005 1.125*
1.206** 1.089* 1.139*** 1.209*** 1.122*** 1.257*** 1.254*** 1.185*** 1.176*** 1.247*** 1.212*** 1.157***
.898 .915 .934 1.003 .999 .972 .929 .965 .946 .837* .959 1.010
1.134* .914 .961 1.020 1.009 .972
1.111* .942* 1.047 .990 1.009 1.036
.990 .985 1.084 1.136* .873* .886*
Group C: legal services food/condiments/beverages investment choices travel arrangements apparel/clothing/shoes sunglasses/personal items real estate insurance services automobiles/motorcycles jewelry/watches household appliances banking/financial services
.884* .920 .942 .907* .911* .898* .877* .875* .912 .925 .898* .905*
*** p
