APIEMS2009
Dec. 14-16, Kitakyushu
E-Learning Usability Measurement－Using Technology Acceptance Model and Usability Test Wen-Chih Chiou † Department of Business Administration National Chin-Yi University of Technology, Taichung, 411, Taiwan Email:
[email protected] Chin-Chao Lin Department of Industrial Engineering and Management Hsiuping Institute of Technology, Taichung, 412, Taiwan Email:
[email protected] Chyuan Perng 1 and Jeng-Teng Tsai 2 Department of Industrial Engineering and Enterprise Information Tunghai University, Taichung, 407, Taiwan
Email:
[email protected] 1
[email protected] 2
Abstract. With the convenience of wireless network and Web page browsing, Personal Digital Assistant (PDA) merging with E-learning has become a new trend for mobile learning. However, the existing E-learning interface is designed for personal computer not a customized interface for portable device. In this study, we recruited 30 management college students for using PDA on campus with pre-specified E-learning tasks. While the participants performed a task, the clicks on web page linkage and eye browsing track were recorded for measuring the performance. At the end of test, the participants were required to fill out a Technology Acceptance Model (TAM) questionnaire for measuring the degree of acceptance level. The study finding shows that the participant’s different experience of using PDA will result in an inconsistency between the belief of technology acceptance and the performance of task execution. The reason can be explained by tester’s different recognition toward a general purpose task and a specific task measurement scope. The existing researches seldom studied on the measurement of usability test and TAM with a specific task execution. As a result, future research may continue to explore and validate the relationship between the belief of technology acceptance and the attributes of usability. Moreover, the list of usability problems of PDA operation, browser operation, and E-learning interface will be valuable reference for practitioner while developing a system interface. Keywords: Mobile learning, technology acceptance model, mobile usability test, personal digital assistance. user’s acceptance of information technology, many researches used two important beliefs (Perceived usefulness, PU, and Perceived ease of use, PEU) of TAM to interpret, diagnose, and forecast user’s attitude, intention, and behavior as toward an information technology（Davis， 1989; Venkatesh & Davis，1996; Calisir & Calisir，2004; Saade & Bahli，2005; Yi et al.，2006; Wu et al.，2007) The study of TAM usually conducts a questionnaire survey to measure user’s subjective perception of using information technology. On the other hand, usability is
1. INTRODUCTION The combination of E-learning with traditional class room teaching provides a personalized learning environment in response to individual capability and various learning context. With the prevalence of wireless communication device and network technology, users are now able to use hand held to process text, voice, or image transformation. It not only brings the convenience to our daily life, but also facilitates E-learning success. To study
558
APIEMS2009
Dec. 14-16, Kitakyushu
typically measured by having a group of test users use an information technology to perform a prespecified set of tasks for assessing the degree of usability (Nielsen，1993). As suggested by Calisir & Calisir (2004) and Henderson & Divett ( 2 0 0 3 ) , the relationship between respondent’s subjective perception of PU and PEU and usability test participant’s objective attributes of a task manipulation should be further explored and compared. Moreover, since a PDA browsing interface for E-learning is still not available in Taiwan, the issue of E-learning usability should be addressed. For the reason, this study recruited participants to perform E-learning task with a PDA. The collected data were compared and measured for the result consistency with a combination approach of TAM and mobile usability test. In additions, the list of usability problems was identified as a valuable reference for practitioner while developing a system interface.
2. LITERATURE REVIEW 2.1 TAM TAM was proposed by Davis (1989) and is depicted in Fig u r e 1. The model indicates external factors will influence user’s PU and PEU (Venkatesh & Davis, 1996; Calisir & Calisir , 2004). External factors include user experience and skill. PU refers to “the degree to which a person believes that using a particular system would enhance his or her job performance”. PEU refers to “the degree to which a person believes that using a particular system would be free of effort”. Research findings show that PU and PEU have a positive impact on attitude and PEU has a positive impact on PU (Wu et al., 2007). Davis et al. (1989) asserted that PU also has a direct impact on intention to use. However, Bruner ＆ Kumar’s (2005) study of portable device acceptance argued that PU has no direct relationship with intention to use. The findings indicated that user perceived convenience of daily life with portable device, but not enough to encourage them to use it. As a matter of fact, attitude toward using has a positive impact on intention to use and intention to use has a positive impact on actual using behavior. Perceived usefulnes External factor
Attitude
Intention
Actual use
Perceived ease of
Figure 1: Technology acceptance model
559
Based on above discussion, this research adopted the TAM questionnaires to measure a participant’s perception, attitude toward to use, and intention to use of PDA while conducting E-learning tasks. Due to the facts that this research is a cross-sectional study and the mobile learning platform is not practically available in learning environment, it is not possible to measure the actual using behavior. The questionnaires were mainly developed by following the base scale of Davis (1989). Moreover, the sales were modified by referring some researches of digital learning (Fu & Chiou, 2004; Saade & Bahli, 2005) and portable device (Lo, 2004; Shih，2005; Bruner & Kumar, 2005; Yi et al., 2006).
2.2 Mobile Usability Test Usability is used to assess how easy user interfaces are to use. Even end-use is not familiar with the system, he or she is able to quickly and pleasantly perform tasks with minimum errors and effective learnablity (Nielsen，1993). Lee et al. (2006) used a Task-based performance method to measure the efficiency and effectiveness of information technology. The study was conducted by giving test users prespecified tasks and analyzing the usability attributes data which were collected during the experiment. The most frequently used usability attributes and criteria include: efficiency (time to complete task) (Killi, 2002; Kjeldskov & Stage, 2004; Lee et al.，2006), effectiveness (task completed ratio) (Zhang & Adipat, 2005), memorability (browsing and searching time for non-regular user) (Zhang & Adipat , 2005 ) , and learnability (number of errors occurred for novice) (Killi, 2002; Lee et al., 2006）. The reviewed studies showed that a combination approach of using multiple usability measurement methods attained more diversified information. For instance, Nielsen (1993) and Lee et al. (2006) used observation method to collect qualitative data and used performance measurement method to collect quantitative data. The synthesis of qualitative and quantitative data was further analyzed to interpret user’s thinking and perception toward system usability. A summary of mobile usability related studies are listed in Table 1. The most common methods are identified as Think-Aloud, Observation, Interviews, Questionnaire Survey, and Performance Measurement. In recent years, eye-tracking is adopted for facilitating usability test by observing the point of gaze at screen and eye movement path (Chang, 2002; Russell, 2005). Eyetracking is the process of observing and recording eye movement for attaining human’s complex visual information. The eye movement record is then analyzed to identify user’s browsing sequence and calculate the fixation time. Two prevalent methods of eye-tracking analysis are fixation and saccade. A fixation is maintaining the gaze in a
APIEMS2009
Dec. 14-16, Kitakyushu
constant direction or slow eye movement around 200-400 milliseconds (ms). On contrary, a saccade is a rapid eye movement between different gazing points, which normally takes about 100-200 ms. In most studies, researchers used visual time and fixation frequency to analyze a tester’s browsing process. The studies show that the total fixation frequency is negatively related to visual searching effectiveness (Tang & Jhuang, 2 0 0 5 ; H u n g , 2 0 0 5 ;
Russell，2005; Sun et al.，2006). Equally, the higher fixation frequency will decrease tester’s visual searching effectiveness. The longer total fixation time represents the interface design is difficult for learning use. As a consequence, user faces the problem of finding target object or comprehending received information (Russell， 2005).
Table 1: Usability test related studies Items Authors (years) Kiili（2002） Öquist & Goldstein（2003） Gulliver et al. (2004) Kjeldskov & Stage（2004） Reuss et al.（2004） Kaikkonen et al.(2005） Kjeldskov et al. (2005) Kushniruk et al. (2005)
Portable devices
Goals／Missions
Testing Place
Functionalities of mobile phone Readability of small PDA screen’s text Visual quality of multiPDA media on PDA PDA and Portable device’s Mobile phone problems Usability of electronic PDA medical record Comparison of testing on Mobile phone laboratory and real environment Usability of mobile PDA navigation Usability of medical PDA hand-held application
Mobile Phone
Laboratory Laboratory Laboratory Laboratory, Real environment Real environment
Observation, Performance measurement Questionnaire, Performance measurement Questionnaire, Performance measurement Think-Aloud, Observation, Questionnaire Think-Aloud, Observation, Interview, Questionnaire
Real environment, Think-Aloud, Observation, Laboratory Laboratory, Real environment
Think-Aloud, Observation, Interview
Laboratory
Think-Aloud, Observation
Zhang & Adipat (2005)
Portable device
Mobile usability test framework
Laboratory, Real environment
Lee et al. (2006)
Mobile phone
Usability of mobile phone
Laboratory
As a portable device’s screen is quite small, it is difficult for video camera to precisely record user’s looking behavior. Thus, this study uses video camera to record tester’s behavior with think-aloud and eye-tracker to record fixation time and frequency while using portable device. In addition, an interview is conducted after the test to clarify the questions which are marked during the test period. The performance measurement criteria include four usability attributes: efficiency, effectiveness, memorability, and learnability. These criteria are used to evaluate PDA’s usability when tester performs requested tasks. Moreover, since testing place has great impact on mobile usability test result, this study arranges tester in real environment to use portable device for performing required tasks. Another key issue of conducting usability is the selection of tester (Kjeldskov & Stage, 2004). For
Methods／Instruments
Think-Aloud, Observation, Interview, Questionnaire, Performance measurement Think-Aloud, Observation, Interview, Questionnaire, Performance measurement
learnability measurement purpose, tester should be selected from new experienced users. On the other hand, memorability test should select non-regular user as participant to measure system using performance again after a period of time. As a consequence, this study selects two types of tester ( n on-regular user and novice) to participating usability test. The corresponded relationship between perceived perceptions of technology acceptance and usability attributes is summarized in Table 2. PU is used to measure the degree of subjective perception which using a particular system would enhance overall performance, productivity, or effectiveness. The corresponded usability attributes which include efficiency and effectiveness can be used to measure the performance of real using a particular system for executing specified tasks. PEU is another measurement
560
APIEMS2009
Dec. 14-16, Kitakyushu
factor of technology acceptance. It measures the degree of subjective perception which using a particular system would reduce user’s effort to complete a task. The corresponded usability attributes which include memorability and learnability can be used to measure the performance of real using a particular system for executing specified tasks. Table 2: The correlated relationship between perceived perception of technology acceptance and usability. Correlated relationship
PU
PEU
Description Similarity: measurement of performance or effectiveness. Usability attributes Difference: 1. Subjective general evaluation compared with (efficiency and objective performance evaluation effectiveness) of actual using system. 2. General purpose compared with particular task. Similarity: measurement of effort reduced and ease of use and learning. Usability attributes Difference: 1. Subjective general (memorability and evaluation compared with objective performance evaluation learnability) of actual using system. 2. General purpose compared with particular task.
2.3 E-Campus and Mobile Learning
assigned tasks purpose. As suggested by the name, mobile learning is a sort of digital learning that involves mobile technologies. It makes learner attain learning purpose through a portable device. Kung et al. (2005) considered mobile learning is one type of digital learning. It decreases limitations of learning location and time by connecting learning resources with a wireless network. Hsiao et al. (2005) and Chan et al. (2006) pointed out some advantages of mobile learning included: portable device’s small size and light weight, unlimited teaching scenario, convenient information, quick information exchange, digital handwriting function, digital camera, and ease of collaborative learning. Regarding the selection of portable device, Chou (2005) suggested PDA because it can be applied in various scenarios and has many advantages, which include light and small, broad application, digitalized data, wireless transformation, stable system, and variety selection of accessories. Wang (2006) conducted a research of integrating mobile learning into class teaching. The study concluded that PDA is more appropriate for mobile learning in different teaching applications compared to Table PC. Furthermore, a study of portable device acceptance (Bruner and Kumar, 2005) showed that PDA is easier to use than other portable devices. In addition, users are more interested in using PDA than other portable devices. Based on above discussion, this study selects PDA as experiment’s portable device.
3. RESEARCH DESIGN This study design a survey of E-Campus using experience and will for joining usability test. The target respondents are undergraduate student of Management College at National Chin-Yi University of Technology. A total of 1525 questionnaires were given out, and 929 complete responses were received. After deducting 49 invalid responses, we have a total of 880 valid responses and 448 respondents were willing to join usability test. The intended participants’ attributes are described in Table 3. Table 3: PDA and E-campus using experience of Intended participants
Considering the selection of participants and the convenience of testing place, this study used the E-Campus learning system of Management College at National ChinYi University of Technology as a testing system. The system’s interface is designed for the browser of desk-top computer. The browsing interface for PDA is currently not available. The system’s functions of student side include three categories: Personalized Data, Personalized Courses, and Help Desk. As a student login the system, he or she will be able to browse this semester’s course introduction and content under the Personalized Course. Course introduction presents basic information, including course name, text book, reference, course schedule, and instructor’s name. The course content, on the other hand, has two functions which are document and discussion. Document function provides instructor’s handbook and related reference. Discussion function enable student to propose a question to instructor for further discussion. In order to conduct our designed experiment, some virtual courses are created on E-Campus system. Participants are assigned as auditors for login and executing
Type
Using Experience
Yes No(Novice) Total Regular User E-Campus Yes Non-regular User using No experience Total
PDA using experience
561
Number
%
36 412 448 103 170 139 412
8 92 100 25 41 34 100
APIEMS2009
Dec. 14-16, Kitakyushu
Since 92% of intended participants do not have using PDA experience, we selected 412 respondents who do not have PDA experience as candidate participants. Among these candidates, 15 of them (6 male and 9 female) were selected from 170 non-regular users. Non-regular users include those respondents who “never”, “seldom”, and “occasionally” use E-Campus within a recent month. Another 15 testers (8 male and 7 female) were selected from 139 novices who do not have any E-campus experience. The participants used a PDA (HP iPAQ hx4700 with window Mobile 2003 and Netfront v3.3 browser) with campus wireless network at the rest area of Management College to conduct a set of E-learning tasks. The learning tasks consist of two stages. In stage one, participants login
an E-learning course (Mobile usability experiment) and read the course’s syllabus. In stage two, participants submit a question to a course lecturer after reading the syllabus. During the testing process, participants are encouraged to use thinking aloud method as they are performing a set of specified tasks. A video camera is also set for recording participant’s operation process and behaviors. Additionally, an eye tracker system is installed for recording participant’s pupil gazing time and frequency. Observers also made some marks on these records when a participant executes tasks having problem. The marked points will be discussed with participants after the test. The test facilities layout and PDA login screen are shown in Figure 2 and 3. Major expected browsing web pages and linking point’s operation processes are listed in Table 4.
Figure 2: Test facilities layout
Figure 3: PDA login screen
Table 4: Main expected browsing web pages / Linking point’s operation processes Tasks
Main operation processes
Note
Stage one: Login “Mobile usability experiment” course and read syllabus
Login > Current semester’s course > Announcement > Content >Course teaching outline > Document content list > Open new web page > Prepare for reading document > Confirm > Save path> Select folder > Complete page > Prepare for reading document > Close web page > Start program > File management > View course outline > Confirmed Start program > Open a browser > Prepare for reading document > Content > Course teaching outline > Reply > Key in question > Confirmed
Expected browsing web pages: 21
Stage two: Submit a question to a course lecturer after reading the syllabus.
The testing time for each participant is about one hour. The first 30 minutes is experiment preparation including check in, agreement signature, think-aloud practice, task scenario explanation, and device tune up. The next ten minutes is task execution time. After the test, participants spend another 20 minutes to fill out a questionnaire and receive an interview. At the end of test, each participant receives NT $100 as a reward. The experiment was
Expected linking points: 24
conducted between 2008/05/26 and 2008/06/30. The measuring variables include using experience, using experience, PEU, PU, attitude, intention, and usability attributes (Learnability, Memorability, Effectiveness, and Efficiency). These variable definitions and measuring methods are listed in Table 5.
4. DATA ANALYSIS
562
APIEMS2009
Dec. 14-16, Kitakyushu
shows the average PU score of non-regular users is significantly lower than that of novice. However, the average PEU score does not attain significant level. The reason for novices with higher PU score can be referred as novice is curious in mobile learning or has higher expectation of mobile learning. On the other hand, the reason for non-regular user with lower PU and PEU score can be explained by bad using experience that results in low PDA use frequency.
4.1 TAM Measurement Result The reliability test result shows that the Cronbach α of PU, PEU, and intention to use are all greater than 0.70. Another Cronbach α of attitude toward to use is between 0.69 and 0.35. All the values of Cronbach α are attained acceptance level. To understand using experience’s impact on user perception, a t-test was conducted to examine the hypotheses and the findings are show in Table 6. The result
Table 5: Variable definitions and measurement Class
Variables
Definitions
Measuring Methods
External Factors
Using experience
Non-regular user and novice
Questionnaire, nominal scale
TAM
PU, PUE, attitude, intention
Usability
1. The time required for accomplishing a task 1. Shorter complete time will have (Classification of task accomplished: Type A higher efficiency. represents two tasks are not complete, Type B Efficiency 2. More fixation time and higher represents one task is complete, Type C frequency will have lower represents two tasks are complete). efficiency. 2. The total fixation time and frequency during the task executing period. 1. Task accomplished ratio = accomplished ratio * accuracy ratio 1. Higher complete ratio will have accomplished ratio = actual browsing page higher effectiveness. number / total expected browsing page Effectiveness 2. Less fixation time and lower numbers frequency will have better accuracy ratio = actual click number / total effectiveness expected click on link 2. The total fixation time and frequency of correct browsing web page. 1. Number of errors includes clinking on the 1. Less number of errors will have wrong link within a expected browsing page higher learnability. and clicking on the wrong link within a wrong Learnability 2. More fixation time and higher browsing web page. frequency will have lower 2. The total fixation time and frequency of wrong learnability. browsing web page. 1. Shorter searching time will have 1. Searching time: searching for expected visiting higher memorability. web page to click on target link. Memorability (Non2. Less fixation time and lower 2. The total fixation time and frequency of regular user) frequency will have higher searching expected visiting web page for memorability. clicking on target link.
Questionnaire, Likert 5 points scale
Table 6: Using experience’s impact on perception Variable PU PEU
Experience (n) non-regular (15) novice (15) non-regular (15) novice (15)
Average St. Dev. 3.70 3.66 3.73 4.08
0.81 0.66 0.62 0.60
Table 7: Correlation analysis of variables
t value PEU
-0.152
PU
2.385*
Attitude **：p
