written on bibliographic instruction using a computer as the method of delivery [2-7], but there is far less information on the use of the Web as a mechanism of.
BRIEF COMMUNICATIONS Implementation of a Web-based tutorial* By Jill E. Foust, M.L.S. Reference Librarian
Nancy Hrinya Tannery, M.L.S. Assistant Director for Information Services
Health Sciences Library System Falk Library of the Health Sciences University of Pittsburgh Pittsburgh, Pennsylvania Ellen Gay Detlefsen, D.L.I.S. Associate Professor
School of Information Sciences University of Pittsburgh Pittsburgh, Pennsylvania INTRODUCTION
While digital libraries provide users with information about library services and collections as well as access to commercial databases, the next likely step for digital libraries is the delivery of instructional services. One of the many responsibilities of reference librarians in academic health sciences libraries is user education. Traditional bibliographic instruction and large-scale programs of information management education are provided to faculty, staff, and students and are ongoing throughout the year. A part of many library skills sessions or orientations is instruction in the library's online public access catalog (OPAC). An early study of OPAC use states, "those who receive at least some initial training and assistance are more satisfied and successful than those who do not" [1]. Because the Web offers libraries an opportunity to develop and implement Web-based tutorials, a trial Web-based tutorial was developed to teach incoming students how to use the University of Pittsburgh's OPAC, known as PITTCAT. The use of PITTCAT has long been taught as part of the library orientation in the Health Sciences Library System (HSLS) at the University of Pittsburgh, but reference and user education staff were interested in devising and testing new methods for introducing and orienting users to PITTCAT. An experimental study was undertaken to determine if a PITTCAT Web-based tutorial could be used This paper is based on a presentation at the Ninety-eighth Annual Meeting of the Medical Library Association, Philadelphia, Pennsylvania, May 26, 1998.
*
Bull Med Libr Assoc 87(4) October 1999
as an effective alternative to the traditional library lecture. LITERATURE REVIEW
A review of the literature revealed that much has been written on bibliographic instruction using a computer as the method of delivery [2-7], but there is far less information on the use of the Web as a mechanism of providing instruction [8-11]. The literature does disclose several significant advantages and disadvantages in using Web-based tutorials as a method of learning. Advantages of Web-based instruction include the ability to teach remote users, those who lack the time or means to come to the library for instruction [12]. In addition, a Web-based tutorial can be updated from a single location and distributed quickly [13]. The nature of the Web allows for global access as well as the ability to develop a Web-based tutorial in one location and have it successfully used across campus or at other institutions [14]. On the other hand, there are disadvantages to using Web-based tutorials in library instruction. The Web format cannot reach users who lack Internet experience or access [15]. The absence of personal contact can be a problem for some users [16]. Web-based tutorials are not updated or refined as often as they should be. The response time on the Web appears to be slowing and computer laboratories often have poor ergonomics that discourage users seeking bibliographic instruction [17]. The disadvantages notwithstanding, a formal exploration of the use of a Web-based tutorial was necessary.
METHODOLOGY Tutorial A Web-based tutorial, developed by reference librarians in the Health Sciences Library System at the University of Pittsburgh, was designed to teach users how to access and use PITTCAT. The tutorial required access to the Internet and a Web browser. Images from the PITTCAT screen were captured and used in almost each section of the tutorial. The pages were linked to each other using HTML coding, allowing users to go back to a previous page and forward to the next page or to jump sections if needed. The tutorial explained, with examples, the three basic methods of PITTCAT use: title, author, and keyword searching. Test population Two different test populations were used in this study. In order to reach a "naive" audience, which would be likely to include individuals with little or no exposure to PITTCAT, the researchers included in the first test 477
Brief communications Table 1 Test scores by population MLIS students n = 67
Lecture # questions wrong #students 0 1 2 3 4 5 6 7 8 9 10
12 12 6 3 0 0 0 0 0 0 0
Governor's School n = 133
Tutorial # questions wrong #students 0 1 2 3 4 5 6 7 8 9 10
5 10 8 8 0 1 1 1 0 0 0
population some sixty-seven first-year master of library and information science (MLIS) degree students from the School of Information Sciences at the University of Pittsburgh. The study was conducted at the first meeting of their core class, "Organization of Information," in January 1998. The second test population included 150 high school juniors and seniors from the Pennsylvania Governor's School of the Health Sciences. The Governor's School, a six-week summer program, provides gifted and talented high school students the opportunity to decide if they are interested in pursuing careers in the health sciences. The second study was conducted in July 1998.
Study Students from each group were randomly divided into two sections; one section received a classroom lecture and the other viewed the Web-based tutorial in a laboratory setting. A reference librarian gave the classroom lecture session. As part of the lecture, the librarian demonstrated the use of PITTCAT by means of a projector connected to a computer terminal. The content of the lecture was identical to the Web tutorial, as the reference librarian memorized and delivered exactly the same material. The lecture lasted fifteen minutes. Students were allowed to take notes and ask questions of the reference librarian. The tutorial group was given fifteen minutes to view the Web tutorial. The students were allowed to take notes and a reference librarian was available to answer questions.
Posttest The posttest consisted of ten basic multiple choice questions. The test questions were taken from examples covered in both the lecture and tutorial. The questions on the test related to author, title, and keyword searching, as well as interpreting information on the 478
Lecture # questions wrong #students 0 1 2 3 4 5 6 7 8 9 10
Tutorial # questions wrong # students
6 18 16 10 8 3 1 1 1
0 1 2 3 4 5 6 7 8 9 10
1 0
3 8 18 21 8 7 0 1 2 0 0
screens such as volume, issue, and document location. The time allotted for taking the test was ten minutes. Students were permitted to use notes taken during the lecture and tutorial. Each test also contained a selfrating scale. Students used the self-rating scale to mark their level of expertise with PITTCAT, with a scale that ranged from novice to professional. RESULTS AND DISCUSSION As shown in Table 1, students from both the lecture and tutorial group missed, on average, between one and three questions on the posttest. This result was true for both test populations. The number of students who missed four or more posttest questions was lower for the MLIS test population than for the Governor's School group. Table 1 also shows that in the MLIS population of sixty-seven students, no one in the lecture group and three in the tutorial group missed four or more questions. The majority of students (64), therefore, answered at least seven of the ten questions correctly. If this test had been given on a pass/fail basis, 100% of the MLIS lecture group and 95.5% of the MLIS tutorial group would have passed. In the Governor's School test population of 133 students, 15 in the lecture group and 18 in the tutorial group missed four or more posttest questions. In this test population, the majority of students (100) also answered at least seven of the ten questions correctly. If this test had been given on a pass/ fail basis, 89% of the Governor's School lecture group and 87% of the Governor's School tutorial group would have passed. The Govemor's School test population was truly a naive group of users. None of these 133 students had ever used PITTCAT. This condition was not true of the MLIS test population. In both the tutorial and lecture groups, the vast majority of students considered themselves to be novice or intermediate users of PITTCAT. Bull Med Libr Assoc 87(4) October 1999
Brief communications
Only 16% of this test population thought of themselves as "professional" users of PITTCAT; a number of the MLIS group were in fact, university library system employees who were part-time MLIS students. They were already users of PITTCAT. The majority of the test population, however, consisted of naive users. The even distribution of levels of users was remarkable, considering both groups were randomly divided. The total number of posttest questions missed by the MLIS test population in the tutorial or lecture group according to their level of expertise was predictable. As expected, students in the novice and intermediate levels missed more total questions than students who rated themselves professional users. The lecture group missed fewer total questions (33) than the tutorial group (68). This result was also true for the Governor's School test population. The lecture group missed a total of 157 questions and the tutorial group missed a total of 197 questions. However, despite the fact that the participants missed questions, students from both groups clearly did learn how to use PITTCAT successfully. CONCLUSION
This paper describes the evaluation of a Web-based tutorial as a method of learning. While the students in the Web tutorial groups of both test populations missed more total questions on a posttest that was designed to determine their comprehension of the material than did students in the lecture group, this result does not devalue the use of the Web tutorial. This finding suggests that students are better able to retain and understand information presented in a traditional lecture style; however, the majority of the students in the tutorial groups of both test populations scored a passing grade on the posttest. Thus, these results imply that they would be able to search PITTCAT effectively. This study, therefore, indicates that Web-based tutorials can serve as an alternative to the more traditional lecture delivery presently used by reference librarians. This finding is extremely important to those who plan programs of bibliographic instruction and information management education. If the Web-based tutorial can do at least as well as a traditional classroom style lec-
Bull Med Libr Assoc 87(4) October 1999
ture, then delivery of Web-based instruction can be an adjunct, if not an outright replacement, form of user education. The advantages of the Web format, delivery to remote or asynchronous users, and the ease of updating content, make it an excellent tool for reference librarians in both large academic libraries and solo practitioner libraries. REFERENCES 1. THOMPSON DM, PASK J, PETERSON B, HAYNES E. Online public access catalogs and user education. RQ 1994 Winter; 34(2):198. 2. CHERRY JM, YUAN W, CLINTON M. Evaluating the effectiveness of a concept-based computer tutorial for OPAC users. Coil Res Libr 1994 Jul;55(4):355-64. 3. CHERRY JM, CLINTON M. An experimental investigation of two types of instruction for OPAC users. Can J Inf Sci 1991 Dec;16(4):2-22. 4. TURNER A. Computer-assisted instruction in academic libraries. J Acad Libr 1990 Jan;15(6):352-4. 5. CALABRETTA N, MIKITA E, WARNER E, BRYANT J, DEVLIN M, LAYNOR B. Watch your language: a CAI approach to teaching MeSH. Med Ref Serv Q 1990 Winter;9(4):1-16. 6. LAWSON VL. Using a computer-assisted instruction program to replace the traditional library tour: an experimental study. RQ 1989 Fail;29(1):71-9. 7. KAPLOWITZ J CONTINI J. Computer-assisted instruction: is it an option for bibliographic instruction in large undergraduate survey classes? Coil Res Libr 1998 Jan;59(1):19-27. 8. SCHOLZ AM, KERR RC, BROWN SK. PLUTO: interactive instruction on the Web. Coil Res Libr News 1996 Jun;57(6): 346-9. 9. VISHWANATHAM R, WILKINS W, JEVEC T. The Internet as a medium for online instruction. Coll Res Libr 1997 Sep; 58(5):433-44. 10. FRIEDMAN RB. Top ten reasons the World Wide Web may fail to change medical education. Acad Med 1996 Sep;71(9): 979-81. 11. SILVA M. Creating electronic environments for learning. Argus 1998 Spring;27(1):24-8. 12. VISHWANATHAM, op. cit. 13. SCHOLZ, op. cit. 14. VISHWANATHAM, op. cit. 15. IBID. 16. SCHOLZ, op. cit. 17. FRIEDMAN, op. cit
Received December 1998; accepted May 1999
479
