past reflections, future possibilities - Wiley Online Library

DOI: 10.1111/j.1471-1842.2007.00750.x

Measuring the value and impact of health library and information services: past reflections, future possibilities Blackwell Publishing Ltd

Joanne Gard Marshall, School of Information and Library Science, University of North Carolina at Chapel Hill, NC, USA

Abstract Objectives: To summarize the context, history and results of research studies conducted on the value and impact of health library and information services by the author since 1975 and to use this as a basis for examining ongoing developments related to evaluation research. To provide a comprehensive bibliography of library value and impact studies. Methods: Literature review and background based on personal involvement in the studies under discussion. Results: The author’s studies demonstrate an ongoing evolution of value and impact studies since the mid-1970s. In health sciences libraries, the approach taken to measuring value and impact has been strongly influenced by the type of research being conducted in the health sciences field as a whole. As a result, health sciences library researchers have become early adopters of methods that incorporate outcome and impact measures and rigorous research designs, and the concept of evidence-based library and information practice. The paper recommends that a range of research approaches from various disciplines be used to guide future evaluation research. Conclusions: Value and impact studies will continue to be important resources for evidence-based practice as health information professionals deal with evolving user needs and new ways of delivering information to a variety of audiences.

Introduction

The value of the clinical librarian

The need to demonstrate the value and impact of library and information services continues to be one of the major challenges for the library and information profession in the 21st century. This issue of Health Information and Libraries Journal presents a range of new articles on the topic as well as an opportunity for reflection. In this article, I will reflect on a body of work on value and impact that has engaged me since the mid-1970s and then link the lessons learned to future possibilities.

My own beginnings in exploring value and impact were very much influenced by the environment in which I found myself in the 1970s. I joined the staff of the Health Sciences Library at McMaster University in Hamilton, Ontario, Canada in 1970. McMaster was the newest medical school in Canada at the time and many educational and research leaders emerged from that environment including Drs John Evans, John Hamilton, Victor Neufeld, Christel Woodward, Susan French, Howard Barrows, Alexander Macpherson and David Sackett. Instead of lectures, students had tutorials and lots of selflearning modules. Patients and family members were considered members of multidisciplinary healthcare teams. There was a general feeling of innovation

Correspondence: Joanne Gard Marshall, Alumni Distinguished Professor, School of Information and Library Science, University of North Carolina at Chapel Hill, 301 Manning Hall, CB#3360, Chapel Hill, NC 27599-3360, USA. E-mail: [email protected]

4

© 2007 The author Journal compilation © 2007 Health Libraries Group. Health Information and Libraries Journal, 24 (Suppl. 1), pp.4–17

Measuring the value and impact of health libraries, Joanne Gard Marshall

and change in the air and the library played a key role in facilitating teaching, learning and research. In addition to its research and education activities, the McMaster Health Sciences Center included a 450-bed hospital and outpatient clinics. Fortunately the library director Beatrix Robinow encouraged her librarians to attend the Medical Library Association (MLA) annual meetings and it was there that I met Gertrude Lamb who was a pioneer in the development of clinical librarianship. When I returned to McMaster from MLA in 1975 it did not take long before I was able to begin a pilot programme in clinical librarianship in the Gastroenterology Program with the support of the director Dr John Hamilton. Our first publication from this period entitled ‘The clinical librarian and the patient: report of a project at McMaster University Medical Centre’1 described the role of the clinical librarian and its impact on patients, the health-care team and the librarian. I used this initial experience as a basis for applying to the Master of Health Sciences programme at McMaster in 1976 so that I could learn more about research methods and clinical practice. At the time, the Clinical Epidemiology Department, led by Dr David Sackett, was running a series of short courses for faculty and students called Critical Appraisal of the Literature. This training for clinicians in literature searching and evaluation of articles based on the research methods employed later became what we know today as ‘evidence-based practice’. I had definitely caught the research bug, and not just any research bug. At McMaster the emphasis was on exploring outcomes and impacts using rigorous research designs and applying the study results to clinical practice. The randomized controlled trial In March 1978, with the support of Dr Victor Neufeld and his colleagues in the Program for Educational Development, I applied for a research grant from the Ontario Ministry of Health to evaluate the clinical librarian. At Dr David Sackett’s suggestion, the proposed research employed a randomized controlled trial (RCT) design to assess the value and impact of two clinical librarians who worked in four different clinical settings at McMaster University Medical Centre for 6 months each.

The intervention was evaluated at the end of the 6-month period, and again 3 months later to see if any effect of the intervention was sustained. A preliminary report of the project was published in the Canadian Medical Association Journal in 19792 and the full report was published in the Bulletin of the Medical Library Association in 1981.3 The results of the study demonstrated that clinical librarians had a variety of positive impacts on the information seeking behaviour and information use of the health-care team members with whom they worked. The impact of the clinical librarians on patients and families was less rigorously evaluated, but the outcomes appeared promising. I did not realize that the RCT design was anything particularly special at the time, it just seemed like the best way to evaluate the clinical librarian programme; however, this experience proved to be a pivotal step along my research path. My interest in patient education and consumer health information continued over the years and I published what might now be thought of as a brief systematic review looking at patient education and health outcomes with Dr Brian Haynes in 1983.4 The review demonstrated that information alone had not been shown to change patient compliance in long-term therapy; however, as information as part of a more complex approach had proved effective according to the well-designed studies, librarians were urged to collaborate with other health professionals in developing patient education programmes. As information alone had been shown to improve compliance with short-term therapy, librarians were urged to work with the various clinics to make appropriate informational materials easily available. The review also found that educating health professionals about compliance with treatment was an effective strategy, suggesting that librarians should highlight such materials in their collection and create finding aids and other supports for health professionals. Although this was not a value study as such, the findings had the potential to increase the value of the information services that librarians provide. Diversifying the approaches to value research In 1983 I left McMaster and began a PhD programme where I focused on information use in


5

6


health-care settings. There I was introduced to a whole new range of social science research methods, some of which employed theory testing and statistical control of data. Such designs did not require the type of control groups used in RCTs, but they could still help to determine value and impact. In my dissertation, I used Diffusion of Innovation theory to examine the adoption and implementation of end-user searching by physicians.5 A subanalysis of the dissertation data using multivariate analysis allowed me to explore the impact of end-user training on search behaviour.6 An argument for diversifying our approaches to value research is that social sciences methods can provide a broader understanding of how and why certain outcomes occur. Randomized controlled trials, however, are focused on demonstrating statistical differences between treatment and control groups in an experimental design. Both types of research have their place, particularly in library settings where we are often as interested in how and why as we are that a difference existed. In the PhD programme I was introduced to methods of programme evaluation such as needs assessment, formative evaluation and summative evaluation. I also discovered that well-conducted programme evaluations take a rigorous approach to gathering and analysing data collected from existing data, surveys, interviews, focus groups and observation. As social scientists (and librarians) are often examining the value and impact of programmes and services that are moving targets, I learned the importance of having an ongoing cycle of evaluation. In such situations it is advisable to bring together data from multiple sources using multiple research methods so that results can be cross-checked—an approach known as triangulation. As my PhD studies took place in a public health sciences programme, I was able to observe the ubiquity of evaluation research in all branches of the health-care system. I also began to realize that value and impact studies were directly related to evaluation research and to the kind of issues that librarians were facing. This experience made me realize that as librarians we need to move towards an approach to our work that makes much greater use of evaluation research, and to apply the results of that research to what we do on a day-to-day

basis. Many health professionals have adopted this approach and called it evidence-based practice—finding and applying the best research-based evidence to professional decision making—and that is what library and information professionals need to do as well. The Rochester study After completing my PhD, I became a professor in the Faculty of Information Studies at the University of Toronto. There were two major projects that I became involved in there that allowed me to use the more broadly based research and evaluation skills that I had acquired in my PhD programme: the development of the Consumer Health Information Service (CHIS) at the Toronto Reference Library,7 and what has become known as the Rochester study.8 In CHIS we used a variety of programme evaluation techniques to guide the development of the Ministry of Health-funded project as well as to measure the value of the service to users. The impetus for the Rochester study, however, emerged directly out of a library practice situation. At the time, the state of New York had decided that it would not continue to require that hospitals have a library because there was no evidence that having a library made any direct contribution to improved patient care. The librarians in the Rochester area took up the challenge and sponsored a study to explore the value and impact of the hospital library. One of the most rewarding aspects of the Rochester study was the partnership that it created between the researchers and the practitioners. The study was sponsored by grants from the New York State Education Department, Division of Library Development and the Rochester Regional Library Council. The support of these organizations was a key factor in gaining the participation of all 15 hospital libraries in the area. We also had outstanding support from health sciences library leaders in Rochester including Lucretia McClure, Bernie Todd-Smith, Julia Sollenberger and June Glaser, as well as Kathleen Miller from the Regional Library Council. All of the hospital librarians in the Rochester area at the time deserve a great deal of the credit for the success of the study.



We began by exploring previous studies and decided that the one that seemed closest to what we thought would be useful in New York was a study that had been conducted by David King in Chicago.9 The King study had asked health professionals to make a request for information from their hospital library related to a current clinical case and then to estimate the value of the information received. We invited David to Rochester and, during his visit, he was very generous in pointing out the things he wished he had done in his study, and we were able to incorporate his suggestions in our own research design. The Rochester study was not an RCT; however, it did incorporate a number of study design characteristics that gave the study a certain level of credibility in the health-care milieu. Subjects for the study were randomly chosen, the study was prospective as opposed to retrospective, all of the hospital libraries in the Rochester area took part, and the librarians did not know the names of the physicians who were participating in the study. One of the many challenges of conducting a study that attempts to measure value and impact is that these terms are difficult to define with any precision. Economists have methods of determining monetary value; however, particularly in a healthcare environment, the concept of value extends beyond the purely monetary level. Typical dictionary definitions of value refer to terms such as worth, desirability, utility or the qualities on which these perceived characteristics depend.10 The key word here is ‘perceived’ and, as we all know, what is perceived as valuable by one person may not necessarily be seen as valuable by another. Put another way, value is in the eye of the beholder. Impact, or the ability of a service to make a difference in behaviour or outcome, is also tricky to measure in a reliable way. Given that the continuing role of library and information services is one of meeting user needs and that these needs are always evolving and changing over time, it is easy to see why measuring value and impact continue to be of great interest to the profession. Our research question in the Rochester study was whether hospital libraries provided information services that were valuable to and had impact on patient care. We made a decision to focus on physicians as key decision makers in the health-care

system and to develop measures that would be relevant for that group. Our measures of clinical value fell into four general categories: quality of the information provided; cognitive value of the information; contribution to patient care; and whether receiving the information saved the physician time. One of the key features of the Rochester study was that it defined specific impacts or changes in key decisions made by physicians, such as diagnosis, choice of tests and choice of treatment. The study also measured the impact of the information on avoidance of adverse patient events. Upon reflection, it appears that one of the main achievements of the Rochester study was that it took the plunge and moved beyond measuring basic inputs, outputs and outcomes to measuring value and impact. Moreover, it showed that, in the eyes of the beholders (in this case the physicians), the library services were valued and that the information provided made a positive difference in patient care. Librarians have continued to use the Rochester study measures in a variety of ways over the years and the study continues to be cited and used to demonstrate the value of hospital libraries. A possible limitation of the Rochester study is that its findings were based on self-report, i.e. the perceptions of the physicians as to whether the information provided by the hospital library was valuable and whether it changed patient outcomes. While it would have been desirable to collect other confirming evidence from patient records or other sources, to do so would have made the study much more difficult and costly. On the plus side, it can be argued that physicians are likely to be in the best position to make accurate estimates of the value and impact of information that they receive in support of clinical decision making. Who else is as closely connected with their patients and most likely to be aware of the impact of their decisions on patient outcomes? Over the years since the Rochester study was published in 1992 there have been a variety of studies that have looked at the value and impact of information on clinical decision making. These studies are listed in the appended bibliography. Another sign that we are accumulating a knowledge base in this area is that systematic reviews are starting to appear such as those on clinical librarianship by Beverly and Winning11 and


7

8


Wagner and Byrd.12 We are also seeing works that focus on the research methodologies used in library and information science research such as Koufogiannakis and Crumley.13 Together this body of research demonstrates that our knowledge base is growing and that there is evidence supporting the value and impact of health library and information services in particular. The corporate sector After the publication of the Rochester study in 1992, I was able to apply a similar research design to examine value and impact in different types of organizational settings. The Special Libraries Association (SLA) provided a grant for a study in the corporate banking environment in Toronto, Ontario, Canada where five major bank libraries were located. Again the study was a partnership between the researchers and library practitioners who worked together to define an appropriate target user group, in this case corporate banking employees, who were responsibility for making decisions on loans of $100 000 or more. The steering group looked at the kind of decisions made by the bankers and attempted to come up with the corporate equivalent of the measures that had worked well for physicians in the Rochester study. Not surprisingly, what we discovered was that the kind of decisions that physicians made were very different from those made in the corporate sector. In the end, our measurement development for the SLA study was guided by Henry Mintzberg’s14 research on the nature of managerial work. Mintzberg identified a number of key tasks that matched our observations of how the corporate bankers operated. The SLA study results were published in a 1993 SLA monograph entitled, The Impact of the Special Library on Corporate Decision Making.15 The results showed that information provided by the library was indeed perceived as valuable for the purposes of making important decisions about whether to lend money. Furthermore, the bankers reported that the information often allowed them to proceed to the next step on a project or task, decide upon a course of action, exploit new business opportunities and, most important of all, avoid loss of funds. We were also able to quantify

the monetary value of the time saved by the library services provided which had not been possible in the Rochester study. The long-term health-care sector In 1997–1998, chronic care had become a key health-care issue. Once again, Bernie Todd-Smith was a key person in facilitating a replication of the Rochester study in libraries that served long-term care facilities. While the outcomes of chronic care are different from those in acute settings, they are still of great importance to health-care managers, providers, patients and families. The unpublished study entitled, The Impact of the Hospital Library on Decision Making in Long-Term Care Settings found that the information provided by hospital librarians had a positive impact on outcomes such as site and level of care; improvement in quality of life for the patient and family; and avoidance of inappropriate care or treatment. The health policy sector Another opportunity to explore the use of the Rochester study methodology came when a federal government ministry, Health Canada, funded a value and impact study of its libraries. A similar collaborative approach between researchers and practitioners was taken to define the areas of decision making in the government sector. A summary to the report submitted to Health Canada may be found in Elizabeth Orna’s 1999 book entitled, Practical Information Policies.16 In addition to some of the impacts measured in the corporate study, the study found that the information provided by the libraries helped the government policy makers to: meet a deadline or deal with an unexpected emergency; improve or approve a procedure, policy or plan; and prevent conflict within the government agency or with another department or jurisdiction. In summary, the success of the Rochester study methodology and its offspring appears to have been related to the ability of the team of researchers and practitioners to devise ways of measuring the value of information services and impact of information on the type of decisions being made in the respective organizational settings. Librarians



have long been aware of the importance of knowing their users and their needs, and it appears that this knowledge is as important to conducting evaluation research as it is to providing effective information services. The Rochester study II Happily, the legacy of this work is continuing. In April 2007, the Regional Advisory Committee of the National Network of Libraries of Medicine, Mid-Atlantic Region, commissioned a planning study that will form the basis for a proposal to measure the value of the health sciences library, library services and librarians. The task of the planning group led by Kathel Dunn, Karen Brewer, Joanne Marshall and Julia Sollenberger is to review the original Rochester study and its offspring and to consider what an appropriate research design would be for the current environment. Part of the work has involved a comprehensive review of value and impact studies conducted by Kathel Dunn which is included as an Appendix I to this article. We would welcome your thoughts on the direction that this initiative should take as we continue to work on the project over the next 2 years. We are currently in the process of conducting key informant interviews with hospital administrators who we see as one of the main audiences for the study results. From these interviews, we will attempt to identify key outcomes to be measured. The future of value and impact studies As health sciences library and information professionals move forward in this area, it is important that we pay attention to what is happening in value and impact research in general. Overall, there appear to have been two major strands of research on the value of libraries. The first strand is composed of economic studies that tend to focus on return-on-investment, cost/benefit analysis and other monetary measures. One of the limitations of the economic models is that they may appear very abstract and, even although they are capable of demonstrating the efficiency of libraries in comparison with other government or industry sectors, the measures can appear to be distant from the actual services provided to users in

particular libraries. The second strand is composed of studies that focus on the use of information by particular groups. The body of research on library user studies falls into this category. In the health environment, where outcome measures for particular populations and evidence-based practice are gaining prominence, the second type of study has been seen as particularly relevant. Another useful exercise is to consider how our past research on the value and impact of health library and information services relates to trends such as evidence-based library and information practice, performance measurement and library valuation. Given the emphasis on evidence-based practice in the health professions, it comes as no surprise that librarians are now looking to apply similar principles in their own work. The Fourth International Conference in Evidence-Based Library and Information Practice was held in 2007 in the USA17 and the next conference is planned for Sweden in 2009. Over 300 librarians and information professionals from many countries attended the 2007 meeting, presenting over 80 research papers and posters. Health Information and Libraries Journal now has a column entitled, ‘Using Evidence in Practice’ edited by Andrew Booth. The Medical Library Association offers the Lindberg Research Fellowship annually. The association also approved revisions to its educational policy statement18 in 2006 and its research policy statement19 in 2007. Both of these policies make substantial references to evidence-based practice and the role of research in the profession. Both Health Information and Libraries Journal and the Journal of the Medical Library Association now use the structured abstract formats that are increasingly found in the journals of the evidence-based professions. A group of Canadian librarians have pioneered a new electronic open-access journal, EvidenceBased Library and Information Practice.20 There is a strong connection between evidence-based library and information practice and value and impact studies because the former is all about linking research to practice. Determining the value and impact of library and information services continues to be one of the major topics which need research evidence. The evidence-based practice movement may well help to support both the


9

10


conduct and use of value and impact research in the future. Performance measurement systems have become increasingly common in both the for-profit and non-profit sectors in recent years and libraries have a substantial record of activity in this area. The US National Information Standards Organization (NISO) has created a comprehensive webliography of performance measurement organizations and initiatives that relate to libraries.21 Many of these initiatives began as ways to gather local, national and international statistics on libraries, but the types of evaluation being undertaken appear to be broadening somewhat, and the potential may exist to incorporate value and impact measures into some of the existing performance measurement instruments. The US Association of Research Libraries (ARL) has created a valid and reliable survey that colleges and universities can use to measure user satisfaction, LibQUAL+,22 now used in a number of countries around the world, including South Africa, New Zealand, Australia and the UK, in addition to the large number using it in North America. ARL also sponsored a Library Assessment Conference in 2006 with a second conference planned for 2008.23 Health sciences librarians have been active in developing benchmarking initiatives that allow libraries to compare their performance as a basis for quality improvement.24,25 The Americans for Libraries Council, an advocacy organization for public libraries, has produced a report entitled, ‘Worth their weight: an assessment of the evolving field of library valuation’.26 This report is part of an initiative called ‘Building Knowledge for Library Advocacy’ that has been carried out by the Council with the support of the Bill and Melinda Gates Foundation. While the report emphasizes the economic evaluation of public libraries, it also points out that such measures are being expanded to include some of the library’s intangible social benefits. A broad vision of value is proposed for the future—one that would ‘redefine monetary value and efficiencies in the context of sustainable, healthy communities when making the case for public libraries’. Health sciences libraries would also do well to consider a broader definition of the value that they provide to their communities. While neither library performance evaluation nor library valuation are equivalent to the value

and impact studies that have been conducted on health library and information services, the links are worth exploring. Each of these initiatives likely has lessons learned that would be valuable to share. The resources required to build valid and reliable measurement tools for use in evaluation studies are substantial and the more we can learn from initiatives in different types of settings, the better. It does seem clear that value and impact studies need to be adapted to their particular environments and that in the health field that means linking to patient care, improved health outcomes and other changes that will benefit the health of the individual and the community in the short- and long-term. It also means giving particular libraries and library systems results that will help to make the case for their libraries in the most compelling way possible. There is certainly a wealth of opportunity here for researchers and practitioners to work together to build the knowledge base that we will need to move us forward with this important task.

Key Messages • A substantial body of knowledge exists about the value and impact of health sciences libraries. • A multiple methods approach that produces both quantitative and qualitative data is needed to fully address value and impact research questions. • Related areas such as evidence-based library and information practice, library performance measurement and valuation research can contribute to our understanding of the value and impact of libraries. • Continuing research on value and impact will be required to address changes in user needs, library services and the health sciences environment. Implications for Policy • Stakeholders such as research funding agencies, government bodies that support libraries, library advocacy organizations and professional associations should prioritize value and impact research and its related areas of evidence-based practice, performance measurement and valuation research in the future.



• Library and information science educators and providers of continuing education should ensure that practical research methods that address key issues such as measuring the value and impact of library and information services are included in the curriculum. • Library and information researchers should practice engaged scholarship by partnering with practitioners to develop major studies on key issues for the field.

9

10 11

12

Implications for Practice 13

• Library and information professionals should be leaders in conducting and using ongoing value and impact studies. • Library and information professionals should partner with researchers and other stakeholders to conduct studies on key issues for the field. • Library directors should support research and evaluation as a major part of the work of professional library staff and provide time and support for librarians to participate in such activities.

14 15

16

17

18

References 1 Marshall, J. G. & Hamilton, J. D. The clinical librarian and the patient: report of a project at McMaster University Medical Centre. Bulletin of the Medical Library Association 1978, 66, 420–5. 2 Marshall, J. G. Clinical librarians join health-care team to offer service directly. Canadian Library Journal 1983, 36, 23–8. 3 Marshall, J. G. & Neufeld, V. R. A randomized trial of librarian educational participation in clinical settings. Journal of Medical Education 1981, 56, 409 –16. 4 Marshall, J. G. & Haynes, R. B. Patient education and health outcomes: implications for library services. Bulletin of the Medical Library Association 1983, 7, 259–62. 5 Marshall, J. G. Diffusion of innovation theory and end-user searching. Library and Information Science Research 1990, 12, 55– 69. 6 Marshall, J. G. The perceived complexity of database searching among end-users: a multivariate analysis. Canadian Journal of Information Science 1987, 12, 89–97. 7 Marshall, J. G. A development and evaluation model for a consumer health information service. Canadian Journal of Information Science 1992, 17, 1–17. 8 Marshall, J. G. The impact of the hospital library on clinical decision making: the Rochester study.

19

20

21

22

23

24

25

Bulletin of the Medical Library Association 1992, 80, 169–78. King, D. N. The contribution of hospital library services to patient care. Bulletin of the Medical Library Association 1987, 75, 291–301. Concise Oxford English Dictionary. Oxford: Oxford University Press, 1976. Beverley, C. A. & Winning, M. A. Clinical librarianship: a systematic review of the literature. Health Information and Libraries Journal 2003, 20, 10–21. Wagner, K. C. & Byrd, G. D. Evaluating the effectiveness of clinical medical librarian programs: a systematic review of the literature. Journal of the Medical Library Association 2004, 92, 14–33. Koufogiannakis, D. & Crumley, E. Research in librarianship: issues to consider. Library Hi Tech 2006, 24, 324–40. Mintzberg, H. The Nature of Managerial Work. New York: Harper & Row, 1973. Marshall, J. G. The Impact of the Special Library on Corporate Decision Making. Washington, DC: Special Libraries Association 1993. Marshall, J. G. Health Canada libraries: past, present and future. In: Orna, E. (ed.). Practical Information Policies. Aldershot: Gower, 1999: 251–65. EBLIP4. Fourth International Conference on EvidenceBased Library and Information Practice. Papers available from: http://www.eblip4.unc.edu/ (accessed 10 September 2007). Medical Library Association. Platform for Lifelong Learning and Professional Success. Available from: http://www.mlanet.org/education/policy/ (accessed 10 September 2007). Medical Library Association. The Research Imperative. Available at http://www.mlanet.org/research/policy/ (accessed 10 September 2007). University of Alberta Learning Services. Evidence-Based Library and Information Practice (open source journal). Available from: http://ejournals.library.ualberta.ca/ index.php/EBLip (accessed 10 September 2007). US National Information Standards Organization. Forum on library statistics and performance measures. Webography of Related Resources. Available from: http://www.niso.org/news/events_workshops/stat-webogr.html (accessed 10 September 2007). Association of Research Libraries (ARL). Libqual+. Charting Library Service Quality. Available from: http://www.libqual.org/ (accessed 10 September 2007). Library Assessment Conference. Available from: http://www.libraryassessment.org/ (accessed 10 September 2007). Dudden, R. F. Using Benchmarking, Needs Assessment, Quality Improvement, Outcome Measurement, and Library Standards: A How-to-Do-It Manual (a Medical Library Association Book), New York: Neal-Schuman, 2007. Marshall, J. G. & Inglis, J. The Benchmarking Tool Kit. Toronto: Canadian Health Libraries Association, 1998.


11

12

Measuring the value and impact of health libraries, Joanne Gard Marshall 26 Americans for Libraries Council. Worth their Weight: An Assessment of the Evolving Field of Library Valuation. Available from: http://www.lff.org/documents/ WorthTheirWeight_002.pdf (accessed 10 September 2007). Received 14 September 2007; Accepted 19 September 2007

Appendix 1 Bibliography on the value and impact of health sciences libraries Compiled by: Kathel Dunn. Corrrespondence to: Kathel Dunn Associate Director National Network of Libraries of Medicine Middle Atlantic Region New York University Medical Center 423 East 23rd Street, Floor 15 South New York, NY 10010, USA [email protected] The following bibliography illustrates the wide variety of approaches that have been taken to measuring the value of library and information services and related areas. The listing of the references is in chronological order starting with the most recent so as to illustrate the development of the field over time. The bibliography was compiled by Kathel Dunn for the value of health sciences libraries study being planned by the US National Network of Libraries of Medicine, Middle Atlantic Region. 2007 Brettle, A., Hulme, C. & Ormandy, P. Effectiveness of information skills training and mediated searching: Qualitative results from the EMPIRIC project. Health Information and Libraries Journal 2007, 24, 24–33. Dudden, R. F. Using Benchmarking, Needs Assessment, Quality Improvement, Outcome Measurement, and Library Standards: A How-To-Do-It Manual (A Medical Library Association Book). New York: Neal-Schuman, 2007. Joubert, D. J. & Lee, T. P. Empowering your institution through assessment. Journal of the Medical Library Association 2007, 95, 46–53.

Perley, C. M., Gentry, C. A., Fleming, A. S. & Sen, K. M. Conducting a user-centred information needs assessment: the Via Christi libraries’ experience. Journal of the Medical Library Association 2007, 95, 173–81. Urquhart, C., Turner, J., Durbin, J. & Ryan, J. Changes in information behaviour in clinical teams after introduction of a clinical librarian service. Journal of the Medical Library Association 2007, 95, 14–22. 2006 Berry JN, III. Blatantberry. humans do a better job: User satisfaction is the most important return on investment. Library Journal 2006, 131, 10. Brettle, A., Hulme, C. & Ormandy, P. The costs and effectiveness of information-skills training and mediated searching: quantitative results from the EMPIRIC project. Health Information and Libraries Journal 2006, 23, 239–247. Brookman, A., Lovell, A., Henwood, F. & Lehmann, J. What do clinicians want from us? An evaluation of Brighton and Sussex University Hospitals NHS Trust clinical librarian service and its implications for developing future working patterns. Health Information and Libraries Journal 2006, 23, 10–21. Bryant, S. L. & Gray, A. Demonstrating the positive impact of information support on patient care in primary care: a rapid literature review. Health Information and Libraries Journal 2006, 23, 118– 125. Chowdhury, G., McMenemy, D. & Poulter, A. Large-scale impact of digital library services: findings from a major evaluation of SCRAN. Research and Advanced Technology for Digital Libraries 2006, 4172, 256–66. Droese, P. & Peterson, N. Utilization of the medical librarian in a state Medicaid program to provide information services geared to health policy and health disparities. Journal of the Medical Library Association 2006, 94, 174–79. Dudden R. F., Corcoran K., Kaplan J., Magouirk J., Rand D. C. & Todd-Smith B. The Medical Library Association Benchmarking Network: development and implementation. Journal of the Medical Library Association 2006, 94, 107–17.



Dudden R. F., Corcoran K., Kaplan J., Magouirk J., Rand D. C. & Todd-Smith B. The Medical Library Association Benchmarking Network: results. Journal of the Medical Library Association 2006, 94, 118–29. Edgar, W. B. Questioning LibQUAL+ TM: expanding its assessment of academic library effectiveness. Portal: Libraries and the Academy 2006, 6, 445–65. Eldredge, J. Evidence-based librarianship: the EBL process. Library Hi Tech 2006, 24, 341–54. Koufogiannakis, D. & Crumley, E. Research in librarianship: issues to consider. Library Hi Tech 2006, 24, 324– 40. Poll, R. & Payne, P. Impact measures for libraries and information services. Library Hi Tech 2006, 24, 547–562. Sen, B. Market orientation: a concept for health libraries. Health Information and Libraries Journal 2006, 23, 23–31. Tennant, M. R., Tobin Cataldo, T., Sherwill-Navarro, P. & Jesano, R. Evaluation of a liaison librarian program: client and liaison perspectives. Journal of the Medical Library Association 2006, 94, 402–09. Urquhart, C., Durbin, J. & Cumbers, B. Evaluation of the KA24 (knowledge access 24) service for health and social care staff in London and the south-east of England. Part 2: qualitative. Health Information and Libraries Journal 2006, 23, 159– 68. 2005 Klein-Fedyshin, M., Burda, M. L., Epstein, B. A. & Lawrence, B. Collaborating to enhance patient education and recovery. Journal of the Medical Library Association 2005, 93, 440–45. Turtle, K. M. A survey of users and non-users of a UK teaching hospital library and information service. Health Information and Libraries Journal 2005, 22, 267–75. Weightman, A. L., Williamson, J., and the Library and Knowledge Development Network (LKDN) Quality and Statistics Group. The value and impact of information provided through library services for patient care: a systematic review. Health Information and Libraries Journal 2005, 22, 4–25.

2004 Abels, E. G., Cogdill, K. W. & Zach, L. Identifying and communicating the contributions of library and information services in hospitals and academic health sciences centres. Journal of the Medical Library Association 2004, 92, 46–55. Allen, C. W. Research. outcomes-based evaluation in a community health library. Journal of Hospital Librarianship 2004, 4, 63–78. Carney, S., Koufogiannakis, D. & Ryan, P. Library services for users of personal digital assistants: a needs assessment and program evaluation. portal: Libraries and the Academy 2004, 4, 393–406. Cullen, R. Evaluating digital libraries in the health sector. Part 2: measuring impacts and outcomes. Health Information and Libraries Journal 2004, 21, 3–13. Gluck, J. C. The contribution of hospital library services to continuing medical education. The Journal of Continuing Education in the Health Professions 2004, 24, 119–23. Gorman, P. N., Yao, P. & Seshadri, V. Finding the answers in primary care: information seeking by rural and non-rural clinicians. Medinfo 2004, 11, 1133–37. Sherwill-Navarro, P. J. & Wallace, A. L. Research on the value of medical library services: does it make an impact in the health-care literature? Journal of the Medical Library Association 2004, 92, 34–45. Sladek, R. M., Pinnock, C. & Phillips, P. A. The informationist: a prospective uncontrolled study. International Journal for Quality in Health Care 2004, 16, 509–515. Wagner, K. C. & Byrd, G. D. Evaluating the effectiveness of clinical medical librarian programs: a systematic review of the literature. Journal of the Medical Library Association 2004, 92, 14–33. 2003 Beverley, C. A. & Winning, M. A. Clinical librarianship: a systematic review of the literature. Health Information and Libraries Journal 2003, 20, 10–21. Cullen, R. Evaluating digital libraries in the health sector. Part 1: measuring inputs and outputs.


13

14


Health Information and Libraries Journal 2003, 20, 195–204. Heath, F., Kyrillidou, M., Webster, D., Choudhury, S., Hobbs, B., Lorie, M. et al. Emerging tools for evaluating digital library services: conceptual adaptations of LibQUAL+ and CAPM. Journal of Digital Information 2003, 4, 8. Sayed, E. N. & Murray, S. D. User satisfaction survey and usage of an electronic desktop document delivery service at an academic medical library. Medical Reference Services Quarterly 2003, 22, 21–29. Weightman, A. & Williamson, J. How do library services impact on patient care? CILIP Health Libraries Group Newsletter 2003, 20, 13. Winning, M. A. & Beverley, C. A. Clinical librarianship: a systematic review of the literature. Health Information and Libraries Journal 2003, 20, 10–21. Yeoman, A. J., Cooper, J. M., Urquhart, C. J. & Tyler, A. The management of health library outreach services: Evaluation and reflection on lessons learned on the VIVOS project. Journal of the Medical Library Association 2003, 91, 426–433. 2002 Abels, E. G., Cogdill, K. W. & Zach, L. The contributions of library and information services to hospitals and academic health sciences centres: a preliminary taxonomy. Journal of the Medical Library Association 2002, 90, 276–84. Calabretta, N. Consumer-driven, patient-centred health care in the age of electronic information. Journal of the Medical Library Association 2002, 90, 32–37. Cook, C., Heath, F. & Thompson, B. Score norms for improving library service quality: a LibQUAL+ study. portal: Libraries and the Academy 2002, 2, 13–26. Klein-Fedyshin, M. S. Consumer health informatics— integrating patients, providers, and professionals online. Medical Reference Services Quarterly 2002, 21, 35–50. O’Connor, P. Determining the impact of health library services on patient care: a review of the literature. Health Information and Libraries Journal 2002, 19, 1–13.

Plutchak, T. S. Determining value. Journal of the Medical Library Association 2002, 90, 273–75. Rodger, E. J. Value and vision: public libraries must create public value through renewal and reinvention. American Libraries 2002, 33, 50–54. Todd-Smith, B.& Markwell, L. G. The value of hospital library benchmarking: an overview and annotated references. Medical Reference Services Quarterly 2002, 21, 85–95. Wolf, D. G., Chastain-Warheit, C. C., EasterbyGannett, S., Chayes, M. C. & Long, B. A. Hospital librarianship in the United States: at the crossroads. Journal of the Medical Library Association 2002, 90, 38–48. 2001 Ash, J. S., Gorman, P. N., Lavelle, M., Lyman, J., Delcambre, L. M., Maier, D. et al. Bundles: meeting clinical information needs. Bulletin of the Medical Library Association 2001, 89, 294–96. Barnett, L., Cmor, D. & Morgan, P. Mediated computer search services relative to instruction services: a survey of one health sciences library. Medical Reference Services Quarterly 2001, 20, 9–21. Helmer, D., Savoie, I., Green, C. & Kazanjian, A. Evidence-based practice: extending the search to find material for the systematic review. Bulletin of the Medical Library Association 2001, 89, 346–52. Richwine, M. P. & McGowan, J. J. A rural virtual health sciences library project: research findings with implications for next generation library services. Bulletin of the Medical Library Association 2001, 89, 37–44. Schacher, L. F. Clinical librarianship: its value in medical care. Annals of Internal Medicine 2001, 134, 717–720. Tennant, M. R., Butson, L. C., Rezeau, M. E., Tucker, P. J., Boyle, M. E. & Clayton, G. Customizing for clients: developing a library liaison program from need to plan. Bulletin of the Medical Library Association 2001, 89, 8–20. 2000 Ali, I. Library provided information and clinical decision making: a study of two hospitals in



Canberra. Australian Academic and Research Libraries 2000, 31, 30–45. Marshall, J. G. Determining our worth, communicating our value. Library Journal 2000, 125, 28– 30. 1999 Dixon-Lee, C. In pursuit of a value-added entity. Journal of AHIMA/American Health Information Management Association 1999, 70, 6–7. Eysenbach, G. & Diepgen, T. L. Patients looking for information on the Internet and seeking teleadvice: motivation, expectations, and misconceptions as expressed in e-mails sent to physicians. Archives of Dermatology 1999, 135, 151–56. Fuller, S. S., Ketchell, D. S., Tarczy-Hornoch, P. & Masuda, D. Integrating knowledge resources at the point of care: opportunities for librarians. Bulletin of the Medical Library Association 1999, 87, 393–403. McCray, J. C. Delivering health information statewide via the Internet in a collaborative environment: impact on individual member institutions. Bulletin of the Medical Library Association 1999, 87, 264–69. Peay, W. J. Strategies and measures for our next century. Bulletin of the Medical Library Association 1999, 87, 1–8. 1998 Marshall J. G. & Inglis, J. The Benchmarking Tool Kit. Toronto: Canadian Health Libraries Association, 1998. Papandrea, V. A. Managing reference services in the electronic age: a competing values approach to effectiveness. The Reference Librarian 1998, 111–26. Stead, W. W. Positioning the library at the epicentre of the networked biomedical enterprise. Bulletin of the Medical Library Association 1998, 86, 26–30. Trinder, V. M. What do health professionals think of their library service? Report of a multiprofessional user satisfaction survey. Health Libraries Review 1998, 15, 205–08. Ward, J. A., Baum, S., Ter Maat, J., Thomsen, C. A. & Maibach, E. W. The value and impact of

the cancer information service telephone service: part 4. Journal of Health Communication 1998, 50–70. 1997 Beilby, J. J. & Silagy, C. A. Trials of providing costing information to general practitioners: a systematic review. Medical Journal of Australia 1997, 167, 89–92. Pifalo, V., Hollander, S., Henderson, C. L., DeSalvo, P. & Gill, G. P. The impact of consumer health information provided by libraries: the Delaware experience. Bulletin of the Medical Library Association 1997, 85, 16–22. Urquhart, C. Performance measurement in library and information services: health advice from the value and EVINCE studies. Library and Information Briefings 1997, 1–12. Urquhart, C. J. & Davies, R. Assessing the impact of information on evidence-based nursing practice: the EVINCE (Establishing the Value of Information to Nursing Continuing Education) project. Topics in Health Information Management 1997, 18, 53–63. Urquhart, C. J. & Davies, R. Assessing the impact of information on evidence-based nursing practice: the EVINCE (Establishing the Value of Information to Nursing Continuing Education) project. Topics in Health Information Management 1997, 18, 53–63. Urquhart, C. J. & Hepworth, J. B. The value to clinical decision making of information supplied by NHS library and information services. Australian Academic and Research Libraries 1997, 28, 175–77. 1996 Urquhart, C. J. & Hepworth, J. B. Comparing and using assessment of the value of information to clinical decision making. Bulletin of the Medical Library Association 1996, 84, 482–89. Urquhart, C. J. & Hepworth, J. B. The value to clinical decision making of information supplied by NHS library and information services. Education for Information 1996, 14, 135–37. Urquhart, C. J. & Hepworth, J. B. Comparing and using assessments of the value of information to


15

16


clinical decision making. Bulletin of the Medical Library Association 1996, 84, 482–89. 1995 Burton, J. E. The impact of medical libraries and literature on patient care in New Zealand. Bulletin of the Medical Library Association 1995, 83, 425–30. Urquhart, C. & Hepworth, J. The value of information supplied to clinicians by health libraries: devising an outcomes-based assessment of the contribution of libraries to clinical decision making. Health Libraries Review 1995, 12, 201–13. Urquhart, C. J. & Hepworth, J. B. The Value of Information Services to Clinicians: A Toolkit for Measurement. London: British Library and Development Report 6206, 1995. 1994 Gorman, P. N., Ash, J. & Wykoff, L. Can primary care physicians’ questions be answered using the medical journal literature? Bulletin of the Medical Library Association 1994, 82, 140–46. Hersh, W. & Hickam, D. Use of a multiapplication computer workstation in a clinical setting. Bulletin of the Medical Library Association 1994, 82, 382–89. Klein, M. S., Ross, F., Adams, D. & Gilbert, C. M. Effect of online literature searching on length of stay and patient care costs. Academic Medicine: Journal of the Association of American Medical Colleges 1994, 69, 489 –95. Marshall, J. The impact of information services on decision making: some lessons from the financial and health-care sectors. In: Feeney, M. & Grieves, M. (eds), Value and Impact of Information. London: published by Bowker Saur for the British Library, 1994: 195–211. Pifalo, V. Outreach to health professionals in a rural area. Medical Reference Services Quarterly 1994, 13, 19–26. 1993 Buchanan, H. S. Library services and health-care administration. Library Trends 1993, 42, 62–82. Kuller, A. B., Wessel, C. B., Ginn, D. S. & Martin, T. P. Quality filtering of the clinical literature by

librarians and physicians. Bulletin of the Medical Library Association 1993, 81, 38–43. Lindberg, D. A., Siegel, E. R., Rapp, B., Wallingford, K. T. & Wilson, S. R. Use of  by physicians for clinical problem solving. Journal of the American Medical Association 1993, 269, 3124–29. Royal, M., Grizzle, W. E., Algermissen, V. & Mowry, R. W. The success of a clinical librarian program in an academic autopsy pathology service. American Journal of Clinical Pathology 1993, 99, 576–81. 1992 Marshall, J. G. The impact of the hospital library on clinical decision making: the Rochester study. Bulletin of the Medical Library Association 1992, 80, 169–78. Smith, B. T. Developing a publicity plan for library research: the Rochester study. Medical Reference Services Quarterly 1992, 11, 75–82. Veenstra, R. J. Clinical medical librarian impact on patient care: a 1-year analysis. Bulletin of the Medical Library Association 1992, 80, 19–22. Veenstra, R. J. & Gluck, E. H. (1992). A clinical librarian program in the intensive care unit. Critical Care Medicine 1992, 20, 1038–42. 1991 Humphries, A. W. & Naisawald, G. V. Developing a quality assurance program for online services. Bulletin of the Medical Library Association 1991, 79, 263–70. Lovas, I., Graham, E. & Flack, V. Health professionals’ use of documents obtained through the regional medical library network. Bulletin of the Medical Library Association 1991, 79, 28–35. Marshall, J. Preliminary results of the impact of the hospital library on clinical decision-making study. Bibliotheca Medica Canadiana 1991, 13, 100–05. Siegel, E. R., Rapp, B. A. & Lindberg, D. A. Evaluating the impact of  using the critical incident technique. In: Clayton, P. D. (ed.), Proceedings of the Symposium on Computer Applications in Medical Care. New York: McGraw-Hill, 1991, 83–87.



1990 Haynes, R. B., McKibbon, K. A., Walker, C. J., Ryan, N., Fitzgerald, D. & Ramsden, M. F. Online access to  in clinical settings: a study of use and usefulness. Annals of Internal Medicine 1990, 112, 78–84. 1987 King, D. N. The contribution of hospital library information services to clinical care: a study in eight hospitals. Bulletin of the Medical Library Association 1987, 75, 291–301. 1986 Barbour, G. L. and Young, M. N. (1986). Morning report: role of the clinical librarian. Journal of the American Medical Association 1986, 255, 1921–22. 1985 Dabanovic, R. How the literature can help in medical treatment. International Journal of Clinical Pharmacology Research 1985, 5, 1–7. Garfield, E. The impact of health information delivery on the quality of patient care: whither medical information science? Health Libraries Review 1985, 2, 159–69. Hardy, M. C., Yeoh, J. W. & Crawford, S. Evaluating the impact of library services on the quality and cost of medical care. Bulletin of the Medical Library Association 1985, 73, 43–46. 1984 Kantor, P. B. Cost and usage of health sciences libraries: economic aspects. Bulletin of the Medical Library Association 1984, 72, 274–286. 1982 Winters, A. A., Sviridov, F. A., Inoue, H., Lenoir, M., Ganz, C., Wilson, T. D. et al. Session B: value of information and impact of information on decision making and economy. Organization and economics of information and documentation.

The Hague, Netherlands: Vibeke Ammundsen, Federation Internationale de Documentation, 1982: 37–107. 1981 Scura, G. & Davidoff, F. Case-related use of the medical literature: clinical librarian services for improving patient care. Journal of the American Medical Association 1981, 245, 50–52. 1978 Greenberg, B., Battison, S., Kolisch, M. & Leredu, M. Evaluation of a clinical medical librarian program at the Yale Medical Library. Bulletin of the Medical Library Association 1978, 66, 319–326. Marshall, J. G. & Hamilton, J. D. The clinical librarian and the patient: report of a project at McMaster University Medical Centre. Bulletin of the Medical Library Association 1978, 66, 420–25. Zachert, M. J. Books and other endangered species: an inquiry into the values of medical librarianship. Bulletin of the Medical Library Association 1978, 66, 381–89. 1977 Tagliacozzo, R. Estimating the satisfaction of information users. Bulletin of the Medical Library Association 1977, 65, 243–49. 1976 Staudt, C., Halbrook, B. & Brodman, E. A clinical librarians’ program—an attempt at evaluation. Bulletin of the Medical Library Association 1976, 64, 236–38. 1975 Harley, A. J. The price and value of information. Proceedings of the Royal Society of Medicine 1975, 68, 691–94. Evans, E., Borko, H. & Ferguson, P. Review of criteria used to measure library effectiveness. Bulletin of the Medical Library Association 1972, 60, 102–10.


17