Planning changes to health library services on ... - Wiley Online Library

DOI: 10.1111/j.1471-1842.2010.00900.x

Planning changes to health library services on the basis of impact assessment Christine Urquhart*, Rhian Thomas*, Jason Ovens†, Wendy Lucking† & Jane Villa‡ *Department of Information Studies, Aberystwyth University, Aberystwyth, UK, †Resource and Information, KRIS, Southmead Hospital, Bristol, UK and ‡Library Service, North Bristol NHS Trust, Frenchay Hospital, Bristol, UK

Abstract Background: Various methods of impact assessment for health library services exist, including a toolkit developed for the UK. The Knowledge, Resource and Information service (KRIS) for health promotion, health service commissioning and public health (Bristol area, UK) commissioned an independent team at Aberystwyth University to provide an impact assessment and evaluation of their services and to provide evidence for future planning. Objective: The review aimed to provide an action plan for KRIS through assessing the impact of the current service, extent of satisfaction with existing services and views on desirable improvements. Methods: Existing impact toolkit guidance was used, with an adapted impact questionnaire, which was distributed by the KRIS staff to 244 users (response rate 62.3%) in early 2009. The independent team analysed the questionnaire data and presented the findings. Results: Users valued the service (93% considered that relevant information was obtained). The most frequent impacts on work were advice to patients, clients or carers, and advice to colleagues. Literature searching and current awareness services saved staff time. Many users were seeking health promotion materials. Conclusion: The adapted questionnaire worked well in demonstrating the service impacts achieved by KRIS, as well as indicating desirable improvements in service delivery.

Key Messages Implications for Practice d

d

d

Commissioners of health services value specialist librarian services in literature searching and current awareness – commissioners appreciate individualised, timely and relevant evidence that saves them time. Those using health promotion resources value responsive and reliable services, with online ordering systems. Information services working with community partners outside the health service need to implement an active outreach programme.

Implications for Policy d

d d

The impact of information obtained from a health library service provides good evidence for service planning. Evaluation should explore the service weaknesses as well as the successful aspects of the service. Validation of impact toolkit guidance should help other health libraries plan similar impact surveys.

Introduction Correspondence: Christine Urquhart, Department of Information Studies, Aberystwyth University, Aberystwyth SY23 3AS, UK. E-mail: [email protected]

In the Bristol area of SW England, UK, restructuring and relocation of health services during the

ª 2010 The authors. Health Information and Libraries Journal ª 2010 Health Libraries Group Health Information and Libraries Journal, 27, pp.277–285

277

278

Planning on the basis of impact evidence, Christine Urquhart et al.

period 2006–2011 required a review of the existing health library provision to meet the new agendas for health service commissioners and providers within the NHS Bristol and South Gloucestershire area. The North Bristol NHS Trust Library and Information Service received funding to provide library services to primary care from the South West SHA and in collaboration with the Knowledge, Resource and Information Service (KRIS) Strategy Group agreed and funded a review of the existing KRIS provision to inform the development of a service suited to commissioning, health service provision and public health. Aberystwyth University’s Department of Information Studies was asked to conduct this independent review. Literature review Guidance exists in the UK for assessing the value and impact of information provided by library services for patient care.1 This guidance was intended to cover all types of services provided by health libraries and was based on previous research and a systematic review. Specific impact evaluation metrics may be necessary to assess specialised services, for example, information literacy programmes, although a systematic review2 on evaluation of information skills training in health libraries found few valid and reliable measures, and those that did exist were more suitable for academic settings, and not for practising professionals. A randomised controlled trial of a librarian consultation service3 used a cognitive impact assessment scale developed for electronic information and knowledge resources.4 This method (information assessment method, IAM) has been used and validated to determine reflection on relevance, cognitive impact, use and health outcomes of objects of information delivered or retrieved in electronic knowledge resources.5 There are strong similarities between the questions developed in the IAM work and the UK impact toolkit questions on impact. Another review on library impact and the adaptation of existing impact frameworks concludes that adaptation is possible, but that impact frameworks need to be specific, relevant to the users, and the types of library services offered.6 Aspects of the KRIS service that were evaluated included the services to public health and outreach

activities. Evidence suggests that research bulletins are valued by public health staff,7 and that the presence of outreach librarians in primary care trusts increases e-resource usage by staff.8 Objectives The review aimed to provide an action plan for KRIS with priorities and recommendations for the next 3 years of service development. The objectives of the research for the review included assessment of the impact of the current services, which included literature search support, database and e-resource training, provision of health promotion resources, and reference enquiries. The research also determined extent of satisfaction with existing services and views of users on desirable improvements to services and service delivery. Setting KRIS is unusual among health library services in the UK as the service combines a health promotion resources service together with a service to commissioners of health services, public health and all other staff employed by Bristol PCT. At the time of the review, the service was managed by the Learning and Development Department under Bristol Community Health, and services were provided from a site in central Bristol (South Plaza) and another at North Bristol NHS Trust teaching hospital on the northern side of Bristol (Southmead). Services were also provided to South Gloucestershire PCT under a service level agreement. Methods The questionnaire design was based on a toolkit1 from a previous NHS Library Service impact study (Appendix for the questionnaire used). The questionnaire was adapted to make it suitable for staff other than health staff, as teachers and youth workers also used the service (for health promotion materials). Those surveyed were asked to reflect on the most recent occasion they had used the KRIS service, and asked to provide details about the purpose(s) of use, the elements of the KRIS service used, satisfaction with the service and ⁄ or information provided, immediate impact on their



work, and probable contribution to future work. The final set of questions queried views on desirable improvements and how the KRIS service contributed to the respondents’ work and continuing professional development. A questionnaire was sent to all library service users and a selection of health promotion service users from both bases. A total of 244 questionnaires were sent out from both sites (n = 121 Southmead, n = 123 South Plaza) in May 2009. KRIS staff managed the distribution process and copies of the questionnaire were sent out by post or e-mail to prospective respondents. The sample was a convenience sample, as the health promotion service users targeted were largely those that used the service during May 2009, although some of this group might also be regular library service users. Interviews with KRIS staff were conducted by one of the research team (RT) and these provided background information on the organisation of the service. Interviews were conducted face-to-face individually, and asked staff for their views on the history of the service and future developments. Results Response rate The response rate was 62.3% (152 ⁄ 244) overall. Response for each site was similar (60.2%, 74 ⁄ 123

South Plaza, 64.5%, 74 ⁄ 121 Southmead). Community nurses, and midwives were the largest group of respondents (n = 31, 20.4%), closely followed by managers and administrators (n = 24, 15.8%), others (mostly support staff; n = 21, 13.8%), health promotion specialists and allied health professionals (both n = 19, 12.5%). The profile of respondents at each site varied. The Southmead site obtained no responses from consultants, but there were proportionally more responses from community nursing, midwifery and health visiting staff, whereas the South Plaza site obtained proportionally more responses from managers, health promotion specialists and allied health professionals. There were some responses from users who were not health professionals (n = 8; mostly among the Southmead respondents). Purpose of service use Health promotion activities dominated, and most of that activity emanated from the Southmead site, traditionally the main health promotion service site. Conversely, over two-thirds of KRIS usage for continuing professional development, service development planning and research came from the South Plaza respondents. This split also reflects the skill mix distribution of KRIS staff (Table 1). Health promotion leaflets and resources for loan were the main service elements used (Table 2). However, more specialist librarian skills are in

Table 1 Purpose of service use

Reason 1 2 3 4 5 6 7 8 9 10 11

Health promotion activities Direct patient or client care Teaching ⁄ supervision CPD Service development and planning Personal research ⁄ interest Clinical governance ⁄ guidelines Audit Research (funded) Other* Legal ⁄ ethical issues

Total n = 152

%Total n = 152

South Plaza n = 74

Southmead n = 78

85 47 45 40 29 27 12 10 6 5 4

55.9 30.9 29.6 26.3 19.1 17.8 7.9 6.6 3.9 3.3 2.6

24 17 17 31 22 23 10 10 6 4 4

61 30 28 9 7 4 2 0 0 1 0

*Other includes writing for publication; use of email and Internet, IT(2); leaflets for health visitors; CPD, continuing professional development. ª 2010 The authors. Health Information and Libraries Journal ª 2010 Health Libraries Group Health Information and Libraries Journal, 27, pp.277–285

279

280

Planning on the basis of impact evidence, Christine Urquhart et al. Table 2 KRIS service element used

Reason 1 2 3 4 5 6 7 8 9 10 11 12 13

Health Promotion leaflets and posters Health Promotion loanable resources Literature searching by or with library staff References for essays or assignments Loans or photocopies from other libraries Information for a patient or client Up-to-date information for CPD General Policy documents Background information Training (e.g. On use of e-resources) Contact, organisational details Guidelines Other*

Total n = 152

%Total n = 152

South Plaza n = 74

Southmead n = 78

94 57 46 29 27 21 18 14 14 13 7 7 4

61.8 37.5 30.3 19.1 17.8 13.8 11.8 9.2 9.2 8.6 4.6 4.6 2.6

22 17 43 25 24 4 14 10 9 9 6 6 3

72 40 3 4 3 17 4 4 5 4 1 1 1

*Equipment (2), viewing, access to IT.

high demand as well as literature searching which was the third most popular element. Satisfaction with service delivery The service provided clearly met the expectations of the users, as 93% (n = 141) of the respondents reported that their needs were completely met. Only 11 people reported incomplete service provision – three of those down to time constraints and no-one reported complete dissatisfaction with the service. The resources provided or obtained met the needs of 75% (n = 114) of the respondents, 17 reported that the information was incomplete (19 respondents made no comment on this question). Information was delivered in time – 97.4% (n = 148) reported that they were wholly satisfied. Generally, most respondents were happy with the quality of the service (76.3%, n = 116 were very satisfied and 21.1%, n = 32 were satisfied). Dissatisfaction was associated mostly with accessibility to one of the sites, which was at the time closed at lunchtimes every working day, and on Tuesdays all day. Many respondents took time to write in comments and the following summarises the opinions of many respondents: ‘The KRIS Team are always so pleasant and helpful, even when given very tight timescales to complete a work request. They are excellent!’ (Community Worker)

Impact of information obtained Respondents valued the information obtained for its relevancy, timeliness and accuracy. Nearly 70% of respondents were distributing the information or resources to other people (as might be expected for the proportion of health promotion resources obtained) and nearly 68% were sharing or discussing information provided with colleagues. Only just over a third of respondents claimed information obtained improved their decision-making, but nobody claimed that they had found ‘little or nothing of value’ (Table 3). For the immediate contribution to their work (Table 4), the most frequent impact categories were Table 3 Immediate cognitive impact of information obtained Immediate (cognitive) impact of service(s) provided on work

Yes

%Total n = 152

1 2 3 4 5

141 115 109 105 103

92.8 75.7 71.7 69.1 67.8

74 59 56 54 7 0

48.7 38.8 36.6 35.5 4.6

6 7 8 9 10 11

I I I I I

obtained relevant information obtained current information obtained accurate information will distribute information to others will share or discuss information with colleagues I saved time I substantiated my prior knowledge I refreshed my memory of information I can make a better decision Other I found little or nothing of value


Planning on the basis of impact evidence, Christine Urquhart et al. Table 4 Impact on current work

Type of contribution now

Yes

%Total* n = 152

1 2 3 4 5 6 7 8 9 10

104 84 74 51 37 29 27 24 19 18 8

68.4 55.3 48.7 33.6 24.3 19.1 17.8 15.8 12.5 11.8 5.3

Advice or information to patients or clients or carers Advice to colleagues Healthier lifestyle choices Improved quality of life for patient or client Organisational development (e.g. of work team) Revision of care pathway or clinical guidelines Changes to health or social care service delivery Commissioning or possible service redesign Minimisation of risks of treatment Legal and ethical issues Other

A = 74

B = 78

45 33 23 22 24 21 18 23 15 15

59 41 51 29 13 8 9 1 4 3

*No responses, n = 10.

advice to patients, clients or carers (as might be expected, given the proportion of health promotion materials supplied) and advice to colleagues (as might be expected of support for commissioning). Next were healthier lifestyle choices and improved quality of life for patients or clients. Comments included:

‘It was information in the form of leaflets provided to training participants - so may help in their service delivery’.

‘I was able to use the evidence provided to build a business case for the development of services’. (Manager or Administrative Respondent)

A total of 106 respondents failed to give an answer to the question asking about improvements – 46 from South Plaza and 60 from Southmead. Some (n = 13) commented that they could not identify any required improvements in the service:

‘My students BTEC Early Years used the information and resources for health promotion mini teach sessions’. (Teacher) The pattern of estimated impact on future work (Table 5) was very similar to that for current work.

Desired improvements

‘Cannot think of anything more they could provide’. Others listed specific and often specialist requirements, reflecting their own individual needs,

Table 5 Impact on future work activities

Type of contribution in the future

YES

%Total* n = 152

A = 74

B = 78

1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th

78 63 61 46 36 35 32 30 15 13 4

51.3 41.4 40.1 30.3 23.7 23.0 21.1 19.7 9.9 8.6 2.6

31 30 17 20 25 32 24 22 11 9 1

47 33 44 26 11 3 8 8 4 4 3

Advice or information to patients or clients or carers Advice to colleagues Healthier lifestyle choices Improved quality of life for patient or client Changes to health or social care service delivery Commissioning or possible service redesign Revision of care pathway or clinical guidelines Organisational development (e.g. of work team) Legal and ethical issues Minimisation of risks of treatment Other

*No responses, n = 31. ª 2010 The authors. Health Information and Libraries Journal ª 2010 Health Libraries Group Health Information and Libraries Journal, 27, pp.277–285

281

282


but the range reflects the wide range of health promotion materials that might be required, suitable for different audiences. General comments on the service The final open-ended question asked for opinions on respondents’ experience of the KRIS contribution to their work and continuing professional development. Some 53 respondents chose not to make any comment (19 from South Plaza and 34 from Southmead). Comments could be grouped, loosely, into the following categories, although there was some overlap: satisfaction with service responsiveness (36 favourable comments); physical access and availability (9 comments, several noting problems with access, opening times and parking); alerting and literature searching (24 comments, mostly noting that how the service was effective and saved time); services for expansion (12 comments, including suggestions for online ordering of health promotion materials and more outreach visits to promote the service); accessibility, marketing and promotion of the service (19 comments). Comments on the general range of services included the following representative comment: ‘KRIS provides a very good service for me - helpful, timely. The availability of information leaflets is essential for my work with community-based organisations. Campaign materials and posters etc., are also very useful as are display boards and electronic equipment that are available for training and community events. The service also offers help with searches for information ⁄ leaflets etc. that may be available elsewhere. Help with literature searches... also very useful’. Comments on services that required specialist librarian skills included the following representative comments: ‘It is simply not possible to develop informed and defensible commissioning policy without using evidence and much of the relevant clinical evidence is contained in limited access journals which I can only obtain with help from KRIS. It is also important on occasions to have assistance with searching because of the scale or complexity of the search or for validation’.

‘I am periodically sent literature searches and then I request from these as applicable to me. This enhances my practice by reading current literature and is invaluable for my CPD’. Specific comments on desirable improvements included the following: ‘Regular updates on searching databases would be beneficial for the rest of my team’.

‘We really need KRIS to come to more teacher events and show what is on offer. It is hard to find the time for teachers to come here’.

‘Would like to see resources online - photographed so could choose when planning groupwork rather than coming and looking always’.

Discussion Main strengths of the service The research was intended to identify the main impacts of the service on users’ work and professional practice. The findings identified that the main strengths of the service were the perceived reliability of the service, the current awareness and alerting services run from one of the sites and the literature searching support for commissioning and public health enquiries. Users frequently commented on the time saved by delegating some of their literature searching, and the main cognitive impact was the perceived relevance of the information obtained. Overcoming main weaknesses The survey also identified some of the weaknesses in service delivery. Unsurprisingly, given that these were hospital ⁄ health service sites, there were accessibility issues (parking problems, security access, opening hours and staffing cover). The findings confirmed anecdotal observations of the KRIS team about user dissatisfaction, and provided evidence to support plans for an interactive



website and catalogue ordering. The main opportunities for service development were outreach capacity, to help in service promotion, and the strategic working with community partners, public and patients. This type of activity is required by the primary care NHS trust itself to support future commissioning activities. KRIS has made a start on this, and probably works more closely with community partners (through the health promotion and public health activities) than many hospital libraries that primarily serve healthcare staff and students. To expand outreach, staff may need to expand existing roles, and the service provision model reviewed, but there is evidence8 that shows e-resource usage is greater at primary care trusts if there are outreach librarians (without a physical library) or a dedicated trust library. Investment in library staff to train other staff ensures that e-resources are used appropriately. Developing and evaluating services Other services supporting public health professionals have produced research bulletins to save the time of public health staff in locating and using evidence.7 Although these research bulletins have not been formally evaluated, the user feedback suggested that information professionals can assist in knowledge transfer and dissemination of this variety, and the KRIS findings confirm that service planners appreciate high quality and individualised alerting. The KRIS review also recommended using simple balanced scorecard measures to develop a set of key performance indicators to suit the type of services offered, and user groups. This would require improved statistics on the number and type of user groups served, to help continuing evaluation of services to be expanded. The following indicates the type of measures that could be used: (i) customer focus (x% of public health ⁄ health promotion staff served by alerting service, visits made to y% of schools in a certain target area); (ii) learning and growth (number of literature searches conducted to support commissioning, induction training, plus measurement appropriate to skill development among KRIS staff); (iii) internal processes (percentage of materials available online for ordering, percentage of resources reviewed for currency); (iv) financial indicator (measure for income

generation and contribution to commissioning decisions). The survey only targeted existing service users, and although the results indicated that the information obtained had a much wider audience than the original requester, the expected impacts for current nonusers or less frequent users might be different. McKnight9 describes how in customer value research, the value and irritants are both assessed. The review identified some of things that irritated the current users, such as the opening hours, and by addressing these, the service should be able to attract and retain new users introduced through outreach and promotion work, assuming that some of the present irritants are deterring people from using the service. Adapting impact frameworks for other library settings The survey questionnaire was adapted from the existing toolkit guidance.1 Librarians replicating the study in other primary care trust library services might integrate the current and future impacts on practice (question 8) as the responses were so similar. Perhaps, the future impacts might best be explored during interviews or focus groups. Respondents to this survey were very generous in their open-ended comments, allowing the collection of examples of impact, but in other impact surveys, some of this detail on impact might need to be collected at interview. The IAM5 and customer value discovery9 both allow for identification of specific negative impacts (or irritants) as well as positive impacts on practice. For future development of the toolkit, such questions would help to identify whether too much (or too little) information was obtained, or whether the information was occasionally deemed harmful. Assessing whether staff time was saved is important in assessing the cost benefits of a library and information service, although such estimates are inevitably subjective3 and a straight skill substitution is not necessarily desirable. An evaluation of a clinical librarian service10 found that doctors were prepared to search for longer after introduction of the clinical librarian service, but were also more prepared to delegate urgent searches for patient care problems. One explanation might be that doctors, once trained, were more confident of finding


283

284


answers, and were more prepared to search, but also more aware of the timescales required, and able to delegate when necessary. These types of impacts are more subtle than the impact of e-knowledge resources that the IAM is designed to assess, but very important for health libraries trying to find a balance between empowering users to do their own searches, and providing a mediated searching service, for the more complex and important queries. Actions taken by KRIS As recommended in the action plan, the immediate main change initiated was to change the opening times, and both KRIS bases are now open throughout Monday–Friday 8.30–17.00. Other recommendations in the action plan were to increase the marketing and promotion of the service to potential users in the community – for example, other schools and colleges, and the KRIS team is doing this. Another feature of the service, to make it easier for users to plan their visits, is the development of the interactive website with an online booking system for health promotion resources. The KRIS team is also acting on the recommendation that the staff skill mix be reviewed to enable both bases to share staff, with training for some staff in literature searching, and outreach activities. Conclusion Adaptation of impact questions from an existing toolkit provided a clear assessment of the effectiveness of a primary care health information service that serves a broad spectrum of users, health professionals, health promotion specialists, community workers and educators. Respondents also indicated what they valued about the service, and how they thought the service should be improved, in the safety of anonymous reporting to an independent review team. A randomised sample might provide more robust evidence of the service impacts, and delineation of negative impacts, or weaknesses. However, finding out what customers value is helpful to those planning service changes as a result of an impact survey.

Acknowledgements The authors thank all the questionnaire respondents and the KRIS strategy group for their support. We also thank the reviewers for their helpful and constructive comments. Supporting Information Additional Supporting Information may be found in the online version of this article: Appendix S1. Impact study of the Knowledge, Resource and Information Service. Please note: Wiley-Blackwell are not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article. References 1 Weightman, A., Urquhart, C., Spink, S. & Thomas, R. The value and impact of information provided through library services for patient care: Developing guidance for best practice. Health Information and Libraries Journal 2009, 26, 63–71. 2 Brettle, A. Evaluating information skills training in health libraries. Health Information and Libraries Journal 2007, 24(Suppl. 1), 18–37. 3 McGowan, J., Hogg, W., Campbell, C. & Rowan, M. Justin-time information improved decision-making in primary care: a randomized controlled trial. PLoS ONE 2008, 3, e3785. doi:10.1371/journal.pone.0003785. 4 Pluye, P., Grad, R. M., Stephenson, R. & Dunikowski, L. A new impact assessment method to evaluate knowledge resources. AMIA 2005 Symposium Proceedings 2005, 609– 613. 5 Information Technology Primary Care Research Group. The Information Assessment Method. Department of Family Medicine, McGill University. 2009-05-19. Accessible at http://iam2009.pbworks.com/FrontPage. (Archived by WebCite at http://www.webcitation.org/5gsxVzICm) 6 Bawden, D., Calvert, A., Robinson, L., Urquhart, C., Bray, C. & Amosford, J. Understanding our value; assessing the nature of the impact of library services. Library and Information Research 2009, 33(105), 62–89. 7 Wilkinson, A., Papaioannou, D., Keen, C. & Booth, A. The role of the information specialist in supporting knowledge transfer: a public health information case study. Health Information and Libraries Journal 2009, 26(2), 118–125. 8 Bell, K., Glover, S. et al. The impact of library services in primary care trusts in NHS North West England: a largescale retrospective quantitative study of online resource


Planning on the basis of impact evidence, Christine Urquhart et al. usage in relation to types of service. Health Information and Libraries Journal 2009, 26(2), 136–142. 9 McKnight, S. Acquisition and cataloguing processes: changes as a result of customer value research. Evidence Based Library and Information Practice 2007, 2, 22–36.

10 Urquhart, C., Turner, J., Durbin, J. & Ryan, J. Changes in information behavior in clinical teams after introduction of a clinical librarian service. Journal of the Medical Library Association 2007, 95, 14–22. Received 5 February 2010; Accepted 14 May 2010


285