evaluation of Project SHARE, a teen health ... addition, adolescents' orientation toward the future ... Project SHARE implementation and evaluation 73.
72 CASE STUDY DOI: dx.doi.org/10.5195/jmla.2019.351
Lessons learned from multisite implementation and evaluation of Project SHARE, a teen health information literacy, empowerment, and leadership program Alla Keselman, PhD, MA; Rachel Anne Chase, MPH; Jennifer Rewolinski, BS; Yulia Chentsova Dutton, PhD; Janice E. Kelly, MLS, FMLA See end of article for authors’ affiliations.
Background: This case study describes the implementation and evaluation of a multisite teen health information outreach program. The objectives of the program were to increase health knowledge, health information literacy, interest in health careers, community engagement, and leadership skills of teens in disadvantaged communities. Case Presentation: Teens at six sites across the country participated in a multi-week curriculum that focused on various aspects of health literacy, information literacy, and leadership. Lesson topics addressed personal health, social determinants of health, information quality, and communication and advocacy skills. Program evaluation included both quantitative and qualitative components and focused on multiple knowledge and skills outcome variables. Results suggested that while teens at all sites showed improvement, particularly with respect to engagement and interest in the topics, the degree of gains in knowledge and information literacy measures varied significantly from site to site. Conclusion: On-site implementation planning, cohesive integration of added activities, and emphasis on retention can contribute to implementation and evaluation effectiveness. This work also underscores the limitation of a purely quantitative approach to capturing the impact of health information and stresses the importance of supplementing numerical scores and statistics with qualitative data.
This article has been approved for the Medical Library Association’s Independent Reading Program . See end of article for supplemental content.
BACKGROUND Helping adolescents navigate health information has many potential benefits. Many US adolescents, particularly those from disadvantaged backgrounds, have limited health literacy [1], which is associated with negative health behaviors and outcomes. As many lifetime health habits form during adolescence [2], it is important to target this age group in outreach efforts. Adolescence is also the time of career choices, and health information programs are a way to raise awareness about health careers with a
Journal of the Medical Library Association
goal to diversify the health care workforce. In addition, adolescents’ orientation toward the future and interest in social action make them potentially great partners for health-related community outreach and advocacy. Supporting health information outreach programs for adolescents is an important mission of the National Library of Medicine (NLM). Programs typically aim to produce several outcomes. For example, two programs conducted in low-income, primarily minority schools—Vital
107 (1) January 2019
jmla.mlanet.org
P r o j e c t S H A R E i m p l e m e n t a t i o n a n d e v a l ua t i o n DOI: dx.doi.org/10.5195/jmla.2019.351
Information for a Virtual Age (¡VIVA!) Peer Tutor Project (with the University of Texas Health Sciences Center at San Antonio) and the South Carolina Teen Health Leadership Program (with the Medical University of South Carolina)—trained groups of high school students to develop and conduct health information outreach in their schools and larger communities [3–5]. These programs had positive impacts on the participants’ health literacy, sense of empowerment, communication, leadership skills, and interest in health careers. ¡VIVA! also had an impact on the broader school community. Health information outreach projects usually occur in community settings and are carried out by organizations with limited budgets, time, and staff, often without evaluation expertise. In a review of published studies evaluating health information outreach, the authors found that few evaluations involve pretest-posttest designs, test for statistical significance, or measure long-term project impacts [6]. Projects often report effectiveness in terms of numbers of attendees and rarely measure health knowledge, information literacy, and behaviors. Therefore, we recommended that outreach programs strengthen their evaluation planning and the breadth of their outcome measures (e.g., to include a range of information behaviors and attitudes). STUDY PURPOSE This paper describes and evaluates the implementation of a health information outreach program aiming to improve health knowledge, information literacy, interest in health careers, community engagement, and leadership and communication skills of teens in disadvantaged communities. The program was a collaboration between NLM and the National Area Health Education Centers Organization (NAO). The protocol was approved by the National Institutes of Health Office of Human Subjects Research (survey) and the National Institute of Child Health and Human Development Institutional Review Board (focus groups).
jmla.mlanet.org
107 (1) January 2019
CASE PRESENTATION Project SHARE teen health literacy and leadership curriculum The NLM-NAO collaboration involved an adaptation of the Student Health Advocates Redefining Empowerment (Project SHARE) health literacy and leadership curriculum that was developed by the Health Sciences and Human Services Library of the University of Maryland, Baltimore, with NLM funding. The six-module (nineteen individual lessons) program was a combination of lectures, group discussions, and student-led health promotion and advocacy activities (supplemental Appendix A) that aligns with the “National Health Education Standards.” Module I introduced teens to health disparities and social determinants of health and prompted them to discuss concerns in their communities. Module II focused on health literacy, information seeking, and evaluation and built foundational skills for later advocacy projects. In module III, students learned about important strategies for maintaining personal health, such as prevention, awareness of family history, and doctor-patient communication. Module IV was about healthy nutrition, including food labels and meal planning. Modules V and VI focused on essential community health advocacy competencies: effective communication, leadership, health policy, and advocacy/outreach. National Library of Medicine (NLM)–National Area Health Education Centers Organization (NAO) Project SHARE adaptation and implementation process To carry out the project, NLM contracted NAO to select six local Area Health Education Centers (AHECs) from low-income and minority communities that were interested in adapting and testing the curriculum: •
Boston: urban community
•
Brooklyn-Queens-Long Island (BQLI): urban community
•
Northeastern Colorado (NE CO): rural community
•
Southwestern Colorado (SW CO): rural, Native American tribal territory
Journal of the Medical Library Association
73
74
Keselman et al. DOI: dx.doi.org/10.5195/jmla.2019.351
•
Eastern Connecticut (E CT): Native American tribal territory
•
Montana (MT) AHECs: rural, Native American tribal territory
While Project SHARE modules that were chosen for implementation varied somewhat among sites (supplemental Appendix A), all sites included an overview of health disparities and quality health information (modules I and II). All but one site emphasized introducing student participants to health or health information careers. Finally, all sites included additional student leadership activities that aimed to give opportunities to draw upon the information from earlier lessons. For example, students in Boston produced videos about the importance of cultural competence in health care, and students in Colorado participated in healthy cooking competitions that parents and community members attended. Sites also conducted field trips and hosted guest speakers. Staff from all sites participated in biweekly conference calls with one another and with NAO and NLM in which they shared experiences and ideas and discussed any needed adjustments.
located. In MT, the program was conducted in a school, and the AHEC staff inadvertently scheduled the posttest after the participating seniors graduated and could not be reached. Only paired data for the students who completed both pretests and posttests were used in the analysis. Because of low participant numbers in NE and SW CO, findings from those sites should be interpreted with caution. Evaluation instruments and procedure Evaluation procedures and instruments were developed by an NLM evaluator in collaboration with the AHEC teams and included quantitative (survey) and qualitative (focus group) components in accordance with health information outreach evaluation guidelines [7, 8]. The quantitative component involved pretest and posttest surveys of variables aligned with our framework for evaluating health information outreach [5, 6]. The variables, organized into eight sections or clusters, were as follows (second-level bullets denote variables within the cluster): •
Participants Table 1 shows the number of students participating in various evaluation activities at each site. The greatest challenge of the evaluation, common in community research settings, was obtaining data from all the students. In SW CO, a very-near car accident caused the pretest to be rescheduled. In NE CO, the program lead changed jobs soon after the posttest, and some of the posttests could not be
•
Cluster 1: Knowledge of health disparities and social determinants of health ○
Number of factors recognized as social determinants of health
○
Average proportion of possible explanations per recognized health determinant
○
Proportion of possible reasons explaining a local disparity
Cluster 2: Understanding of the importance of knowing one’s family history ○
Understanding of health relevance of one’s family history
Table 1 Participants in evaluation activities (numbers represent individual students)
Posttest
Quantitative analysis (students with both pretest and posttest)
Boston
14
8
14
9
8
8
BQLI
12
8
12
8
8
8
MT
16
16
12
11
8
13
NE CO
15
9
12
3
3
3
SW CO
12
11
10
7
6
3
E CT
26
26
21
24
20
18*
Total
95
78
81
62
53
53
Site
Started program
Completed program
Pretest
Focus group
BQLI=Brooklyn-Queens-Long Island; NE CO=Northeastern Colorado; SW CO=Southwestern Colorado; E CT=Eastern Connecticut; MT=Montana. * Two groups.
Journal of the Medical Library Association
107 (1) January 2019
jmla.mlanet.org
P r o j e c t S H A R E i m p l e m e n t a t i o n a n d e v a l ua t i o n DOI: dx.doi.org/10.5195/jmla.2019.351
•
•
•
Cluster 3: Knowledge of health risk factors ○
Knowledge of health risk factors one can control
○
Knowledge health risk factors one cannot control
Cluster 4: Knowledge of preventive health ○
Awareness of diseases that are public health concerns in the United States
○
Average number of known preventive health measures per disease
○
Preventive measure recognition
Cluster 5: Knowledge of nutrition ○
•
•
•
Knowledge of nutritional groups and the basics of food labels
Cluster 6: Information evaluation skills ○
Recognition of information quality markers of a hoax website
○
Recognition of information quality markers of an authoritative website
○
Knowledge of general online information quality criteria
Cluster 7: Awareness of quality health information resources ○
Awareness of quality health information sites
○
Awareness of MedlinePlus
Cluster 8: Knowledge of and interest in health careers ○
Number of health occupations known
○
Average knowledge score per known health occupation
○
Number of health occupations of interest
Students at each site answered survey sections pertaining to lessons presented at their sites. The qualitative component involved post-project focus group discussion of students’ experience. The full survey can be found in supplemental Appendix B, and supplemental Appendix C contains detailed description of the outcome variables and their correspondence to Project SHARE lessons and survey questions.
as explanations of social determinants of health or information quality criteria, involved narrative answers and were coded and scored by comparing students’ answers against gold standard models obtained from expert response or existing guidelines (e.g., Evaluating Internet Sources: A Library Resource Guide). Two coders coded a subset of all narrative data establishing “substantial” to “almost perfect” levels of inter-rater reliability as evidenced by Cohen’s kappa values >0.61 [9]. To reduce the number of comparisons while assessing the significance of variables across sites, continuous outcome variables from each cluster were included in repeated measure multivariate analysis of variance (MANOVA) accounting for participants being nested within sites. If MANOVA showed a significant main effect of time (pretest to posttest) or time by site interaction for a cluster, further analysis was performed to identify which variables drove the difference. In addition, we conducted paired sample t-tests for within-site comparisons. As the results specified which variables drove significant main or interaction effects, Bonferroni correction for multiple comparisons was not applied. Categorical variables were analyzed using McNemar tests. Details of the statistical analysis are described in supplemental Appendix D. Due to data attrition, the analyses are underpowered and, likely, overly conservative. QUANTITATIVE RESULTS: SURVEY ANALYSIS There were some significant improvements in six out of eight clusters. However, they were limited to specific cluster variables and individual sites. A summary of statistically significant pretest to posttest improvements for clusters, overall variables across sites, and variables within individual sites is presented in Table 2. The findings suggest overall across-sites improvements in Cluster 4: Knowledge of preventive health and Cluster 6: Information evaluation skills. In addition, the analysis suggests significant time by site interactions, which indicates differential program impact on different sites, for all clusters except Cluster 3: Knowledge of health risk factors and Cluster 7: Awareness of quality health information resources.
Survey coding and statistical analysis Some of the variables were simple counts of correct responses to multiple choice questions. Others, such
jmla.mlanet.org
107 (1) January 2019
Journal of the Medical Library Association
75
76
Keselman et al. DOI: dx.doi.org/10.5195/jmla.2019.351
Table 2 Summary of pretest-posttest improvements for individual sites Time
Time × Site
Boston
BQLI
MT
NE CO
SW CO
E CT
S
Knowledge of health disparities and social determinants of health Number of factors recognized as social determinants of health
S
Average proportion of possible explanations per recognized health determinant
S
S
S
Proportion of possible reasons explaining a local disparity
S
S
S
Awareness of the importance of knowing one’s family health history Awareness of health relevance of one’s family history
S
NA
NA
NA
NA
S
Knowledge of health risk factors one can control
NA
NA
NA
NA
S
Knowledge of health risk factors one cannot control
NA
NA
NA
NA
S
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
S
NA
NA
NA
NA
S
Knowledge of health risk factors
Knowledge of preventive health Awareness of diseases that are public health concern in the United States
S
S
S
S
Average number of preventive health measures per disease Preventive measures recognition*
S
S
S
Knowledge of nutrition Knowledge of nutritional groups and the basics of food labels Information evaluation skills
S S
S
Recognition of information quality markers of a hoax site
S
Recognition of information quality markers of an authoritative site
S
Knowledge of general online information quality criteria
S
S
S S
S S
S
S
S
Awareness of quality health information resources Awareness of quality health information sites Awareness of MedlinePlus*
S S
Knowledge and interest in health careers
S
S
S
Number of health occupations known
NA
NA
Average knowledge score per known health occupation
NA
NA
Number of health occupations of interest
S
NA
S
S
S S
NA
Key: Time=Time (pre-post) effect, Time × Site=time by site interaction effect, S=Statistically significant at p
