Testing the Reliability of a Measure of Motivation to ...

2 downloads 0 Views 985KB Size Report
Alexis R. Dewar, Tyler P. Bull, Jessica M. Sproat, Natalie P. Reyes, Donna M. Malvey, and. James L. ... boomers (Colby & Ortman, 2014) or military veterans.
Proceedings of the Human Factors and Ergonomics Society 2016 Annual Meeting

1114

TESTING THE RELIABILITY OF A MEASURE OF MOTIVATION TO ENGAGE WITH TELEHEALTH TECHNOLOGY Alexis R. Dewar, Tyler P. Bull, Jessica M. Sproat, Natalie P. Reyes, Donna M. Malvey, and James L. Szalma University of Central Florida Telehealth systems and devices allow for the exchange of important healthcare data between users and practitioners (Malvey & Slovensky, 2014). To date, there has been little research on the motivation underlying engagement and adoption of telehealth systems. Given this, the present study tested a new measure, the mHealth Technology Engagement Index (mTEI), across two student samples (N = 191). Participants interacted with two types of telehealth devices or two types of telehealth interfaces, then reported their motivation to engage with these devices or interfaces. The purpose of this research was to establish the reliability of the mTEI and validate its usefulness across differing devices and interfaces. We also discuss the importance of developing a measure based on strong theoretical support.

Copyright 2016 by Human Factors and Ergonomics Society. DOI 10.1177/1541931213601261

INTRODUCTION The delivery of successful health interventions has rapidly increased in recent years for several reasons. The first and most obvious reason is an enormous increase in the human population, along with this is the increase in special populations, such as retiring baby boomers (Colby & Ortman, 2014) or military veterans (National Center for Telehealth and Technology, 2013). To address the health needs of an ever-growing population were telehealth or mHealth (short for mobile health) systems. A telehealth system is a technological device (i.e., computer, tablet, mobile phone application, etc.) that facilitates the exchange of important healthrelated information between a healthcare provider and patient in real-time through electronic means (Malvey & Slovensky, 2014). Telehealth systems also improve access to quality care (Wootton, 2012). Telehealth systems have been used successfully to supplement face-to-face patient-practitioner interactions, which in turn, has substantially reduced the cost of medical- and psychological-related visits to hospitals (Force & Brady, 2013; Healthcare Intelligence Network, 2013; Hilty, Luo, Morache, Marcelo, & Nesbitt, 2002). Telehealth systems have also improved the affordability of healthcare in general (Chong & Moreno, 2012; Melenhorst, Rogers, & Bouwhuis, 2006). Given this, the American Telemedicine Association (2009) anticipates that the amount of telehealth device usage will at least double, or quite possibly triple within the next five years. Subsequently, telehealth technology will become part of a patient’s routine and lifestyle, which will require consistent usage of the technology to ensure the effectiveness of and adherence to their individualized care plan.

Hence, it is important to understand how patients engage with these systems and what motivates them to use the system from a human factors perspective. To test the motivation underpinning human interaction with telehealth systems, we developed the mHealth Technology Engagement Index, or the mTEI (Dewar, Bull, Malvey, & Szalma, 2016). The mTEI is a 16-item measure that draws from theories of human motivation and engagement (Ryan & Deci, 2008; Ryan, Patrick, Deci, & Williams, 2008), as well as the technology acceptance literature (Schepers & Wetzels, 2007; Venkatesh & Davis, 2000; Venkatesh, 2000). THE PRESENT STUDY The present study seeks to further establish the reliability of this measure in addition to testing the validity of the mTEI across two studies. One limitation of the mTEI is that it has not been tested across multiple types of telehealth devices (e.g., computers, laptops, mobile devices, tablets, etc.) or telehealth interfaces (e.g., mobile applications, desktop applications, etc.). Therefore, the present set of studies seeks to compare the reliability of the mTEI to previous studies and demonstrate reliability across interfaces and device types. STUDY ONE Due to differences in the design of telehealth systems, some devices or applications are inherently more interesting and engaging from the perspective of motivational affordances (Szalma, 2014). However, the mTEI should not be sensitive to differences in telehealth application type because it is tapping a latent, individualized trait: motivation to engage with the system. To test this hypothesis, we selected two of the

Proceedings of the Human Factors and Ergonomics Society 2016 Annual Meeting

most frequently downloaded telehealth applications: one that affords engagement, the T2 Mood Tracker™ mobile application (National Center for Telehealth and Technology, 2013), and one that affords less engagement, the What’s My M3?™ mobile application (Gaynes et al., 2010), respectfully. What’s My M3?™ is an application used to rapidly diagnose anxiety, depression, bipolar disorder, and post-traumatic stress disorder. It also immediately connects the user to mental health services in their area. On the other hand, the T2 Mood Tracker™ is a mobile application designed to monitor and measure emotional health in military personnel over time. This application is now widely available for free in the civilian sector and is used by many counselors outside of the military. We hypothesized that the mTEI should not be sensitive to differences in telehealth application type because we have developed a measure of a stable trait, motivation to use telehealth systems. Method Participants. A sample of 80 students (Sample A) enrolled in a psychology course agreed to participate in Study One for either course credit or extra credit. All participants reported owning at least one mobile phone and at least one computer, with many students additionally reporting that they owned at least one tablet. Individuals that had never interacted with or owned a computer or mobile phone were not allowed to partake in the study. The characteristics of Sample A are shown in Table 1. Table 1. Participant Demographics Across Samples A and B Sample Characteristic A B Cronbach’s alpha (α) .929 .884 N 80 111 Age range (in years) 18-25 18-41 Age mean (in years) 18.56 19.84 Female % 68.4 73.9 Male % 31.6 26.1 African American % 11.4 12.6 Asian % 7.6 7.2 Caucasian % 56.2 49.5 Hispanic % 16.5 22.5 Native American/Alaskan Native % 0.0 0.9 Biracial/Multiracial % 5.0 5.4 Other ethnicity % 1.3 1.8 Note. Four participants did not indicate their gender and one participant did not indicate their preferred ethnicity in Sample A.

1115

Measures. In addition to including demographic information, students completed several measures related to telehealth device acceptance and usage (reported elsewhere, Dewar et al., 2016), as well as the mTEI. All surveys used in this study were administered electronically on a desktop computer. Materials. Participants completed the experiment on a Windows 7 Dell™ desktop computer using Android emulation software (BlueStacks App Player™). This software allowed participants to interact with the two telehealth applications used in this study. Procedure. Upon arrival to the experiment, participants were randomly assigned to one of two conditions: 1) interaction with the What’s My M3? ™ application, or 2) interaction with the T2 Mood Tracker™ application, respectfully. After freely using and exploring the telehealth application on the computer for ten minutes, participants then completed the mTEI. All measures were presented in a random order and counterbalanced to control for order effects. Based on our analyses, no order effects were observed). Results and Discussion Intercorrelations. The condition to which participants were assigned was not significantly correlated with mTEI score (r = .020). Additionally, the number of technological devices owned was not significantly correlated with mTEI score (r = -.015), nor was the frequency of usage of these technological devices (r = -.024). Because Android emulating software was used in this study, it is possible that the ownership of a specific brand (i.e., Apple™, Windows™, Android™) could influence the motivation of participants to interact with the computer and subsequent software, indicating a variable that will be important to control for in future studies. However, we found that all bivariate correlations for each type of device owned and mTEI score were not significant with the one exception of Sample B (see Table 2). Table 2. Correlations of Device Ownership with mTEI Score Across Samples A and B Device Type iPhone Android Phone Windows Phone iPad Android Tablet Windows Tablet Overall number of devices owned Average usage of all devices Note. * p < .05, ** p < .01.

Sample(s) A, B A, B A, B A, B A, B A, B A, B A

Correlation(s) -.057, -.151 .004, .173 -.147, .002 -.129, -.066 .158, .188* .147, -.020 -.015, .076 0.24

Proceedings of the Human Factors and Ergonomics Society 2016 Annual Meeting

Statistical Tests. An independent-samples t-test between the T2 Mood Tracker™ application (M = 51.75, SD = 11.96) and What’s My M3? ™application (M = 52.21, SD = 11.23) revealed no significant difference in mTEI scores across interface type, t(78) = -.173, p = .863, Cohen’s d = .04. These results indicate that the mTEI is resistant to differences in application type and favorability of brand names, thus providing more evidence that motivation to engage with telehealth systems is a unique individual difference. Reliability. In previous studies (Dewar et al., 2016), the reliability of the mTEI was found to range between α = .885 - .898. In Study One, reliability was quite high with Cronbach’s α = .929 (16 items). STUDY TWO In Study One, we found that the mTEI was not susceptible to differences based on interface type. However, it was unknown if the mTEI is sensitive to differences based on device type (e.g., computer, phone, tablet, etc.). In Study Two, we expand upon the results of Study One by testing whether scores on the mTEI vary as a function of the type of device used (i.e., computer-based or mobile-based). Again, we hypothesized that mTEI scores would be similar between device types because motivation to use telehealth devices should not vary as a function of a systematic motivational affordance (Szalma, 2014), rather it should be stable across devices because it is an individual difference. Method Participants. A new sample of 111 (Sample B) students enrolled in a psychology course at the same university agreed to participate in Study Two for either course credit or extra credit. Again, all participants reported owning at least one mobile phone, computer, or tablet device. Those that had never interacted with or owned a computer or mobile phone were not allowed to participate in the study. See Table 1 for a summary of participant demographics. Materials. Participants completed the experiment on either a Windows 7 Dell™ desktop computer using Android emulation software (BlueStacks App Player™) or an Android Tablet. The user interfaces were identical on both devices, which allowed participants to interact with the T2 Mood Tracker™ (National Center for Telehealth and Technology, 2013) in the same way (with the exception of using the computer mouse to navigate the computer). The T2 Mood Tracker™ application was selected for this study

1116

because it affords more engagement. All surveys were administered electronically on the desktop computer. Procedure. In this experiment, participants were randomly assigned to interact with the T2 Mood Tracker™ application on either an Android tablet (i.e., mobile-based condition) or on a desktop computer (i.e., computer-based condition). After freely exploring and using the telehealth application for seven minutes, participants then completed the mTEI and demographics. Results and Discussion Intercorrelations. The condition to which participants were assigned was not significantly correlated with mTEI score (r = -.111). Additionally, the number of technological devices owned was not significantly correlated with mTEI score (r = .076). Because Android emulating software and an Android tablet were used in this study, it is again possible that the ownership of a specific brand (i.e., Apple™, Windows™, Android™) could influence the motivation of participants to interact with the computer and subsequent software, which could bias our results. Nearly all bivariate correlations for each type of device owned and mTEI score were not significant, with the exception of Android tablet ownership, which was weakly, negatively correlated with mTEI score (see Table 2). This indicates partial evidence for some bias in motivation to use telehealth devices based on brand familiarity. Statistical Tests. An independent-samples t-test between the mobile-based device condition (M = 52.87, SD = 10.09) and computer-based device condition (M = 54.94, SD = 8.05) showed no significant difference in mTEI scores, t(109) = 1.170, p = .245, Cohen’s d = .22. Of the 11 participants who owned Android tablets, those in the mobile-based device condition (N = 5, M = 57.00, SD = 6.12) did not significantly rate their motivation to engage with the Android tablet any higher those who did not own an Android tablet (N = 57, M = 52.51, SD = 10.33), t(60) = .953, p = .344, Cohen’s d = .24. Generally, these results indicate that the mTEI is resistant to differences in device type and favorability of specific brand names. Reliability. In Study Two reliability was not quite as high as previous tests of the mTEI, but was still within the acceptable range for a reliable measure with Cronbach’s α = .884 (16 items). GENERAL DISCUSSION The purpose of these studies was to show that a newly developed measure of motivation and engagement

Proceedings of the Human Factors and Ergonomics Society 2016 Annual Meeting

is reliable across device and interface types. We have also demonstrated that this measure is reliable across time and valid in student samples. Future studies should aim to test the mTEI in more specialized populations, such as hospitals or counseling centers. The mTEI also needs to be normed across a population, not just a sample. Importantly, the mTEI supplements current theory on acceptance, engagement, and usability by incorporating, rather than assuming, motivation to engage with telehealth systems. We believe that the mTEI will help user experience researchers and designers measure baseline motivation to use telehealth systems. Significant changes in pre- and post-mTEI scores would indicate that participants were more autonomously motivated to use the telehealth system after interaction. This is crucial in patient well-being. Furthermore, the mTEI can be indicative of a general lack of motivation from individuals. If motivation to engage with telehealth technology is initially low, it could be important to intervene with individualized training or education to increase motivation to utilize telehealth devices. To summarize, by studying the motivation underlying the acceptance and usage of telehealth technology, we better understand why users engage with telehealth systems, not just who or how many people use these devices. This is an important step forward for both human factors, as well as the field of telehealth: it is critical to consider individual differences in technological design, especially for these designs to be useful, usable, and satisfying. REFERENCES American Telemedicine Association. (2015). Research Outcomes: Telemedicine’s Impact on Healthcare Cost and Quality. Retrieved from: www.americantelemed.org/docs/default source/policy/examples-of-research-outcomestelemedicine's-impact-on-healthcare-cost-andquality.pdf Chong, J., & Moreno, F. (2012). Feasibility and Acceptability of Clinic-Based Telepsychiatry for Low-Income Hispanic Primary Care Patients. Telemedicine and eHealth, 18(4), 297-304. Colby, S. L., & Ortman, J. M. (2014). The Baby Boom Cohort in the United States: 2012 to 2060. Population Estimates and Projections, Washington, DC: US Census Bureau. Dewar, A.R., Bull, T.P., Malvey, D.M., & Szalma, J.L. (2016). Developing a Measure of Engagement with Telehealth Systems: The mTEI. Journal of Telemedicine and Telecare, [ePub ahead of print]. doi: 10.1177/1357633X16640958 Force, S., & Brady, C. (2013). Case study: Lee Memorial

1117

Health System the role of telehealth in an integrated health delivery system. Caring: National Association for Home Care Magazine, 32(8), 28. Gaynes, B.N., DeVeaugh-Geiss, J., Weir, S., Gu, H., MacPherson, C., Schulberg, H.C., Culpepper, L., & Rubinow, D.R. (2010). Feasibility and Diagnostic Validity of the M3 Checklist: A Brief, Self-Rated Screen for Depressive, Bipolar, Anxiety, and PostTraumatic Stress Disorders in Primary Care. Annals of Family Medicine, 8(2), 160-169. Healthcare Intelligence Network. 2013 Healthcare Benchmarks: Telehealth and Telemedicine, Third Annual Edition. Sea Girt, NJ: Healthcare Intelligence Network. Retrieved from: http://www.hin.com Hilty, D.M., Luo, J.S., Morache, C., Marcelo, D.A., & Nesbitt, T.S.(2002). Telepsychiatry: An Overview for Psychiatrists. Central Nervous System (CNS) Drugs, 16(8), 527-548. Malvey, D., & Slovensky, D.J. (2014). mHealth: Transforming Healthcare. New York, NY: Springer. Melehhorst, A.S., Rogers, W.A., Bouwhuis, D.G. (2006). Older adults’ motivated choice for technological innovation: evidence from benefit-driven selectivity. Psychology and Aging, 21(1), 190-195. National Center for Telehealth and Technology. (2013). 2013 Annual Report. Joint Base Lewis-McChord, WA: Defense Centers of Excellence for Psychological Health and Traumatic Brain Injury. Ryan, R.M., & Deci, E.L. (2008). Self-Determination Theory and the Role of Basic Psychology Needs in Personality and the Organization of Behavior. In O.P. John, R.W. Robins, & L.A. Pervin (Eds.), Handbook of personality theory and research, 3rd ed. (pp. 654678). New York, NY: Guilford Press. Ryan, R.M., Patrick, H., Deci, E.L., & Williams, G.C. (2008). Facilitating healthy behaviour change and its maintenance: Interventions based on SelfDetermination Theory. The European Health Psychologist, 10, 2-5. Schepers, J., & Wetzels, M. (2007). A meta-analysis of the technology acceptance model: Investigating subjective norm and moderation effects. Information & Management, 44, 90-103. doi: 10.1016/j.im.2006.10.007 Szalma, J.L. (2014). On the Application of Motivation Theory to Human Factors/Ergonomics: Motivational Design Principles for Human-Technology Interaction. Human Factors, 56(8), 1453-1471. Venkatesh, V., & Davis, F.D. (2000). A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Management Science, 46(2), 186-204. Venkatesh, V. (2000). Determinants of Perceived Ease of Use: Integrating Control, Intrinsic Motivation, and Emotion into the Technology Acceptance Model. Information Systems Research, 11(4), 342-365. Williams, G.C. (2002). Improving Patients’ Health Through Supporting the Autonomy of Patients and Providers.

Proceedings of the Human Factors and Ergonomics Society 2016 Annual Meeting

In E.L. Deci and R.M. Ryan (Eds.), Handbook of Self-Determination Research (pp. 233-254). Rochester, NY: The University of Rochester Press. Wootton, R. (2012). Twenty years of telemedicine in chronic disease management- an evidence synthesis. Journal of Telemedicine and Telecare, 18, 211-220. doi: 10.1258/jtt.2012.120219

1118