Maturity of Hospital Information Systems: Most Important Influencing Factors João Vidal Carvalho1, Álvaro Rocha2, António Abreu3 1
Politécnico do Porto, ISCAP, S. Mamede de Infesta, Portugal
Departamento de Engenharia Informática, Universidade de Coimbra, Coimbra, Portugal
Politécnico do Porto, ISCAP, S. Mamede de Infesta, Portugal
Abstract – Maturity models facilitate organizational management, including information systems (IS) management, with hospital organizations no exception. The present paper puts forth a study carried out with a group of experts in the field of hospital information systems (HIS) management with a view to identifying the main influencing factors to be included in an encompassing maturity model for HIS management. This study is based on the results of a literature review, which identified maturity models in the health field and relevant influencing factors. The development of this model is justified to the extent that the available maturity models for the HIS management field reveal multiple limitations, including lack of detail, absence of tools to determine their maturity and lack of characterization for stages of maturity structured by different influencing factors. Keywords – Maturity Models, Hospital Information Systems, Maturity Influencing Factors
1. Introduction Health institutions, in line with government organizations, are starting to realize that the reasons behind a certain inability to properly manage health processes are directly related to the limitations of technological infrastructures and the lack of efficiency in their management [1, 2]. HIS managers often look at the mistakes experienced in these organizations and wonder what could have been done to avoid them . The fact is that these errors are usually natural symptoms of the growth and maturation of organizations, which often derive from the development that brought the organization to its current maturity stage . This phenomenon of change, experienced by an organization from inception to maturity, reflects the principles underlying stages of growth theory and the current context of IS in health organizations. Based on this presupposition, which highlights the significance of maturity models in the HIS field, we are developing an investigation project with the purpose of creating a maturity model that is specially adapted to the needs of HIS management. To develop this new model, we initially carried out a systematic literature review to identify the main models currently existing in this field; we identified 14 maturity models in the health field. We found that the different identified models had a more specific character, representing health subsystems, and a broader scope, that is, they
represented HIS as a whole . It is also important to note that, from the identified maturity models comprising multiple dimensions or influencing factors, not all explicitly discriminated their characteristics according to each stage of maturity. In none of the models did the authors assign a different weight to each influencing factor; in other words, when accessing the general maturity of a health IS, all associated influencing factors were equally significant . As a result of this study, we observed that none of the identified models was sufficiently broad in its scope to include all areas and subsystems of a health organization. In this sense, we believe there is room for the development of a holistic maturity model that includes a comprehensive set of influencing factors. In the present paper, we begin by describing the approach that led to the identification of 12 influencing factors connected to maturity models in the health field. Following the identification of these influencing factors, we offer a brief description of each. Subsequently, we describe and analyze a survey carried out with a group of experts in the HIS management field, with a view to identifying the main factors that must be included in an encompassing maturity model for HIS management. To facilitate understanding on the part of the respondents, influencing factors were designated as subareas of IS and technologies (IST) for healthcare.
2. Characterization of the Influencing Factors As previously mentioned, following a strategy based on a broad and structured literature review , we tried to identify and systematize a group of hospital IST-related maturity models. Simultaneously, we wanted to identify and characterize, in the aforementioned models, dimensions that could be viewed as influencing factors in an encompassing maturity model. By the end of the process, 12 influencing factors were considered relevant. These will be presented and described in this paper. To help understand why we opted for or selected these 12 influencing factors, it is important to mention three assumptions adopted in our investigation work in the course of this identification. Our first assumption was to define influencing factors based on the lines of development for HIS, as identified by Haux . Table 1 displays Haux’s seven lines of development, as well as their respective influencing factors. Table 1 – Connection between the lines of development for HIS and influencing factors adopted in this investigation work Lines of development for HIS (1) The shift towards computer-based processing and storage, as well as the increase in data processing
Dimensions/influencing factors Electronic medical records
(2) The shift from local to global IS architectures
Systems and IT infrastructure Cooperation Interoperability Information security People Usability
(3) The inclusion of patients and health consumers as HIS users, besides healthcare professionals and administrators (4) The usages of the data not only for patient care and administrative purposes, but also for healthcare planning and clinical research (5) The shift of focus from mainly technical HIS problems to those of change management, as well as strategic information management (6) A shift from (predominantly) alphanumerical data to clinical images, and now on data at the molecular level (7) The steady increase of new technologies to be included in order to enable continuous monitoring of the health status of patients
Data analysis Strategy PACS mHealth Telemedicine
As to the attribution of influencing factors according to the lines of development by Haux, it is important to justify the association of mobility and telemedicine to line of development (7). Of all the influencing factors identified in our literature review, these are the only ones that fail to directly fit a line of development in IST for healthcare. However, as the area of action of this line of development congregates new patient health monitoring technologies, we believe that they fit this area and should thus integrate the influencing factors list to be analyzed. As for line of development (2), it is thought to be rather encompassing, with the capacity to comprise several influencing factors in hospital IS architecture, namely, systems and IT infrastructure, cooperation, interoperability and IS. The second assumption was to avoid, if possible, the definition of influencing factors that could integrate, even partially, common characteristics or activities. This assumption was difficult to manage in a number of situations. We decided to maintain influencing factors with common traits whenever we were unable to define different influencing factors without losing meaning or identity. For instance, the systems and IT infrastructure factor involves several characteristics that can be observed in other influencing factors, namely, collaboration and interoperability. Similarly, the people factor incorporates a number of the characteristics of usability. Finally, the third assumption was to identify and define the most encompassing influencing factors we could in order to group potentially similar or related factors. In the course of our literature review , we observed that several maturity model factors referred to the same type of characteristic or activity, yet differed in the field of action. For instance, the NHS Infrastructure Maturity Model, the NEHTA Interoperability Maturity Model and the Hospital Cooperation Maturity Model display organizational, informational and technological influencing factors. In these three cases, we adopted more encompassing influencing factors, namely, systems and IT infrastructure in the first case, interoperability in the second, and collaboration in the third. Similarly, the Picture Archiving and Communication System (PACS) Maturity Model incorporates several influencing factors from the specific PACS field. However, we adopted PACS as the most encompassing influencing factor. The same procedure was applied to the following influencing factors: data analysis, usability, telemedicine and electronic medical records. All these examples include specific influencing factors that are circumscribed to their respective fields of action. Faced with this diversity of potential dimensions in different influencing factors, we thought it would be more useful and practical to consider a single factor encompassing all factors relating to the same area. This led to a significant reduction in the number of factors and, consequently, a lesser complexity of the model to be developed. We will now list and describe the 12 influencing factors in IST for healthcare maturity models, which emerged from our literature (see Table 2). For each influencing factor under consideration, we defined: (a) a designation, (b) a short definition or description, which was included in our survey questionnaire with the purpose of helping experts understand these influencing factors (Section 3 of this paper); (c) the identification of the IST for healthcare maturity models where these influencing factors were signaled; (d) and the identification of the main references, which allowed us to identify the influencing factor and the context in which it is referred.
Table 2 – Early list of influencing factors Influencing factors designation Data analysis Cooperation Strategy
Interoperability mHealth PACS People
Electronic medical records
Information security Systems and IT infrastructure
Health IST maturity models Healthcare Analytics Adoption Model  Hospital Cooperation Maturity Model  Forrester Model  Hospital Cooperation Maturity Model  PACS Maturity Model  NEHTA Interoperability Maturity Model  Forrester Model  IDC Mobility Maturity Model for Healthcare  PACS Maturity Model  Healthcare Usability Maturity Model  Telemedicine Service Maturity Model  PACS Maturity Model  EMRAM  Forrester Model  EPR Maturity Model  Quintera Maturity Model  NHS Infrastructure Maturity Model  IDC Healthcare IT Maturity Model  Healthcare Usability Maturity Model  Telemedicine Service Maturity Model  Telemedicine Service Maturity Model  Healthcare Usability Maturity Model 
It is important to note that the debate surrounding influencing factors in maturity models is, in some cases, quite limited due to the lack of information on this subject in the literature of reference. In this investigation work, we tried to base our description of influencing factors on information available from bibliographical sources, thereby avoiding explanations stemming from the understanding of the author on this matter. However, it is also important to mention that not all authors present evidence supporting their claims regarding these influencing factors.
3. Description of the Influencing Factors We will now describe the identified influencing factors. Data analysis – Health field-related process that uses and transforms raw data to produce knowledge and information that can be used in decision-making processes. Cooperation – The activities carried out by several hospitals or hospital departments, where certain resources and skills are shared with a view to optimizing results, which benefits all intervening parties. Strategy – An action plan that leads the healthcare organization to a new position in order to achieve its mid-/long-term goals. Interoperability – The capability of IT systems comprising several programs and software applications, which are used to communicate, exchange data with accuracy, and share information effectively and consistently. mHealth – Practice of medicine and public health via mobile devices.
PACS – A picture archiving and communication system is associated with diagnostic imaging, which allows for immediate access to medical images in a digital format within any hospital sector. People - Skills, knowledge and abilities of people involved in the use, management and development of IST for healthcare. Electronic medical records – Electronic records containing information on the health of an individual, which can be created, congregated, managed and consulted by authorized physicians, medical staff and users inside a health organization. Information security – The protection of a set of information in order to preserve the value it holds for an individual or a health organization. Systems and IT infrastructure – This concerns the hardware, software, network resources and necessary services to operate and manage an IT environment in the health field. Telemedicine – A group of technologies and applications that allow for the provision of medical care at a distance. Usability – The measure of a product adopted by healthcare users to achieve specific goals effectively, efficiently and satisfactorily within the context of hospital and health Systems.
4. Description of the Survey The goal behind the administration of a survey via the use of questionnaire was to answer two research questions posed by this investigation project. These research questions tried to identify the most important influencing factors to be adopted in an IST for healthcare maturity model (RQ1) and confirm if the maturity of these influencing factors can be evaluated (RQ2). The answers to these questions were obtained from the analysis of opinions offered by a group of experts in IST for healthcare, which concerned 12 influencing factors (IST for healthcare subareas) identified in the course of our systematic literature review. In order to guarantee the validity and consistency of results, we tried to identify and invite a group of experts in the field under study, whose knowledge could ensure the credibility of the obtained answers. Diversity within the expert panel was also a concern, as we wanted to ensure heterogeneity, which is necessary for the success of the study. Thus, through a selection process involving a broad range of personalities, we initially invited 188 Portuguese experts in IST for healthcare, of which only 144 effectively received the invitation to participate (in other words, 44 of the 188 emails sent with regard to this survey were returned as failed to be delivered). Of the 144 experts that received the invitation to participate in the survey, 58 agreed to participate (40.3%). However, of these 58 experts, only 46 went onto effectively participate in our study by fully answering our questionnaire with validated answers. Besides these 46 experts who successfully completed the questionnaire, another 12 began to complete it but failed to do so. This way, the effective participation rate was 79%. Since this domain is characterized by a small number of experts and considering the participation rates obtained in similar studies, our participation rate can be considered to be quite reasonable. Moreover, for samples with more than
30 elements in each group under study, the violation of normality and homoscedasticity assumptions does not call into question the conclusions [19, 20]. The administration of the survey began on January 18, 2016, and the questionnaire was available for a period of 10 days (until January 28, 2016). The 144 IST for healthcare experts were notified by email of the survey dates. Besides the notification and a description of the study, an online link to access the questionnaire1 was also made available. As previously mentioned, of the 144 experts we contacted, 46 successfully completed the questionnaire within the established period.
5. Characterization of the Expert Panel Regarding the characterization of the expert panel, it is important to note that, in the present study, we opted for a heterogeneous panel, that is, a panel comprising experts from the health field with different profiles and experiences within the IST context. The characterization of the participants was considered important in terms of putting the results into context and discussing them. With this in mind, our questionnaire included a group of questions concerning profile characterization, which the respondents had to complete on the first page of the online questionnaire. In this group of questions, each expert had to provide the following information: professional category and years of experience in terms of jobs or positions relating to IST for healthcare. Where professional experience was concerned, all experts were asked to indicate if their professional experience in the field of IST for healthcare involved management, consultancy, teaching or ‘other’. In global terms, from the group of experts who participated in our study, 29 (63%) stated that their professional experience involved management or consultancy in the IST field, eight experts (17%) mentioned teaching or research, five experts (11%) mentioned experience in unit/department management, and four experts (9%) indicated other areas. Table 3 shows the distribution of participants according to their professional category. Table 3 – Characterization of the respondents in terms of professional category Professional category IS manager IST consultant Unit/department manager Professor/researcher Other
Number 22 7 5 8 4
Percentage 47.82% 15.22% 10.87% 17.39% 8.70%
Another aspect considered important in the characterization of the participants was their experience of jobs or positions involving IST for healthcare. Thus, the experts were asked to indicate if their experience corresponded to less than three years, three to six years, or more than six years. In the group of 46 experts who actively participated in the study, the vast majority were largely experienced in this field. Indeed, 39 experts (85%) indicated more than six years of experience in IST for healthcare, while only one (2%) indicated less than three years of experience, with the remaining six (13%) indicating three to six years of experience.
Table 4 shows the distribution of experts by years of experience in the IST for healthcare field.
Prepared with the LimeSurvey freeware.
Table 4 – Characterization of the respondents in terms of years of experience in the health field Experience in the health field Less than three years Three to six years More than six years
Number Percentage 1 2.17% 6 13.04% 39 84.78%
Bearing in mind the results of this respondent profile analysis, and the high percentage of consultants and managers, in both units/departments and IS (74%), as well as their significant experience in the health field, as confirmed by 85% of the respondents having more than six years of experience, it is noteworthy that the group of respondents who participated in our survey represent a significant part of the knowledge available in this field.
6. Importance of HIS Subareas After characterizing the respondents’ personalities, we turned to the main focus of our study. In a second group, we introduced questions to identify the most important IST subareas. We should mention that the 12 IST subareas included in our proposed list were presented in alphabetical order, so as to minimize a possible bias in the responses of the participants. Additionally, along with a designation, each subarea included a short description or definition in order to facilitate the interpretation of the items (IST for healthcare subareas) used in this study. The first question in this group concerned the possibility of evaluating each of the 12 subareas, while the second question, assuming a Likert scale format, asked the opinion of the experts with regard to the importance of each subarea. Moreover, each participant was allowed the option to propose new IST subareas for the study. With this in mind, a specific field was added so that each expert could propose, without restriction, new subareas, which they considered necessary, by simply entering a name. From the obtained individual rankings, data were compiled to create a global ranking that represented the general opinion of all the participants in our survey. To prepare the global ranking for the first question, we considered the level of importance of each subarea (scored from 1 to 5). Then, the scores for each subarea were added and, finally, the global ranking was created2 by sorting the sums of all subareas, from the highest (the most important, placed at the top of the ranking) to the lowest (the least important, placed on the bottom of the list). Table 5 shows the global ranking for Question 1, where, besides the value corresponding to the sum of the positions obtained in individual rankings, we present other values often used in studies of this nature, such as mean, variance (Var), standard deviation (SD) and coefficient of variation (CV). In the CV analysis, we observe that the dispersion around the mean displayed acceptable values for most subareas, although the last three revealed higher dispersion rates (>30%). In Table 5, we also present the results concerning the possibility (or impossibility) of evaluating IST for healthcare subarea maturity. We observe that the majority came close to 100%. The exceptions that obtained lower values were cooperation (67.39%) and mHealth (58.70%). As we will confirm further ahead, these two subareas will not be included in our new maturity model proposal, as they displayed relatively low mean values of importance, while the possibility of evaluating their maturity, according to experts, fell under 70%. Bearing in mind that the subareas, which were considered most important (and are included in our new model proposal), scored high 2
Designated as Rk1 in this study.
with regard to the possibility of being evaluated in terms of maturity, we can conclude that we obtained a categorical answer to our previously posed research question (RQ2). Table 5 – Subarea ranking in IST for healthcare Position 1 2 3 4 5 6 7 8 9 10 11 12
Subarea (influencing factor) Electronic medical records People Systems and IT infrastructure Data analysis Interoperability Information security PACS Strategy Cooperation Usability Telemedicine mHealth
Yes (%) 91.30 84.78 93.48 89.13 91.30 91.30 86.96 76.09 67.39 80.43 78.26 58.70
No (%) Sum Mean Var SD CV 8.70 210 4.56 0.65 0.81 17.7% 15.22 204 4.43 0.92 0.96 21.6% 6.52 197 4.28 0.65 0.81 18.9% 10.87 196 4.26 0.46 0.68 16.0% 8.70 195 4.24 1.30 1.14 26.9% 8.70 195 4.24 1.30 1.14 26.9% 13.04 190 4.13 1.18 1.09 26.3% 23.91 188 4.09 1.06 1.03 25.2% 32.61 166 3.61 0.73 0.86 23.7% 19.57 161 3.50 1.81 1.35 38.5% 21.74 148 3.22 1.55 1.25 38.7% 41.30 132 2.87 1.32 1.15 40.0%
Still in this group of questions, besides the characterization of subareas in terms of importance, we asked the experts to suggest new subareas, which, in their opinion, were relevant in the IST for healthcare context. Of the 46 experts who participated in our study, about one in four (11 respondents) proposed the introduction of new subareas. To analyze and select among these, we carefully studied all 11 proposals for new subareas in order to identify those that could be included in our new maturity model proposal. The analysis of these new subareas was based on four criteria: the first criterion was to check if the proposed subarea fell within the concept of the IST for healthcare subarea adopted in this study; the second criterion was to check if the proposed subarea could somehow be included in one of the preexisting subareas; the third criterion was to check for similar or repeated proposals; and, lastly, as a fourth criterion, we established that we would only accept proposals with several expert proponents. This latter criterion was based on the fact that the goal of this study was to identify the most important subareas, such that, if a proposed subarea was indeed important, it would be suggested by more than one expert. Consequently, nine new subareas were suggested and discarded for not complying with these previously defined criteria. As seen in Table , most of the proposals had only one proponent, while four of these proposals could be included in previously identified areas. The only proposal with more than one proponent was ‘clinical decision support’, which justifies its inclusion in ‘data analysis’. Table 6 – New subareas proposed for consideration in a new maturity model Proposed subarea name Clinical information archive Suitability for use Workflow Clinical decision support Operational programs EHR (dif. medical) ITIL/ISO 20000 R&D in IS for healthcare Dematerialization procedures
Number of proposers 1 1 1 3 1 1 1 1 1
Subarea to be included Electronic medical records Usability Data analysis Electronic medical records
Still referring to the main questionnaire, by following the selection sort3 methodology, we asked the experts to rank the main 12-item list (IST subareas) according to their perception of the eight most important in IST for healthcare. Fundamentally, each expert was asked to complete the designated list, which in practice would allow us to define the individual ranking of the IST subareas identified in our initial list. This new ranking would work as a reinforcement and validation of the first ranking, resulting from the importance of each subarea (based on a Likert scale). Based on individual rankings, we carried out data compilation to obtain a second global ranking4, which represented the general opinion of all those who participated in the last question of the survey. To elaborate the global ranking for the last question, each subarea was consulted in terms of its position in each individual ranking and scored accordingly, from eight points given to the first to one point given to the eighth. Next, the scores of each subarea were added, after which a global ranking was generated by sorting the sums of every subarea, from the highest value (most important, which was placed at the top of the ranking) to the lowest (least important, which was placed at the bottom of the list). In Table 77, we present the global ranking of the last question, in which, besides the value corresponding to the sum of the positions obtained in individual rankings, we present other values often used in studies of this nature, such as mean, variance (Var) and standard deviation (SD). The last column of this table displays the change in position observed between the first question global ranking and the last for each subarea. The obtained results show that the top eight subareas maintain their top positions despite slight changes in their order. On the other hand, the bottom four subareas maintain their positions in the importance rankings. However, it is important to note that three subareas registered significant changes. While ‘strategy’ climbed six positions (from 8 to 2) between the first and the second ranking, ‘electronic medical records’ went down three positions, while ‘data analysis’ and ‘interoperability’ went down two positions. As we can observe, with the exception of these four cases, all other subareas maintained their positions, changing one position at most. Table 7 – Evolution of the global importance ranking of the IST subareas
∆ Pos. ▲1 ▲6
Position 1 2
Subarea (influencing factor) People Strategy
Sum 268 224
Mean Var SD PreRk 5.83 2.92 1.71 2 4.87 7.35 2.71 8
Systems and IT infrastructure Electronic medical records
4.37 2.09 4.50 2.12
5 6 7 8
Information security Data analysis Interoperability PACS
193 176 109 92
4.20 3.83 2.37 2.00
3.23 4.55 2.26 4.34
1.80 2.13 1.50 2.08
5 4 5 7
⚌0 ▼-2 ▼-2 ▼-1
10 11 12
72 15 3
1.57 0.33 0.07
3.57 1.89 1.27 1.13 0.00 0.00
10 12 11
⚌0 ▲1 ▼-1
Selection sort is a sorting algorithm, which places the highest vector value in the first position (or lowest, depending on the order), followed by the second highest in the second position, and so forth for the (n-1) remaining elements, until the last two values are derived. 4 Designated as Rk2 in this study.
7. Expert Consensus and Level of Stability Analysis The consensus analysis, according to the parameters defined in this study, involved the analysis of two criteria: the level of agreement between individual rankings by the experts who participated in the study, based on Kendall’s coefficient of concordance (W); and the level of stability attached to the opinion of the experts between the global rankings of the two questions, measured with Spearman’s correlation coefficient (rho). Starting with the consensus analysis between individual rankings, Kendall’s coefficient of concordance scored W=0.250 (p