Predicting Emergency Department Visits Among ...

Proceedings of the 2017 Industrial and Systems Engineering Conference K. Coperich, E. Cudney, H. Nembhard, eds.

Predicting Emergency Department Visits Among Patients Receiving Primary Care Using Risk Scores Mandana Rezaeiaharia, Srikanth Porankia and Mohammad Khasawneha aDepartment

of Systems Science and Industrial Engineering State University of New York at Binghamton Binghamton, New York 13902, USA Abstract

Because of the reform to value-based care delivery system, healthcare organizations are working toward reducing costs and improving healthcare outcomes by identifying high-risk patients. The process of accurately targeting highrisk patients followed by effective design of interventions has been recently sought by many studies. However, most risk-stratification related studies only consider comorbid conditions and demographic data as influential factors to stratify patients by risk. This paper provides an easy-to-implement and effective tool to identify patients based on other additional factors: inpatient admissions, mental health status, number of medications taken, insurance, neurological conditions and total number of outpatient visits. To validate the accuracy of the proposed tool, all adults in primary care (11,960 patients) at a community hospital are studied. Several classification methods (e.g., support vector machine (SVM), AdaBoost.M1, J48 Decision Tree, and Neural Networks) are used to compare the accuracy of the model with and without the inclusion of the risk score. Based on the results, inclusion of risk factor improves the model discrimination on average by 26%. Also, J48 Decision Tree, when risk factor is included, performs relatively better than the other methods. Keywords: risk-stratification, comorbidity, care coordination, value-based care delivery system, population health management

1. Introduction With Fee-For-Service (FFS) payment system, providers are willing to gain more benefit by performing high cost procedures rather than well managed care and other cost saving services. However, with the provisions by Accountable Care Act, healthcare organizations are facing away from FFS model to outcome-based payment systems to enhance healthcare outcomes. The outcome-based payment systems emphasize on both improving the quality and cost of care. The outcome-based health systems should be organized to create value for the patient. Value for the patient is defined as good healthcare outcomes that are achieved by efficient resource allocations rather than limiting provision of health services. To this end, “value-based health delivery systems” are needed which emphasize restructured delivery system. To achieve a value-base delivery system, true healthcare outcomes should be adjusted and measured over the continuum of care cycle rather than separate interventions for each patient [1]. One recent mainstream in healthcare to achieve value-based health delivery system is risk-stratification. Classifying a patient population based on their conditions into high-risk and low-risk is called risk-stratification. Risk-stratification would enable healthcare organizations to identify high-risk patients and plan required interventions accordingly [2]. There are six common models in the literature for stratifying population by risk. These models categorize the diseases selected from ICD codes into clinical groups and assign risk scores to each category, as follows: (1) Adjusted Clinical Groups (ACGs) developed by Johns Hopkins University is a prevalent risk-stratification model being used to determine the medical resource utilization. ACGs works based on administrative data on disease, age and sex and assign each patient a score from 1 to 93; (2) Minnesota Tiering (MN) classifies the patients into five distinct tiers based on number of conditions: (tier 0): 0 conditions; basic (tier 1): 1 to 3 conditions; intermediate (tier 2): 4 to 6 conditions; extended (tier 3): 7 to 9 conditions; and complex (tier 4): 10 or more conditions; (3) In Hierarchical Condition Categories (HCCs), 70 categories are built based on ICD-9-CM (International Classification of Diseases, Ninth Revision, Clinical Modification) diseases and demographic data to adjust expenditure of enrollees in Medicare Advantage Plan; (4) Elder Risk Assessment (ERA) Index is used to calculate a hospitalization and ED visit risk rate

Rezaeiahari, Poranki and Khasawneh

for patients over 60 years. The risk rate is weighted score of medical conditions (diabetes, coronary artery disease, congestive heart failure, stroke, chronic obstructive pulmonary disease, and dementia), age, sex, marital status, and number of hospitalization days in the last two years; (5) Chronic Condition Count (CCC) works based on counting comorbid conditions and groups the total counts into six groups: 0, 1, 2, 3, 4, and 5 or more; (6) Charlson Comorbidity Index (CCI) uses the presence or absence of 17 conditions and assign each patient a score from 1 to 20 with 20 being patients with multiple comorbidities [3]. To date, risk-adjusted models that take into account mental health status of patients are rare. Therefore, in this study a tool is developed to stratify patient risk based on the commonly used criteria such as comorbid conditions, number of inpatient admissions, number of medications taken, insurance, total number of outpatient visits plus a new factor for mental health status. The remainder of the manuscript is structured as follows. Section 2 provides a literature review on riskstratification models and interventions for care coordination of high-risk patients. Section 3 explains the components of the risk-stratification tool, the framework for validating the performance of the tool in predicting ED visits and computational results. Section 4 provides conclusions and discussion of future research.

2. Literature Review Quality of ambulatory care can affect number of hospitalizations and mortality since many complex medical problems are treated outside of the hospital nowadays. ACG was first developed by Weiner et al. [4] to provide applications for provider payment system. ACG system is based on patient age, sex and ICD-9-CM diagnosis group. First, ICD-9-CM codes are grouped in to 34 ambulatory diagnostic groups based on persistence/recurrence and intensity of services. Then similar Ambulatory Diagnostic Groups (ADGs) are collapsed to 12 ADGs and are finally placed into 25 mutually exclusive major ambulatory categories (MAC). They applied multivariate regression analysis on several sets of independent variables: age, sex, presence/absence of ADGs, number of different ADGs, and final ACGs to predict total number of ambulatory visits, total number of ambulatory visits to specialists, ancillary charges associated with visits and etc. Many research studies related to risk-adjustments models at outpatient setting such as Case-Mix System, Ambulatory Patient Groups, and Ambulatory Visit Groups have been revolving around predicting resource utilization. Only few studies have addressed risk-adjusted models for prediction of mortality in outpatient settings. For instance, Tierney et al. [5] developed a risk-adjusted regression model to predict mortality only for patients with heart disease. Selim et al. [6] developed a risk-adjusted regression model to predict mortality with specific consideration to different case-mix at an outpatient setting. The risk factors of this study were sociodemographic (age, gender, and race), diagnoses (based on ICD-9-CM coding scheme and Charlson weighting system), and functional status including physical component summary (PCS) and the mental component summary (MCS). Billing et al. [7] developed a new case finding tool for patients at risk of hospital readmissions based on hospital episode statistics within five years. Two reference conditions; congestive heart disease or chronic obstructive pulmonary disease were the focus of their study. One year of the data was considered as the triggering event data from which diagnostic data was extracted. Other information such as characteristics of the patient, hospital of admission and etc. was extracted from three years prior to the triggering year to predict the risk of readmission in the next 12 months. In another study, a risk-stratification tool was developed for in-hospital mortality of patients with Acute Decompensated Heart Failure (ADHF). As part of their results, they found the odds-ratio for mortality between high and low risk equal to 12.9 [8]. Boyd et al. [9] developed a risk-identification tool (BRIGHT; Brief Risk Identification for Geriatric Health Tool) for identifying residents with declining functionality within 10 days after any ED visit. They concluded that BRIGHT is more predictive than comprehensive geriatric assessment to identify older adults in the ED who have functional deficits (AUC of 0.83 compared to 0.73). The outcome variables of their study include instrumental activity of daily living (IADL) cognitive performance scale (CPS), and activities of daily living (ADL). Haas et al. [3] evaluated the accuracy of six common risk-stratification models: ACG, MN, HCCs, ERA Index, CCC, CCI in prediction of inpatient hospitalization, ED visits without hospitalizations, 30-day readmission, and highcost users. In their study, high cost patients are top 10% of all patients in terms of total healthcare costs. They concluded that ACG provides the highest accuracy in predicting all healthcare outcomes. The result of their study implied that ACG outperforms other five methods in predicting the healthcare outcomes and they used this method to identify high-risk patients for better care coordination. Wang et al. [10] identified patients with high risk of hospitalization or death in primary care Veteran Health Administration using multinomial logistic regressions. The predictors in their study include socio-demographics, medical conditions, vital signs, prior year use of health services,


medications, and laboratory tests. Cohen et al. [11] combined predictive modeling and high risk patient identification by physician to accurately targeting high risk patients for care coordination. In their study, a panel of 6 primary care providers surveyed to identify inclusion criteria for care coordination program; medical complexity, patient characteristics and social reasons were identified as criteria for good fit. Then risk scores from ACG was used to predict high cost patients. Most of the aforementioned studies are limited to identifying high-risk patients for a specific condition; however, it is recognized that patients with multiples health conditions are exposed to higher risk. Thus, the proposed riskstratification tool in this research, focuses not only on several comorbid conditions and other commonly used factors in the literature such as number of inpatient admissions, number of medications taken, insurance, total number of outpatient visits but it also includes a new factor related to mental health status.

3. Proposed Risk-stratification Tool and Validation In this section, principal and amplifying risk factors in the proposed risk-stratification tool are explained. The proposed tool is a real time identification tool which captures the level of risk of patient who are already in the contact by the system. The complexity variables contributing to various levels of risk associated with the studied population are as follows: (1) Inpatient admissions: patients with only one inpatient admission due to chronic conditions are more likely to be readmitted [12], (2) Comorbid conditions: According to Muenchberger and Kendall [12], patients with more than three, and five and more than five comorbid conditions are twice and three times more likely to be hospitalized than patients with a single condition, (3) Mental health: patients with mental health problems have poor adherence to treatment due to difficulties for self-care [13]. Inclusion of mental health and specifically depression in patient riskstratification is directly correlated with health care cost reduction. According to Mulrow et al. [14], patients with depression spend more time with their physicians and costs associated with patients with depression is close to $43.7 billion,(4) Medications: number of daily medication intakes has been proven to be a predictor of hospital admission. Patient with more complex conditions take more medications [15], (5) Insurance: in a study by O’Malley [16], it was found newly insured patients had greater rate of hospitalization compared to previously insured patients. This implies to the fact that absence of insurance lead to hospitalization [12], (6) Neurological conditions: in addition to chronic diseases, neurological conditions will similarly lead to hospitalization [17], and (7) Total outpatient visits: according to Agency for Clinical Innovation [17], ideally risk-stratification should be applied across the whole of the healthcare settings, however the complexity variables must be weighted to the setting in which they are implemented, so that proposed interventions are more appropriate to the specific population. The distribution of scores for low, medium and high risk patients within the proposed tool are presented in Table 1. Table 1. Risk categories and levels Category Comorbid Conditions (Diabetes, CHFa, HTNb, COPD, CADd, Stroke, Dementia) Inpatient Admissions Mental Health (Schizophrenia, Bipolar, Borderline Personality Disorder, Anxiety) Medications Insurance Neurological Disorders (Alzheimer’s Disease, Dementia, MGe, MSf, CPg, Mental Retardation, Parkinson’s Disease, ALSh)

Low Risk

Medium Risk

High Risk

0 (0 Point)

1-2 (3 point)

3+ (5 point)

0 in 12 months (0 point)

1 admission in 12 months (2 point)

2+ admissions in 12 months (5 points)

0 (0 point)

NA

1 (5 points)

0-5 (0 point) Any insurance (0 point) 0 (0 point)

6-10 (2 point) NA

NA

10+ (3 points) No insurance (3 points) 1 or more (3 points)

No visit in last 12 months or seen > 7/year (3 points) a Congestive Heart Failure; b Hypertension; c Chronic Obstructive Pulmonary Disease; d Coronary Artery Disease; e Myasthenia Gravis; f Multiple Sclerosis; g Cerebral Palsy; h Amyotrophic Lateral Sclerosis Total Outpatient Visits

4 or less (0 point)

5-7 (1 point)


The risk thresholds and points given to each category of low, medium and high risk are based on group of physicians’ assessments. The risk score obtained from Table 1 is categorized into three classes: (1) low risk patient (𝑟𝑖𝑠𝑘 𝑠𝑐𝑜𝑟𝑒 < 8)), (2) medium risk patient (8 ≤ 𝑟𝑖𝑠𝑘 𝑠𝑐𝑜𝑟𝑒 < 14) and, (3) high risk patient (𝑟𝑖𝑠𝑘 𝑠𝑐𝑜𝑟𝑒 ≥ 14). In order to validate the tool, risk score of 11,960 primary care patients at Upstate New York were calculated based on Table 1. Then, the risk scores together with some additional demographic data such as gender, marital status, and age were used to predict the number of yearly Emergency Department (ED) visits. Based on the discussion with the group of physicians, two classes were defined for the yearly number of ED visits: (1) ED visits more than 3 times a year and (2) ED visits less than or equal to 3 times a year. Among the independent variables, age and risk scores were imported as numeric values while gender and marital status were considered as categorical variables. The purpose of the validation is to evaluate the effect of risk score in predicting the ED visits. Of the 11,960 patients included in the analysis, 8,367 (70%) were used for training set and 3,588 were used for testing set across all classification methods. Figure 1 represents the summary of the data. The description of the categorical variables in Figure 1 for parts (a) and (b) are as follows: In part (a), “0” and “1” refer to “female” and “male” and in part (b), “0”, “1”,”2” and “3” denote “single”, “married”, “divorce” and “widow”, respectively. Table 2 also provides additional information for numerical variables; age and risk score. Table 2. Dataset Summary Male 5,537 Gender Female 6,423 Single 6,927 Married 3,479 Marital status Divorce 633 Widow 921 Mean St. Deviation Age 45.5 24.9 Risk Score 4.0 4.7

(a) gender

(b) marital status


(c) age

(d) risk score Figure 1 Bar plots of patient characteristics

Binary Logistic Regression, Support Vector Machine (SVM) (Sequential Minimal Optimization (SMO) is used to train the support vector classifier), AdaBoost.M1, J48 Decision Tree, and Neural Network (Levenberg algorithm) are applied for both cases when the risk factor is included and not included in the prediction models. Since in this research, the risk scores are used to identify the high risk patients for the care coordination purposes, False Positive Rate (FPR) in addition to sensitivity and C statistic are used to characterize the models. Care coordination programs have usually a specific capacity, thus resources should be allocated to those individuals that will most likely benefit from the program. Thus, lower FPR ensures fitter individuals for the care coordination program. Table 3Table 3 shows the results of the prediction models for the testing set. Table 3. Performance of classification methods in prediction of ED visits with/without risk factor Models Methods Sensitivity False Positive Rate Accuracy C Statistic Binary Logistic Regression 0.67 0.21 0.73 0.79 0.60 0.18 0.71 0.71 Risk Factor SVM AdaBoost.M1 0.79 0.33 0.73 0.77 Included J48 Decision Tree 0.76 0.24 0.76 0.81 Neural Network 0.70 0.23 0.74 0.78 Binary Logistic Regression 0.37 0.37 0.51 0.55 0.18 0.09 0.55 0.54 Risk Factor SVM AdaBoost.M1 0.22 0.12 0.55 0.57 Excluded J48 Decision Tree 0.75 0.51 0.63 0.67 Neural Network 0.64 0.34 0.64 0.73 As shown in Table 3, the model discrimination (expressed as C statistic) is higher when the risk factor is included compared to the same models when the risk factor is not included. Also, among all models that include risk factor, J48 decision tree is the strongest model (C statistics > 0.80). Although other measures of performance; sensitivity and false positive rate, in J48 decision tree do not outperform all of the four other methods, the relative performance of this method is still better than the others.

4. Conclusions and Future Work A risk-stratification tool was designed by a group of physicians at a local hospital in Upstate New York to identify the high risk patients for care coordination programs. Patients with higher level of risk have more frequent ED visits. Thus, the objective of this paper is to validate the performance of the risk scores obtained from the developed tool in identifying the ED visits. Binary logistic regression, support vector machine, AdaBoost.M1, J48 decision tree, and neural network were applied to predict a binary outcome for ED visits. It was found that inclusion of risk factor improves the prediction performance of the models. Additionally, considering all performance measures; sensitivity, false positive rate and C statistic, J48 decision tree performed relatively better than the other classification methods.


In terms of future work, other healthcare outcomes such as cost and readmission rate can be studied as well. Also, a standardized method for patient selection that takes into account several healthcare outcomes can be studied further.

References [1] [2] [3]

[4] [5]

[6]

[7] [8]

[9]

[10]

[11] [12] [13]

[14] [15]

[16]

[17]

M. E. Porter, “A strategy for health care reform--toward a value-based system,” N. Engl. J. Med., vol. 361, no. 2, pp. 109–12, 2009. B. E. Just, “Understanding risk stratification , comorbidities , and the future of healthcare,” pp. 1–7, 2014. L. R. Haas, P. Y. Takahashi, N. D. Shah, R. J. Stroebel, M. E. Bernard, D. M. Finnie, and J. M. Naessens, “Risk-stratification methods for identifying patients for care coordination,” Am. J. Manag. Care, vol. 19, no. 9, pp. 725–32, 2013. J. P. Weiner, B. H. Starfield, D. M. Steinwachs, and M. Laura, “Development and application of a populationoriented measure of ambulatory,” vol. 29, no. 5, pp. 452–472, 2015. W. M. Tierney, B. Y. Takesue, D. L. Vargo, and X. H. Zhou, “Using electronic medical records to predict mortality in primary care patients with heart disease: prognostic power and pathophysiologic implications,” J Gen Intern Med, vol. 11, no. 2, pp. 83–91, 1996. A. J. Selim, D. R. Berlowitz, G. Fincke, A. K. Rosen, X. S. Ren, C. L. Christiansen, Z. Cong, A. Lee, and L. Kazis, “Risk-adjusted mortality rates as a potential outcome indicator for outpatient quality assessments,” Med. Care, vol. 40, no. 3, pp. 237–45, 2002. J. Billings, J. Dixon, T. Mijanovich, and D. Wennberg, “Case finding for patients at risk of readmission to hospital: development of algorithm to identify high risk patients,” BMJ, vol. 333, no. 7563, p. 327, 2006. G. C. Fonarow, K. F. Adams, W. T. Abraham, C. W. Yancy, and W. J. Boscardin, “Risk stratification for inhospital mortality in acutely decompensated heart failure: classification and regression tree analysis,” Jama, vol. 293, no. 5, pp. 572–80, 2005. M. Boyd, J. Koziol-McLain, K. Yates, N. Kerse, C. McLean, C. Pilcher, and G. Robb, “Emergency department case-finding for high-risk older adults: The Brief Risk Identification for Geriatric Health Tool (BRIGHT),” Acad. Emerg. Med., vol. 15, no. 7, pp. 598–606, 2008. L. Wang, B. Porter, C. Maynard, G. Evans, M. Mcdonell, K. Frisbee, C. Nielson, and F. Kirkland, “Predicting risk of hospitalization or death among patients receiving primary care in the veterans health administration,” vol. 51, no. 4, pp. 368–373, 2013. C. J. Cohen, N. Flaks-manov, and M. Low, “High-risk case identification for use,” vol. 18, no. 1, pp. 15–23, 2015. H. Muenchberger and E. Kendall, “Determinants of avoidable hospitalization in chronic disease: Development of a predictor matrix,” 2008. T. Croghan and J. Brown, “Integrating mental health treatment into the patient centered medical home,” Prep. by Math. Policy Res. under Contract No. HHSA290200900019ITO2. AHRQ Publ. No. 10-0084, no. 10, pp. 1–14, 2010. C. D. Mulrow, J. W. W. Jr, M. B. Gerety, G. Ramirez, O. M. Montiel, and C. Kerber, “Case-finding instruments for depression in primary care settings,” pp. 913–922, 1995. E. Wallace, E. Stuart, N. Vaughan, K. Bennett, T. Fahey, and S. M. Smith, “Risk prediction models to predict emergency hospital admission in community-dwelling adults a systematic review,” Med. Care, vol. 52, no. 8, pp. 751–765, 2014. A. S. O’Malley, H. H. Pham, D. Schrag, B. Wu, and P. B. Bach, “Potentially avoidable hospitalizations for COPD and pneumonia: the role of physician and practice characteristics,” Med. Care, vol. 45, no. 6, pp. 562– 70, 2007. Agency for Clinical Innovation (ACI) NSW Agency for Clinical Innovation, “Risk-stratification a discussion paper for nsw health’s approach to risk-stratification,” 2014. [Online]. Available: http://www.aci.health.nsw.gov.au/__data/assets/pdf_file/0015/253005/Risk_Stratification_discussion_paper. PDF.