Risk stratification for in-hospital mortality after ... - Semantic Scholar

ORIGINAL ARTICLE

European Journal of Cardio-Thoracic Surgery 46 (2014) 840–848 doi:10.1093/ejcts/ezt657 Advance Access publication 30 January 2014

Risk stratification for in-hospital mortality after cardiac surgery: external validation of EuroSCORE II in a prospective regional registry Domenico Paparellaa,†*, Pietro Guidab,†, Giuseppe Di Eusanioc, Sergio Caparrottid, Renato Gregorinic, Mauro Cassesee, Vitantonio Fanellif, Giuseppe Spezialeg, Valerio Mazzeid, Salvatore Zaccariah, Luigi De Luca Tupputi Schinosaa and Tommaso Fiorei a b c d e f g h i

Division of Cardiac Surgery, Department of Emergency and Organ Transplant, University of Bari Aldo Moro, Bari, Italy Puglia Health Regional Agency, Bari, Italy Department of Cardiac Surgery, Città di Lecce Hospital, Lecce, Italy Department of Cardiac Surgery, Villa Bianca Hospital, Bari, Italy Department of Cardiac Surgery, Santa Maria Hospital, Bari, Italy Department of Cardiac Surgery, Villa Verde Hospital, Taranto, Italy Department of Cardiac Surgery, Anthea Hospital, Bari, Italy Department of Cardiac Surgery, Vito Fazzi Hospital, Lecce, Italy Division of Anesthesia, Department of Emergency and Organ Transplant, University of Bari Aldo Moro, Bari, Italy

* Corresponding author. Division of Cardiac Surgery, Department of Emergency and Organ Transplant, University of Bari ‘Aldo Moro’, Piazza Giulio Cesare 11, 70100 Bari, Italy. Tel: +39-0805595075; fax: +39-0805595076: e-mail: [email protected] (D. Paparella). Received 11 September 2013; received in revised form 22 November 2013; accepted 26 December 2013

Abstract OBJECTIVES: To evaluate performance of the European System for Cardiac Operation Risk Evaluation (EuroSCORE II), to assess the influence of model updating and to derive a hierarchical tree for modelling the relationship between EuroSCORE II risk factors and hospital mortality after cardiac surgery in a large prospective contemporary cohort of consecutive adult patients. METHODS: Data on consecutive patients, who underwent on-pump cardiac surgery or off-pump coronary artery bypass graft intervention, were retrieved from Puglia Adult Cardiac Surgery Registry. Discrimination, calibration, re-estimation of EuroSCORE II coefficients and hierarchical tree analysis of risk factors were assessed. RESULTS: Out 6293 procedures, 6191 (98.4%) had complete data for EuroSCORE II assessment with a hospital mortality rate of 4.85% and EuroSCORE II of 4.40 ± 7.04%. The area under the receiver operator characteristic curve (0.830) showed good discriminative ability of EuroSCORE II in distinguishing patients who died and those who survived. Calibration of EuroSCORE II was preserved with lower predicted than observed risk in the highest EuroSCORE II deciles. At logistic regression analysis, the complete revision of the model had most of reestimated regression coefficients not statistically different from those in the original EuroSCORE II model. When missing values were replaced with the mean EuroSCORE II value according to urgency and weight of intervention, the risk score confirmed discrimination and calibration obtained over the entire sample. A recursive tree-building algorithm of EuroSCORE II variables identified three large groups (55.1, 17.1 and 18.1% of procedures) with low-to-moderate risk (observed mortality of 1.5, 3.2 and 6.4%) and two groups (3.8 and 5.9% of procedures) at high risk (mortality of 14.6 and 32.2%). Patients with low-to-moderate risk had good agreement between observed events and predicted frequencies by EuroSCORE II, whereas those at greater risk showed an underestimation of expected mortality. CONCLUSIONS: This study demonstrates that EuroSCORE II is a good predictor of hospital mortality after cardiac surgery in an external validation cohort of contemporary patients from a multicentre prospective regional registry. The EuroSCORE II predicts hospital mortality with a slight underestimation in high-risk patients that should be further and better evaluated. The EuroSCORE II variables as a risk tree provides clinicians and surgeons a practical bedside tool for mortality risk stratification of patients at low, intermediate and high risk for hospital mortality after cardiac surgery. Keywords: Cardiac surgery • EuroSCORE II • Risk analysis/modelling

INTRODUCTION The European System for Cardiac Operation Risk Evaluation (EuroSCORE) is one of the possible tools to assess the operative †

The first two authors contributed equally to this study.

mortality risk [1–3]. The first EuroSCORE, initially based on an additive system derived by a logistic regression model, was developed on data collected in 1995 and reported in 1999 to predict in-hospital or 30-day mortality [1, 2]. Recently, in order to improve the poor calibration and to optimize EuroSCORE usefulness, an updated version of this model has been proposed for the

© The Author 2014. Published by Oxford University Press on behalf of the European Association for Cardio-Thoracic Surgery. All rights reserved.

D. Paparella et al. / European Journal of Cardio-Thoracic Surgery

MATERIALS AND METHODS Study population Consecutive adult patients who had undergone on-pump cardiac surgery or off-pump CABG in Puglia region from 1 January 2011 to 31 December 2012 were considered. Pre-, intra- and postoperative information were gathered from Puglia Regional Adult Cardiac Surgery Registry, which involves the seven adult cardiac surgery centres in Puglia: University of Bari, Policlinico Hospital (Coordinating Centre), Anthea Hospital in Bari, Santa Maria Hospital in Bari, Villa Bianca Hospital in Bari, Villa Verde Hospital in Taranto, Vito Fazzi Hospital in Lecce and Città di Lecce Hospital in Lecce. Health regional agency personnel guaranteed data accuracy and quality control procedures. All cases performed in each centre were checked from operating theatre documents, ensuring that all patients who had undergone heart operations were in the registry. Cleaning consisted of minimizing the amount of missing data through interactions with hospitals and auditing consisted of inspection of data in hospital medical records.

European System for Cardiac Operation Risk Evaluation II All risk factors included in EuroSCORE II were available in the registry: patient’s age, gender, creatinine clearance, extracardiac arteriopathy, poor mobility secondary to musculoskeletal or neurological dysfunction, previous cardiac surgery, chronic lung disease, active endocarditis, critical preoperative state, diabetes on insulin, New York Heart Association (NYHA) functional classification, Canadian Cardiovascular Society (CCS) grade IV of angina, left ventricular ejection fraction, recent myocardial infarction, systolic pulmonary artery pressure, urgency of operation, weight of the intervention and surgery on thoracic aorta [3]. EuroSCORE II was derived for all patients based on the original equation described by Nashef et al. [3]. Hospital mortality was considered as death occurring at any time after surgery during the period in hospital in which the operation was performed.

Ethical considerations The Adult Cardiac Surgery Registry is an initiative of the Health Regional Agency of Puglia with the aim to evaluate the postoperative outcome of patients receiving major cardiac surgery in the region. According to institutional review board policy, it was determined that research does not require informed consent. Data were collected and stored in an anonymous fashion with the patients only identified by their medical record number.

Statistical analysis Data are shown as mean values ± standard deviation and categorical variables are given as percentages. Variables were compared among groups by using Student’s t-test or χ 2 test. EuroSCORE II was calculated at the time of data analysis for each patient evaluating the prognostic index (PI) as the linear predictor of variables included in the original model: X PI ¼ b0 þ bi xi where β0 is the intercept and βi is the coefficient of each single risk factor coded as a binary variable that indicates presence (xi = 1) or absence (xi = 0). The logistic equation Predicted mortality ¼

expðPIÞ ð1 þ expðPIÞÞ

provided the EuroSCORE II assessment of the risk for hospital mortality [3]. We fitted a logistic regression model for in-hospital mortality with the intercept as the only free parameter and the linear predictor based on the PI as an offset variable (i.e. the slope was fixed at unity) and a model incorporating the linear predictor PI as the only covariable (update of both the intercept and the overall calibration slope). The intercept, which should be zero, indicated if the predictions were systematically lower or higher than mortality and the calibration slope, which should be 1, indicated the relation between EuroSCORE II linear predictor and mortality. Moreover, each risk factor of EuroSCORE II was introduced into a logistic regression model with re-estimation of all coefficients. We compared re-estimated coefficients with those in the original model considering the statistic that follows a normal distribution under the hypothesis of equality of the two

ADULT CARDIAC

assessment of cardiac surgical risk [3]. The new score, named EuroSCORE II, was better calibrated than the original model in detecting hospital mortality preserving high discrimination [3]. Risk evaluation with EuroSCORE systems was developed as a logistic regression model for short-term mortality, which is a dichotomous outcome. A logistic model may be used to provide predictions of events for individual patients at a centre different from where the model was developed [4]. The validity of predictions can be assessed by simple comparison between observed outcomes and predicted probabilities. To date, several studies have evaluated performance of EuroSCORE II in patients who had undergone major cardiac surgery [5–11], isolated coronary artery bypass graft (CABG) [12, 13], valve surgery [14, 15], emergency or high-risk cardiac surgery [16, 17]. Most of these studies were retrospective [6, 8–10, 12–17]. External validity of a model prediction is important to test the generalizability of a prediction rule over time in future patients [18]. Differences in population characteristics may affect the predictive accuracy of a scoring system and the validation process requires evaluation of the predictive model according to local circumstances. The external validity may be assessed considering the model performance as simple discrimination (concordance between predicted probabilities and outcome) and calibration (agreement between predicted and observed frequencies) or, to improve predictions, as recalibration (re-estimation of the intercept or slope of the linear predictor) and model revision (re-estimation of regression coefficients). Moreover, hierarchical trees approach allows non-parametric modelling of the relationship of several risk factors and mortality by selecting subgroups that are internally as homogeneous as possible with respect to the outcome and externally as separated as possible. No study has evaluated the EuroSCORE II risk factors by a recursive tree-building algorithm. The aim of this study was, in a large prospective cohort of consecutive adult patients who had undergone major cardiac surgery, to evaluate the performance of EuroSCORE II, to assess its calibration, to re-estimate logistic equation in comparison with the original model and to derive a risk tree to partition patients into categories of low-, intermediateand high-risk of hospital mortality.

841

842


coefficients, calculated as difference between coefficients divided by its standard error (SE). Considering the independence of the two samples, the SE of the difference between coefficients was calculated as the square root of the sum of the two squared SEs. A Cox-Snell graph was used to assess observed vs predicted cumulative hazard and the Hosmer–Lemeshow statistic to evaluate calibration [19]. Observed/expected (O/E) ratio of mortality was calculated: a value >1 indicates that the model underestimated mortality, whereas a value 0.05 or the number of patients in a subgroup 55 mmHg Urgency Elective Urgent Emergency Salvage Surgery of the thoracic aorta Weight of the intervention Isolated CABG 1 non-CABG 2 procedures 3 procedures


844

CI: 0.828–0.873) and the Hosmer–Lemeshow test was not significant (χ 2 = 5.45 with P = 0.709). Recalculated coefficients were compared with those derived in the original EuroSCORE II: most of re-estimated coefficients were not statistically different from those used to calculate EuroSCORE II. Age, urgent and emergency operation had significantly greater coefficients in comparison with the original prediction model. Only creatinine clearance ≤50 ml/ min showed a significantly lower coefficient. Table 4 shows observed mortality, EuroSCORE II expected mortality, O/E ratio, AUC of ROC curve and the recalibration slope of EuroSCORE II according to urgency of operation and different types of cardiac surgery. Non-elective procedures were associated to significant greater mortality than those predicted. The risk in subgroups of urgent and emergency operations was significantly underestimated, whereas elective procedures did not show deviation between expected and observed mortality rates. Discrimination was very high with the AUC of the ROC curve ranging between 0.773 and 0.903, with good calibration in each subgroup of cardiac surgery considered.

Missing EuroSCORE II In comparison with patients with EuroSCORE II evaluable (Table 1), the 102 with one or more missing EuroSCORE II risk factors had similar mean age (66.5 ± 9.5 years; P = 0.450), proportion of females (31.4%; P = 0.558), weight of intervention (isolated CABG 39.2%, one procedure 33.3%, two procedures 18.6% and three procedures 8.8%; P = 0.377) with a greater proportion of emergency and salvage operations (8.8%; P = 0.020). Fifteen patients died after the surgery: the incidence of hospital mortality was significantly higher than in patients with EuroSCORE II (14.7%; P < 0.001). When missing EuroSCORE II were replaced with the mean value according to urgency and weight of intervention, the risk score confirmed the high discrimination (AUC of ROC curve 0.827; 95% CI: 0.804–0.852) and calibration (Hosmer–Lemeshow test was χ 2 = 16.47 with P = 0.051) over the entire sample.

Hierarchical tree Figure 3 shows the final tree generated on the entire sample (6293 procedures with 315 patients died) by a recursive tree-building algorithm that hierarchically created subgroups on the basis of variables included in the EuroSCORE II. Advanced NYHA class and emergency/salvage procedures were the first two predictors with the highest association to mortality. The stratification was improved considering the presence of depressed left ventricular ejection fraction (≤30%), active endocarditis, creatinine clearance ≤50 ml/min or preoperative dialysis, female gender, previous cardiac surgery, critical preoperative state, age >72 years, presence of extracardiac arteriopathy, systolic pulmonary artery pressure >55 mmHg and urgent operation. Patients at low risk (≤1.5%) were 3467 (55.1%), at low-intermediate risk (1.6–5.0%) were 1074 (17.1%), at intermediate risk (5.1–10.0%) were 1140 (18.1%), at intermediate-high risk (10.1–20.0%) were 239 (3.8%) and at high risk (>20%) were 373 (5.9%). The five subgroups, easily identifiable at the terminal node of tree showed in Fig. 3, had observed mortality rates of 1.5, 3.2, 6.4, 14.6 and 32.2%. The EuroSCORE II expected mortality rates were, respectively, 2.2, 3.3, 6.2, 9.9 and 18.8%.

DISCUSSION In this study of more than 6000 patients over 2 years from the seven cardiac surgical units of a Southern Italian region, we have shown that the external performance of EuroSCORE II in predicting hospital mortality was good with high concordance between predicted probabilities and outcome. The model showed acceptable calibration with no significant differences between predicted and observed frequencies. There were deviations in absolute risk predictions with overestimation of survival in patients at higher risk. The complete model revision showed that most of the re-estimated regression coefficients were not significantly different from those in

Table 4: Performance of EuroSCORE II according to different subgroups of procedures n

Elective Urgent Emergency/salvage Overall CABG surgery Overall valve procedure Overall surgery of the thoracic aorta Isolated CABG surgery One procedure Aortic valve Mitral valve Thoracic aortic surgery Two procedures CABG and valve Two valves Valve and thoracic aortic surgery Three procedures

4990 944 257 3424 3131 899 2605 1596 769 497 182 1362 502 317 287 628

Hospital mortality

Discrimination

Recalibration

Observed (%)

Expected (%)

O/E ratio (95% CI)

c-statistic (95% CI)

Slope (P-value)

3.1 9.0 23.7 4.7 5.6 7.6 3.0 3.8 2.1 3.8 7.7 7.1 7.8 7.9 3.5 10.0

3.4 6.3 16.4 4.0 5.5 8.3 2.7 3.1 2.7 2.5 5.3 6.5 6.3 6.5 6.4 10.4

0.90 (0.76–1.04) 1.43 (1.14–1.72) 1.45 (1.13–1.76) 1.17 (0.99–1.35) 1.01 (0.86–1.15) 0.91 (0.7–1.12) 1.12 (0.88–1.37) 1.25 (0.94–1.56) 0.76 (0.39–1.13) 1.53 (0.85–2.21) 1.46 (0.72–2.20) 1.10 (0.89–1.32) 1.23 (0.86–1.61) 1.22 (0.76–1.68) 0.54 (0.21–0.88) 0.96 (0.74–1.19)

0.792 (0.757–0.827) 0.822 (0.780–0.864) 0.740 (0.669–0.810) 0.846 (0.812–0.875) 0.811 (0.777–0.846) 0.837 (0.790–0.886) 0.830 (0.790–0.870) 0.829 (0.773–0.884) 0.783 (0.648–0.918) 0.792 (0.688–0.896) 0.869 (0.783–0.955) 0.808 (0.762–0.854) 0.790 (0.721–0.859) 0.773 (0.668–0.878) 0.903 (0.849–0.957) 0.792 (0.734–0.851)

1.12 (0.123) 0.92 (0.435) 0.68 (0.014) 1.12 (0.098) 1.01 (0.912) 1.13 (0.297) 1.20 (0.064) 1.19 (0.106) 1.15 (0.489) 1.05 (0.808) 1.27 (0.316) 1.06 (0.573) 1.00 (0.992) 0.99 (0.961) 1.30 (0.356) 1.03 (0.800)

Recalibration slope refers to the coefficient of EuroSCORE II linear predictor included in a logistic model for hospital mortality with P-values that compare the estimated slope to unity under the hypothesis of perfect calibration. O/E: observed/expected ratio.

845

ADULT CARDIAC


Figure 3: Predictors of in-hospital mortality and risk stratification after cardiac surgery: each node is based on available data for each predictive variable in the EuroSCORE II model. Owing to low prevalence of some high risk categories, a salvage procedure was analysed with emergency procedure and patients with left ventricular ejection fraction ≤20% with those 21–30% (unique group with values ≤30%). CC = creatinine clearance; LVEF = left ventricular ejection fraction; NYHA = New York Heart Association.

the original model. Moreover, the recursive analysis of EuroSCORE II variables allowed developing a practical user-friendly bedside tool for risk stratification of hospital mortality in patients receiving cardiac operations. Risk prediction models play an important role in current cardiac surgical practice. The EuroSCORE II model was developed to help clinicians and surgeons in estimation of the absolute mortality risk of patients undergoing cardiac surgery and it was conceptually created by a combination and modification of previous scoring systems [1–3]. The new EuroSCORE II was developed in order to

improve calibration of the score preserving the high discrimination reported by the old model of EuroSCORE [3]. The performance of the recently updated EuroSCORE II has been assessed in several studies including patients who had undergone major cardiac surgery [5–11], isolated CABG or valve surgery [12–15], emergency or high-risk procedures [16, 17]. Generalization of results assessing the role of EuroSCORE II, however, is limited by small sample sizes of prospective studies conducted after EuroSCORE II enrolment or by retrospective design of larger studies, often single centre, that have analysed


846

Table 5: Comparison of recent EuroSCORE II studies Study

Time period

Sample size

Centres

Procedures

AUC

Calibration

Nashef et al. [3] Di Dedda et al. [5] Chalmers et al. [6] Grant et al. [7] Barili et al. [8] Carnero-Alcázar et al. [9] Kirmani et al. [10] Borde et al. [11] Biancari et al. [12] Kunt et al. [13] Zhang et al. [14] Wang et al. [15] Grant et al. [16] Howell et al. [17]

May to July 2010 September 2010 to October 2011 January 2006 to March 2010 July 2010 to March 2011 2006–2011 January 2005 to December 2010 February 2001 to March 2010 December 2011 to October 2012 June 2006 to April 2011 June 2004 to March 2012 January 2006 to December 2011 January 2008 to December 2011 April 2008 to March 2011 April 2006 to March 2011

22 381 1090 5576 23 740 12 325 3798 15 499 498 1027 428 3479 11 170 3342 933

154 1 1 41 3 1 1 1 1 1 1 4 41 2

Major cardiac surgery Major cardiac surgery Major cardiac surgery Major cardiac surgery Major cardiac surgery Major cardiac surgery Major cardiac surgery CABG and valve surgery Isolated CABG Isolated CABG Valve surgery Valve surgery Emergency procedures High-risk cardiac surgery

0.810 0.81 0.79 0.808 0.82 0.85 0.818 0.69 0.852 0.72 0.685 0.72 0.690 0.67

0.051 0.22