Author + information
- Received January 18, 2017
- Revision received February 13, 2017
- Accepted February 15, 2017
- Published online March 5, 2018.
- Tessa S.S. Genders, MD, PhDa,
- Adrian Coles, PhDa,
- Udo Hoffmann, MD, MPHb,
- Manesh R. Patel, MDa,
- Daniel B. Mark, MD, MPHa,
- Kerry L. Lee, PhDa,
- Ewout W. Steyerberg, PhDc,
- M.G. Myriam Hunink, MD, PhDd,e,
- Pamela S. Douglas, MDa,∗ (, )
- on behalf of the CAD Consortium and the PROMISE Investigators
- aDuke Clinical Research Institute, Duke University School of Medicine, Durham, North Carolina
- bDepartment of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts
- cDepartment of Public Health, Erasmus University Medical Center, Rotterdam, the Netherlands
- dDepartment of Epidemiology and Radiology, Erasmus University Medical Center, Rotterdam, the Netherlands
- eDepartment of Health Policy and Management, Harvard T.H. Chan School of Public Health, Harvard University, Boston, Massachusetts
- ↵∗Address for correspondence:
Dr. Pamela S. Douglas, Duke Clinical Research Institute, 7022 North Pavilion DUMC, P.O. Box 17969, Durham, North Carolina 27715.
Objectives This study sought to externally validate prediction models for the presence of obstructive coronary artery disease (CAD).
Background A better assessment of the probability of CAD may improve the identification of patients who benefit from noninvasive testing.
Methods Stable chest pain patients from the PROMISE (Prospective Multicenter Imaging Study for Evaluation of Chest Pain) trial with computed tomography angiography (CTA) or invasive coronary angiography (ICA) were included. The authors assumed that patients with CTA showing 0% stenosis and a coronary artery calcium (CAC) score of 0 were free of obstructive CAD (≥50% stenosis) on ICA, and they multiply imputed missing ICA results based on clinical variables and CTA results. Predicted CAD probabilities were calculated using published coefficients for 3 models: basic model (age, sex, chest pain type), clinical model (basic model + diabetes, hypertension, dyslipidemia, and smoking), and clinical + CAC score model. The authors assessed discrimination and calibration, and compared published effects with observed predictor effects.
Results In 3,468 patients (1,805 women; mean 60 years of age; 779 [23%] with obstructive CAD on CTA), the models demonstrated moderate-good discrimination, with C-statistics of 0.69 (95% confidence interval [CI]: 0.67 to 0.72), 0.72 (95% CI: 0.69 to 0.74), and 0.86 (95% CI: 0.85 to 0.88) for the basic, clinical, and clinical + CAC score models, respectively. Calibration was satisfactory although typical chest pain and diabetes were less predictive and CAC score was more predictive than was suggested by the models. Among the 31% of patients for whom the clinical model predicted a low (≤10%) probability of CAD, actual prevalence was 7%; among the 48% for whom the clinical + CAC score model predicted a low probability the observed prevalence was 2%. In 2 sensitivity analyses excluding imputed data, similar results were obtained using CTA as the outcome, whereas in those who underwent ICA the models significantly underestimated CAD probability.
Conclusions Existing clinical prediction models can identify patients with a low probability of obstructive CAD. Obstructive CAD on ICA was imputed for 61% of patients; hence, further validation is necessary.
- computed tomography angiography
- coronary artery disease
- invasive coronary angiography
- prediction models
Every year, millions of patients in the United States with stable chest pain undergo noninvasive diagnostic testing to investigate the presence of obstructive coronary artery disease (CAD) (1). The decision to proceed to invasive coronary angiography (ICA) is often based on the results of such noninvasive tests. However, 59% of stable symptomatic patients referred for elective ICA in the United States are free of obstructive CAD (2). A better strategy to select patients who might benefit from invasive testing is needed, which should begin by better risk-stratifying patients who should undergo noninvasive testing (3).
The clinical value of a diagnostic test for CAD depends on the pre-test probability of CAD (4–7). Current guidelines uniformly recognize this and recommend considering the pre-test probability before deciding whether to test. However, due to a lack of evidence on comparative effectiveness of imaging strategies, for a given pre-test probability and other factors, the test of choice may vary across countries (8–10). The traditional Diamond and Forrester model (11), which in combination with the model based on the CASS (Coronary Artery Surgery Study) study data (12) is currently recommended by the American College of Cardiology Foundation/American Heart Association stable ischemic heart disease guidelines (8), significantly overestimates the pre-test probability of obstructive CAD (13–15). Improved estimates of the pre-test probability may be obtained using updated prediction models that were developed by the CAD Consortium (13), and they can potentially help clinicians make better decisions as to which patients should undergo noninvasive testing. The current study aims to externally validate the CAD consortium prediction models for the presence of obstructive CAD in chest pain patients from the United States.
Our study population consisted of patients enrolled in the PROMISE (Prospective Multicenter Imaging Study for Evaluation of Chest Pain) trial, which has been described in detail previously (16,17). In brief, the PROMISE trial was a pragmatic, multicenter randomized trial that compared outcomes of initial anatomic testing with the use of coronary computed tomography angiography (CTA) versus initial functional testing (exercise electrocardiography, nuclear stress testing, or stress echocardiography) for patients with suspected CAD. Enrollment began on July 27, 2010, and was completed on September 19, 2013. Patients were symptomatic outpatients; men were ≥55 years of age, or ≥45 to 54 years of age with ≥1 cardiac risk factor (diabetes, peripheral arterial disease, cerebrovascular disease, current or past smoking, hypertension, or dyslipidemia) and women were ≥65 years of age, or ≥50 to 64 years of age with ≥1 cardiac risk factor. Patients with a history of acute myocardial infarction, known CAD, or revascularization were excluded. For the current study we selected PROMISE trial patients who were assigned to the anatomic testing strategy, presented with chest pain, and underwent CTA, ICA, or both.
The Duke University Health System Institutional Review Board approved this study. A waiver of informed consent was granted for the current analysis. Patients previously consented for enrollment in the PROMISE trial.
Risk factor definitions in the PROMISE trial
Chest pain symptoms were defined as typical, atypical, or noncardiac. Typical chest pain was defined as: 1) substernal chest pain or discomfort; that was 2) provoked by exertion or emotional stress; and 3) relieved by rest or nitroglycerine. Atypical chest pain was defined as 2 of the previously mentioned criteria. If 1 or none of the criteria was present, chest pain symptoms were categorized as noncardiac (18). Hypertension was defined as a blood pressure >140/90 mm Hg on at least 2 occasions (>130/80 mm Hg for patients with diabetes or chronic kidney disease) or requiring antihypertensive treatment. Diabetes was defined as a history of diabetes, an elevated fasting serum glucose >126 mg/dl (7 mmol/l), or the need for antidiabetic agents. Dyslipidemia was defined as an elevated cholesterol level (total cholesterol >200 mg/dl [5.18 mmol/l], low-density lipoprotein >130 mg/dl [3.37 mmol/l], or high-density lipoprotein <40 mg/dl [1.04 mmol/l] in men and <50 mg/dl [1.30 mmol/l] in women) or treatment with cholesterol-lowering medications. Smoking was defined as current or past smoking. The coronary artery calcium (CAC) score was determined using the Agatston method (19). Risk factor definitions were essentially the same as those used during the development of the prediction models (13) (Table 1).
We validated 3 logistic regression models as previously developed and reported by the CAD Consortium (13). The basic model included age, sex, and type of chest pain. The clinical model included the variables of the basic model and additionally diabetes, hypertension, dyslipidemia, and smoking. The clinical + CAC score model included the CAC score in addition to the variables of the clinical model. For reference we also considered the validity of the updated Diamond and Forrester model (14).
Obstructive CAD was defined as ≥1 stenosis of ≥50% in 1 or more vessels (≥2.0 mm diameter) on ICA (within 90 days of the index test). Assessment of ICA results was blinded for neither clinical information nor the CTA result. CTA and ICA interpretation was based on site-specific reads. ICA is often considered to be the reference standard for diagnosing CAD. CTA is an excellent test with a sensitivity of 96% to 100% and specificity of 85% to 92% compared with ICA (20,21). Given its high sensitivity for diagnosing obstructive CAD, a negative CTA (<50% stenosis) virtually excludes the presence of obstructive CAD on ICA, especially in low-risk populations (20). We assumed that patients with a completely normal CTA (0% stenosis and CAC score of 0) would be free of obstructive CAD on ICA, which is supported by the very low risk of adverse events in this subgroup (22).
Sensitivity analyses without imputation
For the primary analysis we relied on imputed data for patients who did not undergo ICA. In 2 sensitivity analyses we eliminated the need to rely on imputed ICA data. In the first sensitivity analysis we used CTA as the outcome of interest and in the second we restricted the analysis to the subgroup of patients who underwent ICA (Figure 2⇓⇓, see Online Figure 2 for additional sensitivity analyses).
We used multiple imputation methods with chained equations (23) to account for missing clinical variables (24) and to impute missing ICA results (Online Appendix, Online Table 1) (25–28). We assumed missing clinical variables occurred at random, and missing variables were predicted using the CTA result, ICA result, CAC score, age, sex, type of chest pain, hypertension, dyslipidemia, diabetes, smoking, and body mass index. McFadden’s R2 for the imputation model ranged from 0.52 to 0.59 across iterations, which indicates excellent model fit. Twenty datasets were created that contained identical information except for the imputed missing variables, for which the variability across datasets reflects the uncertainty associated with imputations. We used previously published logistic regression model equations (Online Table 2) to calculate predicted probabilities for each patient.
For each prediction model, we calculated the C-statistic (discrimination), constructed calibration plots (29), and performed a detailed assessment of model validity (Online Appendix, Online Table 3, Online Figure 1) (28–31). The first validation step consisted of assessing calibration in the large, which compares the mean predicted probability and the mean observed frequency of obstructive CAD in the validation data. The ideal value is zero difference. The second validation step was assessment of the overall predictive effect, which included graphical assessment in a calibration plot and estimation of a calibration slope. A slope different from 1 indicates that the overall combined predictive effect of the predictors in the model was different from the overall effect as observed in the validation data. The last validation step was re-estimating the predictor effects in the validation data and calculating the difference with the predictor effects in the prediction models. A p value <0.05 was considered statistically significant. All analyses were performed using STATA/SE version 14.1 (StataCorp, College Station, Texas). Calibration plots were constructed using R version 3.3.0 (R Project for Statistical Computing, Vienna, Austria).
Of the 4,996 patients assigned to the anatomic testing (CTA) strategy in the PROMISE trial, 3,468 patients presented with chest pain and underwent CTA, ICA, or both (1,663 men, 1,805 women; mean 58 and 61 years of age, respectively). Nearly all patients (99.6%) underwent CTA, and 3,335 (96%) had interpretable CTA results, showing ≥50% stenosis in 779 patients (23%). Of those who underwent CTA, 441 were referred for ICA, which showed obstructive CAD in 75% (n = 332). Of those patients with <50% on CTA (n = 2,556), 38 patients (1.5%) were referred for ICA, which demonstrated obstructive disease in 13.
When compared with the patient characteristics of the derivation cohort (13), the PROMISE trial patients were more likely to have cardiac risk factors such as diabetes, hypertension, dyslipidemia, and smoking (Table 1). The distribution of type of chest pain was different as well, with fewer patients with typical and noncardiac chest pain in the PROMISE trial. The proportion of patients with ≥50% stenosis on CTA was 23% in the PROMISE trial and 25% in the derivation cohort. Overall 452 (13%) patients in the PROMISE trial were referred for ICA and 848 (19%) patients were referred for ICA in the derivation cohort. The proportion of ICA results showing absence of obstructive CAD was 25% in the PROMISE trial and 48% in the derivation cohort.
Some patients had missing values for body mass index (n = 34, 1%), CAC score (n = 375, 11%), CTA (n = 133, 4%), and ICA (n = 3,016, 87%). Of the patients without ICA results, 910 had a completely normal CTA (0% stenosis and CAC score = 0). For the primary analysis we assumed that these patients would be free of obstructive CAD on ICA, leaving 2,106 (61%) patients with missing ICA results. Of these, 1,608 had <50% stenosis on CTA for whom almost all imputed ICA results were negative (range 93% to 98% across imputations), 289 had moderate CAD (≥50% to 70% stenosis) on CTA for whom most imputed ICA results were positive (range 59% to 72% across imputations), and 127 had severe CAD (≥70% stenosis or ≥50% left main stenosis) for whom the majority of imputed ICA results were positive (range 73% to 89% across imputations).
The externally validated C-statistic was 0.69 (95% confidence interval [CI]: 0.67 to 0.72) for the basic model, 0.72 (95% CI: 0.69 to 0.74) for the clinical model, and 0.86 (95% CI: 0.85 to 0.88) for the clinical + CAC score model (Figure 1). Overall calibration was satisfactory for all 3 models. The clinical model overestimated for higher predicted probabilities, which was most pronounced in women. The basic and clinical model were less well calibrated in women versus men; however, calibration of the clinical + CAC score model was near perfect regardless of sex (Figure 1). The overall predictor effects were slightly smaller in the PROMISE trial (calibration slope <1). Specifically, the presence of typical chest pain or diabetes was less predictive (p = 0.001 and p = 0.04, respectively) in the PROMISE trial as compared with the effect as suggested by the prediction models, and the predictive effect of the CAC score was stronger (p = 0.02) (Online Table 3). The updated Diamond and Forrester model (14) significantly overestimated the probability of CAD (poor calibration), and its predictor effects were stronger than observed in the validation data (Online Figure 3).
The potential clinical usefulness of the 3 prediction models is illustrated with a categorization of patients as low (≤10%), intermediate (>10% to <90%), and high probability (≥90%) (Table 2, Online Tables 4 and 5) for the predicted probability of obstructive CAD. The basic and clinical model did not predict any probabilities ≥90%, whereas the clinical + CAC score model predicted a probability of ≥90% for <1% of patients. The clinical model was able to identify 31% of patients as having a low probability, of whom 7% had obstructive CAD, whereas the clinical + CAC score model identified 48% as having a low probability, of whom only 2% had obstructive CAD (Table 2).
Sensitivity analyses without imputation
We used the CTA result as the outcome of interest in one sensitivity analysis and in another the subgroup of patients who underwent ICA, eliminating the need to rely on imputed data in both cases (Figure 2). The prediction models underestimated the probability of obstructive CAD on CTA, which is expected because the prediction models were developed to predict CAD on ICA. Given the relatively less favorable positive predictive value of CTA, especially in low-probability populations (20), the prevalence of CAD based on CTA is generally higher than based on ICA. The prediction models significantly underestimated the probability of disease in the subgroup of patients who underwent ICA, which is expected due to the effect of referral bias.
Summary of key findings
The CAD consortium prediction models provided overall accurate estimates of the pre-test probability in the PROMISE trial, even though some predictors (i.e., typical chest pain and diabetes) appeared to be less predictive and the CAC score appeared to be more predictive than expected. The models can be used to estimate the pre-test probability in patients with chronic stable angina who are considered for noninvasive imaging tests. Including the CAC score significantly improved the prediction of obstructive CAD, suggesting that the CAC score could add important information to the diagnostic work-up. For example, patients with a low probability based on the clinical + CAC score model (e.g., those with CAC score of 0) could potentially defer further diagnostic imaging. The CRESCENT (Calcium Imaging and Selective CT Angiography in Comparison to Functional Testing for Suspected Coronary Artery Disease) trial compared a tiered computed tomography strategy (CTA was deferred for patients with a low probability and a CAC score of 0) with functional testing showing that this was safe and effective (32).
Traditional prediction tools such as the Diamond and Forrester model (11,14,15) and the Duke Clinical Score (13,33) have been shown to overestimate the probability of CAD. The European Society of Cardiology adopted the updated Diamond and Forrester model, because it significantly reduces the overestimation by the traditional Diamond and Forrester model for patients referred to ICA (14,34). However, contemporary patient populations who are considered for noninvasive testing have an even lower probability of disease (15,16). Based on our findings, a future revision of the 2012 American College of Cardiology Foundation/American Heart Association stable ischemic heart disease guideline (8) may review the current recommendation of using the traditional Diamond and Forrester model (11) and CASS study model (12). We believe the CAD consortium prediction models should receive strong consideration, as these models were developed and validated in contemporary patient populations that better represent the low-risk patients who are currently considered for noninvasive testing.
Interestingly, model calibration was better in men compared with women, which could be related to sex differences in symptom presentation, prevalence of risk factors (35,36), and overall prevalence of disease (obstructive CAD on CTA was 30% in men and 18% in women). We observed a significantly smaller predictive effect of the presence of typical chest than expected. Patients with typical chest pain are generally considered to have a very high likelihood of CAD and are oftentimes directly referred for ICA. Given the fact that the PROMISE trial included participants whose physicians believed that nonurgent, noninvasive cardiovascular testing was necessary, a substantial proportion of patients with typical chest pain might not have been included in the PROMISE trial but instead might have been referred directly to ICA (2).
First of all, only 13% of this subset of PROMISE trial patients underwent ICA. For the primary analysis we relied on imputed data for missing ICA results. In a sensitivity analysis we restricted the analysis to patients who underwent ICA, even though this analysis is subject to verification bias (37). In another sensitivity analysis we used the CTA result as the outcome; however, the presence or absence of CAD can be misclassified by CTA as a result of false negative and false positive test results, respectively. In a population with a 20% prevalence of obstructive CAD, the positive predictive value of ≥50% stenosis on CTA is estimated to be approximately 70% (20). This means that a non-negligible proportion of patients with ≥50% stenosis on CTA will be free of obstructive CAD on ICA. Using the CTA result as the outcome will overestimate the true prevalence of obstructive CAD. We therefore opted to use an alternative method for the primary analysis in which we assumed absence of obstructive CAD on ICA for those with a completely normal CTA (0% stenosis and CAC score of 0). We believe this is justified based on the well-established negative predictive value of a negative (<50% stenosis) CTA, which is >99% regardless of the CAC score (20) and the very low probability (<1%) of ≥50% stenosis for patients with a CAC score of 0 (38,39). In our assumption we were as restrictive as possible and assumed a negative ICA result only for those patients with both 0% stenosis on CTA and a CAC score of 0. We subsequently multiply imputed the remaining missing ICA results based on the CTA result and clinical variables.
Second, although the clinical + CAC score model demonstrated the best model performance, the CAC score is not currently recommended in the diagnostic work-up for patients with stable chest pain. Therefore, the CAC score is often not available when the pre-test probability is calculated, which currently limits the clinical usefulness of the clinical + CAC score model and suggests that the CAC score should be considered in future algorithms.
Third, the overall CAD prevalence in the PROMISE trial was low, which limited our ability to assess calibration at higher disease prevalence. However, the aim of the current analysis was to show validity of predictions for patients who are being considered for noninvasive testing, which, in the contemporary era, are generally low-probability populations.
Finally, the current analysis focused on diagnosing obstructive CAD and did not consider prognostic information. Simultaneous prediction of cardiovascular adverse event rates potentially improves decision making regarding noninvasive diagnostic testing.
Future research should focus on studying new predictors and updating existing predictors (especially those that are most uncertain [e.g., type of chest pain, sex differences, CAC]) so these can be updated to further improve model discrimination and calibration.
Although it is widely recognized that the pre-test probability should be used to determine which patient should be considered for noninvasive testing, the method of determining this varies across countries and across guidelines as does the test of choice (8–10). Our prediction model could be used in future comparative effectiveness trials and cost-effectiveness analyses to determine the optimal imaging strategy, taking into account long-term outcomes and comparing multiple diagnostic strategies (40).
Our results suggest that the basic model, clinical model, and the clinical + CAC score model can be used to reliably estimate the pre-test probability of CAD in low-risk patients who present with stable chest pain and who do not have a history of CAD (41).
The European Society of Cardiology (10) and National Institute for Health and Care Excellence (9) guidelines recommend considering other causes of chest pain as opposed to further testing for CAD in patients with an estimated probability of <10% to 15% for CAD. Using the clinical model, 31% of patients are identified as having a probability of ≤10% for CAD, in whom one may consider refraining from further testing for CAD, potentially reducing unnecessary procedures and costs. Including the CAC score significantly improved model performance, which suggests that the CAC score may be useful to guide further management (9,32,42).
Clinical prediction models for the presence of CAD provide well-calibrated predictions and can help clinicians to identify patients with a low probability of CAD.
COMPETENCY IN MEDICAL KNOWLEDGE: In patients with stable chest pain suspected of having obstructive CAD traditional prediction models tend to overestimate the probability of disease. The CAD consortium prediction models provide better estimates of the probability of disease for low-risk patients who are being considered for noninvasive testing. The clinical model was most useful for identifying patients at very low risk. The CAC score significantly improves the accuracy of predictions and should be considered in the diagnostic work-up for low-risk patients with chest pain.
TRANSLATIONAL OUTLOOK: This study externally validated the CAD consortium prediction models in a low-risk population in which the majority of patients did not undergo ICA. The primary analysis relied on imputed data for the presence of obstructive CAD on ICA; therefore, further validation studies remain necessary. Although the CAC score improved the accuracy of predictions, the CAC score may not be readily available, which limits the clinical applicability.
The PROMISE trial was supported by grants R01HL098237, R01HL098236, R01HL98305, and R01HL098235 from the National Heart, Lung, and Blood Institute. The authors are solely responsible for the design and conduct of this study, all study analyses, the drafting and editing of this paper, and its final contents. This paper does not necessarily represent the official views of National Heart, Lung, and Blood Institute. Dr. Hoffmann has received research grants (significant) from the American College of Radiology Imaging Network, HeartFlow, Siemens Healthcare, Pfizer, and Genzyme. Dr. Patel has received research grants (significant) from AstraZeneca, Janssen, and HeartFlow; and has served on the advisory board for AstraZeneca, Janssen, Bayer, and Genzyme. Dr. Mark has served as a consultant for Medtronic, CardioDx, and St. Jude Medical; and has received research grants (significant) from Eli Lilly and Company, Medtronic, Bristol-Myers Squibb, AstraZeneca, Merck & Company, Oxygen Therapeutics, and Gilead. Dr. Hunink has received royalties from Cambridge University Press; grants and nonfinancial support from the European Society of Radiology; and nonfinancial support from the European Institute for Biomedical Imaging Research. Dr. Douglas has received research grants (significant) from GE and HeartFlow. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- coronary artery calcium
- coronary artery disease
- confidence interval
- computed tomography angiography
- invasive coronary angiography
- Received January 18, 2017.
- Revision received February 13, 2017.
- Accepted February 15, 2017.
- 2018 American College of Cardiology Foundation
- Ladapo J.A.,
- Jaffer F.A.,
- Hoffmann U.,
- et al.
- Fihn S.D.,
- Gardin J.M.,
- Abrams J.,
- et al.
- Cooper A.,
- Timmis A.,
- Skinner J.,
- Guideline Development Group
- Chaitman B.R.,
- Bourassa M.G.,
- Davis K.,
- et al.
- Genders T.S.,
- Steyerberg E.W.,
- Hunink M.G.,
- et al.
- Cheng V.Y.,
- Berman D.S.,
- Rozanski A.,
- et al.
- Diamond G.A.
- Agatston A.S.,
- Janowitz W.R.,
- Hildner F.J.,
- Zusmer N.R.,
- Viamonte M. Jr..,
- Detrano R.
- Villines T.C.,
- Hulten E.A.,
- Shaw L.J.,
- et al.
- Steyerberg E.W.
- Hemal K.,
- Pagidipati N.J.,
- Coles A.,
- et al.
- Pagidipati N.J.,
- Hemal K.,
- Coles A.,
- et al.
- Haberl R.,
- Becker A.,
- Leber A.,
- et al.
- Hunink M.G.,
- Weinstein M.C.,
- Wittenberg E.,
- et al.
- ↵Kattan M. Cleveland Clinic Foundation, Department of Quantitative Health Sciences. Available at: http://www.r-calc.com/. Accessed January 1, 2017.