Author + information
- Received February 8, 2010
- Revision received August 31, 2010
- Accepted September 7, 2010
- Published online November 1, 2010.
- Leslee J. Shaw, PhD⁎,⁎ (, )
- Peter W.F. Wilson, MD⁎,
- Rory Hachamovitch, MD, MS†,
- Robert C. Hendel, MD‡,
- Salvador Borges-Neto, MD§ and
- Daniel S. Berman, MD∥
- ↵⁎Reprint requests and correspondence:
Dr. Leslee J. Shaw, 1256 Briarcliff Rd NE, Suite 1-N, Emory University School of Medicine, Atlanta, Georgia 30306
Objectives We compared analytical approaches to estimate the added value of myocardial perfusion single-photon emission computed tomography (MPS) variables in estimating coronary artery disease (CAD) outcomes.
Background Stress MPS markers of regional ischemia are strong estimators of prognosis. Evidence published to date has not compared analytical methods to establish the added value of stress MPS and to define a clinically meaningful approach to detect improve classification of risk.
Methods A total of 4,575 patients were consecutively and prospectively enrolled in the Myoview Prognosis Registry. Multivariable Cox proportional hazards model were employed to estimate CAD death or myocardial infarction (MI). Risk reclassification methods were also calculated.
Results In risk-adjusted models (including age, sex, presenting symptoms, stress type, CAD history, and risk factors), stress MPS ischemia, rest and post-stress left ventricular ejection fraction (LVEF) (all p < 0.0001) were all significant estimators of CAD death or MI. In this multivariable model, 34% of the model chi-square was contributed by MPS ischemia. In receiver-operating characteristic curve analysis, the area under the curve increased from 0.61 to 0.66 when rest and post-stress LVEF were combined with pre-test CAD likelihood (p < 0.0001), increasing to 0.69 for MPS ischemia (p < 0.0001). The net reclassification improvement (NRI) by adding the Duke Treadmill Score (DTS) to a model including pre-test CAD likelihood was 0.112. The cost per NRI was $57 for the exercise test as compared with an office visit for risk stratification purposes. Further, the NRI by adding MPS ischemia to a model with the DTS and pre-test CAD likelihood was 0.358. The cost per NRI was $615 for the stress MPS as compared with an exercise test.
Conclusions Stress-induced ischemia is independently predictive of near-term CAD outcomes. Analytical approaches that establish the reclassification of events provide a unique approach and may serve as a quality imaging metric for estimation of improved health outcomes for stress MPS as well as for comparison to other imaging modalities.
The concept of risk stratification using clinical and laboratory markers has been a major focus of observational registries over the past several decades. Risk prediction that integrates clinical variables into a prognostic model results in the identification of cardiac risk factors with a, generally, graded impact on prognosis (1,2). The development of a prognostic model traditionally includes the evaluation of the independent contribution of variables with consideration of covarying factors that may confound risk stratification (3). Yet, the multivariable modeling process fails to define how a given risk marker improves the classification of risk, that is, the enhanced detection of low- or high-risk patients. Improvements in discrimination may be accomplished using receiver-operator characteristic (ROC) curves, yet they fail to quantify the proportion of patients and the direction of improvement in classifying low- to high-risk patients. Risk reclassification methods that enumerate newly defined patient subsets have clinical appeal and an ease of understanding where other analytical approaches do not.
Few reports have explored the comparative ability of stress myocardial perfusion single-photon emission computed tomography (MPS) risk markers using varied iterative, incremental, and risk classification approaches (4,5). The aim of the current study was to estimate coronary artery disease (CAD) death or myocardial infarction (MI) using traditional approaches of prognostication to more recent methods from a large prospective registry of stable chest pain patients undergoing stress MPS, focusing on improved risk detection with ischemia as compared with electrocardiographic and pre-test likelihood measurements (6–8).
The methods of the Myoview Prognosis Registry were previously published (6–8). Of the 7,849 patients enrolled, 4,575 patients had rest and post-stress left ventricular ejection fraction (LVEF) and MPS. Study procedures were approved by each center's institutional review board.
Stress testing protocols
Forty percent of patients underwent exercise testing using the modified Bruce protocol. Heart rate, blood pressure, and electrocardiographic changes were monitored. Testing was discontinued at maximal stress, fatigue, or due to: ≥3 mm of ST-segment depression or ≥1 mm of ST-segment elevation (in a non–Q-wave lead); ventricular tachycardia or fibrillation; exertional hypotension, chronotropic incompetence, or worsening chest pain. The remaining patients underwent pharmacologic stress; with monitoring procedures similar to exercise testing.
Rest and stress gated MPS
Each scan was interpreted by a nuclear cardiologist blinded to the patient's stress test and clinical data. However, the patient's sex was accessible to the interpreter. MPS procedures were standardized across participating sites (9). Patients underwent either rest Tl-201 or Tc-99m. Representative short-axis and horizontal long-axis images were segmented into 20 regions for interpretation of myocardial perfusion. Each segment was scored from normal uptake (score = 0) to absent perfusion (score = 4). Total scores at rest and post-stress were summed. To define percentage ischemic myocardium, the total stress score was subtracted from the total rest score and divided by 80 to get the percentage ischemic myocardium.
Patients were prospectively followed using a scripted interview performed over the telephone or during clinic visits; ∼1% of patients were lost. Outcome data collection included documentation of coronary revascularization procedures. Death information was confirmed by review of the patient's death certificate or medical records. Fatal MI was defined for patients who died in-hospital within 24 h of admission following enzyme elevation. Patients with end-stage CAD with heart failure noted as a cause of death were coded as CAD deaths. A witnessed sudden cardiac death was coded as CAD death. Median follow-up time for surviving patients was 1.6 years (25th to 75th percentile = 1.2 to 2.0 years).
Categorical comparisons were calculated using the chi-square statistic, whereas continuous measures were compared using the nonparametric Kruskal-Wallis statistic. Time to CAD death or MI was estimated using Cox proportional hazard models. Patients were censored at the time of revascularization or at the time of a primary end point, whichever occurred first. Models were analyzed for univariable associations as well as multivariable associations. Model overfitting procedures were considered, and the proportional hazards assumption was met.
For the risk-adjusted model, we a priori included age, sex, risk factors, symptoms, stress type, and CAD history as covariates. The proportion of excess risk explained was calculated using the equation: [HRU − HRA] / [HRU − 1], where HR is the hazard ratio, HRU is an unadjusted HR, and HRA is a risk-adjusted HR (10).
Additional models, which examined the incremental value of stress MPS variables of LVEF and ischemia, were calculated by defining unadjusted models followed by additional models including: 1) age and sex adjustment; 2) age, sex, CAD history, stress type, and cardiac symptom adjustment; and 3) age, sex, CAD history, cardiac symptoms, stress type, and cardiac risk factors. The ROC curve areas (95% confidence intervals [CI]) were calculated.
We then applied the reclassification methodologies of Pencina et al. (11,12). We compared: 1) the addition of the Duke Treadmill Score (DTS) in a model including the CAD likelihood; and 2) the addition of the MPS ischemia in a model including the DTS. We applied risk categories of <1%, 1% to 3%, 3.01% to 5.00%, and >5% predicted CAD death or MI rate per year. For those with events, the number reclassified upward and those reclassified downward was enumerated, followed by a subtraction of the number with events upwardly reclassified minus those downwardly reclassified, then divided by the total number of events. For those without events, the total number who were classified upwards and those reclassified downward was enumerated, followed by a subtraction of patients without events downwardly reclassified minus those upwardly reclassified, then divided by the total number of nonevent patients. The net reclassification improvement (NRI) was then calculated by summing the calculations from these latter 2 sentences (13).
We calculated the incremental costs of: 1) exercise test–outpatient visit; and 2) MPS–exercise test. Procedural costs were applied to estimate cost per NRI (14,15). Procedural costs were determined from the 2010 Medicare Physician Fee Schedule for Current Procedural Terminology (CPT) codes 93015 and 78452 (16) and from the Medicare Outpatient PC Pricer System using APC code 601 for a midlevel clinic visit (17).
Clinical characteristics of the Myoview Prognosis Registry
Patients were typical of those referred to a stress imaging laboratory (Table 1). They were on average 62 years of age, mostly male, with the majority referred for evaluation of stable chest pain symptoms.
The stress MPS results (Table 2) reveal that the majority of patients underwent pharmacologic stress or exercised through stage II of the modified Bruce protocol. ST-segment depression ≥1 mm occurred frequently in this cohort (range = 32% to 38%). Approximately 9% to 20% of low- to high-likelihood patients had ischemia ≥5% of the myocardium. Few patients had rest or stress LVEF ≤45%; except 13% to 15% of high-likelihood patients had LVEF measures ≤45%. We present blood pressure data on the whole cohort; higher resting systolic blood pressure was reported with pharmacologic stress (mean: 146 mm Hg for pharmacologic stress vs. 136 mm Hg for exercise), whereas higher peak exercise systolic blood pressure was reported with peak exercise (mean: 187 mm Hg for exercise vs. 158 mm Hg for pharmacologic stress).
Cox survival curves estimating time to CAD death or MI
In risk-adjusted models, stress MPS ischemia (p < 0.0001), rest (p < 0.0001), and post-stress LVEF (p < 0.0001) were significant estimators of time to CAD death or MI (Fig. 1).
Prognostic models estimating mortality with stress MPS ischemia and LVEF
Table 3 depicts a multivariable model including all 3 MPS variables, noting that all were highly significant estimators of death or MI, even in adjusted models that included an array of symptom, risk factor, and historical variables. The multivariable HRs for CAD death or MI were 1.6 (95% CI: 1.3 to 2.0), 1.5 (95% CI: 1.2 to 1.9), and 2.0 (95% CI: 1.6 to 2.4) for rest LVEF ≤45%, post-stress LVEF ≤45%, and stress perfusion ischemia ≥5% of the myocardium, respectively (each p < 0.0001). Significant collinearity did not exist among the 3 variables (r < 0.5). Interestingly, in a subset analysis of patients with diabetes, the HRs were similar (MPS ischemia: 2.0, p < 0.0001); rest LVEF: 1.5, p = 0.009), with the exception post-stress LVEF was of borderline significance (HR: 1.4, 95% CI: 1.0 to 1.9; p = 0.05).
In stepwise models including age, sex, presenting symptoms, stress type, risk factors, and CAD history as well as the 3 MPS variables, ischemia was ranked first whereas rest LVEF ≤45% was ranked second and post-stress LVEF ≤45% was ranked eighth. Factors chosen before post-stress LVEF included anginal symptoms, smoking, diabetes, age, and hypertension.
Increment value analyses
When comparing the relative contribution of each MPS variable to the death or MI models, the total percentage contribution was greatest for stress ischemia, yielding 34% of prognostic content (Fig. 2). That is, when considering the estimation of risk with age, sex, risk factors, presenting symptoms, CAD history, and stress type, 34% of the chi-square was contributed by ischemia.
Unadjusted and risk-adjusted HRs are presented for rest and post-stress LVEF ≤45% and ≥5% ischemic myocardium (all p < 0.0001). In Figure 3, unadjusted hazards ranged from 2.3 to 2.5. In models estimating CAD death or MI, the proportion of excess risk explained by clinical factors was minimal (4.5%) for stress ischemia, whereas excess risk estimates were in the range of 33% to 35% for LVEF; supporting a strong independent contribution of ischemia to prognostic models estimating CAD death or MI.
ROC curve areas
Despite being significant in the risk-adjusted models, the ability of each MPS marker to classify CAD death or MI varied (Table 4). Areas under the ROC curve were 0.61 for pre-test CAD likelihood; when adding rest and post-stress LVEF measurements, this increased to 0.66 and to 0.69 when adding stress MPS (p = 0.0001).
NRI of 2-year death or MI
The NRI was 0.112 when the DTS was added to a model containing CAD likelihood (Table 5). Further, the NRI was 0.358 when MPS ischemia was added to a model containing the DTS and CAD likelihood. In the latter model including MPS, newly identified cases (i.e., events upwardly classified) more often underwent pharmacologic stress imaging (82.1%), were at intermediate-high pre-test CAD likelihood (75.5%), or had an intermediate DTS (89.3%). Newly identified controls (i.e., nonevents downwardly classified) more often underwent exercise testing (41.6%), had 0 to 1 cardiac risk factors (56.1%), and did not have angina as a presenting symptom (89.8%). Figure 4 plots the cumulative frequency of exercise times for newly identified cases and controls, noting higher total exercise times with newly defined controls as compared with cases.
Incremental cost effectiveness analyses
The incremental national payment rate for an exercise test was an added $6.33 when compared with an outpatient visit (Table 6). The incremental payment was $220.20 for MPS when compared with an exercise test. The cost per NRI was $56.52 for an exercise test compared with an outpatient visit and $615.08 for a stress MPS as compared with an exercise test.
Tremendous growth in cardiac imaging has fostered numerous policy efforts aimed toward containing utilization and devising metrics to define quality cardiovascular (CV) imaging. Recent efforts from both public and private payers have focused on extending test evaluation criteria beyond that of traditional performance characteristics (e.g., diagnostic sensitivity and specificity) to include measures defining improvements in health outcome (14,15). The development of quality imaging metrics is now paramount for evaluation of a test's added value in order to justify the referral decision. In an era where health care resources are limited and the perception of imaging utilization is that of excess, the development of analytical approaches are important to define a test's “value for money spent” (18). In our report, we evaluated both conventional risk stratification methods as well as new innovative approaches that may prove valuable to assess a test's value in clinical decision making. Our results reveal that beyond near-term risk stratification, important trends were identified, emphasizing the importance of stress ischemia as a core measure of risk. Regardless of the analytical approach, 5% or more of ischemic myocardium was an important measure of estimating 2-year risk of CAD death or MI. In fact, this measure was effective at reclassifying a sizeable proportion of patients.
The use of reclassification methods as a novel marker of a test's incremental value has appeal because it can be easily performed using available observational registries as well as in administrative datasets. This type of analysis may be one method to devise net improvement in outcome, a recent standard put forth in technology evaluation criteria (14,15). Within recent technology evaluations, the standard of improving health outcomes was defined as a test's ability to improve the quantity or quality of patient's life (14,15). From the epidemiologic literature, a new method for devising improved risk stratification has been proposed (12) that includes the calculation of net reclassification in risk. Several reports have noted this as an effective method for evaluating the added value of high sensitivity C-reactive protein beyond that of the Framingham risk score (12,13,19,20). Although net reclassification in risk has been applied in the setting of CV screening of apparently healthy individuals (19,20), prior reports have not utilized this analytical approach for the evaluation of risk in symptomatic patients. With the incorporation of net reclassification of risk, we propose that this may be one methodology to evaluate a test's ability to note improvements in outcome detection beyond the pre-test risk evaluation.
Incremental cost effectiveness analyses
Using this approach, should the test reclassify few patients, then its value would be minimal. Conversely, should a test reclassify a sizeable proportion of patients, then the cost of the procedure may be justified. We devised a simple cost model to examine the impact of NRI as a novel effectiveness measure. In this report, exercise testing was roughly similar in cost to an outpatient visit but newly reclassified nearly 11% of patients using risk data from the DTS, resulting in a cost per NRI of $57 when compared with an outpatient visit. There are no metrics or league tables for this novel cost-effectiveness metric, yet these analyses may provide 1 venue for evaluating value or discerning appropriate utilization of clinical resources. For the exercise test, it is clear that the cost per NRI of $57 is low and is far less than that for a midlevel clinic visit. However, stress MPS cost more than $200 but did so within the context of reclassifying nearly one-third of patients; resulting in a cost per NRI of ∼$615 when compared with an exercise test. It remains challenging to interpret this value, but when compared with stress echocardiography (a cheaper modality), the NRI would only have to be 20.3% (given a 2010 Medicare payment rate of $203) in order to achieve a cost per NRI of <$615. Moreover, if the NRI for stress echocardiography was similar to that of MPS (i.e., 35.8%), then the cost per NRI would be $347. Of course, the inclusion of induced costs is paramount to further development of this novel cost effectiveness metric. Moreover, employing the calculation of cost per NRI in diverse patient populations and across multiple modalities would help to validate this metric and to define a league table of comparative cost-effectiveness measures. The result could be a simple measure that may be applied to guide focused resource utilization.
Many have argued that reliance upon standard cost-effectiveness measures of cost per life-year saved fits more within therapeutic strategies and that testing does not save a life but rather detects risk. Any given test only indirectly impacts outcome, with the ensuing treatment saving lives. Of course, this latter statement remains controversial and is the subject of ongoing discussions regarding CAD imaging. Importantly, randomized trials evaluating the comparative effectiveness, including discerning the economic value of testing versus nontesting strategies, are an important part of ongoing discussions on developing broader, high-quality evidence with CAD testing. As well, the inclusion of more than just procedural cost data would provide greater improvement in creating value and efficiency for noninvasive testing.
Covarying risk markers and estimating imaging risk
Few prior reports have examined the gamut of analysis presented herein. From our analyses, it is now apparent that much of the estimated CV risk from an imaging risk marker overlaps with cardiac history and comorbidity (1,19–22). Nearly one-third of risk was encumbered by clinical covariates known prior to the stress MPS. Similarly, in a full-risk model, stress ischemia provided 34.2% of novel information content. This sizeable influence of pre-test cardiac risk, unless accounted for analytically, will result in an inflated value of any test. In this large registry, post-stress LVEF ranked below several historical factors including angina, smoking, and diabetes. This further illustrates the importance of a unified analytical approach to assessing a test's added value. The relative contribution of stress MPS variables in stepwise modeling reveal that improvement in risk prediction remains possible, especially when tailored to disease-specific effects of aging or diabetes or symptom stability and frequency, as may be the case with new metabolic or molecular imaging agents (23–30).
We attempted to use strict criteria for classification of death; however, it remains that a proportion of all-cause deaths may be CAD in origin (31). Patients' lost during follow-up had a similar clinical risk profile as compared with the included cohort. The inclusion of other variables, such as transient ischemic dilation and increased lung uptake, may alter our presented results but were not uniformly available. Finally, a longer time period for follow-up may have revealed varying results than those presented herein.
Tremendous growth in cardiac imaging has fostered numerous policy efforts aimed toward containing utilization and devising metrics to define quality CV imaging. The development of quality imaging metrics is now paramount for evaluation of a test's added value in order to justify the referral decision. In the current report, we evaluated both conventional risk stratification methods as well as new innovative approaches that may prove valuable to assess a test's value in clinical decision making. Our results reveal that many patients may be risk reclassified and that this analytical method may prove useful for assessing a test's impact on health outcomes. Although comparative data are not available for other imaging modalities, the development of a league table of net reclassification indexes across modalities may prove useful for evaluation of comparative effectiveness. Moreover, the cost per NRI may prove useful for evaluating referral decisions. Given that not all evidence may be derived from large clinical trials, strategies need to be devised to evaluate “real world” effectiveness. The evaluation of risk reclassification may serve as 1 measure for evaluation of large administrative databases or for registry purposes. (2,11).
Partial grant support, in the form of an unrestricted grant ≥$10,000, for this database was provided by GE Healthcare. The authors report they have no other relationships to disclose. H. William Strauss, MD, served as Guest Editor for this paper.
- Abbreviations and acronyms
- coronary artery disease
- Duke Treadmill Score
- left ventricular ejection fraction
- myocardial infarction
- myocardial perfusion single-photon emission computed tomography
- net reclassification improvement
- receiver-operator characteristic
- Received February 8, 2010.
- Revision received August 31, 2010.
- Accepted September 7, 2010.
- American College of Cardiology Foundation
- Wilson P.W.F.,
- D'Agostino R.B.,
- Levy D.,
- Belanger A.M.,
- Silbershatz H.,
- Kannel W.B.
- Cook N.R.
- Sharir T.,
- Germano G.,
- Kang X.,
- et al.
- Shaw L.J.,
- Hendel R.C.,
- Cerquiera M.,
- et al.
- Shaw L.J.,
- Hendel R.,
- Lauer M.S.,
- et al.
- Holly T.A.,
- Abbott B.G.,
- Al-Mallah M.,
- et al.
- Wilson P.W.F.,
- Pencina M.,
- Jacques P.,
- Selhub J.,
- D'Agostino R. Sr..,
- O'Donnell C.J.
- ↵Decision Memo for Computed Tomographic Angiography (CAG-00385N). Centers for Medicare & Medicaid Services website https://www.cms.hhs.gov/mcd/viewdecisionmemo.asp?from2=viewdecisionmemo.asp&id=206&. Accessed March 12, 2008.
- Matchar D.B.,
- Mark D.B.,
- Patel M.R.,
- et al.
- ↵http://www.acc.org/advocacy/advoc_issues/AllMemberCall2010RuleSlides.pdf. Accessed June 15, 2010.
- ↵PC Pricer. http://www.cms.gov/PCPricer/. Accessed October 11, 2010.
- Mark D.B.,
- Shaw L.J.,
- Lauer M.S.,
- O'Malley P.,
- Heidenreich P.
- Shaw L.J.,
- Berman D.S.,
- Maron D.J.,
- et al.
- Hachamovitch R.,
- Hayes S.W.,
- Friedman J.D.,
- Cohen I.,
- Berman D.S.
- Greenland P.,
- Bonow R.O.,
- Brundage B.H.,
- et al.
- Jaffer F.A.,
- Libby P.,
- Weissleder R.
- Fuster V.,
- Moreno P.R.,
- Fayad Z.A.,
- Corti R.,
- Badimon J.J.
- DiCarli M.F.,
- Hachamovitch R.
- Bax J.J.,
- Beanlands R.S.,
- Klocke F.J.,
- et al.
- Min J.K.,
- Shaw L.J.,
- Devereux R.B.,
- et al.
- Lauer M.S.,
- Blackstone E.H.,
- Young J.B.,
- Topol E.J.