Author + information
- Received April 10, 2014
- Revision received July 9, 2014
- Accepted August 6, 2014
- Published online November 1, 2014.
- Sergey V. Nesterov, MD, PhD, PMP∗,†∗ (, )
- Emmanuel Deshayes, MD, MSc‡,§,
- Roberto Sciagrà, MD‖,
- Leonardo Settimo, MD‖,
- Jerome M. Declerck, PhD¶,
- Xiao-Bo Pan, PhD¶,
- Keiichiro Yoshinaga, MD, PhD#,
- Chietsugu Katoh, MD, PhD#,
- Piotr J. Slomka, PhD∗∗,
- Guido Germano, PhD∗∗,
- Chunlei Han, MD, PhD∗,
- Ville Aalto, MSc∗,
- Adam M. Alessio, PhD††,
- Edward P. Ficaro, PhD‡‡,
- Benjamin C. Lee, PhD§§,
- Stephan G. Nekolla, PhD‖‖,
- Kilem L. Gwet, PhD¶¶,
- Robert A. deKemp, PhD, PEng, PPhys##,
- Ran Klein, PhD##,
- John Dickson, PhD∗∗∗,
- James A. Case, MD, PhD†††,
- Timothy Bateman, MD, PhD†††,
- John O. Prior, MD, PhD‡ and
- Juhani M. Knuuti, MD, PhD∗
- ∗Turku PET Centre, University of Turku and Turku University Hospital, Turku, Finland
- †IM Sechenov Institute of Evolutionary Physiology and Biochemistry RAS, St. Petersburg, Russia
- ‡Lausanne University Hospital, Lausanne, Switzerland
- §Regional Cancer Institute of Montpellier (ICM)—Val d’Aurelle, Montpellier, France
- ‖University of Florence, Florence, Italy
- ¶Siemens Molecular Imaging, Oxford, United Kingdom
- #Hokkaido University Graduate School of Medicine, Sapporo, Japan
- ∗∗Cedars-Sinai Medical Center, Los Angeles, California
- ††University of Washington, Seattle, Washington
- ‡‡University of Michigan Health Systems, Ann Arbor, Michigan
- §§INVIA Medical Imaging Solutions, Ann Arbor, Michigan
- ‖‖Department of Nuclear Medicine, Technical University, Munich, Germany
- ¶¶Advanced Analytics LLC, Gaithersburg, Maryland
- ##National Cardiac PET Center, University of Ottawa Heart Institute, Ottawa, Ontario, Canada
- ∗∗∗University College London, London, United Kingdom
- †††Cardiovascular Imaging Technologies, Kansas City, Missouri
- ↵∗Reprint requests and correspondence:
Dr. Sergey V. Nesterov, University of Turku and Turku University Hospital, Turku PET Centre, Kiinamyllynkatu 4-8, 20520 Turku, Finland.
Objectives The purpose of this study was to compare myocardial blood flow (MBF) and myocardial flow reserve (MFR) estimates from rubidium-82 positron emission tomography (82Rb PET) data using 10 software packages (SPs) based on 8 tracer kinetic models.
Background It is unknown how MBF and MFR values from existing SPs agree for 82Rb PET.
Methods Rest and stress 82Rb PET scans of 48 patients with suspected or known coronary artery disease were analyzed in 10 centers. Each center used 1 of 10 SPs to analyze global and regional MBF using the different kinetic models implemented. Values were considered to agree if they simultaneously had an intraclass correlation coefficient >0.75 and a difference <20% of the median across all programs.
Results The most common model evaluated was the Ottawa Heart Institute 1-tissue compartment model (OHI-1-TCM). MBF values from 7 of 8 SPs implementing this model agreed best. Values from 2 other models (alternative 1-TCM and Axially distributed) also agreed well, with occasional differences. The MBF results from other models (e.g., 2-TCM and retention) were less in agreement with values from OHI-1-TCM.
Conclusions SPs using the most common kinetic model—OHI-1-TCM—provided consistent results in measuring global and regional MBF values, suggesting that they may be used interchangeably to process data acquired with a common imaging protocol.
Measuring myocardial blood flow (MBF) in absolute terms with positron emission tomography (PET) is now possible in clinical routine practice (1). These measurements at rest and under stress can be completed quickly (2,3), and the reconstructed dynamic images can be analyzed in a few minutes by the majority of the available software packages (SPs) (4). The analysis produces left ventricle (LV) absolute MBF values measured in ml/min/g at rest and under stress as well as the myocardial flow reserve (MFR)—the ratio of stress to rest MBF expressed as a unitless number. These values provide unique information regarding diagnosis and monitoring of coronary artery disease (CAD), microvascular health (5), multivessel CAD (6), and risk stratification (7). Although recent studies have shown the diagnostic and prognostic value of MBF quantification over the standard relative image analysis (6,8,9), and use of the generator-produced rubidium-82 (82Rb) (10,11) has brought MBF quantification closer to the clinic, its integration into clinical routine practice remains underutilized (5).
To convert imaging data to quantitative MBF parameters, measured radioactivity concentration values need to be transformed into milliliters of blood per minute per gram of myocardial tissue (ml/min/g) by applying tracer kinetic modeling to dynamic PET images. Thus, any numerical value that any professional receives from 82Rb PET is a result of this transformation. At least 8 different models have been proposed (12–19) for 82Rb. Although deKemp et al. (20) and Tahari et al. (21) had addressed the reproducibility of 82Rb PET analysis methods for MBF quantification, they had focused on a limited number of methods; therefore, a comprehensive comparison study was needed to analyze the current situation in 82Rb PET quantification to help establish common and robust methods to support collaborative multicenter clinical trials.
The objective of the RUBY project was to compare all currently available SPs that can analyze 82Rb PET MBF studies. The criteria for inclusion were the presence of the software in the peer-reviewed literature (16,18,19,22–26) and the willingness of the development team to collaborate according to same ground rules, including blind analysis of the same selected patient datasets. For further details on the 10 compared SPs, please see Table 1 and “The Evaluated Software Packages” section in the Online Appendix; for the side-by-side comparison of the packages, see Table 1 in Saraste et al. (4).
All 82Rb PET studies were performed at the Department of Nuclear Medicine of the University Hospital of Lausanne (Switzerland), according to the routine clinical practice. The study protocol was approved by the local ethics committee. Written informed consent was obtained from each patient prior to the study. Forty-eight patients with suspected or known CAD underwent rest and adenosine-induced stress 82Rb PET. Patients were studied after an overnight fast and were instructed to refrain from caffeine- or theophylline-containing products or medications for 24 h before the 82Rb PET study. During the study, patients were instructed to breathe normally. For further details about the PET image acquisition, please see the Online Appendix.
The reconstructed rest and stress images were delivered to 10 facilities located in 10 centers across 7 countries. Each investigator used 1 SP and, by the rules of this project, had been blinded to results of the image analysis of the other readers before sharing his or her results (see the Online Appendix for details of the study design).
In general, all of the 10 packages implemented variations of a 1-tissue compartment model (TCM) (27). A total of 7 packages implemented by the Ottawa Heart Institute 1-TCM model (OHI-1-TCM) (14). An eighth package also used this model; however, it used a shorter 2.5-min dynamic sequence (8×12s, 2×27s) interpolated from the original image data. Additionally, 1 SP implemented an axially-distributed blood flow model (18)—AD_Ref18, and another used a 2-TCM (12)—2-TCM_Ref12 (Table 1). The image analysis process in all packages consisted of image reorientation, segmentation of both LV myocardium and cavity, and tracer kinetic modeling. Several packages enabled automatic reorientation and segmentation; others depended on the operator to influence segmentation of regions where modeling would be done. Please see “The Evaluated Software Packages” section in the Online Appendix for details of the image analysis process.
Image analysis resulted in estimated values for 3 parameters: rest MBF, stress MBF, and MFR on global and regional levels. Global presented the average LV value, and regional presented values for the 3 vascular territories in the regions of coronary arteries: the left anterior descending, left circumflex, and right coronary artery (RCA). The vascular territories were in agreement with the 17-segment American Heart Association standard model (28).
The large number of models compared prohibited the use of standard approaches to measure agreement between 2 methods (29), so a custom linear mixed model for the repeated measures (30) was applied to the dataset. The statistical model output included 2 main agreement metrics—intraclass correlation coefficient (ICC) and difference between the values from the implemented kinetic models—both calculated pairwise. The pairwise agreement between models was considered sufficient if the difference was <20% of the median across all programs and with the corresponding ICC being ≥0.75. The criteria for ICC was based on Khorsand et al. (31), and the difference was greater than the pre-defined 20% standard. We also expressed the values as a percent of corresponding medians to demonstrate the scale of differences.
The paired Student t test (Microsoft Excel 2013, Redmond, Washington) was used to evaluate the differences between hemodynamic parameters of patients at rest and at pharmacological stress.
To visualize the large number of results of the RUBY-10 comparisons, we developed a custom biplot relating the 2 defined metrics—the differences and the ICC values of compared pairs. In this plot, the x-axis shows pairwise differences between the model values and the y-axis shows corresponding pairwise values of 1 − ICC. In this biplot the origin (x = 0 and y = 0) is the point of identity between the compared values, where there is no difference and the intraclass correlation = 1. Thus, values further from the origin are less in agreement: either showing increasing difference or reduced ICC. The pre-defined criteria of agreement were defined as a rectangular region on the biplot. Thus, this biplot visualizes in an intuitive way our pre-defined criteria of agreement—the pairs inside of these borders were considered to have high pre-defined agreement.
Patient characteristics and hemodynamics
The study population demographic and hemodynamic characteristics are in Table 2. During the pharmacological stress test, heart rate increased (p < 0.001), whereas blood pressure showed a mild decrease (p < 0.05), resulting in a rate pressure product net increase (p < 0.01). All 48 patients—including the 1 with 70/30 mm Hg stress blood pressure—tolerated the stress test well.
Absolute values of MBF at rest and during adenosine stress and MFR
Average MBF and MFR values (Table 3) showed marked variation between models. Differences (p < 0.0001) between highest and lowest values for any studied parameter were always greater than a factor of 1.5 times. For rest MBF, the ratios between extreme values were 1.7 globally and ∼1.8 regionally; for stress MBF, the ratios ranged from 1.9 globally to ∼2.2 regionally; and for MFR, the ratios were 1.5 globally and ranged from 1.9 to 2.3 regionally.
Agreement of global LV MBF measurements
The biplots (Figure 1) demonstrated several consistent patterns. The first pattern was that OHI-1-TCM implementations (green elements) in 8 SPs tended to concentrate close to the origin (14). The second pattern was that 1-TCM_Ref19 (purple elements) provided results that differed greatly from other models on all studied levels for both MBF and MFR (19). The third pattern was that 1-TCM_Ref17 (red elements) provided MBF values much higher than the others, both at rest and stress (17). Note also that RT_Ref13 (yellow elements) was within the pre-defined difference limits globally at rest (up to 19.8% of the median), but showed higher values for stress (up to 35.0% of the median) and for MFR (up to 24.5%) (13).
Agreement of regional LV MBF measurements
Regional values generally showed larger differences: up to 41.5% of the median for RCA. Also, over one-half (60%) of ICC values did not fulfill the pre-defined criteria for agreement. RT_Ref16 (pink elements) was within the pre-defined limits globally for MBF and MFR values and also regionally in the left anterior descending and left circumflex arteries, but had somewhat larger differences in RCA (up to 28.5%), and almost all (97%) of the ICC values did not fulfill the criteria of agreement (16).
2-TCM_Ref12 (brown elements) exhibited a pattern similar to RT_Ref16: all of the global differences were below the pre-defined limit, as well as the regional differences except for the RCA values, which were up to 48.3% of the median, yet again almost all the ICC values (97%) did not fulfill the criteria of agreement (12).
Differences using 1-TCM_Ref15 (light blue elements) were within the pre-defined limits globally and regionally, with the exception of MFR in the RCA where the difference was 30.0% of the median (15). ICC values in 38% of comparisons were below pre-defined limits; however, discarding 2-TCM and both the retention models, ICC values fulfilled the agreement criteria in 80% of remaining comparisons.
Differences between the axially-distributed model (AD_Ref18), and the other models were generally within the pre-defined limits, yet occasionally were above: 23.5% of the median at rest and 22.5% at stress on the global level (18). Almost all (95%) of the ICC values were >0.75.
Agreement of LV MBF measurements for OHI-1-TCM
Because the OHI-1-TCM was the most commonly applied model in the evaluated SPs, specific biplots for comparisons between its implementations in 8 SPs were created and are displayed in Figure 2; red elements demonstrate two implementations of the model that were added later to the RUBY project. Globally, all of the stress differences were well within the pre-defined limits of agreement, <20% of the median value, and the majority of rest differences were also within this limit. Similar patterns were observed regionally: the majority of stress MBF values were well within the pre-defined limits. However, in general, regional differences seemed to be larger in the RCA region. Values of the largest differences between implementations of OHI-1-TCM are shown in Table 4.
RUBY-10 is the first and currently the only study aimed at comparing all existing software tools—used both in clinical cardiology and in the research setting—for analyzing MBF and MFR with the most widely used cardiac PET tracer: 82Rb.
The positive finding of our study is that OHI-1-TCM, the model described by Lortie et al. (14)—commonly found in most PET analysis programs—provided results generally close enough to be used interchangeably, if dynamic time binning protocols are the same. We must emphasize that without an absolute reference standard—such as microsphere data—we cannot infer the diagnostic or quantitative accuracy of any of the methods considered. Despite this, our results do demonstrate that applying the same kinetic model to the same 82Rb PET data, the received MBF and MFR values are independent of the SP within the specified agreement tolerances.
The negative finding is that different kinetic models currently used in 82Rb PET produce different values for the same PET data. The finding is not new: in 2005, Khorsand et al. (32) found differences comparing 1-TCM with 2-TCM for 13N-ammonia PET. New is the magnitude of possible differences: in the referred study; global differences were up to 13% for MBF and up to 26% for MFR, and our results demonstrate that for 82Rb PET global differences can be up to 90% for MBF and 50% for MFR. Regional differences can be up to 130% for both MBF and MFR.
The causes of differences can vary. In some cases, smoothing of the data can result in higher MBF (33) for factor-analysis-based methods such as Sitek et al. (17) and El Fakhri et al. (15), and minimal filtering is recommended for improved MBF estimates. In others, the difference in prompt-gamma corrections for 82Rb between the PET computed tomography scanner used to perform the current study and the PET studies used originally to validate the models could be the cause of the difference (34). Notwithstanding the causes, the practical implication is clear: values of MBF or MFR presented without reference to the kinetic model cannot be directly compared, neither for pooling of patient data, nor for following up the same patients.
Two metrics, derived from our statistical model, were used to indicate the agreement—ICC and differences between the compared MBF and MFR values. The benefit of using ICC was clear: it avoids the limitation of standard correlation coefficients—often met in comparison studies—when a linear relationship is mistaken for agreement. However, like other correlation coefficients, ICC depends on the range of variables measured, and this can explain its lower value for rest MBF and MFR compared with stress. The choice of limits of agreement is critical, and for ICC we used recommended (31) values—a cutoff for excellent agreement at over 0.75. For the differences, the choice of appropriate limit is not that straightforward, and we chose to use <20% difference in studied parameters as acceptable, as it is similar to the test-retest repeatability of 20% to 25% for rest MBF and MFR reported recently using 82Rb PET (35).
Increasing the number of compared models geometrically increases the results, which makes the analysis and display of these results challenging. For the measured global and regional values, there were 2,520 differences (210 × [3 + 9]) and 1,260 ICC values; listing all of these values is impractical. The biplot binds these values, and with pre-defined cutoffs informs on the relative agreement of the model results. Therefore, the developed biplots were enabled to handle the complexity of the data inherent in a cross-comparison of this scale.
The analysis of a dynamic PET scan goes through several steps—reorientation, myocardial segmentation, selection of the input function, kinetic modeling, and polar plot generation—each of which could significantly affect the results. We designed our study to simulate the clinical routine practice as much as possible and treated the workflow inside of each SP as a “black box” being only interested in input (the patient PET images) and the output (the results in milliliters [MBF] or ratio units of MFR). As all of the studied SPs were operated either by their developers or under their close supervision, we believe that the tools were used appropriately.
The most significant limitation of this study is that there was no gold standard used, and thus, no claim of quantitative accuracy of a particular model can be inferred by these results. Another consideration is that one of the 1-TCM programs used interpolated dynamic image frames to produce a dataset compatible with this implementation of OHI-1-TCM. The shortened dynamic sequence may tend to exaggerate any differences from later uptake and washout frames that were used by the other OHI-1-TCM implementations. Last, 2 of 8 OHI-1-TCM programs were added after receiving preliminary (study average) results of RUBY. These decisions were made for the sake of comprehensiveness, because it would have been practically impossible to repeat the study de novo, so we chose to include these analyses in the primary results. However, these analyses were still performed blinded to the individual results of the other software programs.
We do not consider a limitation the fact that we used only 82Rb data coming from 1 center, acquired on 1 scanner, reconstructed with 1 algorithm, and so on, because introducing these new variables into our combinatorial study would have led to a practical impossibility to carry out the project.
MBF and MFR values obtained by 82Rb PET must be interpreted together with information on their computational origin. The most important part of such information may not be the software program used to obtain these values, but rather the mathematical tracer kinetic model implemented within the software. The most widely implemented model for 82Rb PET is the OHI-1-TCM (14) available in 8 software tools out of the studied 10. When different implementations of this kinetic model are used to analyze the same data, the results appear to be independent of the particular SP utilized. The quantitative blood flow results agree well between these analysis programs and may be used interchangeably for the benefit of large multicenter trials.
The authors thank Vesa Oikonen (Turku, Finland) for his everyday advice on kinetic models; Dr. Kim Holmberg (Turku, Finland) for his effort to develop the network analysis method for RUBY-10; Dr. Cyril Burger (Zürich, Switzerland) for his instruction in using PMOD; and Drs. Shana Elman and James Caldwell at the University of Washington (Seattle, Washington) for their expertise in analyzing the data.
For a supplemental methods section, please see the online version of this article.
This study was conducted within the Finnish Centre of Excellence in Cardiovascular and Metabolic Diseases supported by the Academy of Finland, University of Turku, Turku University Hospital, and Åbo Akademi University; and was supported in part by grants from the Japanese Ministry of Education, Science and Culture (No. 1959135), Northern Advancement Center for Science & Technology (H23-S2-17), and the U.S. National Institutes of Health grant K25-HL086713. Cedars-Sinai receives royalties from the licensing of QPET software, a minority of which is shared with developers, including Drs. Slomka and Germano. Dr. Slomka has received grant support from Siemens Healthcare. Dr. Alessio has received a research grant from GE Healthcare; and has served as a consultant for Lantheus Medical Imaging. Dr. Ficaro has received revenue shares from the sale of Corridor4DM and is the owner of INVIA Medical Imaging Solutions. Dr. Lee has received financial support from and is an employee of INVIA Medical Imaging Solutions. Drs. deKemp and Klein have received revenue shares from the sale of FlowQuant; and have served as consultants to and received revenue shares from Jubilant-DraxImage. Dr. deKemp has received royalties from technologies licensed to Jubilant DraxImage and INVIA Medical Imaging Solutions. Drs. Case and Bateman are owners of Cardiovascular Imaging Technologies, which licenses and sells ImagenQ. Dr. Bateman has served on the advisory board of Lantheus Medical Imaging, GE, and FluoroPharma. Dr. Knuuti has served as a consultant to Lantheus Medical Imaging. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose. Drs. Nesterov and Deshayes contributed equally to this work as first authors. Drs. Prior and Knuuti contributed equally to this work as senior authors.
- Abbreviations and Acronyms
- coronary artery disease
- intraclass correlation coefficient
- left ventricle
- myocardial blood flow
- myocardial flow reserve
- right coronary artery
- software package
- tissue compartment model
- Received April 10, 2014.
- Revision received July 9, 2014.
- Accepted August 6, 2014.
- American College of Cardiology Foundation
- Gould K.L.
- Schindler T.H.,
- Schelbert H.R.,
- Quercioli A.,
- Dilsizian V.
- Dorbala S.,
- Di Carli M.F.,
- Beanlands R.S.,
- et al.
- Farhad H.,
- Dunet V.,
- Bachelard K.,
- Allenbach G.,
- Kaufmann P.A.,
- Prior J.O.
- Yoshinaga K.,
- Chow B.J.,
- Williams K.,
- et al.
- Herrero P.,
- Markham J.,
- Shelton M.E.,
- Bergmann S.R.
- Yoshida K.,
- Mullani N.,
- Gould K.L.
- El Fakhri G.,
- Kardan A.,
- Sitek A.,
- et al.
- Dekemp R.A.,
- Declerck J.,
- Klein R.,
- et al.
- Saha K. Automated quantification of rubidium-82 myocardial perfusion images using wavelet based approach [PhD dissertation]. University of Missouri-Columbia, 2007; Columbia, MO.
- Slomka P.J.,
- Alexanderson E.,
- Jácome R.,
- et al.
- Coxson P.G.,
- Huesman R.H.,
- Borland L.
- Cerqueira M.D.,
- Weissman N.J.,
- Dilsizian V.,
- et al.
- Davis C.S.
- Rosner B.
- Lee B.C.,
- Moody J.B.,
- Sitek A.
- Renaud J.M.,
- Mylonas I.,
- McArdle B.,
- et al.