Author + information
- Received December 24, 2013
- Revision received April 29, 2014
- Accepted April 30, 2014
- Published online September 1, 2014.
- R. Sacha Bhatia, MD, MBA∗,
- David M. Dudzinski, MD, JD†,
- Rajeev Malhotra, MD†,
- Creagh E. Milford, DO†,
- Danita M. Yoerger Sanborn, MD†,
- Michael H. Picard, MD† and
- Rory B. Weiner, MD†∗ ()
- ∗Women’s College Hospital, Institute for Health System Solutions and Virtual Care, Toronto, Ontario, Canada
- †Cardiology Division, Department of Medicine, Massachusetts General Hospital, Boston, Massachusetts
- ↵∗Reprint requests and correspondence:
Dr. Rory B. Weiner, Cardiology Division, Massachusetts General Hospital, Yawkey Suite 5B, 55 Fruit Street, Boston, Massachusetts 02114.
Objectives This study sought to prospectively study the impact of an appropriate use criteria (AUC)-based educational intervention on outpatient transthoracic echocardiography (TTE) ordering by physicians-in-training.
Background AUC were developed in response to concerns about inappropriate utilization. It is unknown whether an educational intervention can reduce inappropriate outpatient TTE.
Methods We conducted a randomized control trial in which physicians-in-training were randomized to an AUC-based educational intervention or a control group at an academic medical center in Boston, Massachusetts. The primary endpoints were the rates of inappropriate and appropriate TTE.
Results For the cardiology physicians-in-training, the proportion of inappropriate TTE was significantly lower in the intervention than in the control group (13% vs. 34%, p < 0.001). As a corollary, the proportion of appropriate TTE ordered by the intervention group was significantly higher than that of the control group (81% vs. 58%, p < 0.001). The odds of ordering an appropriate TTE in the cardiology intervention group was 2.7 (95% confidence interval [CI]: 1.5 to 5.1, p = 0.002) relative to the control group. The internal medicine physicians-in-training ordered a small number of TTE overall, and there was a trend toward significant odds of ordering an appropriate TTE in the intervention group relative to the control group (odds ratio [OR]: 8.1, 95% CI: 0.95 to 69.0, p = 0.055). Six clinical scenarios accounted for 75% of all inappropriate TTE, with the 3 most common inappropriate indications being routine surveillance (<1 year) of known cardiomyopathy without a change in clinical status, routine surveillance of known small pericardial effusion, and routine surveillance of ventricular function with known coronary artery disease and no change in clinical status.
Conclusions In cardiology fellows with a high rate of ordering inappropriate TTE, an AUC-based educational and feedback intervention reduced the proportion of inappropriate outpatient TTE and increased the proportion of appropriate outpatient TTE. (Educational Intervention to Reduce Outpatient Inappropriate Transthoracic Echocardiograms; NCT01944202)
Transthoracic echocardiography (TTE) is an important tool in the diagnosis and management of cardiovascular disease. However, there has been concern in recent years regarding the rapid growth of TTE utilization, which has been estimated at 6% to 8% per year (1,2). In response, the American College of Cardiology Foundation, along with other subspecialty societies, developed appropriate use criteria (AUC) for TTE in 2007, and updated AUC were published in March 2011 (3,4).
AUC has become an important quality improvement tool, both for the purpose of echocardiography laboratory accreditation, and as a focus for professional societies and healthcare payers in an effort to reduce healthcare delivery costs (5–8). Previous attempts using AUC to reduce inappropriate utilization of diagnostic testing have shown varying degrees of success (9–12), although none were in the form of a randomized control trial. We previously reported a time-series analysis in which an AUC-based education and feedback intervention successfully reduced inappropriate TTE on an inpatient academic medical service, where the baseline inappropriate rate was about 13% (9).
It is documented that the proportion of inappropriate TTE is highest in the ambulatory environment, where it has been reported as high as 30% (13–15). Therefore, the potential to improve TTE utilization may be greatest in the outpatient setting. However, no study to date has evaluated whether an AUC-based educational intervention can reduce inappropriate TTE in this setting. We therefore designed and conducted the first randomized control trial of an AUC-based educational and feedback intervention aimed at reducing inappropriate outpatient TTE ordered by physicians-in-training.
This study was conducted in ambulatory practice environments at Massachusetts General Hospital, a quaternary care academic medical center in Boston, Massachusetts. The study participants were physicians-in-training. The first environment was the ambulatory cardiology fellows’ clinic, a longitudinal practice where fellows evaluate patients, supervised by a rotating staff preceptor. In this setting, cardiology fellows see new consults referred from primary care and other specialties, follow up with patients discharged from the hospital, and provide longitudinal care for patients over the course of their fellowship (minimum of 36 months). Patients typically have a wide variety of cardiovascular conditions with a high degree of complexity. The fellows determine management plans for their patients. The plan of care is discussed with the preceptor; however, the fellows practice with a high degree of autonomy. The preceptor is a member of the cardiology division attending staff and is most engaged in decision making for invasive procedures (i.e., cardiac catheterization). During the study period, the same attending preceptor may have supervised fellows in both the intervention and control groups. Cardiology fellows are typically in clinic for one-half day per week. A TTE is not “pre-ordered” on the day of an initial visit unless the physician who will be seeing the patient has reviewed the medical records and agreed with the need for TTE. Referral for pre-operative evaluation for noncardiac surgery is not a frequent reason cardiology fellows see outpatient consults, as these patients tend to be triaged to attending physicians at our institution.
The second practice environment consisted of 2 ambulatory internal medicine practices in which internal medicine residents evaluate patients in a hospital-based, outpatient primary care environment. The residents maintain their longitudinal clinic throughout residency training and are typically in clinic one-half day per week. Residents are supervised by a single preceptor, although each resident practices with significant autonomy.
The study protocol was reviewed and approved by the Partners Healthcare Institutional Review Board. All study physicians provided verbal consent prior to study initiation. Fellows and residents from all years of training participated in this study.
We conducted a randomized, controlled trial of an AUC-based educational and feedback intervention designed to reduce inappropriate TTE in the previously defined cardiology and internal medicine outpatient practices. A random number generator divided 24 cardiology fellows and 88 internal medicine residents into control and intervention arms. The study period was from August 1, 2012, to April 30, 2013.
During the study, physicians in the intervention arm received a multifaceted educational intervention on TTE appropriateness: 1) a lecture at the beginning of the study period, which described the AUC for echocardiography and highlighted common clinical scenarios for which outpatient TTE are ordered; 2) an electronic “pocket card” via e-mail that provided tips on appropriate ordering of TTE (Figure 1); and 3) an individualized monthly feedback report via e-mail. The feedback reports contained the number of TTE ordered during the month and how many were classified as appropriate, inappropriate, or uncertain based on the 2011 AUC. A description of all inappropriate TTE and the rationale for the inappropriate classification was provided. Although inappropriate TTE are now more accurately described as “rarely appropriate,” this study began prior to the release of the new terminology and the term inappropriate was used. The physicians in the control arm had their TTE orders tracked and classified, but they did not receive any feedback. Although study participants were not blinded to which arm of the study they were in, they were blinded to which arm of the study their colleagues were in. Study participants were instructed to avoid discussing the study with their peers in an effort to preserve blinding, although this could not be monitored in a systematic fashion. Clinic preceptors were also blinded to which study participants were in the control or intervention arms.
All study participants received a pre-study and post-study knowledge assessment survey.
TTE ordering information was determined from a review of the electronic medical record (EMR). Signs and symptoms and the reason(s) for the TTE were abstracted from both the EMR and the echocardiogram order (13,16,17). Outpatient TTE are ordered on a paper-based system, and there is no decision support tool. The EMR is a comprehensive database capturing the entirety of clinical activity at the hospital. The EMR also contains TTE reports from all hospitals within the healthcare network, and typically outside reports are scanned in. Patient demographics and comorbidities were determined through EMR review and classified according to the International Classification of Diseases, Ninth Revision codes.
Transthoracic echocardiogram classification
Two study investigators (R.S.B., R.B.W.) independently reviewed all TTE ordering information and classified each TTE as appropriate, inappropriate, or uncertain according to the 2011 AUC for TTE (4). If the reason for a TTE did not have a corresponding indication in the 2011 AUC, it was considered unclassifiable. If initial agreement on TTE appropriateness designation was not achieved, the 2 study investigators classifying TTE reviewed the EMR together in order to achieve consensus; a third study investigator (M.H.P.) was available to review cases if consensus could not be achieved. Agreement was defined as achieving the same appropriateness designation (i.e., appropriate, inappropriate, or uncertain). Investigators were blinded to whether the TTE was ordered by a physician in the intervention or control group. This was necessary to help avoid potential inherent biases in TTE classification. Outside TTE reports were taken into consideration when classifying study TTE if the outside reports were complete and contained sufficient information to answer the clinical question. If the physician’s note indicated that the information from the outside study (either the report or images) was adequate to answer the clinical question, then a TTE performed at our institution for the same clinical indication was classified as a repeat TTE.
The primary outcome measures in this study were the rates of inappropriate and appropriate TTE. Secondary outcomes included the number of TTE ordered, common appropriate and inappropriate TTE indications, and pre- and post-study knowledge assessment scores.
Using previous retrospective data from our institution, the rate of inappropriate TTE in the ambulatory cardiology practice is 30%. Two randomized groups of 10 cardiology fellows were required for 80% power to determine statistical significance with a reduction in the inappropriate TTE ordering rate to 15%, using alpha = 0.05. Similarly, for internal medicine, using a historically inappropriate TTE rate of 21%, 2 groups of 14 physicians were required to see a reduction of the TTE inappropriateness rate to 10%, which would be statistically significant with 80% power, using alpha = 0.05.
Data were tested for normality, and nonparametric tests were used for non-normal data. Categorical variables for ordering characteristics, patient demographics, and appropriateness ratings were compared using chi-square or Fisher exact test, as required. Continuous variables are reported as mean ± SD or median (interquartile range [IQR]) as appropriate and were compared using analysis of variance or Kruskal-Wallis test. Because each provider ordered multiple TTE, we performed conditional logistic regression stratified by the individual physician to account for any associations within providers. In this logistic regression analysis, ordering an appropriate TTE was defined as the outcome (dependent) variable and the intervention represented the independent variable. Statistical significance was indicated by a 2-tailed p value <0.05.
Study participants (physicians-in-training)
In total, 88 internal medicine residents and 24 cardiology fellows were randomized. During the 9-month study period, 292 TTE (32 TTE/month) were ordered by the study physicians. Of patients who had a TTE during the study period, 27% also had a previous TTE (inpatient or outpatient) in the previous year. During the study period, the majority of TTE were ordered by cardiology fellows (10.6 ± 5.8 TTE/physician in the intervention vs. 7.6 ± 4.2 TTE/physician in the control group, p = 0.16). Internal medicine residents ordered few TTE (median: 1 [0, 1] TTE/physician in the intervention vs. 0 [0, 1] TTE/physician in the control group, p = 0.51). Because cardiology fellows accounted for the majority of TTE, the primary reported results are for this group. Data for the internal medicine physicians are in Online Tables 1 to 3.
Prior to study initiation, 18 of 24 (75%) cardiology fellows completed the knowledge survey (Table 1). There was no difference in the correct response rate to the case-based questions among the intervention and control arms (78% vs. 76%, p = 0.45). There was no difference between the pre-study attitudes toward AUC or consideration of costs when ordering diagnostic tests. In the post-study case-based assessment, there was no difference between the scores achieved by cardiology fellows (85% vs. 80%, p = 0.44). The mean year of cardiology fellowship was similar for the intervention and control groups (2.3 vs. 2.7, p = 0.52).
From August 1, 2012, to April 30, 2013, 613 patients were seen by cardiology fellows in the intervention group versus 600 patients in the control group. This represents 51.1 ± 22.7 patients/physician in the intervention group compared with 50 ± 17.8 patients/physician in the control group (p = 0.64). The clinical characteristics of the patients seen by cardiology fellows are in Table 2. Patients in the intervention group had higher rates of previous myocardial infarction (29% vs. 22%, p = 0.01) and hospitalization in the past year (63% vs. 53%, p < 0.001). All other clinical characteristics were similar.
Number of TTE and appropriateness of TTE
Table 3 details the TTE ordered by cardiology fellows. The control group ordered a total of 91 TTE and the intervention group ordered 127 TTE. The mean number of TTE ordered per physician was not significantly different between the intervention and control groups.
Nearly all TTE ordered were classifiable by the 2011 AUC. Agreement in TTE appropriateness designation between the 2 study investigators was 94.9%, and all initially discrepant TTE classifications were resolved by consensus discussion between the 2 study investigators, without the need for involvement of a third investigator. For the cardiology fellows, the proportion of inappropriate TTE was significantly lower in the intervention than in the control group (13% vs. 34%, p < 0.001). As a corollary, the proportion of appropriate TTE ordered by the intervention group was significantly higher than that of the control group (81% vs. 58%, p < 0.001). The odds of ordering an appropriate TTE in the cardiology intervention group was 2.7 (95% confidence interval [CI]: 1.5 to 5.1, p = 0.002) relative to the control group. The raw numbers and percentages of appropriate, inappropriate, and uncertain TTE are in Table 3. The total number of TTE, number of inappropriate TTE, rate of inappropriate TTE, and rate of appropriate TTE ordered per month in the cardiology intervention and control groups are displayed in Figures 2A to 2D, respectively. Analysis of individual TTE ordering behavior showed that ≥1 inappropriate TTE was ordered by the majority of cardiology fellows. In the control group, 11 of 12 fellows ordered ≥1 inappropriate TTE (6 fellows each ordered 4, 1 ordered 3, and 4 ordered 1 inappropriate TTE). In the intervention group, 8 of 12 fellows ordered ≥1 inappropriate TTE (1 ordered 4, 1 ordered 3, 3 ordered 2, and 4 ordered 1 inappropriate TTE). Table 4 details common appropriate, inappropriate, and uncertain indications for all TTE ordered in this study.
This study reports the results of the first prospective, randomized trial of an AUC-based educational intervention aimed at ordering of TTE in the outpatient setting. The education and feedback intervention reduced ordering of inappropriate TTE and increased ordering of appropriate TTE in the intervention versus control group of cardiology fellows at an academic medical center.
Growth in the use of cardiovascular testing has led to efforts by the American College of Cardiology Foundation to create AUC. Studies investigating AUC-guided efforts to improve utilization of diagnostic testing are relatively sparse and have yielded mixed results (9–12). We previously reported a prospective, time-series analysis of an AUC-based educational intervention aimed at medical residents on the inpatient medical service. This produced a 62% reduction in inappropriate TTE (9). However, it has been documented that the inappropriate TTE rate is higher in the outpatient setting, and, therefore, methods to improve appropriate TTE ordering in this environment are needed (14,15). Additionally, until the present study, to our knowledge, there has never been a randomized control trial of an AUC-based educational intervention. Our study focused on cardiology fellows at an academic center, with a relatively high rate of inappropriate TTE observed in the control group.
It is important to note that only 6 clinical indications accounted for 75% of all inappropriate TTE in our study. These clinical scenarios centered on common situations including follow-up assessment (i.e., surveillance) of left ventricular function, pericardial effusion, and valvular heart disease. The higher reported rate of inappropriate TTE in the outpatient setting (13) is frequently due to such “surveillance” studies. Determination of common reasons for inappropriate tests in a local practice environment is likely an important prerequisite to the effective design of a successful intervention. The educational lecture and supportive material (Figure 1) developed for this study was aimed at the highest yield areas (the most common inappropriate indications at our institution). It is also important to recognize that the baseline inappropriate TTE rate in our study is higher than that reported in other studies (15). In the study by Mansour et al. (15), which also used the 2011 AUC to classify TTE, the rate of inappropriate outpatient TTE was 16%. The reasons for the differences are speculative, but may involve institutional practice patterns for ordering repeat surveillance studies. At centers or practice environments with a lower baseline TTE inappropriate rate, it may be more difficult to show benefit with the type of intervention used in our study. Furthermore, in our study, there was month-to-month variation in total TTE ordered and inappropriate TTE rates. For example, the fewest number of TTE were ordered in February 2013 (Figure 2). This may in part be explained by February being the shortest month of the year and fellows having fewer of their weekly outpatient clinics. Additionally, there were no inappropriate TTE in February 2013, and in September 2012, there was actually a higher proportion of inappropriate TTE in the intervention. Such month-to-month variation may not be completely unexpected in a real-world clinical practice and supports the need for relatively long periods of study.
Previous literature on interventions to reduce inappropriate testing has produced mixed results. Recent studies of educational interventions to reduce inappropriate single-photon emission computed tomography and stress echocardiograms were negative (11,12). The intervention in the single-photon emission computed tomography study consisted of a presentation at medical grand rounds, publication in the staff newsletter, meetings with administrators, and presentations to ordering physicians, whereas in the stress echocardiogram study, the intervention was a grand rounds lecture and distribution of a list of common inappropriate indications. Neither of these interventions provided direct, personal feedback to providers (11,12). In contrast, AUC-based and other studies that have incorporated individualized provider feedback have been shown to be effective in improving ordering of testing (9,18–19). It appears that feedback is an important component of an intervention designed to alter physician behavior.
Unlike our previous inpatient AUC-based intervention (9), the current study was performed in a randomized controlled fashion. This design overcomes many of the limitations of that previous AUC-based educational intervention study. Also, the ambulatory study setting required modification of the delivery of the intervention to take into account the several practice locations, variability in schedules, and extended period of study. Also, efforts were made to minimize the potential for cross-talk and information sharing between study groups. This was addressed by making our predominant mode of communication with physician study participants electronic. As an example, the “pocket card” was not printed and rather was e-mailed to all physicians in the intervention group as a “virtual pocket card.” Intervention group subjects were instructed not to discuss the pocket card or other aspects of the study.
Similar to our previous inpatient study, feedback was provided via e-mail, although at a less frequent interval (i.e., monthly) in this study. This was mainly due to the lower volume of ordering of TTE in the outpatient environment. We did nonetheless show an improvement in appropriate ordering, indicating that it is possible to positively affect behavior without making large intrusions on physicians’ time. This has important implications in terms of the ability of others to reproduce this type of intervention. Furthermore, there is potential for components of this educational intervention to be automated, including the integration of the virtual pocket card into an electronic decision support tool at the time of order entry. Additionally, classification of TTE and e-mail feedback to providers could be automated (20), which would decrease the resources and time required to deliver this type of intervention.
It is noted that the agreement in classification of TTE by the 2 study investigators in our study was high. This may be due to several factors. First, agreement in classification was defined as achieving the same appropriateness designation (i.e., appropriate, inappropriate, or uncertain), as opposed to requiring the same exact AUC indication. Second, the EMR at our institution is a robust system that contains all physicians’ notes. A typical outpatient visit note produced by a physician-in-training is detailed and describes the thought process for ordering tests. In fact, in the cardiology intervention group, we encountered situations where the physicians mentioned the specific AUC indications that they were following. Whether attending staff-level physicians with more established practice patterns would respond to an AUC-based intervention in this manner requires study. Additionally, study investigators were blinded to whether a TTE was ordered by a physician in the intervention or control group. This was necessary to avoid potential bias, although it is recognized that investigators were aware of the study hypothesis, and this represents a limitation of this type of analysis.
First, the study was aimed at physicians-in-training, and the impact of performing this type of intervention on attending physicians is unknown and represents an area of ongoing investigation. Second, this study was performed at an academic medical center, and therefore our findings may not be generalizable to other environments. Parallel studies in community practices are warranted. Third, the study was underpowered to determine differences in TTE ordering volumes, primarily due to the low volume of TTE ordered per physician. However, although the results are not statistically significant, it is noteworthy that there was a trend toward higher TTE ordering in the cardiology intervention group (Table 3). This can raise the theoretical concern that cardiology intervention group physicians may have ordered “extra” TTE that they knew would be considered appropriate, even if studies were unlikely to affect patient care. However, it is important to note that the cardiology intervention group evaluated patients with higher rates of previous myocardial infarction and recent hospital admission, suggesting that higher acuity patients may have influenced TTE ordering volume. The number of TTE ordered by internal medicine residents was low, limiting the analysis of this subgroup and indicating that this population is not a large contributor to inappropriate outpatient TTE. Fourth, the impact of this type of intervention on longer-term ordering practices is not well known. In a recent analysis of the sustainability of the educational intervention we previously employed on the inpatient medical service, after completion of the intervention the rate of inappropriate TTE increased and regressed back to pre-intervention levels (21). For the current study, due to graduation/matriculation of physicians-in-training shortly after completion of the study, analysis of the “post-intervention period” was limited. However, for the 2-month period following completion of the intervention and prior to the academic year transition, we classified TTE ordered by the cardiology fellows. Although there was a nonsignificant trend toward a lower inappropriate TTE rate in the intervention group, overall we found no significant differences in the rates of appropriate and inappropriate TTE ordered by the 2 groups. This supports the need for continued education/feedback to produce sustained benefits in physician ordering behavior. Pre-specifying such a post-intervention analysis should be a part of future study designs of physicians-in-training. Fortunately, as investigations move toward attending level physicians, transition points in the academic year should not play as large a role. Additionally, the potential impact of preceptors on decision making of study physicians will not be an issue in studies of attending physicians as their own decision making will be the focus. Fifth, the impact of this type of intervention on patient outcomes and healthcare costs is needed. Sixth, it is possible that physicians in the intervention group documented their rationale for ordering TTE more clearly. For instance, they may have provided information/language in the visit note that linked a TTE to an appropriate indication, regardless of the patient’s actual signs/symptoms. This would indicate a change in documentation, as opposed to actual ordering behavior, and to what extent this phenomenon was present cannot be determined. Finally, we acknowledge that other strategies to reduce inappropriate ordering of TTE may be complimentary and have value.
In cardiology fellows with a high rate of ordering inappropriate TTE, an AUC-based educational and feedback intervention reduced the proportion of inappropriate outpatient TTE and increased the proportion of appropriate outpatient TTE. The intervention had no significant impact on internal medicine residents with a lower rate of inappropriate TTE. This study may provide a template for others interested in investigating methods to educate physicians to improve ordering of diagnostic testing. A larger scale, multicenter trial of this type of intervention directed at attending staff level physicians and/or physician extenders is warranted to determine whether this approach will be successful in other practice environments.
For supplemental tables, please see the online version of this paper.
This study was supported by a Partners Healthcare Center of Expertise in Patient Care Quality and Safety Research Grant, however, they had no role in the design/conduct of the study; data collection/analysis; or preparation, review, or approval of the manuscript. All authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- appropriate use criteria
- confidence interval
- electronic medical record
- interquartile range
- transthoracic echocardiography
- Received December 24, 2013.
- Revision received April 29, 2014.
- Accepted April 30, 2014.
- Blecker S.,
- Bhatia R.S.,
- You J.J.,
- et al.
- Pearlman A.S.,
- Ryan T.,
- Picard M.H.,
- Douglas P.S.
- Douglas P.S.,
- Khandheria B.,
- Stainback R.F.,
- et al.
- Douglas P.S.,
- Garcia M.J.,
- Haines D.E.,
- et al.
- ↵Intersocietal Accreditation Commission Echocardiography Standards. Available at: http://www.intersocietal.org/echo/main/echo_standards.htm. Accessed June 20, 2013.
- American Board of Internal Medicine (ABIM) Foundation. Choosing Wisely. Available at: http://www.choosingwisely.org/doctor-patient-lists/american-society-of-echocardiography/. Accessed June 20, 2013.
- Shaw L.J.,
- Min J.K.,
- Hachamovitch R.,
- et al.
- Bhatia R.S.,
- Milford C.E.,
- Picard M.H.,
- Weiner R.B.
- Chinnaiyan K.M.,
- Peyser P.,
- Goraya T.,
- et al.
- Gibbons R.J.,
- Askew J.W.,
- Hodge D.,
- Kaping B.,
- Carryer D.J.,
- Miller T.
- Willens H.J.,
- Nelson K.,
- Hendel R.C.
- Bhave N.M.,
- Mansour I.N.,
- Veronesi F.,
- Razi R.R.,
- Lang R.M.,
- Ward R.P.
- Bhatia R.S.,
- Dudzinski D.M.,
- Milford C.E.,
- Picard M.H.,
- Weiner R.B.