Author + information
- Received December 10, 2012
- Accepted January 23, 2013
- Published online May 1, 2013.
- R. Sacha Bhatia, MD, MBA,
- Creagh E. Milford, DO,
- Michael H. Picard, MD and
- Rory B. Weiner, MD⁎ ()
- ↵⁎Reprint requests and correspondence:
Dr. Rory B. Weiner, Division of Cardiology, Massachusetts General Hospital, Yawkey Suite 5B, 55 Fruit Street, Boston, Massachusetts 02114
Objectives This study sought to prospectively study the impact of an appropriate use criteria (AUC)-based educational intervention on transthoracic echocardiography (TTE) ordering among house staff on the inpatient general internal medicine service at an academic medical center.
Background AUC for TTE were developed in response to concerns about inappropriate use of TTE. To date, educational interventions based on the AUC to reduce inappropriate use of TTE have not been prospectively studied.
Methods A prospective, time series analysis of an educational intervention was conducted and then compared with TTE ordering on the same medical service during a control period. The intervention consisted of: 1) a lecture to house staff on the 2011 AUC for TTE; 2) a pocket card that applied the AUC to common clinical scenarios; and 3) biweekly e-mail feedback regarding ordering behavior. TTE ordering was tracked over the intervention period on a daily basis and feedback reports were e-mailed at 2-week intervals. The primary outcome was the proportion of inappropriate and appropriate TTE ordered during the intervention period.
Results Of all TTEs ordered in the control and study periods, 99% and 98%, respectively, were classifiable using the 2011 AUC. During the study period, there was a 26% reduction in the number of TTE ordered per day compared with the number ordered during the control period (2.9 vs. 3.9 TTE, p < 0.001). During the study period, the proportion of inappropriate TTE was significantly lower (5% vs. 13%, p < 0.001) and the proportion of appropriate TTE was significantly higher (93% vs. 84%, p < 0.001).
Conclusions A simple educational intervention produced a significant reduction in the proportion of inappropriate TTE and increased the proportion of appropriate TTE ordered on an inpatient academic medical service. This study provides a practical approach for using the AUC to reduce the number of inappropriate TTE. Further study in other practice environments is warranted.
Transthoracic echocardiography (TTE) is an important tool in the diagnosis and management of cardiovascular disease. There are concerns regarding the usage and growth of TTE, and the attendant implications for increased healthcare costs (1–3). Previous studies have found that the growth of TTE volume is estimated between 6% and 8% annually (2,4). Furthermore, whereas cardiologists interpret the majority of TTE, noncardiologists, particularly primary care and internal medicine physicians, order the majority of TTE (4).
In response to the growing demand and use of echocardiography, the American College of Cardiology Foundation, along with the American Society of Echocardiography and other subspecialty societies, developed appropriate use criteria (AUC) for TTE in 2007, and updated AUC were published in March 2011 (5,6). Adherence to AUC has become a quality improvement focus for echocardiography laboratories, and tracking and efforts to improve appropriateness is now required for accreditation (7). In addition to accreditation bodies, appropriate use has become a focus of professional societies and healthcare payers, in an attempt to contain the increasing costs of health care (7–9).
Prior retrospective studies have determined that the proportion of inappropriate TTE ranges from 10% to 15%, with variations seen in different practice settings (8,10–14). To our knowledge, there have been no studies to prospectively evaluate whether an AUC-guided educational intervention can reduce the proportion of inappropriate TTE. Therefore, we conducted a prospective time series analysis to evaluate an educational intervention aimed at reducing inappropriate TTE in a large academic general medical service.
This interrupted time series analysis was conducted on the teaching general medicine service at the Massachusetts General Hospital, a 907 bed quaternary care academic medical center. The teaching medical service is composed of 5 medical teams. Each team consists of 1 to 2 hospitalist attending staff, 1 junior resident, 4 interns, and 1 to 2 Harvard medical students. Patients typically have a wide variety of medical conditions with a high degree of medical complexity and comorbidities. Medical decision making is shared by each of the team members, as rounds are conducted in a “team style,” meaning every physician knows all of the patients. Diagnostic tests can be ordered by any physician team member, although they are typically ordered by an intern.
Each rotation on the medical service is 2 weeks long, and residents usually complete 2 consecutive rotations to comprise a 1-month-long rotation. Attending physicians typically spend 2 weeks on the service at a time.
The study protocol was reviewed and approved by the Partners Healthcare Institutional Review Board. All medical house officers provided verbal consent prior to study initiation.
From February 23, 2012, to June 24, 2012 (the intervention period), all medical house staff were given the following multifaceted educational intervention on TTE appropriateness: 1) a lecture at the beginning of the study period and the beginning of every rotation that described the AUC for echocardiography and highlighted the common clinical scenarios for which inpatient TTE are ordered; 2) a laminated pocket card providing tips on appropriate ordering of TTE (Fig. 1); and 3) a pre-intervention questionnaire on knowledge of the AUC and attitudes toward appropriate ordering and a post-intervention questionnaire to test understanding of the AUC. TTE ordering was tracked over the intervention period on a daily basis, and a feedback report was e-mailed to the teams at 2-week intervals. This feedback report contained the number of TTE ordered during the prior 2-week period and how many of those were classified as appropriate, inappropriate, or uncertain based on the 2011 AUC. Descriptions of all inappropriate TTE and the rationales for the inappropriate classifications were provided. Overall, the estimated time to design and implement the educational intervention was 18 h, or 1 h per week on average.
TTE ordering information was determined from a review of the electronic medical record (EMR). Signs and symptoms and the reason(s) for the TTE were abstracted from both the EMR and TTE order, as has been described previously (12). Inpatient TTE are ordered electronically. The inpatient ordering system contains an electronic prompt regarding the date of a patient's most recent TTE; otherwise, there is no decision support tool. The EMR is a comprehensive database that includes physician notes, imaging results, and laboratory findings, and it captures the entirety of clinical activity at the hospital. The EMR contains TTE reports from all hospitals within the healthcare network, and, typically, outside written TTE reports are scanned into the system.
Patient demographics and comorbidities were determined through review of the EMR and were classified according to the International Classification of Diseases-Ninth Revision codes.
Transthoracic echocardiogram classification
Two study investigators (R.S.B., R.B.W.) independently reviewed all TTE ordering information and classified each TTE as appropriate, inappropriate, or uncertain according to the 2011 AUC for TTE (6). If the reason for a TTE did not have a corresponding indication in the 2011 AUC, it was considered unclassifiable. If agreement on TTE classification was not achieved, then a third study investigator (M.H.P.) would review the order to achieve consensus; however, this was not required in this study. Outside TTE reports were taken into consideration when classifying Massachusetts General Hospital's TTE if the outside studies were complete and contained sufficient information to answer the clinical question. Data collection, classification of TTE, and feedback to house staff during the intervention period required approximately 25 h (1.4 h/week). Orders for stress echocardiograms and transesophageal echocardiograms were excluded from this study. Unlike TTEs, all inpatient stress echocardiograms and transesophageal echocardiograms are screened by an echocardiography fellow, which we felt could introduce a bias into our study.
Assessment for potential underutilization of TTE
To assess for the unintended consequence of underuse of appropriate TTE during the intervention period, we evaluated TTE ordering in 50 randomly selected patients from the control and intervention groups who were hospitalized with a primary diagnosis of heart failure (HF) (International Classification of Diseases-Ninth Revision code #428). Through review of the EMR, we determined whether the patient had a TTE performed during the admission, and if one was conducted, what the appropriateness rating of that TTE was. If the patient did not have a TTE, we assessed what the appropriateness rating would have been if a TTE had been performed. Methodology for determining appropriateness in patients who did not actually undergo TTE has been previously published (11).
The primary outcome measures in this study were the rate of inappropriate TTE, rate of appropriate TTE, and the total number of TTE ordered. Secondary outcome measures included common appropriate and inappropriate TTE indications, and pre- and post-intervention knowledge assessment scores among the medical house staff.
Categorical variables for ordering characteristics, patient demographics, and appropriateness ratings using the 2011 AUC were compared using chi-square or Fisher exact tests as required. Continuous variables are reported as mean ± SD and were compared using analysis of variance. Statistical significance was indicated by a 2-tailed p value <0.05.
During the pre-intervention period of January 1, 2011, to November 30, 2011, 5,623 patients were admitted to the teaching medical service, and during the intervention period of February 23, 2012, to June 24, 2012, 1,711 patients were admitted. The average number of admissions per day was 16.8 in the pre-intervention period and 13.9 in the intervention period. Patient characteristics are detailed in Table 1. There were statistically but not clinically significant differences between the patients analyzed in the 2 time periods.
Number of TTE and appropriateness of TTE
During the pre-intervention period, 56.5 TTE were ordered during each 2-week rotation, or 3.9 TTE per day. In the intervention period, 38.3 TTE were ordered during each 2-week rotation, or 2.9 TTE per day. Comparing the pre-intervention and intervention periods, there was a 26% reduction in the number of TTE ordered per day (p < 0.001) and the number of TTE per 100 admissions decreased by 15% in the intervention period (Table 2). The total number of TTE per 2-week rotation is shown in Figure 2A.
Appropriateness classifications of TTE are shown in Table 2. The vast majority of TTE in the pre-intervention and intervention periods were classifiable by the 2011 AUC. The percentage of inappropriate TTE decreased by 62% when the pre-intervention period was compared with the intervention period (13% vs. 5%, p < 0.001). As a corollary, the percentage of appropriate TTE increased (84% vs. 93%, p < 0.001). The number of inappropriate TTE and the rate of inappropriate and appropriate TTE per 2-week rotation block are displayed in Figures 2B to 2D.
Most common clinical indications for TTE
The most common appropriate, inappropriate, and uncertain indications for TTE are detailed in Table 3. Common appropriate indications included assessment of symptoms of conditions potentially related to suspected cardiac etiology (indication #1) or when there was prior testing concerning for heart disease or structural abnormality (indication #2). These 2 indications were among the most common in both the pre-intervention and educational intervention periods. Assessment of patients with sustained or nonsustained atrial fibrillation, supraventricular tachycardia, or ventricular tachycardia (indication #5) was the second and fourth most common indication in the pre-intervention and intervention periods, respectively. Other common appropriate indications during both time periods were initial evaluation of suspected endocarditis with positive blood cultures or new murmur (indication #52), evaluation of syncope when there are no other symptoms or signs of cardiovascular disease (indication #9), the initial evaluation of known or suspected heart failure (indication #70), or the re-evaluation of known heart failure with a clinical change without a clear precipitating change in medication or diet (indication #71).
The most common inappropriate indications in the pre-intervention period were transient fever without evidence of bacteremia or new murmur (indication #53) and transient bacteremia with a pathogen not typically associated with infective endocarditis and/or a documented nonendovascular source of infection (indication #54). Other common inappropriate TTE in the pre-intervention period included lightheadedness/pre-syncope with no other symptoms or signs of cardiovascular disease (indication #8), routine surveillance of left ventricular (LV) function in patients with known coronary disease (indication #11), evaluation of LV function in patients with prior ventricular function evaluation showing normal function and no clinical change (indication #12), and establishing diagnosis of pulmonary embolism (indication #28). The overall number of inappropriate TTE was low during the educational intervention, with the most common inappropriate indication being transient fever without evidence of bacteremia or no murmur (indication #53).
When the inappropriate indications for TTE to evaluate for infective endocarditis (indications #53, #54, and #56) were pooled, there were significantly fewer of these inappropriate TTE in the educational intervention period than in the pre-intervention period (7% vs. 3%, p = 0.002). Furthermore, when the inappropriate indications regarding routine LV assessment were pooled (indications #10, #11, #12, #13) there were significantly less of these inappropriate TTE during the educational intervention (2.2% vs. 0.5%, p = 0.05).
Knowledge and attitudes of medical house officers
Over the course of the educational intervention, 61 different interns rotated through the medical service for at least 2 weeks. In total, 49 interns (80%) completed pre- and post-intervention questionnaires, which included a 4-question, case-based knowledge assessment of TTE appropriateness based on the AUC and also questions regarding attitudes toward ordering of diagnostic testing. Overall, 47 of 49 interns (96%) felt physicians ordered too many tests, and all 49 interns said appropriate ordering is either “very important” or “somewhat important.” However, at pre-intervention assessment, 65% of interns were aware of the existence of AUC and only 45% thought about costs of tests they order “all” or “most” of the time. Interns' knowledge of the AUC and appropriateness improved over the course of the educational intervention, as the initial percentage of correct responses on the case-based assessment was 74.5% and the post-intervention correct response rate was 83.5% (p = 0.02).
Assessment of TTE use in HF patients
Results of a retrospective EMR review of HF patients admitted in both the pre-intervention and educational intervention periods are found in Table 4. The rate of ordering TTE on HF patients was similar in the pre-intervention and intervention periods (54% vs. 52%, p = 0.84). There was no difference in the rate of appropriate TTE in HF patients between pre-intervention and intervention (93% vs. 96%, p = 0.58). Additionally, in HF patients that did not have a TTE during hospital admission, it would have been appropriate to perform a TTE in 74% of pre-intervention cases and 54% of the intervention cases (p = 0.16).
This study describes the impact of an AUC-based educational intervention on the ordering of inpatient transthoracic echocardiograms by medical house officers at an academic medical center. The educational intervention significantly reduced the rate of inappropriate TTE, increased the rate of appropriate TTE, and also resulted in a reduction in the overall number of TTE. Specifically, this intervention produced a 62% reduction in the rate of inappropriate TTE, and the total number of TTE ordered per day fell by 26% and the rate of appropriate TTE increased by 11%. The medical house staff participating in this study demonstrated greater knowledge of the AUC on self-assessment after the educational intervention. Additionally, analysis of ordering of TTE for a subset of patients diagnosed with HF was similar in the pre-intervention and intervention periods, suggesting that appropriate TTE was not underused during the intervention period. To our knowledge, this study is the first to document that an AUC-based educational intervention can reduce the rate of inappropriate TTE.
The growth of cardiac imaging, including echocardiography, has led to efforts by the American College of Cardiology and other subspecialty organizations to create AUC. The goal of AUC is to allow physicians to continue the privilege of self-regulation by identifying clinical situations in which it is appropriate (and inappropriate) to pursue diagnostic testing or therapeutic interventions (15). For TTE, numerous retrospective “implementation” studies have evaluated practice patterns and identified an inappropriate rate of TTE of around 15% in the inpatient setting (11–13,16). Though the aim of the AUC is not to have an inappropriate rate of 0%, the data from the implementation studies indicate that there is room for practice improvement. Our study, the first AUC-based educational intervention study for TTE, shows that physician behavior can change and result in a reduction in the rate of inappropriate TTE and increase in the rate of appropriate TTE.
In addition to improving TTE ordering practices, our study provides valuable lessons to guide future efforts at improving the appropriate use of any diagnostic imaging modality, not just TTE. First, our study highlights the importance of identifying local practice patterns when designing and conducting an “appropriateness intervention.” The 2011 AUC for TTE contains 98 clinical indications (excluding transesophageal echocardiography and stress echocardiography); however, analysis of our pre-intervention period found that a select few indications are responsible for the majority of inappropriate studies. This same phenomenon has been shown in previous implementation studies (10,16–18). In the current study, 56% of all inappropriate TTE were for evaluation of low pre-test probability endocarditis or routine follow-up of known endocarditis, 11% for cases of pre-syncope with no other cardiac signs or symptoms, and 13% for routine surveillance of LV function. Thus, 80% of all inappropriate TTE in our control pre-intervention period were related to 3 main clinical situations. With this knowledge in hand, our educational intervention was designed to focus on these scenarios in order to target the educational lecture and supportive material (Fig. 1) to the highest yield areas. Thus, determination of common reasons for inappropriate tests in a given practice environment is an important prerequisite to the effective design of an intervention program.
Another important aspect of the educational intervention in our study was providing feedback to the medical house officers regarding their TTE ordering. Feedback to providers has been shown to be effective in reducing inappropriate ordering of laboratory and other radiology tests (19–21), but to date has not been studied in the context of an AUC-based educational intervention for echocardiography. A prior study has also shown that increasing the frequency of feedback and attaching educational messages to that feedback may enhance its effectiveness (20). Given the relatively high volume of TTE ordering on the inpatient medical service, we provided feedback on a biweekly basis, and this included specific details regarding the inappropriate TTE. The feedback may have enhanced the effectiveness of the intervention by reinforcing concepts that were taught in the initial lecture and addressed in the educational pocket cards. Additionally, providing feedback at 2-week intervals gave the house staff the opportunity to further refine their ordering practices for the remainder of that month. The biweekly feedback reports were e-mailed to the medical teams, and rarely we received responses further clarifying reasons and decision making for ordering TTE that were classified as inappropriate. These reasons varied from requests from specialty physicians, patient requests, requests from outside physicians, and clinical judgment. Although we did not include specialty consultant services in our educational intervention, this may be an area for future study.
Our study evaluated overall appropriateness rates of TTE ordering. Whereas some health-care payers, such as insurance companies, are using radiology benefits managers to screen individual TTE requests, and in certain cases refusing authorization for that study, our approach focuses on improving the global rate of appropriate ordering. To date, we are not aware of any studies that demonstrate that radiology benefits managers reduce the rate of inappropriate TTE, and there is evidence to suggest that there is limited agreement between radiology benefits manager pre-authorization and AUC rating (14). Another proposed strategy to improve ordering is physician pre-screening, which has been shown to reduce inappropriate stress echocardiograms (22) and may account for higher appropriate rates of transesophageal echocardiography (23). However, given the high volume of TTE, this is likely not practical for this modality. Future studies comparing strategies to reduce inappropriate use are required before an approach is widely adopted.
There are distinct implications of our findings with regard to health policy. A recent Institute of Medicine report found that 30% of healthcare spending in 2009—approximately $750 billion—was wasted, with $250 billion of that spent on unnecessary medical services (24). Furthermore, the American Board of Internal Medicine, along with numerous professional societies, created the “Choosing Wisely” campaign to bring public attention to potentially medically unnecessary procedures and help physicians be better stewards of finite healthcare resources (25). Therefore, the results of our study should be of interest to a wide spectrum of parties, including payers (government and insurance companies), policymakers, physicians, and patients. All of those with the goal to reduce unnecessary healthcare expenditures are eager to identify methods to do so, and our study provides a real-world example of such a method. Additionally, our subanalysis of HF patients helps address the concern that the reduced volume of TTE may result in failure to perform appropriate studies needed for patients. This was not the case in this exploratory analysis. Further investigation of the cost-effectiveness of our approach and the impact on healthcare savings will be required.
It is also important to recognize that parallel efforts are underway for other forms of cardiac imaging. A recent study by Gibbons et al. (26) found that a quality improvement project that used the AUC did not reduce the proportion of inappropriate stress single-photon emission computed tomography imaging. However, that study may have been limited by sample size and the lack of personalized, real-time feedback to ordering clinicians. Recently, a statewide continuous quality initiative funded by a large insurance company improved appropriate ordering of cardiac computed tomography and produced a decrease in inappropriate computed tomographies (27). Importantly, whereas that study provided evidence of a jurisdictional system of improving appropriateness, our study, aimed at health providers, provides a model for individual hospitals or practices to improve appropriate ordering.
We believe that the educational intervention performed in our study could be implemented and evaluated in other practice environments (i.e., outpatient practices) and for other types of cardiac testing. Provided that there is an infrastructure to track appropriateness of cardiac testing (both before implementation to focus the intervention and during the assessment period), this type of relatively straightforward intervention could have a significant impact on use of cardiac testing while preserving the privilege of physician self-regulation. Use of computerized programs to automate the determination of appropriateness ratings, which have been recently tested (28), would reduce the manual work required. Ultimately, educational information regarding appropriateness of all cardiology and radiology testing, not just echocardiography, may be available through point-of-care decision aids embedded in electronic ordering systems. Incorporating regular feedback reports could enhance the success of such programs.
Given the nature of the team-based system for medical residents and the potential for cross talk between teams, a randomized controlled trial was not feasible in this practice environment. The study was aimed primarily at medical residents, and it is unknown if this educational intervention would be successful among attending staff physicians, who may have more ingrained practice patterns and may be more resistant to change. Finally, this study was conducted at an academic medical center and so further study is required to determine if this intervention would be successful in private practices and community hospitals, where some clinicians may have financial incentives to order TTE.
This prospective study using a simple educational and feedback intervention based on the 2011 AUC for echocardiography reduced the rate of inappropriate TTE on an academic inpatient medical service. The rate of appropriate TTE also increased, and the total volume of TTE decreased. This study provides a practical approach of adapting the AUC to local practice patterns in an effort to improve use of medical resources and to reduce the number of unnecessary cardiac imaging tests.
This study was funded by the Massachusetts General Physicians Organization Clinical Innovation Award. The authors have reported that they have no relationships relevant to the contents of this paper to disclose.
- Abbreviations and Acronyms
- appropriate use criteria
- electronic medical record
- heart failure
- left ventricular
- transthoracic echocardiography or transthoracic echocardiograms
- Received December 10, 2012.
- Accepted January 23, 2013.
- American College of Cardiology Foundation
- Medicare Payment Advisory Commission (MedPAC)
- Alter D.A.,
- Stukel T.A.,
- Newman A.
- Lucas F.L.,
- DeLorenzo M.A.,
- Siewers A.E.,
- Wennberg D.E.
- Pearlman A.S.,
- Ryan T.,
- Picard M.H.,
- Douglas P.S.
- Douglas P.S.,
- Khandheria B.,
- Stainback R.F.,
- et al.
- Douglas P.S.,
- Garcia M.J.,
- Haines D.E.,
- et al.
- Intersocietal Commission for the Accreditation of Echocardiography Laboratories
- Shaw L.J.,
- Min J.K.,
- Hachamovitch R.,
- et al.
- Ballo P.,
- Bandini F.,
- Capecchi I.,
- et al.
- Willens H.J.,
- Hendel R.C.,
- Inhaber F.R.,
- et al.
- Patel M.R.,
- Wolk M.J.,
- Allen J.M.,
- Dehmer G.J.,
- Brindis R.G.
- Ward R.P.,
- Mansour I.N.,
- Lemieux N.,
- Gera N.,
- Mehta R.,
- Lang R.M.
- Parikh P.B.,
- Asheld J.,
- Kort S.,
- American College of Cardiology, American Society of Echocardiography
- Bhatia R.S.,
- Carne D.M.,
- Picard M.H.,
- Weiner R.B.
- Institute of Medicine
- ABIM Foundation
- Gibbons R.J.,
- Askew J.W.,
- Hodge D.,
- Kaping B.,
- Carryer D.J.,
- Miller T.
- Chinnaiyan K.M.,
- Peyser P.,
- Goraya T.,
- et al.
- Bhave N.M.,
- Mansour I.N.,
- Veronesi F.,
- Razi R.R.,
- Lang R.M.,
- Ward R.P.