- Research article
- Open Access
- Open Peer Review
Development and validation of a bedside risk score for MRSA among patients hospitalized with complicated skin and skin structure infections
BMC Infectious Diseasesvolume 12, Article number: 154 (2012)
Methicillin-resistant Staphylococcus aureus (MRSA) is a frequent cause of complicated skin and skin structure infections (cSSSI). Patients with MRSA require different empiric treatment than those with non-MRSA infections, yet no accurate tools exist to aid in stratifying the risk for a MRSA cSSSI. We sought to develop a simple bedside decision rule to tailor empiric coverage more accurately.
We conducted a large multicenter (N=62 hospitals) retrospective cohort study in a US-based database between April 2005 and March 2009. All adult initial admissions with ICD-9-CM codes specific to cSSSI were included. Patients admitted with MRSA vs. non-MRSA were compared with regard to baseline demographic, clinical and hospital characteristics. We developed and validated a model to predict the risk of MRSA, and compared its performance via sensitivity, specificity and other classification statistics to the healthcare-associated (HCA) infection risk factors.
Of the 7,183 patients with cSSSI, 2,387 (33.2%) had MRSA. Factors discriminating MRSA from non-MRSA were age, African-American race, no evidence of diabetes mellitus, cancer or renal dysfunction, and prior history of cardiac dysrhythmia. The score ranging from 0 to 8 points exhibited a consistent dose–response relationship. A MRSA score of 5 or higher was superior to the HCA classification in all characteristics, while that of 4 or higher was superior on all metrics except specificity.
MRSA is present in 1/3 of all hospitalized cSSSI. A simple bedside risk score can help discriminate the risk for MRSA vs. other pathogens with improved accuracy compared to the HCA definition.
Although rates of serious infections with pathogens such as methicillin-resistant Staphylococcus aureus (MRSA) have been rising over recent years , there is encouraging news from the Centers for Disease Control and Prevention (CDC) indicating that this rise may be abating for healthcare-associated pathogens, though not for community-acquired ones . Much of the emergent resistance can be attributed to selection pressures created by overuse of broad-spectrum antibiotics [3, 4]. Empiric treatment of an infection becomes a balancing act for physicians concerned with containing the spread of resistance vs. not covering broadly enough, impacting clinical response and potentially outcomes [5–11].
Complicated skin and skin structure infections (cSSSI) are a common reason for hospitalization. S. aureus-related cellulitis hospitalizations, for example, have risen 4-fold in a recent 6-year period to over 90,000 discharges , and MRSA is the cause of cellulitis in 15% of all cSSSI cases . While there is indirect evidence that inappropriate choice of empiric therapy may impact clinical outcomes among patients hospitalized with cSSSIs , this has not been confirmed broadly . However, what does appear to be affected by the initial antibiotic choice is the duration of hospitalization , an outcome of utmost importance given the fiscal constraints in the healthcare system. For this reason, an early risk stratification tool for the presence of MRSA infection would be useful because a) prompt targeted treatment would balance appropriate spectrum coverage with concerns for resistance emergence and b) conventional laboratory testing takes 48–72 h to yield culture results.
We set out to compare current epidemiology and outcomes of patients hospitalized with MRSA cSSSI to those hospitalized with non-MRSA. Based on these differences, we developed and validated a clinical decision tool to help guide appropriate antimicrobial treatment at presentation.
This retrospective cohort study used data collected from 62 hospitals in the Health Facts electronic medical record database (Cerner Corporation, Kansas City, MO). Health Facts is a database built from hospitals’ comprehensive clinical records including pharmacy, laboratory, admission, and billing information from all affiliated patient care locations. All admissions, medication orders, and laboratory orders and collections are labeled with date and time. Hospital billing/registration and encounter data are integrated with clinical information relating to the drugs dispensed and the results of diagnostic laboratory testing. Cerner Corporation has established Health Insurance Portability and Accountability Act (HIPAA)–compliant operating policies and procedures using statistical methods to establish de-identification.
Included patients were at least 18 years of age, hospitalized between April 2005 and March 2009, and met our diagnostic criteria in one of two ways: 1) Admission had a primary diagnosis from Additional file 1: Appendix A OR 2) Admission had a secondary diagnosis from Additional file 1: Appendix A, a primary diagnosis of sepsis (International Classification of Diseases, Clinical Modification [ICD-9-CM] codes 020.2, 038.xx, 790.7, 995.91, or 995.92), and no other source of infection documented by ICD-9-CM code. Patients were also required to have both 1) collection of a blood or skin culture that was positive for bacteria and 2) treatment with any intravenous (IV) antibiotic within 48 h of presentation. The positive index culture had to contain at least one organism that was not considered to be a skin contaminant. Cultures of organisms considered to be skin contaminants (Corynebacterium/diphtheroids, Staphylococcus epidermidis or other coagulase-negative Staphylococcus sp., Propionibacterium, Streptococcus viridians, Aerococcus spp, Bacillis spp [except B. anthracis]) were excluded from the cohort. Other exclusions were pregnancy or complications of childbirth, major trauma, and inpatient admission within 30 days with primary or secondary diagnosis of cSSSI (Additional file 1: Appendix A).
Microbiology results were prioritized to reduce the likelihood of colonizing microorganisms, though included patients could have polymicrobial infection. For patients with multiple culture results within the 48-hour timeframe, cultures from a skin source or venipuncture were prioritized over blood cultures drawn from a line, and certain sources of skin cultures with a high risk of contamination such as “swabs” were excluded. Lower extremity infections were identified using the 4th digit of the ICD-9-CM diagnosis code and/or the anatomic location of the skin culture, if present. Healthcare-associated (HCA) infection was defined based on the presence of at least one of the following factors: 1) admission from a chronic care facility, or 2) hospitalization within the previous 180 days or recent outpatient surgery, or 3) hemodialysis within 90 days or end-stage renal disease (ESRD) diagnosis code, or 4) chronic dependence on mechanical ventilation. Impaired immune function, based on specific ICD-9-CM codes and/or medication use, was examined separately. Chronic comorbidities present during admission or 12 months prior were identified by ICD-9-CM codes; baseline (within 48 hours of admission) laboratory variables were also recorded. Organ dysfunction measures at or before the time of the index culture were designed to be consistent with a Sepsis-related Organ Failure Assessment (SOFA) score equal to 2 or greater . These included cardiovascular, respiratory, hepatic, hematologic, and renal dysfunction. Unadjusted hospital mortality served as the primary endpoint, while length of stay (LOS) among survivors was the secondary outcome.
Based on statistical and clinical considerations, we built and explored the usefulness of a simple summary score to identify infection with MRSA vs. a non-MRSA organism.
Our aim was to develop a simple summary score to be used at the bedside to serve as a clinical decision tool to distinguish patients with MRSA from those with an infection with another pathogen (non-MRSA). To accomplish this, we first compared patients admitted with MRSA to those with non-MRSA with regard to baseline demographic, clinical and hospital characteristics. To test for differences between groups, we used t-tests for continuous variables and the chi square test or Fisher’s exact test for categorical variables. When constructing the summary scores, potential covariates were included in the models if they reached significance at alpha=0.2 in the univariate analyses, or if a priori they were determined to have clinical relevance (Additional file 1: Appendix B) . Variables were included only if they were present on admission or could be known within the first 24 hours of admission. The rationale for the 24-hour window was that inappropriately targeted empiric antimicrobial coverage within this time frame is associated with worsened outcomes [5–11]. In the selection of laboratory variables, we only considered tests for which at least 95% of values were obtained within 24 hours of admission.
A split sample approach was used, whereby 80% of the patients were randomly selected for model development and the remaining 20% were used for model validation. To identify the most predictive variables of MRSA vs. non-MRSA infection, we constructed a series of backward stepwise logistic regression models on bootstrapped with replacement (simulated) datasets from the development set and created a frequency table indicating the number of times the eligible predictors entered these stepwise regression models. The variables appearing most frequently were then considered for inclusion in the final summary score . The score was designed a priori to rely on no more than 5 predictors for ease of use in clinical practice (which compelled us to exclude highly predictive but rare variables from the final model). The final model’s predictors were included in a logistic regression model and the coefficients were fully standardized to a unit variance. The predictors in the summary score were then assigned integer-valued weights proportional to these standardized coefficients.
We calculated the prevalence of MRSA infection at every score value in the development cohort and compared these results to the prevalence in the validation cohort to assess calibration. A simple, overall classification rate is often misleading when the overall prevalence of a condition is either quite high or low. To better assess the predictive ability of the models, we calculated specificities and sensitivities at various threshold levels of the score(s) to derive likelihood ratios. With these statistics, we constructed Fagan’s nomograms to show a patient’s posterior probability of having MRSA if the score was above or below a particular threshold (i.e., being above a threshold level was analogous to a positive “test”) .
To augment usefulness at the bedside and to benchmark the score against the HCA risk factors, an additional assessment of score characteristics was done calculating sensitivity, specificity, positive and negative predictive values for each threshold score along the scale. These characteristics were then compared to the performance of the HCA definition as a way to discriminate the risk for MRSA from that for non-MRSA infection.
Among the 7,183 patients hospitalized with cSSSI, MRSA was cultured in 2,387 (33.2%), with the remainder (n=4,796) infected with non-MRSA (Table 1). Compared to patients with non-MRSA, those with MRSA were younger, less likely to be Caucasian, more likely to be on the medical service, and had lower comorbidity burden. Additionally, fewer of them had any risk factors for HCA infection (MRSA 32.6%, non-MRSA 36.8%, p<0.001) or a history of immune suppression (MRSA 22.9%, non-MRSA 28.9%, p<0.001).
Unadjusted hospital mortality was lower in the MRSA (3.0%) than the non-MRSA (4.7%) group, p<0.001. Similarly, the median hospital LOS was shorter in the MRSA (4.2 days) than in the non-MRSA (5.7 days) group, p <0.001.
MRSA prediction score
The MRSA score was constructed on a random sample of 80% of the patients (development set, N=5,736) and validated on the remaining 20% (validation set, N=1,447). Factors differentiating MRSA infection from other organisms, based on the logistic regression, resulted in a score ranging from 0 to 8 points (Table 2). Those factors and their corresponding points were: age (range 0–3 points), African-American race (1 point), no evidence of diabetes mellitus (1 point), cancer (1 point) or renal dysfunction (1 point), and prior history of cardiac dysrhythmia (1 point). Increasing scores corresponded well to increased likelihood of MRSA infection in both development and validation cohorts (Figure 1).
Table 3 presents the prevalence of actual MRSA isolates among patients with a particular score. While approximately 20% of all cases of MRSA had the score of 0–1, among those scoring 7–8 nearly 2/3 had the organism. Between these extremes, the score exhibited a consistent dose–response relationship.
We additionally examined the test characteristics of various score thresholds for detecting MRSA, and compared those to the HCA definition (Table 4). A MRSA score of 5 or higher turned out to be superior to the HCA classification in all characteristics, while that of 4 or higher was superior on all metrics except specificity. Using the threshold of 5 as an example for the score, if a clinician is most interested in excluding the probability of MRSA, the negative predictive value (the probability of scoring below 5 given that there was no MRSA present) was 72.3%. This represents a 12% relative improvement over the negative predictive value of the HCA definition (64.3%).
A Bayesian analysis was performed via Fagan’s nomogram on the validation set (Figure 2). The threshold classification results are based on the scores of 3 (T-3), 5 (T-5) and 7 (T-7). Using the pre-test probability of MRSA of 33% (the overall prevalence in our cohort), at a threshold score of 3 or higher the posterior probability of having MRSA is 34.5%. Similarly, for the score of 5 it becomes 43.0%, and for the score of 7 it is 48.2%. Alternatively, if the score is below 3, the posterior probability of having MRSA diminishes to less than 25%.
In this large multi-center cohort study, we have demonstrated that MRSA is present in one-third of all cSSSI patients hospitalized for their infection. There was a distinct age difference with regard to the risk for MRSA, wherein younger age groups were more likely to be infected with it relative to the older ones. Similarly, the burdens of comorbidities and risk factors for healthcare-associated infection were lower in the MRSA-infected group compared to those harboring non-MRSA. Commensurately, both hospital mortality and LOS among the MRSA-infected group were lower when compared to the non-MRSA group. Finally, we developed a bedside risk score to aid in the early identification of MRSA as a potential pathogen. Although improving somewhat the post-test probability of MRSA, the usefulness of the score may be limited by the high pathogen prevalence in the cohort.
Over the last decade, the increasing frequency of resistant organisms among persons presenting with infections from the community prompted a re-evaluation of the populations at risk for harboring such pathogens. Out of this effort, the HCA infection definition was born to signify a patient presenting from the community who is nevertheless put at risk for resistant pathogens by his/her ongoing contact with the healthcare system . Some of the commonly acknowledged risk factors for HCA infection are recent hospitalization, immune suppression, need for hemodialysis and nursing home residence. Because the HCA risk factors were neither developed nor validated prospectively, several authors have expressed concern that such imprecise definition would lead to over-diagnosis of resistance, and thus overtreatment, with its sequelae of promoting further resistance development. For example, in a cohort of patients with HCA pneumonia (HCAP), Shorr et al. calculated the specificity of the HCAP risk factors for a resistant pathogen to be below 50% . In an effort to sharpen accuracy, the investigators focused on identifying specific risk factors for resistant pathogens overall and for MRSA individually, with the view to developing a simple bedside risk score [17, 18].
Our findings are important on several accounts. The risk factors we found that were strongly associated with the presence of a MRSA infection differed from the broadly used HCA definition. In fact, HCA was less likely to be present in the setting of MRSA than non-MRSA infection. Although this differs from other reports that found 2/3 of all MRSA cSSSIs to have underlying HCA risk factors, the discrepancy is likely due more to the lower prevalence of HCA overall in the current study rather than to fundamental microbiologic differences . That is the prior study noted ~75% prevalence of HCA in the cohort of 717 patients admitted with a cSSSI, while in the current analysis it was only 35%. Given that the previous result came from a single-center analysis, the current data are more generalizable. Additionally, although the systematically derived score performed somewhat better than the consensus definition, it was far from perfect, thus shielding some, but not all, of those with susceptible pathogens from exposure to overly broad coverage. A similar finding was reported by Schreiber et al. in a group of patients with pneumonia and respiratory failure .
In the realm of cSSSI, the evidence-based guideline recommendations shed limited light on how to evaluate a patient’s risk for harboring MRSA . Compounding this difficulty further is the emergence of distinct community-acquired genotypes: though they require expanded antimicrobial coverage, their risk factors diverge from the traditional HCA paradigm. Although we were unable to examine the genetic make-up of the organisms in our study, the finding that MRSA was more frequent in the younger age groups may be consistent with the high prevalence of CA-MRSA. This underscores the need for a better risk stratification tool in this area than the HCA definition alone can provide.
Our current effort extends the findings of previous investigators on the challenges of identifying resistant organisms upon presentation and prior to the availability of culture results. In our study, the HCA definition identified only 32.0% of patients with MRSA correctly, while mislabeling an additional 35.5% as having MRSA, thus potentially prompting a high volume of overuse of broad-spectrum coverage (Table 4). In contrast, different values for our MRSA score provide flexibility for balancing sensitivity with specificity, and, therefore the negative with the positive predictive values. Namely, if excluding the possibility of MRSA is of primary interest, then the MRSA score of 2 or higher provides a higher degree of confidence than the HCA definition, given its substantially higher negative predictive value. However, when accuracy in general is sought, it is clear that higher MRSA scores (e.g., 4 or more) possess better test characteristics than the HCA definition, thus leading to more targeted prescription and less potential misuse of antibiotics.
Our study has a number of strengths and limitations. As a large multicenter cohort representing over 60 US hospitals, it has a high degree of generalizability. At the same time, as a retrospective observational study it is subject to a number of threats to validity, including a selection bias. We mitigated the possibility of this by setting a priori cohort definitions and including all consecutive patients meeting our selection criteria. Given that we relied on administrative coding to define our cohort, the study is prone to some degree of misclassification. However, the same approach has been used widely by multiple investigators, and this type of misclassification is likely to be either minor or non-differential [11, 12, 22]. Finally, while we focused on predictors that had a strong association with MRSA and were common, it is possible that other variables not in our database would have yielded a more accurate model, though we note that the list of predictors examined was quite comprehensive.
In summary, although our simple bedside score did improve on the specificity of the HCA definition, it left room for further improvement. Ultimately, what is needed is a bedside diagnostic tool that can identify MRSA and other resistant organisms rapidly and with a high degree of accuracy, so as to tailor treatment appropriately. Until such tools are available, and depending on the local patterns of resistance, empiric broad-spectrum coverage followed by prompt de-escalation in response to microbiology data may need to remain the standard of care.
This study was funded by Astellas Pharma US (APUS), Inc, Deerfield, IL. Drs. Zilberberg and Shorr have received research and consulting funds from APUS. Dr. Chaudhari is an employee and a stock holder in APUS. Drs. Campbell and Emons, Ms. Fiske and Mr. Hays work for Cerner Corporation, which has received research funding from APUS. Dr. Nathanson works for OptiStatim, LLC, which has received funds from Cerner Corporation.
Klein E, Smith DL, Laxminarayan R: Hospitalizations and deaths caused by methicillin-resistant Staphylococcus aureus, United States, 1999–2005. Emerg Infect Dis. 2007, 13 (12): 1840-1846. 10.3201/eid1312.070629.
Kallen AJ, Mu Y, Bulens S, Reingold A, Petit S, Gershman K, et al: Health care-associated invasive MRSA infections, 2005–2008. JAMA. 2010, 304 (6): 641-648. 10.1001/jama.2010.1115.
Dellit TH, Owens RC, McGowan JE, Gerding DN, Weinstein RA, Burke JP, et al: Infectious Diseases Society of America and the Society for Healthcare Epidemiology of America guidelines for developing an institutional program to enhance antimicrobial stewardship. Clin Infect Dis. 2007, 44 (2): 159-177. 10.1086/510393.
Spellberg B, Guidos R, Gilbert D, Bradley J, Boucher HW, Scheld WM, et al: The epidemic of antibiotic-resistant infections: a call to action for the medical community from the Infectious Diseases Society of America. Clin Infect Dis. 2008, 46 (2): 155-164. 10.1086/524891.
Micek ST, Kollef KE, Reichley RM, Roubinian N, Kollef MH: Health care-associated pneumonia and community-acquired pneumonia: a single-center experience. Antimicrob Agents Chemother. 2007, 51 (10): 3568-3573. 10.1128/AAC.00851-07.
Schramm GE, Johnson JA, Doherty JA, Micek ST, Kollef MH: Methicillin-resistant Staphylococcus aureus sterile-site infection: The importance of appropriate initial antimicrobial treatment. Crit Care Med. 2006, 34 (8): 2069-2074. 10.1097/01.CCM.0000227655.41566.3E.
Ibrahim EH, Sherman G, Ward S, Fraser VJ, Kollef MH: The influence of inadequate antimicrobial treatment of bloodstream infections on patient outcomes in the ICU setting. Chest. 2000, 118 (1): 146-155. 10.1378/chest.118.1.146.
Alvarez-Lerma F: Modification of empiric antibiotic treatment in patients with pneumonia acquired in the intensive care unit. ICU-Acquired Pneumonia Study Group. Intensive Care Med. 1996, 22 (5): 387-394.
Iregui M, Ward S, Sherman G, Fraser VJ, Kollef MH: Clinical importance of delays in the initiation of appropriate antibiotic treatment for ventilator-associated pneumonia. Chest. 2002, 122 (1): 262-268. 10.1378/chest.122.1.262.
Zilberberg MD, Shorr AF, Micek ST, Mody SH, Kollef MH: Antimicrobial therapy escalation and hospital mortality among patients with health-care-associated pneumonia: a single-center experience. Chest. 2008, 134 (5): 963-968. 10.1378/chest.08-0842.
Zilberberg MD, Shorr AF, Micek ST, Chen J, Ramsey AM, Hoban AP, et al: Hospitalizations with healthcare-associated complicated skin and skin structure infections: impact of inappropriate empiric therapy on outcomes. J Hosp Med. 2010, 5 (9): 535-540. 10.1002/jhm.713.
Lipsky BA, Weigelt JA, Gupta V, Killian A, Peng MM: Skin, soft tissue, bone, and joint infections in hospitalized patients: epidemiology and microbiological, clinical, and economic outcomes. Infect Control Hosp Epidemiol. 2007, 28 (11): 1290-1298. 10.1086/520743.
Vincent JL, Moreno R, Takala J, Willatts S, De Mendonca A, Bruining H, On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine, et al: The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. Intensive Care Med. 1996, 22 (7): 707-710. 10.1007/BF01709751.
Austin PC, Tu J: Bootstrap methods for developing predictive models. Am Stat. 2004, 58: 131-137. 10.1198/0003130043277.
Guidelines for the management of adults with hospital-acquired ventilator-associated, and healthcare-associated pneumonia. Am J Respir Crit Care Med. 2005, 171 (4): 388-416.
Fagan TJ: Letter: Nomogram for Bayes theorem. N Engl J Med. 1975, 293 (5): 257-
Shorr AF, Zilberberg MD, Micek ST, Kollef MH: Prediction of infection due to antibiotic-resistant bacteria by select risk factors for health care-associated pneumonia. Arch Intern Med. 2008, 168 (20): 2205-2210. 10.1001/archinte.168.20.2205.
Shorr AF, Zilberberg MD, Reichley R, Kan J, Hoban A, Hoffman J, et al: Validation of a clinical score for assessing the risk of resistant pathogens in patients with pneumonia presenting to the emergency department. Clin Infect Dis. 2012, 54 (2): 193-198. 10.1093/cid/cir813.
Zilberberg MD, Shorr AF, Micek ST, et al: Epidemiology and outcomes of hospitalizations with complicated skin and skin structure infections: Implications of healthcare associated infection risk factors. Infect Control Hosp Epidemiol. 2009, 30: 1203-10. 10.1086/648083.
Schreiber MP, Chan CM, Shorr AF: Resistant pathogens in nonnosocomial pneumonia and respiratory failure: is it time to refine the definition of health-care-associated pneumonia?. Chest. 2010, 137 (6): 1283-1288. 10.1378/chest.09-2434.
Stevens DL, Bisno AL, Chambers HF, Everett ED, Dellinger P, Goldstein EJ, et al: Practice guidelines for the diagnosis and management of skin and soft-tissue infections. Clin Infect Dis. 2005, 41 (10): 1373-1406. 10.1086/497143.
Edelsberg J, Berger A, Weber DJ, Mallick R, Kuznik A, Oster G: Clinical and economic consequences of failure of initial antibiotic therapy for hospitalized patients with complicated skin and skin-structure infections. Infect Control Hosp Epidemiol. 2008, 29 (2): 160-169. 10.1086/526444.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1471-2334/12/154/prepub
This study was funded by Astellas Pharma US (APUS), Inc, Deerfield, IL. Drs. Zilberberg and Shorr have received research and consulting funds from APUS. Dr. Chaudhari is an employee and a stockholder in APUS. Drs. Campbell and Emons, Ms. Fiske and Mr. Hays work for Cerner Corporation, which has received research funding from APUS. Dr. Nathanson works for OptiStatim, LLC, which has received funds from Cerner Corporation.
MDZ participated in the conception and design of the study, data interpretation and drafting and revising of the manuscript. PC participated in design of the study and revising of the manuscript. BHN participated in the design of the study, analyzed the data and revised the manuscript critically for intellectual content. RSC participated in the design of the study, study analyses and drafting of the manuscript. MFE participated in the conception and design of the study and revising of the manuscript. SF participated in data management and analysis. HDH participated in data management and analysis. AFS participated in the conception and design of the study, data interpretation and revising of the manuscript for important intellectual content. All authors read and approved the final manuscript.