Agar Disk Diffusion and Automated Microbroth Dilution Produce Similar Antimicrobial Susceptibility Testing Results for Salmonella Serotypes Newport,Typhimurium,and 4,5,12:i-,But Differ in Economic Cost

Isolate selection

Non-typhoidal Salmonella enterica subsp. enterica isolates from bovine (n=150) and human (n=72) sources representing serotypes Typhimurium (n=76), 4,5,12:i:- (n=32), and Newport (n=114) were collected as part of a previous study (Hoelzer et al., 2010). All isolates were collected between 2004 and 2005 from cases with clinical signs of salmonellosis; human-source isolates were obtained from the Wadsworth Center, New York State Department of Health, whereas bovine-source isolates were collected from dairy cattle on 57 farms in New York and Vermont and were obtained via the New York State Animal Health Diagnostic Center. Isolation was performed using standard culturing techniques as previously described (Cummings et al., 2010), and confirmed Salmonella isolates were sub-streaked to ascertain presence of pure cultures.

Agar disk diffusion based antimicrobial susceptibility testing

Agar disk diffusion based testing was performed at the Field Disease Investigation Unit, Washington State University College of Veterinary Medicine. Susceptibility was determined as described previously (Bauer et al., 1966). Isolates were cultured in brain heart infusion broth (Hardy Diagnostics, Santa Maria, CA) and adjusted to a McFarland Standard of 0.5. Bacterial lawns were streaked onto Mueller-Hinton agar plates (Hardy Diagnostics). Antibiotic disks (Hardy Diagnostics) containing the following antimicrobial agents were used: ampicillin [10 μg], amoxicillin-clavulanic acid [20 and 10 μg], chloramphenicol [30 μg], kanamycin [30 μg], gentamicin [10 μg], streptomycin [10 μg], trimethoprim-sulfamethoxazole [1.25 and 23.75 μg], tetracycline [30 μg]. After overnight incubation at 37°C inhibition zone diameters were read using a Fowler digital caliper (Newton, MA). Results have been reported previously (Hoelzer et al., 2010). Isolates were classified as resistant or susceptible using resistance breakpoints as reported by CLSI (Table 1) (NCCLS, 2003a, 2003b).

Automated microbroth dilution-based antimicrobial susceptibility testing

Microbroth dilution-based testing was performed at the New York State Animal Health Diagnostic Center (NYSAHDC). MICs for the same eight antimicrobial agents were established using the Sensititre system (TREK Diagnostic Systems, Cleveland, OH) as detailed previously (Cummings et al., 2010). CLSI guidelines were used to interpret MIC values if available, otherwise breakpoints detailed in the National Antimicrobial Resistance Monitoring System (NARMS) were used (Table 1) (CLSI, 2008; CDC, 2009). Weekly quality control was performed using four bacterial reference strains: Escherichia coli ATCC 25922, Staphylococcus aureus ATCC 29213, Enterococcus faecalis ATCC 29212, and Pseudomonas aeruginosa ATCC 27853, representing the standard quality controls used at NYSAHDC. Quality control was performed as recommended by CLSI. Isolates with disagreeing results were re-tested using both tests. If disagreement was resolved upon re-testing, very major and major error rates were calculated as described previously (Stuckey, 2007), but the original data were used for all other analyses.

Statistical analysis

All statistical analyses were performed using SAS version 9.2 (SAS, Cary, NJ). Exact 95% binominal confidence intervals, McNemar's Chi-squared test for paired data (McNemar, 1947), and the kappa statistic were calculated, with kappa defined as follows: \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland,xspace}\usepackage{amsmath,amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6} \begin{document} \begin{align*}= \frac {observed \ agreement - expected \ agreement}{1 - expected \ agreement} \end{align*} \end{document}

Data were visualized using scatter plots. Since MIC values are approximately log-normally distributed (Turnidge and Paterson, 2007), MIC values (except gentamicin and trimethoprim-sulfamethoxazole with most MIC values in the range of 0.1–1) were log-transformed before further analysis. Spearman's rank correlation coefficients were calculated to estimate correlations between zone diameters and MICs. Cox's semi-parametric proportional hazards model (Cox, 1972) was also used, with growth inhibition, the event of interest, and concentration to event (c) replacing time to event (Stegeman et al., 2006). Since the partial likelihood method only depends on event time ranks, log-transformed and original MIC values generate identical outputs (Allison, 1995; Kalbfleisch and Prentice, 2002). Right-censoring was accounted for, but only few observations were left-censored, and because the expected impact of including left censoring was marginal, it was omitted. Ties were handled using the exact method for continuous time data in SAS version 9.2, resulting in models of the general form below (Stegeman et al., 2006).

\documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$$h(c, X ) = h_{0} (c){\rm e}^{\Sigma_{{\rm i} = 1}^{\rm p} \beta{\rm iXi}}$$\end{document} where h ₀(c) is the baseline hazard function, and \documentclass{aastex}\usepackage{amsbsy}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{bm}\usepackage{mathrsfs}\usepackage{pifont}\usepackage{stmaryrd}\usepackage{textcomp}\usepackage{portland, xspace}\usepackage{amsmath, amsxtra}\pagestyle{empty}\DeclareMathSizes{10}{9}{7}{6}\begin{document}$${\sum_{\rm i = 1}^{\rm p}} \beta{\rm iXi}$$\end{document} is a linear function of a set of p fixed covariates. Year of isolation, host species, and serotype were potential covariates, and the final models were determined based on a stepwise backward selection algorithm. Model fit was assessed through influence diagnostics and deviance residuals.

Parametric regression models (i.e., exponential, Weibull, lognormal, loglogistic, and Gamma) were also used. Graphical diagnostic methods (i.e., survivor plots and residual plots) allowed evaluation of model fit, and nested models were compared using generalized likelihood ratio tests. Product-limit estimators were calculated when the data allowed.

Economic cost analysis

Input values to calculate labor, equipment, and consumable costs were estimated based on expert judgment and standard rates applicable at Cornell University during fiscal year 2010. Staff at Cornell University and Washington State University with relevant expertise in antimicrobial susceptibility testing provided cost and time estimates and consulted on the overall cost analysis. All cost calculations were performed in Excel version 2007, with labor costs based on current rates at Cornell University.

Category agreement analysis

In total, 11 of the 222 isolates (4.9%) showed some disagreement in category classification. Category disagreement usually only affected one drug, with the exception of one isolate for which disagreement affected two drugs. For the eight drugs tested, between 0% and 3% of isolates showed some category disagreement (Table 1). Kappa values, measuring better-than-chance agreement between tests, ranged from 0.95 to 0.99 (Table 1), indicating excellent agreement. To explore whether one test was consistently more likely to classify isolates as susceptible than the other, we calculated McNemar's Chi-squared statistic (Table 1). For the combination of amoxicillin and clavulanic acid, we detected evidence of this (p=0.03), with all five disagreeing results classified as resistant by disk diffusion but susceptible by microbroth dilution. We did not detect evidence for such differences for any other drug (Table 1).

All 11 isolates with disagreement were retested. For the combination of amoxicillin and clavulanic acid all isolates remained susceptible by microbroth dilution, whereas they now also appeared susceptible by disk diffusion. If microbroth dilution results were assumed to be correct, this translates into a major error rate of 5%. Disk diffusion results for isolates with disagreement for kanamycin remained unchanged upon re-testing. However, upon re-testing by microbroth dilution two isolates now appeared susceptible, and one isolate now appeared resistant, whereas one isolate remained susceptible. If disk diffusion results were assumed to be correct, this translates into a very major error rate of 1% and a major error rate of 2%. To avoid potential bias introduced by selective re-testing of some isolates, the initial data were used for all other analyses.

Comparison of zone diameters and MICs

MICs and zone diameters were negatively correlated (Fig. 1, Table 2). Correlations were nearly perfect for ampicillin, chloramphenicol, and streptomycin, evidenced by correlation coefficients exceeding −0.95 (Table 2). Correlations were considerably weaker, yet statistically significant, for the remaining drugs, indicating that higher MICs always tended to be associated with smaller zone diameters. MICs were often censored at the resistance breakpoint, leading to many isolates reported as exceeding the breakpoint, which may distort results.

OPEN IN VIEWER

FIG. 1.

Scatter plots showing minimal inhibitory concentrations (MICs) and zone diameters. Resistance breakpoints are indicated by horizontal (MICs) and vertical (zone diameters) lines.

Proportional hazard models can account for such censoring, evaluate the relationship between the two test results, and allow inference whether additional factors need to be considered. These models fitted the data well, with the exception of streptomycin and potentially amoxicillin and clavulanic acid (data not shown). For most drugs, the zone diameter hazard ratios were in the range of 1.2–1.4 (Table 2), indicating that for isolates that were not inhibited at lower concentrations, the hazard of reaching the MIC at a given drug concentration increased 20%–40% for each 1 mm increase in associated zone diameter. For ampicillin, gentamicin, kanamycin, and tetracycline, zone diameters were the only significant predictors (Table 2). For the remaining drugs additional predictors impacted this hazard. For instance, for both chloramphenicol and the combination of amoxicillin and clavulanic acid, isolation year was a significant predictor, indicating that the increase in hazard differed between isolates collected in 2004 and 2005, even if zone diameters were identical.

Parametric models could allow further characterization of this hazard, but their fit was generally questionable. From these models, meaningful estimates of median MICs could only be obtained for gentamicin and trimethoprim-sulfamethoxazole, equaling 0.25 and 0.12, respectively.

Comparison of economic testing costs

We estimated costs for an average monthly throughput of 10 to 1000 isolates. A lower throughput appears economically unfeasible for automated microbroth dilution, and a higher monthly throughput would probably be logistically unfeasible for disk diffusion.

Based on estimates from staff at Cornell University and Washington State University with extensive experience performing antimicrobial susceptibility testing, we assumed consumable costs of approximately $11.46 per sample for microbroth dilution and of approximately $8.94 per sample for disk diffusion (Table 3), assuming the testing volume is sufficient to avoid wasted reagents. We further estimated labor costs of ∼15 minutes per sample to set up the microbroth dilution, and 10 minutes per sample to set up, analyze, and report the disk diffusion tests, translating into labor costs of $5.5 and $3.7 per sample, respectively. The absolute quality control costs are sample-number dependent for disk diffusion but constant for microbroth dilution, significantly contributing to the continuous decline in operational costs for microbroth dilution as sample throughput increases (Table 3). Operational costs are lower for disk diffusion than for microbroth dilution for all sample volumes, but the difference in per-sample cost reduces from $23.86 for an average throughput of 10 samples to $3.5 at an average throughput of 1000 samples per month.

Investment costs were ∼3 times as high for microbroth dilution as for disk diffusion, and because of the need for external training we estimated training costs alone as ∼40 times as high for microbroth dilution as for disk diffusion, even though our cost estimates were conservative ($2000). Our calculation did not account for opportunity cost, service and maintenance costs, or potential other costs. Equipment for microbroth dilution can often also be used for other diagnostic applications, and we also did not account for potential additional revenue streams. Taking investment cost into consideration, the difference in cost reduces from $136.67 for an average throughput of 10 samples per month to $4.64 for an average throughput of 1000 samples per month.

Automated microbroth dilution and agar disk diffusion yield highly comparable isolate classifications

Test results were highly comparable in our study, with category agreement equaling 95%. Such results are in agreement with previous reports, even though only limited evaluations of antimicrobial test correspondence are available (Kohner et al., 1997; Pfaller et al., 2003; Halbert et al., 2005; Jones et al., 2005; Nayak et al., 2007), and some other studies have reported poor agreement between disk diffusion and microbroth dilution (Woolfrey et al., 1981; Lumbiganon et al., 2000). According to the CLSI criteria for laboratory evaluation, a very major error rate of 1.5%, a major error rate of 3%, and an overall category agreement of 90% represent acceptable interpretative errors (Stuckey, 2007). For almost all antimicrobial drugs, our tests satisfied or exceeds these minimum criteria, indicating excellent agreement between CLSI recommended breakpoints and demonstrating adequacy of results generated with either method. A small number of random errors are to be expected with each test, and except for the combination of amoxicillin and clavulanic acid, disagreement appeared non-differential and was generally resolved through re-testing. Error rates similar to those reported here have been reported previously (Edelmann et al., 2007; Nayak et al., 2007). In those studies as well as in ours, error rates varied considerably among drugs, and error rates appeared particularly high for the combination of amoxicillin and clavulanic acid (Edelmann et al., 2007). Disagreement between test results may be due to human error or the spontaneous loss or gain of resistance during culture. However, disk diffusion based methods can also be impacted by numerous external variables including agar pH and depth, varying concentrations of free calcium ions, thymine or thymidine in the agar, uneven inoculation volumes, or inconsistent incubation times and temperatures (CDC, 1999). Microbroth dilution tests can in contrast be affected by deviations in pH and inoculation time (Whithear et al., 1983). Importantly, results for different drug classes are often differentially affected by test procedure deviations.

MIC and zone diameter values are negatively correlated but correlation strength differs between drugs, and simple back-calculation from zone diameter to MIC without accounting for potential confounders appears invalid

A clear negative relationship between MIC and zone diameter values, as detected here, has been reported previously (Bemis et al., 2009). Knowledge of this relationship may be useful, especially in clinical settings. A survey of antimicrobial susceptibility testing practices in 77 rural hospitals located in four U.S. states reported that 20 hospitals used disk diffusion for susceptibility screening, with16 hospitals not confirming results with dilution methods (Stevenson et al., 2003). The survey was performed ∼10 years ago, but similar practices are likely still common in many areas.

Several studies have reported a linear relationship between MIC and zone diameter values (Wood and Shadomy, 1983; Jones et al., 2005; Dimitriu et al., 2006; Bemis et al., 2009). Based on this observation, some researchers have transformed zone diameters into MIC values based on regression analysis and commercial applications exist, with potentially acceptable results for certain bacteria, including Enterobacteriaceae, Pseudomonas, and Streptococcus (D'Amato et al., 1985; Sautter and Denys, 1987; Hubert et al., 1998; Jacobs et al., 1998). However, the semi-parametric Cox regression model represents a more appropriate analysis technique for censored data (D'Amato et al., 1985; Sautter and Denys, 1987; Hubert et al., 1998; Jacobs et al., 1998). In our study, we also detected a drug-specific relationship between MIC and zone diameter values. Yet, for several drugs additional covariates such as serotype, host species, or isolation year also impacted the MIC, indicating that simple back-calculations that do not account for these confounders are often inappropriate.

For laboratories with high sample throughput automated microbroth dilution may be preferable, whereas agar disk diffusion is clearly more economical for laboratories with low sample throughput

Few studies have formally compared the economic feasibility of antimicrobial susceptibility tests. We found that automated microbroth dilution becomes prohibitively expensive if sample throughput is low. This finding is in agreement with a recent study, which concluded that the semi-quantitative E-test was economically more feasible than the dilution based method it was compared to (Valdivieso-García et al., 2009), and the observation also agrees with older studies (Berke and Tierno, 1996). Dilution based tests allow for high sample throughput and are therefore potentially advantageous for large laboratories, and at high sample throughput automated microbroth dilution is only ∼20% more expensive than disk diffusion. However, disk diffusion is clearly appealing for smaller laboratories with low sample throughput. In fact, because of the low investment costs, disk diffusion is feasible for small surveillance or hospital laboratories and, for the latter, may permit in-house susceptibility screening. Collins et al. (2007) analyzed the economic costs associated with different susceptibility testing regimens for Candida glabrata fungemia and found that in-house testing represented a clear cost advantage compared to sending out tests or not testing isolates at all, mainly due to reductions in treatment costs. Disk diffusion might therefore offer additional advantages over microbroth dilution in certain situations, and the most adequate test should be selected based on scientific considerations and the individual laboratory requirements. Given the lower cost and high degree of comparability between results, disk diffusion may, for instance, be an alternative to microbroth dilution for surveillance studies.