Evaluation of Immunoassays and General Biological Indicator Tests for Field Screening of Bacillus anthracis and Ricin

Abstract

There is little published data on the performance of biological indicator tests and immunoassays that could be used by first responders to determine if a suspicious powder contains a potential biothreat agent. We evaluated a range of biological indicator tests, including 3 protein tests, 2 ATP tests, 1 DNA test, and 1 FTIR spectroscopy instrument for their ability to screen suspicious powders for Bacillus anthracis (B. anthracis) spores and ricin. We also evaluated 12 immunoassays (mostly lateral flow immunoassays) for their ability to screen for B. anthracis and ricin. We used a cost-effective, statistically based test plan that allows instruments to be evaluated at performance levels ranging from 0.85 to 0.95 lower confidence bound of the probability of detection at confidence levels of 80% to 95%. We also assessed interference with 22 common suspicious powders encountered in the field. The detection reproducibility for the biological indicators was evaluated at 10⁸ B. anthracis spores and 62.5 μg ricin, and the immunoassay detection reproducibility was evaluated at 10⁷ spores/mL (B. anthracis) and 0.1 μg/mL (ricin). Seven out of 12 immunoassays met our most stringent criteria for B. anthracis detection, while 9 out of 12 met our most stringent test criteria for ricin detection. Most of the immunoassays also detected ricin in 3 different crude castor seed preparations. Our testing results varied across products and sample preparations, indicating the importance of reviewing performance data for specific instruments and sample types of interest for the application in order to make informed decisions regarding the selection of biodetection equipment for field use.

The authors evaluated a range of biological indicator tests, including 3 protein tests, 2 ATP tests, 1 DNA test, and 1 FTIR spectroscopy instrument for their ability to screen suspicious powders for Bacillus anthracis spores and ricin.

Rapidly screening and identifying potential biothreat agents in suspicious samples is a challenge. Numerous biodetection products are commercially available for screening samples for various types of biothreat agents. However, very little test data are available to better understand their performance and guide procurements by end users. The goal of this testing is to address this need by evaluating the ability of currently available commercial off-the-shelf biological indicator tests and immunoassays to detect Bacillus anthracis (B. anthracis) spores and ricin. In general, immunoassays provide more specific identification of biological threats as compared to indicator tests.¹ Many of these detection products are widely used by first responders and other end users. In most cases, the scant performance data for these assays that do exist are supplied directly from the manufacturer, and the data have not been verified by an external, independent assessment.² Our test plan modules included assessments of inclusivity (ability to generate true-positive and not false-negative results), commonly encountered suspicious powders (which can cause potential interferences or false-positives), and estimation of the limit of detection (ie, sensitivity).

Our test plan was guided by published standard method performance requirements from AOAC, which require detection with an estimated 5% lower confidence limit on the probability of detection of 0.95 or higher.^3-7 Our plan was also guided by an ASTM standard specification for handheld point chemical vapor detectors, which requires that detectors achieve a probability of detection of at least 0.85% as specified by an 80% lower confidence bound.⁸

Our plan enables an efficient and cost-effective approach to establishing performance levels of various biodetection products. Rather than simply measuring performance in a pass/fail fashion, we characterized the performance of the assays within a performance range between the ASTM published standard (0.85 probability of detection [POD]/80% confidence level [CL]) and the AOAC standards (0.95 POD/95% CL). While our statistical approach differed slightly from the AOAC and ASTM standards, it was comparable in rigor. Our criteria employ a probability of detection as demonstrated by a lower confidence bound of 0.95 (minimum 0.85) of a one-sided 95% (minimum 80%) confidence interval. We fixed the maximum number of test sample replicates at 79. If an assay failed 10 times, performance of that product fell below the lowest acceptable performance criteria (0.85 POD/80% CL) and the testing was halted. The range of lower confidence bound and confidence level values can be found in Table 1. Further details on our statistical approach can be found in a companion article in this journal issue.⁹

Table 1.

Products were tested to the range of LCB and CL shown in the table below. The highest criterion was 0.95 LCB/95% CL, while the minimum criterion was 0.85 LCB/80% CL.

Confidence Level (CL)	95%	90%	85%	80%
Number of Failed Tests	Lower Confidence Bound (LCB) Values
1	0.95	0.96	0.97	0.97
2	0.93	0.94	0.95	0.96
3	0.91	0.92	0.93	0.94
4	0.89	0.91	0.92	0.92
5	0.88	0.89	0.90	0.91
6	0.86	0.88	0.89	0.90
7	0.84	0.86	0.87	0.88
8	0.83	0.85	0.86	0.87
9	0.81	0.83	0.84	0.85
10	0.80	0.82	0.83	0.84

Total number of tests = 79.

Our inclusivity testing included a single biosafety level 2 (BSL-2) strain of B. anthracis, the Ames35 strain that lacks the pXO2 virulence plasmid. Because of the specific application we addressed (field screening of suspicious visible powders by first responders to support tactical decision making and public/responder safety actions), near-neighbor exclusivity organisms were not deemed to be of value to test because the most common material encountered that could generate false-positive results are suspicious powders and not near-neighbor organisms. We also did not conduct any exclusivity testing with ricin for the same reasons. We did conduct testing using a limited numbers of replicates (6) with 3 different preparations of crude castor bean mash to simulate the types of real-world samples that might be encountered by first responders. Prior to B. anthracis inclusivity testing, we evaluated the impact of strain type and growth media used to generate B. anthracis test material and found significant impacts on immunoassay sensitivity for these 2 variables (data not shown). This article summarizes our evaluation of 28 different general biological indicator and immunoassay products for the detection of B. anthracis and ricin.

Materials and Methods

Bacillus anthracis

Four B. anthracis strains were obtained from BEI Resources (Manassas, VA): Ames35 (NR-10355), Sterne 34F2 (NR-1400), Weybridge (NR-10350), and UM23 (NR-10351). Each strain was grown in identical conditions, as previously described.¹⁰ Briefly, overnight cultures were grown at 37°C in tryptic soy broth, then diluted 1:100 in 1.6% nutrient broth with CCY salts (a commonly used mixture of salts that are added to bacterial growth media) and grown at the appropriate temperature with shaking at 200 rpm. B. anthracis spores were harvested after 72 hours by centrifugation for 10 min at 10,000 × g at 4°C. Spore pellets were resuspended in sterile water and stored at 4°C for 7 days to enhance vegetative cell lysis. Spores were then washed 3 times in sterile water prior to use. Spore purity was evaluated using phase contrast microscopy, and all preparations were >95% phase bright, indicating a pure preparation. Four types of growth media were tested to assess the impact of media on limit of detection: (1) Nutrient Broth (NB; Becton Dickinson, Franklin Lakes, NJ) with CCY salts (Sigma Aldrich, St. Louis, MO), (2) New Sporulation Medium (NSM; Sigma Aldrich, St. Louis, MO), (3) Blood Agar (Becton Dickinson, Franklin Lakes, NJ), and (4) Brain Heart Infusion (BHI) agar (Teknova Inc, Hollister, CA). Initial testing indicated the Ames35 strain was the most consistently detected at lower concentrations (see below). Therefore, this strain was chosen for all subsequent B. anthracis testing and was stored at 4°C as a stock suspension of 10⁸ spores/mL.

On the day of testing, the stock was vortex mixed for 2 minutes immediately prior to making dilutions. Dilutions were made using manufacturer-supplied buffer or ultrapure water if not supplied. Serially diluted samples were vortex mixed in between every dilution to maintain spore suspension. B. anthracis test sample concentrations were verified each day of testing by serially diluting an aliquot of the standard test solutions before and after testing to ∼10³ spores/mL. The aliquot was cultured on agar plates in triplicate, and the resulting colonies were enumerated the next day after overnight growth at 37°C. Limit of detection for the biological indicator tests was determined using the following amounts of B. anthracis spores per test: 10⁸, 10⁷, 10⁶, 10⁵, and 10⁴. These amounts of spores were obtained by using a 50-μL test volume and B. anthracis concentrations ranging from 2 × 10⁹ spores/mL to 2 × 10⁵ spores/mL.

Ricin

Pure ricin toxin was obtained from Vector Laboratories (Burlingame, CA) as a 5-mg/mL stock solution and stored at 4°C. On the day of testing, dilutions were made from the stock using manufacturer-supplied buffer or ultrapure water if buffer was not indicated, and stored on ice prior to and during testing.

In addition to purified ricin, 3 crude ricin preparations were made at PNNL by (1) crushing castor seeds to produce a mash, (2) acetone extraction of oil from mash, and (3) acetone precipitation from seed mash.¹¹ Following are brief summaries of each of the ricin preparation methods.

• Crushed seed mash: Soak seeds in NaOH, rinse in cold water, then air dry and peel the seed hulls. Mash the peeled seeds with a mortar and pestle.

• Acetone extraction: Prepare mash as above, then cover with acetone in a flask, and stir for 72 hours. Filter through a double layer of filter paper, dry, and scrape the dried material into a vial.

• Acetone precipitation: Prepare 5 grams of castor seeds as for the crushed seed mash, but add 10 mL phosphate-buffered saline when grinding with a pestle. Centrifuge the resulting slurry at 10,000 × g for 20 min at 4°C, which results in 3 layers. Recover the middle (aqueous) layer. Add acetone to approximately 6 times the aqueous volume and freeze overnight. Centrifuge the next day at 10,000 × g for 20 min, decant the acetone, and air dry the precipitate.

Crude ricin preparations were stored at −80°C. On the day of testing, dilutions were made using manufacturer-supplied buffer or ultrapure water. To ensure stability, all samples were stored on ice prior to and during testing.

Suspicious Powders

Twenty-two powders were selected that are often encountered by first responders during suspected bioterror events and used to assess the assays (Table 2). Many of these powders are listed in the Environmental Factors Panel in AOAC standards.^4,7 The powders were purchased from Amazon, Target, and a local garden store. They were categorized into 4 groups based on composition: organic biological, organic protein containing, organic no protein, and inorganic (Table 2). Powders were tested at a concentration of 1 mg/mL for biological indicator tests. The PROFILE^® 1 ATP and immunoassays were tested at 0.1 mg/mL.

Table 2.

Commonly encountered suspicious powders analyzed using biological indicator tests and immunoassays

Powder Type	Powder
Organic, biological	Brewer's yeast powder
	Dipel dust
Organic, protein-containing	Milk powder
	Infant formula
	White flour
Organic, no protein	Coffee creamer
	Instant pectin
	Acetaminophen
	Powdered sugar
	Cornstarch
	MiraLAX (Polyethylene glycol 3350)
Inorganic	Toothpaste powder
	Baking powder
	Antacid
	Baking soda
	Epsom salt (magnesium sulfate)
	Gym chalk
	Borax
	Talc
	Road dust
	Kaolin
	Popcorn salt

General Biological Indicator Tests

Brief descriptions of the 7 biological indicator tests that we evaluated as part of this study are given below. We evaluated 3 protein test kits: (1) BioCheck^™ Powder Screening (20/20 Gene Systems^™, Rockville MD), (2) INDIPRO strips (Macherey-Nagel, Bethlehem, PA), and (3) TASKit Bioscreener^™ (Field Forensics, St. Petersburg, FL). We evaluated 2 ATP tests: (1) Clean-Trace^™ Surface ATP (3M^™, St. Paul, MN) and (2) PROFILE^® 1 ATP (New Horizons Diagnostics, Baltimore, MD). We evaluated 1 DNA test, the Prime Alert^® system (GenPrime^™, Spokane, WA). We also evaluated 1 FTIR system, the HazMatID^™ 360 FTIR system (Smiths Detection, Edgewood, MD).

Immunoassay Tests

We tested both single-plex assays (1 target biothreat is detected per sample) and multiplex assays (2 or more biothreat agents are detected simultaneously per sample). Immunoassays require a liquid sample, so B. anthracis and powders were solubilized or suspended prior to testing. Some manufacturers offered optical readers, and, when available, we also used the optical readers (in addition to manually reading the assay results); we report the readout option used in the results tables. The immunoassays tested along with the volume of liquid applied are listed below:

• AdVnt (Phoenix, AZ)

○ BADD^™: single-plex assays for B. anthracis and ricin (200 μL/test)

○ Pro Strips^™: 5-plex assay includes ricin and B. anthracis (600 μL/test)

• Alexeter (Wheeling, IN)

○ BioDetect^™: single-plex assays for B. anthracis and ricin + optional reader (200 μL/test)

○ RAID^™ 5: 5-plex assay includes ricin and B. anthracis (400 μL/test)

○ RAID^™ 8: 8-plex assay includes ricin and B. anthracis (800 μL/test)

• ANP Technologies^® (Newark, DE)

○ NIDS^® 3-Plex 3: 3-plex assay includes B. anthracis + optional reader (120 μL/test)

○ NIDS^® 4-Plex 5: 4-plex assay includes ricin + optional reader (120 μL/test)

• Tetracore^® (Rockville, MD)

○ BioThreat Alert^®: single-plex assays for B. anthracis and ricin + optional reader (150 μL/test)

• Pathsensors (Baltimore, MD)

○ CANARY^® Zephyr: single-plex assay for B. anthracis + required measurement platform; no manual read option (200 μL/test)

• Environics (Tolland, CT)

○ ENVI: B. anthracis and ricin single-plex assays + optional reader (120 μL/test)

• BBI Detection (Madison, WI)

○ IMASS^™: 8-plex assay includes ricin and B. anthracis (2.5 mL/test)

• Response Biomedical (Vancouver, BC, Canada)

○ RAMP^®: single-plex assays for B. anthracis and ricin + required reader (no manual read option) (70 μL/test)

• New Horizons Diagnostics (Baltimore, MD)

○ SMART^™-II: single-plex assays for B. anthracis and ricin (100 μL/test)

• GenPrime^™ (Spokane, WA)

○ Toxin Screen: 3-plex assay includes ricin (250 μL/test)

All of the above technologies are based on visible dyes using a lateral flow assay format, except the CANARY^® Zephyr and the RAMP^®. The CANARY uses an engineered B-cell format and requires a suitcase-sized platform, which contains a laptop, small centrifuge, and luminescence optical reader. The RAMP^® uses fluorescence detection, rather than visible dyes, so a fluorescence optical reader is required.

Results and Discussion

General Biological Indicator Tests

General biological tests detect a broad range of biological and organic materials, but do not confirm the presence of a specific biothreat agent. In general, biological indicator tests have low specificity (ie, potential for a false-positive result), and many have low sensitivity (ie, potential for a false-negative result). Therefore, they are best suited to be used as a screening tool when sufficient material is present and in conjunction with more specific tests. Indicator tests are often designed to measure a solid sample directly, though most can also measure a liquid sample.

Protein tests are based on color-changing indicator dyes and are read by eye. These tests detect any type of protein or amino acid, which are present in all cells (eg, anthrax, human cells, and nonpathogenic bacteria) and also are present in some common consumer products (eg, coffee creamer and powdered infant formula). Some kits include separate tests for pH or starch to further help identify the sample type.

Adenosine triphosphate (ATP) tests can determine if any type of cellular material is present and alive, since ATP is a cellular metabolite present in all living cells. The amount of ATP in a sample is typically proportional to the number of living cells. Because ricin is a protein, it is not detected by this method unless residual live cells from the castor plant (Ricinus communis) associated with ricin are present. These tests require an optical reader.

Deoxyribonucleic acid (DNA) tests detect any type of DNA (eg, human, plant, or animal) and some types of RNA using an optical reader that measures the fluorescence of a dye that binds only to nucleic acids.

Fourier transform infrared (FTIR) spectroscopy is primarily used to rapidly identify the chemical composition of an unknown sample. Proteins (contained in most biological materials) give a unique FTIR spectrum that should be detected in samples if they have a protein content of at least 10%. Identification of a sample is based on comparison of the sample's composite spectrum to a library of known individual component spectra.

Immunoassays

Immunoassays use antibodies, which are proteins designed to bind to a specific agent such as anthrax or ricin. Most immunoassays use a lateral flow assay format similar to a home pregnancy test. A lateral flow assay includes an assay strip containing all the assay components encased in a plastic cartridge. The cartridge has a sample window where the sample is applied to the assay strip and a results window where the results are read manually (with your eye) or using an electronic optical reader. Depending on the specificity of the immunoassay, false-positives can occur if closely related agents are present (like nonpathogenic relatives of B. anthracis).

Limit of Detection Estimation

The limit of detection can be determined in various ways and has a wide range of formal definitions;^12,13 here we simply are using the term to indicate the lowest concentration that consistently produces a positive result (in 3 out of 3 replicates). We performed testing to estimate the limit of detection for general biological indicators and immunoassays with B. anthracis and ricin. This testing was not intended to precisely determine the limit of detection in a statistically rigorous fashion, but rather to provide an estimate to guide selection of a concentration level for inclusivity testing (which should be done at a concentration above the limit of detection). Five concentrations over 5 orders of magnitude were tested in triplicate. If 3 of 3 results were positive at the lowest concentration, successively lower (10-fold) concentrations were tested in triplicate. Because of the 10-fold difference in test sample concentrations, it is likely that the sensitivity of the assay actually lies between the lowest concentration that consistently generated positive results and the next lowest concentration tested.

Impact of Spore Strain Type and Growth Conditions on Immunoassay Limit of Detection

Because initial limit of detection results were several orders of magnitude higher for multiple products than manufacturer claims, we examined several variables that may have affected our results. These variables included strain type and media conditions.

B. anthracis Strain Testing

Four B. anthracis strains were obtained from BEI Resources: Ames35 (NR-10355), Sterne 34F2 (NR-1400), Weybridge (NR-10350), and UM23 (NR-10351). Each strain was grown in identical conditions (in New Sporulation Medium), washed 5 times, and stored wet. We consulted ASTM E2800-11 for guidance on B. anthracis spore characterization and storage.¹⁴ One immunoassay product manufacturer communicated that their testing indicated that the limit of detection was affected by dry (higher LOD) versus wet B. anthracis storage, and the time after initial suspension of B. anthracis (ie, longer wet storage times) also resulted in higher limit of detection (data not shown); therefore, it is possible that storage condition and storage time will affect the estimated limit of detection.

Testing was conducted with 3 of the BEI strains (UM23 did not sporulate and was not used). Three different manufacturer immunoassays were used that had been compared in a previously published study.¹⁵ Although this prior study used only 2 to 6 replicates for each test concentration and did not identify the B. anthracis growth medium or whether spore washing was performed, it did provide some independent comparative test data for the immunoassay spore limit of detection. Our limit of detection testing was done in triplicate starting at 10⁷ spores/mL. If 10⁷ spores/mL was negative, only then was 10⁸ spores/mL tested. Successively lower concentrations (10-fold dilutions) were then tested until negative results were obtained on all 3 replicates. The diluted spore samples were also plated and cultured in triplicate to verify the concentration of spores that were actually applied to the test strips. Phase contrast microscopy was performed on the spore preparations to verify purity and that minimal spore clumping was occurring. A summary of the lowest concentration detected (at least 1 of 3 replicates) for the BADD^™, Smart^™-II, and BioThreat Alert^® immunoassays using the 3 B. anthracis strains tested is given in Table 3. Based on the results (lowest LOD for the greatest number of products) and the ease of obtaining the strain, B. anthracis Ames35 was chosen for all subsequent B. anthracis testing. It should be noted that all of the B. anthracis strains we tested are BSL-2 strains and are not as virulent as BSL-3 strains because they are lacking 1 or both of the virulence plasmids. While it is unlikely a BSL-2 strain would be used in a terror attack, working with BSL-2 strains is far less costly, and these strains are typically used by other researchers when testing immunoassay products. As our initial data show, the sensitivity of the products may be different depending on the B. anthracis strain.

Table 3.

LOD estimates for immunoassay B. anthracis spore detection using 3 different strains of B. anthracis spores, all grown in New Sporulation Medium, using 3 different immunoassays. Ames35 was chosen for all subsequent B. anthracis LOD and inclusivity testing.

	B. anthracis Strain
Immunoassay Product	34F2	Ames35	Weybridge
BADD^™	10⁸ (3/3)	10⁷ (3/3)	>10⁸
Smart^™-II	10⁷ (3/3)	10⁷ (3/3)	10⁷ (1/3); 10⁸ (0/3)^a
BioThreat Alert^®	10⁸ (2/3)	10⁸ (3/3)	10⁸ (1/3)

Inconsistent results are not uncommon at concentrations near the LOD.

B. anthracis Growth Media Testing

As another quality check and to assess the influence of growth media, we tested the impact of different growth media used to generate the B. anthracis test samples on the immunoassay limits of detection for B. anthracis Ames35. Three different immunoassay products (the same products that were used to test the difference in LOD for B. anthracis strains) were used in triplicate for B. anthracis.¹⁵ Test sample concentrations ranged from 10⁵ to 10⁹ spores/mL.

While there was significant dependence of B. anthracis limit of detection on the type of growth media (up to 3 orders of magnitude), the results did not show that one growth medium resulted in lower limits of detection for multiple products. The results summarized below are for instances where 3 of 3 replicates were positive.

• BADD^™ detected 10⁷ spores/mL in NSM, and 10⁸ spores/mL in BHI, blood agar, and NB with CCY salts;

• Smart^™-II detected 10⁶ spores/mL in NB with CCY salts, 10⁸ spores/mL in blood agar, and was unable to detect 10⁸ spores/mL in BHI or NSM; and

• BioThreat Alert^® detected 10⁸ spores/mL in blood agar and NB with CCY salts, and was unable to detect 10⁸ spores/mL in BHI or NSM.

We ultimately chose to use NB with CCY salts based on the good results across platforms with this medium, as well as good results using NB with CCY salts reported in the literature.¹⁰

General Biological Indicator Tests Limit of Detection Estimation

A small, concentrated liquid sample (50 μL) was used for B. anthracis, pure ricin toxin, and powder testing rather than a solid sample to enable the application of a precise amount of test material. Therefore, the amount applied to the test is listed in spores or μg, not a concentration. A 2 × 10⁹ spores/mL stock suspension was used to prepare serially diluted B. anthracis standards. For pure ricin, the stock 5 mg/mL solution was used as received from the manufacturer.

Bacillus anthracis

B. anthracis was tested using the following amounts per test: 10⁸, 10⁷, 10⁶, 10⁵, and 10⁴. These amounts of spores were obtained by using a 50-μL test volume and B. anthracis concentrations ranging from 2 × 10⁹ spores/mL to 2 × 10⁵ spores/mL. The PROFILE^® 1 ATP test, which was designed for screening suspicious powders and includes an incubation step to germinate spores and increase their number, was the most sensitive of the biological indicator tests, with an estimated limit of detection of 10⁵ spores, although this test required multiple manual manipulation steps, making it more cumbersome and time-consuming to perform than the other biological indicator or immunoassay tests. The BioCheck^™ and INDIPRO protein tests could both detect 10⁸ spores, which was the inclusivity test concentration for biological indicator tests. In addition to the commercially available BioCheck^™ test kit, a prototype variant of the BioCheck^™ test kit that included a bead-beating step to disrupt the spores was determined to have an estimated limit of detection of 10⁷ spores. The TASKit^™ BioScreener^™ protein test was able to detect 10⁹ spores. The Clean-Trace^™ ATP test, which was not designed for this application but rather for verifying the cleanliness of food contact surfaces without any incubation step, was unable to detect 10⁹ spores. The Prime Alert^® DNA test also could not detect 10⁹ spores. Our compiled estimated limit of detection test data for B. anthracis is shown in Table 4, along with manufacturer limit of detection claims (where available) and limit of detection data from a peer-reviewed literature publication.¹⁶ Poore et al found similar limits of detection as we did for the biological indicator tests that were assessed and also showed that increased spore washing can raise the limit of detection for protein and DNA tests by more than an order of magnitude.¹⁶

Table 4.

LOD estimates for B. anthracis spore and ricin detection using general biological indicator tests

		Protein Tests			ATP Tests		DNA Test
Sample Subtype	Information Source	BioCheck^™	TASKit^™ BioScreener^™	INDIPRO Strips	Clean-Trace^™ Surface ATP	PROFILE^® 1 ATP	Prime Alert^®
Ba Ames35 (# spores)	PNNL	10^8a	>10⁹	10⁸	>10⁹	10⁵	10⁹
Pure ricin (μg)	PNNL	62.5	625	62.5	N/A	N/A	N/A
Ba spores or protein^b	Vendor	100 μg protein	NR	50 μg protein	NR	10⁴ spores	10⁶ spores
Ba (# spores)	Poore 2009¹⁶	10⁸ (4X wash)	N/A	N/A	N/A	10⁴ (2X and 4X wash)	2 × 10¹⁰ (4X wash)
		10⁷ (2X wash)					10⁹ (2X wash)
Pure ricin (μg)	Poore 2009¹⁶	100	N/A	N/A	N/A	N/A	N/A

A kit that included a bead-beating step to disrupt spores was also tested and achieved an estimated LOD of 10⁷ spores (not currently commercially available).

100 μg is ∼10⁸ spores.

NP = Not performed.

N/A = Not applicable.

Ricin

Pure ricin toxin was tested at the following concentrations: 250 μg, 62.5 μg, 15.6 μg, 3.9 μg, and 0.98 μg. Only the protein tests were evaluated with pure ricin toxin, since no DNA or ATP is expected to be present in pure ricin. The BioCheck^™, INDIPRO, and TASKit^™ BioScreener^™ protein tests all had an estimated limit of detection of 62.5 μg of ricin toxin, which was the inclusivity test concentration that we used for biological indicator tests. The ricin limit of detection test data are also shown in Table 4.

Immunoassay LOD Estimation

The test sample volumes applied to each product varied from 70 μL/test to 2.5 mL/test, depending on manufacturer instructions. (Volumes are listed in the Materials and Methods section.) A B. anthracis stock concentration of 10⁸ spores/mL was used to prepare 10-fold serially diluted standards. For pure ricin, the stock 5 mg/mL solution was used as received from the manufacturer. Some immunoassays had optional optical readers, in which case both visual (manually read by eye) and optical reader results were reported. The RAMP^® and CANARY^® Zephyr both required optical readers and could not be read manually. Based on our studies showing up to 3 orders of magnitude impact to limit of detection depending on the type of B. anthracis growth medium and strain, it is not surprising that limit of detection values reported by manufacturers and in the literature vary widely. However, our limit of detection results are in much closer agreement with published literature values compared to manufacturer-reported specifications. Manufacturer-reported limit of detection values were typically 2 to 3 orders of magnitude lower than we found for B. anthracis and up to 2 orders of magnitude lower for ricin. The compiled test data, manufacturer limit of detection claims, and other published data are shown in Tables 5A and 5B.

Table 5A.

LOD estimates for B. anthracis spore and ricin detection using immunoassays

		BADD^™		Pro Strips^™	BioDetect^™ (Reader & Manual)		RAID^™ 5	RAID^™ 8	NIDS^® 3 (Ba) (Reader & Manual)	NIDS^® 4 (Ricin) (Reader & Manual)
Sample Subtype	Information Source	B. anthracis	Ricin	Multiplex	B. anthracis	Ricin	Multiplex	Multiplex	Multiplex	Multiplex
B. anthracis Ames35 (spores/mL)	PNNL	10⁷	N/A	>10⁸	>10⁸ (R, M)	N/A	>10⁸	>10⁸	10⁸ (R)^a 10⁷ (M)	N/A
Pure ricin (μg/mL)	PNNL	N/A	0.4	0.1	N/A	0.025 (R) 0.1 (M)	0.1	1.6^b	N/A	0.025 (R, M)
B. anthracis (spores/mL)	Manufacturer	1.5-8.3 × 10⁴	N/A	1.5-8.3 × 10⁴	1-5 × 10⁶	N/A	1 × 10⁵	1 × 10⁵	NR	N/A
Pure ricin (μg/mL)	Manufacturer	N/A	0.01	0.01	N/A	0.005-0.01	0.006	0.006	N/A	NR
B. anthracis UD III-7 (spores/mL)	Zasada 2015¹⁷	>10⁸	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
B. anthracis (spores/mL)	Peckham 2013¹⁸	5 × 10⁶	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
B. anthracis (spores/mL)	King 2003¹⁵	5 × 10^5c	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A
Pure ricin (μg/mL)	Slotved 2014¹⁹	N/A	>0.02	>0.02	N/A	N/A	>0.02	>0.02	N/A	N/A

2/3 positive at 10⁷ spores/mL with reader.

2/3 positive at 0.1 μg/mL; 1/3 positive at 0.4 μg/mL (inconsistent results are not uncommon at concentrations near the LOD).

Value shown in table has been modified from published value (# spores) to reflect actual concentration tested.

N/A = not applicable.

NR = not reported by manufacturer.

R = reader; M = manual.

Table 5B.

LOD estimates for B. anthracis spore and ricin detection using immunoassays

		IMASS^™ (Multiplex)		ENVI (Reader & Manual)		Toxin Screen	Smart^™-II		CANARY^® Zephyr		RAMP^®		BioThreat Alert^® (Reader & Manual)
Sample Subtype	Information Source	Ba	Ricin	Ba	Ricin	Ricin	Ba	Ricin	Ba	Ricin	Ba	Ricin	Ba	Ricin
B. anthracis Ames35 (spores/mL)	PNNL	10⁷	N/A	10⁶ (R, M)	N/A	N/A	10^7a	N/A	10³	N/A	10⁷	N/A	10⁸ (R, M)	N/A
Pure ricin (μg/mL)	PNNL	N/A	0.025	N/A	0.1 (R) 0.025 (M)	1.6	N/A	0.025	N/A	0.003	N/A	1.6^b	N/A	0.006 (R) 0.025 (M)
B. anthracis (spores/mL)	Manufacturer	10⁴	N/A	10⁵	N/A	N/A	10⁵	N/A	100-500	N/A	6 × 10⁵	N/A	10⁴-10⁶	N/A
Pure ricin (μg/mL)	Manufacturer	N/A	0.001	N/A	0.005	0.4	N/A	0.01	N/A	0.0005	N/A	0.1	N/A	0.002-0.005
B. anthracis UD III-7 (spores/mL)	Zasada 2015¹⁷	N/A	N/A	N/A	N/A	N/A	10⁸	N/A	N/A	N/A	N/A	N/A	10⁸	N/A
B. anthracis (spores/mL)	Peckham 2013¹⁸	N/A	N/A	N/A	N/A	N/A	5 × 10⁶	N/A	N/A	N/A	N/A	N/A	5 × 10⁶	N/A
B. anthracis (spores/mL)	King 2003¹⁵	N/A	N/A	N/A	N/A	N/A	10^6c	N/A	N/A	N/A	N/A	N/A	7 × 10^6c	N/A
B. anthracis (spores/mL)	Hoile 2007²⁰	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	6 × 10⁵	N/A	N/A	N/A
Pure ricin (μg/mL)	Slotved 2014¹⁹	N/A	0.010	N/A	0.01-0.02	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	0.005
Pure ricin (μg/mL)	Hodge 2013²¹	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	0.004

While LOD was estimated to be 10⁶ spores/mL during growth media testing, during B. anthracis strain testing and later actual LOD testing the LOD was found to be 10⁷ spores/mL.

2/3 positive at 0.4 μg/mL.

Value shown in table has been modified from published value (# spores) to reflect actual concentration tested.

N/A = not applicable.

NR = not reported by manufacturer.

R = reader; M = manual.

Bacillus anthracis

Immunoassay test sample concentrations ranged from 10⁸ to 10⁴ spores/mL. If an assay result was still positive at 10⁴ spores/mL, successively lower concentrations were tested in triplicate (ie, 10³_, 10² spores/mL). If an assay failed to give 3 positive results at 10⁸ spores, then the limit of detection was reported as >10⁸ spores. The CANARY^® Zephyr achieved an estimated limit of detection of 10³ spores/mL, which was 4 orders of magnitude lower than any other immunoassay tested. Of the other 11 immunoassay products, 6 had limit of detection estimates of <10⁷ spores/mL (the inclusivity test concentration for immunoassays), while the remaining 5 products had estimated limits of detection of 10⁸ or higher.

Ricin

Each of the immunoassays was also tested with pure ricin toxin at concentrations ranging from 1,600 to 6.25 ng/mL using 4-fold serial dilutions. If a test result was still positive at 6.25 ng/mL, successively lower concentrations were tested in triplicate. The CANARY^® Zephyr achieved the lowest limit of detection (3 ng/mL). Six products had estimated limits of detection of 25 ng/mL, while 4 of the 13 ricin immunoassays had estimated limits of detection >0.1 μg/mL, which was the inclusivity test concentration for immunoassays.

Inclusivity Testing

Immunoassays and biological indicators were tested with B. anthracis, pure ricin toxin, and common powders using the PNNL-developed test plan to determine if a given assay met the lower confidence bound of the probability of detection of 0.95 at the test concentration with 95% confidence, or fell within the range of performance of 0.95 LCB/95% CL to 0.85 LCB/80% CL. To achieve this, 47 samples had to be tested with no failures, or up to 79 samples were tested if between 1 and 9 failures occurred during the course of 79 tests. If there were 10 failures during any point in the testing, testing was halted for a given product. Because we tested 22 different suspicious powders, we chose to test each powder in triplicate (66 samples) at a minimum. If there were any failures (up to 10 failures allowed before testing was stopped), 79 samples were tested. In addition, we conducted limited replicate testing of 3 different crude ricin preparation test samples, which allowed demonstration of a 0.85 LCB/80% CL for that product if no failures resulted.

The exception to this testing plan was the general biological indicator testing of powders. We tested each of the 22 powders in triplicate with each of the indicator tests for a total of 66 tests per product to characterize the biological indicator test performance (ie, to understand which common powders generated positive vs negative results), rather than evaluating the products using a more statistically rigorous test plan that specifies confidence bounds and levels. As an example, because some of these products test for protein, and some of the powders (eg, milk powder) contain protein, a “positive” result is expected. However, although the kits are performing as intended, the outcome in those cases is essentially a false alarm for first responders.

For inclusivity testing, we chose to use test concentrations that were appropriate for visible powder screening and for the 2 technology “families” (general biological indicator tests and immunoassays). We also considered the sensitivity of the technology based on results of the limit of detection estimate study, manufacturer claims, and published literature. We chose inclusivity test concentrations of 10⁷ spores/mL for B. anthracis and 100 ng/mL for pure ricin toxin for immunoassays. We arrived at these values by considering the following information and possible scenario:

• 1-10 mg is the typical amount of residual powder remaining for field screening assays after the bulk sample is collected for Federal Bureau of Investigation (FBI) or Laboratory Response Network (LRN) use per ASTM E2458-10 (data not shown).²²

• 10⁹ pure spores weigh approximately 1 mg.²³

• Assume that a benign suspicious powder is “spiked” to contain only 1% by weight of a biothreat agent (∼10 μg of biothreat agent in 1 mg of powder).

• Assume that only 1 mg of powder (containing 1% biothreat agent) is collected, suspended in 1 mL, and used for field screening.

• The resulting sample would have a concentration of 10⁷ spores/mL.

In the case of ricin we make similar assumptions, except a crude ricin mash is spiked into a benign suspicious powder, which would equate to ∼100 ng of active ricin in 1 mg of powder. Crude ricin preparations contain low percentage amounts of active ricin, and the ricin content can vary based on the castor cultivar and preparation method. As a general guide, crushed seed mash contains approximately 1% active ricin, acetone extraction contains approximately 2% active ricin, and acetone precipitation contains approximately 4% active ricin. As a result, crude ricin preparations were tested using much greater amounts than pure ricin. Ten mg/mL crude ricin was used for biological indicator tests, while 0.1 mg/mL was used for immunoassays. Pure ricin was used to estimate the limit of detection and for the inclusivity studies. The scoping studies using crude ricin preparation included a limited number of replicates of each preparation, to obtain general information about the ability of the platforms and tests to detect of the types of ricin samples that could be encountered in the field.

For biological indicators, we tested quantities of material (rather than concentrations) that were determined based on the results of the limit of detection estimate studies. These levels (10⁸ spores [∼100 μg] or 62.5 μg pure ricin toxin) are still reasonable amounts of material to be expected for a visible suspicious powder.

Inclusivity Testing: General Biological Indicator Tests

No spore testing or pure ricin testing was performed with the HazMatID^™ 360 FTIR. The 2 ATP tests and the DNA test were not tested with pure ricin because these components (ATP and DNA) are not expected to be present in pure ricin. The FTIR system was tested only with dry suspicious powders and crude ricin preparations, and ∼1 mg of material was used, which was just enough to cover the sampling optics. Summaries of the general biological indicator inclusivity data are shown in Tables 6 and 7.

Table 6.

Inclusivity results for general biological indictor B. anthracis spore and ricin detection. Unshaded cells met the 0.95 LCB/95% CL criteria. Shaded cells did not meet 0.85 LCB/80% CL minimum performance criteria. Data are reported as number of positive results/total tests performed.

	Protein Tests			ATP Tests		DNA Test
Sample Subtype	BioCheck^®	TASKit^™ BioScreener^™	INDIPRO Strips	Clean-Trace^™ Surface ATP	PROFILE^® 1 ATP	Prime Alert^®
B. anthracis Ames35 (10⁸ spores)	47/47	1/11	47/47	0/10	47/47	0/10
Pure ricin (62.5 μg)	47/47	0/10	47/47	N/A	N/A	N/A

N/A = not applicable.

Table 7.

Limited screening of 3 crude ricin preparations using biological indicator tests. Unshaded cells met the minimum 0.85 LCB/80% CL performance criteria. Shaded cells did not meet this criterion. Data are reported as number of positive results/total tests performed.

	Protein Tests			ATP Tests		DNA Test	Chemical/ Spectroscopic
Sample Subtype	BioCheck^®	TASKit^™ BioScreener^™	INDIPRO Strips	Clean-Trace^™ Surface ATP	PROFILE^® 1 ATP	Prime Alert^®	HazMatID^™ 360 FTIR^a
AP: Native castor acetone protein precipitate (0.5 mg)	6/6	6/6	6/6	6/6	0/6	0/6	6/6
AE: Native castor acetone extract (0.5 mg)	6/6	6/6	6/6	5/6	0/6	0/6	6/6
CM: Native castor control mash (0.5 mg)	6/6	6/6	6/6	6/6	0/6	0/6	6/6

Approximately 1 mg of material was used for FTIR system testing, which was just enough material to cover the sampling optics.

Bacillus anthracis

A quantity of 10⁸ spores (50 μL of 2 × 10⁹ spores/mL) was used. The BioCheck^™ and INDIPRO strip protein tests and the Profile^® 1 ATP test achieved a 0.95 LCB/95% CL with no failures for 47 replicates of the B. anthracis test samples. The TASKit^™ BioScreener^™ protein test, Clean-Trace^™ ATP test, and Prime Alert^® DNA test all failed to meet the minimum criteria of 0.85 LCB/80% CL for B. anthracis test samples.

Ricin

The pure ricin toxin inclusivity test amount was 62.5 μg. Both the BioCheck^™ and the INDIPRO strip protein tests achieved a 0.95 LCB/95% CL with no failures in 47 replicates, while the TASKit^™ BioScreener^™ protein test failed to meet the minimum criteria of 0.85 LCB/80% CL.

A limited number of 3 different crude ricin preparations were analyzed using 50-μL aliquots of 10 mg/mL suspension. All 3 of the protein tests and the FTIR resulted in positive detections for 6 of 6 samples in each of the 3 crude ricin preparations and met the 0.85 LCB/80% CL criteria. The Clean-Trace^™ ATP test produced positive results for 6 of 6 replicates of acetone precipitation and crushed seed mash ricin preparations (achieving the 0.85 LCB/80% CL criteria) but did not meet these criteria for the acetone extraction preparations (only 5 of 6 replicates positive). The Profile^® 1 ATP test and the Prime Alert^® DNA test did not have any positive detections for any replicates of the 3 different ricin preparations. The results of the ricin testing for the general biological indicators are summarized in Table 7.

Inclusivity Testing: Immunoassays

Bacillus anthracis

Inclusivity testing was performed using a test concentration of 10⁷ spores/mL. Seven of 12 products met the 0.95 LCB/95% CL criteria, with only 1 product (Smart^™-II) requiring 79 tests due to 1 assay failure. The other 5 products failed the first 10 replicate assays and, therefore, did not meet the minimum criteria (0.85 LCB/80% CL). Data are shown in Table 8A.

Table 8.

Inclusivity results for B. anthracis spore (A) and ricin (B) detection using immunoassays. Unshaded cells met the 0.95 LCB/95% CL performance criteria. Shaded cells did not meet 0.85 LCB/80% CL minimum performance criteria. Data are reported as number of positive results/total tests performed.

A.
Sample Type	Sample Subtype	BADD™	CANARY® Zephyr	ENVI (Reader and Manual)	IMASS™ (Multi-plex)	NIDS® 3 Ba (Reader and Manual)	RAMP® (Reader and Manual)	Smart™-II	BioDetect™ (Reader and Manual)	BioThreat Alert® (Reader and Manual)	Pro Strips™ (Multi-plex)	RAID™ 5 (Multi-plex)	RAID™ 8 (Multi-plex)
Agent	Ba Ames35 (10⁷ spores/mL)	47/47	47/47	47/47 (R, M)	47/47	47/47 (R, M)	47/47 (R, M)	78/79	0/10 (R, M)	0/10 (R, M)	0/10	0/10	0/10

B.
Sample Type	Sample Subtype	BioDetect™ (Reader and Manual)	BioThreat Alert® (Reader and Manual)	ENVI (Reader and Manual)	IMASS™ (Multi-plex)	NIDS® 4 Ricin (Reader and Manual)	RAID™ 5 (Multi-plex)	RAID™ 8 (Multi-plex)	RAMP® (Reader and Manual)	Pro Strips™ (Multi-plex)	BADD™	Smart™ II	Toxin Screen
Toxin	Pure ricin (0.1 μg/mL)	47/47 (R, M)	47/47 (R, M)	47/47 (R, M)	47/47	47/47 (R, M)	47/47	47/47	47/47 (R, M)	47/47	0/10	25/35	0/10

R = reader; M = manual.

Ricin

Inclusivity testing was done using a pure ricin concentration of 0.1 μg/mL. Nine of 12 products met the 0.95 LCB/95% CL criteria and had no failures in the first 47 assays. The other 3 products failed to meet the minimum criteria of 0.85 LCB/80% CL. BADD^™ and Toxin Screen failed the first 10 replicate assays, while Smart^™-II failed 10 of 35 assays. Data are shown in Table 8B.

We also tested the immunoassays using a limited number of the 3 different crude ricin preparations at a concentration of 0.1 mg/mL. As shown in Table 9, all but 2 of the 12 products tested produced positive results for all 6 replicates of each of the 3 crude ricin preparations. The RAID^™ 8 and the Toxin Screen both produced 6 of 6 positive results for the acetone precipitation and acetone extraction samples, but for the crushed seed mash sample, the RAID^™ 8 generated 5 of 6 positive results and the Toxin Screen generated 4 of 6 positive results.

Table 9.

Limited screening of 3 preparations of crude ricin using immunoassays. Unshaded cells met the minimum 0.85 LCB/80% CL performance criteria. Shaded cells did not meet this criterion. Data are reported as number of positive results/total tests performed.

	BADD^™	BioDetect^™ (Reader & Manual)	BioThreat Alert^® (Reader & Manual)	RAMP^®	Smart^™-II	ENVI (Reader & Manual)	IMASS^™	NIDS^® 4 Ricin (Reader & Manual)	Pro Strips^™	RAID^™ 5	RAID^™ 8	Toxin Screen
Sample Subtype	Ricin	Ricin	Ricin	Ricin	Ricin	Ricin	Multiplex	Multiplex	Multiplex	Multiplex	Multiplex	Ricin
AP: Native castor acetone protein precipitate (0.1 mg/mL)	6/6	6/6 (R, M)	6/6 (R, M)	6/6	6/6	6/6 (R, M)	6/6	6/6 (R, M)	6/6	6/6	6/6	6/6
AE: Native castor acetone extract (0.1 mg/mL)	6/6	6/6 (R, M)	6/6 (R, M)	6/6	6/6	6/6 (R, M)	6/6	6/6 (R, M)	6/6	6/6	6/6	6/6
CM: Native castor seed mash (0.1 mg/mL)	6/6	6/6 (R, M)	6/6 (R, M)	6/6	6/6	6/6 (R, M)	6/6	6/6 (R, M)	6/6	6/6	5/6	4/6

R = reader; M = manual

Suspicious Powder Testing

While the assumption of equivalency for powders does not at first consideration seem to be valid, for relatively specific assays like immunoassays, suspicious powders can be considered a class of material that should not generate any false-positives (ie, a type of exclusivity panel). Assuming equivalency in a statistical sense allows the powder samples to be considered as a single group of test samples. Twenty-two powders were initially tested in triplicate to yield a total of 66 tests per immunoassay. If a false-positive result occurred in any of the initial 66 tests, then additional tests were conducted, up to a total of 79 tests per immunoassay, and the additional tests beyond 66 were selected at random. If more than 10 failures (false-positives) occurred, testing of the powders was halted.

Because general biological indicator tests have poor specificity, the powders cannot be grouped together for statistical considerations when testing these products. Nevertheless, triplicate tests were conducted for each of the 22 powders to provide general information about indicator test performance and limitations with these sample types.

Suspicious Powder Testing: General Biological Indicator Tests

The BioCheck^™ and TASKit^™ protein tests also include another test for pH and starch, respectively, with the goal of providing additional information to guide interpretation of the protein test result. While suspensions of biothreat agents are often a neutral pH, an acidic or basic result does not necessarily exclude the presence of a biothreat. And while a positive starch result is indicative of certain foodstuffs, it does not necessarily mean there is no biothreat agent present.

Most of the indicator tests generated positive results with powders, many of which are to be expected (eg, protein tests generating positive results for milk or flour products, ATP or DNA tests generating positive results for brewer's yeast). However, some powders with no known protein content and inorganic powders also generated positive results. While the number of “false” positive results ranged from 3 of 66 (Prime Alert^® DNA test) to 32 of 66 (INDIPRO Protein test), these results cannot be technically considered false-positives from an analytical perspective because many of them are detecting the component they are designed to detect. However, if a product generates a positive result for a suspicious powder it can be considered a false-positive from an operational perspective. The data are summarized in Table 10.

Table 10.

Biological indicator testing of commonly encountered suspicious powders. It is not appropriate to apply performance criteria to indicator powder tests due to their inherently nonspecific nature, but the results are informative to understand limitations when screening commonly encountered suspicious powders. Data are reported as number of positive results/total tests performed.

		BioCheck^™		TASKit^™ BioScreener^™
Class of Powder	Powder Type	Protein	pH	Protein	Starch	INDIPRO Strips	Clean-Trace^™ Surface ATP	PROFILE^® 1 ATP	Prime Alert^®	HazMatID^™ 360 FTIR
Organic, biological	brewer's yeast powder	3/3	3/3N	3/3	0/3	3/3	3/3	3/3	3/3	3/3
	Dipel dust	0/3	3/3N	0/3	0/3	0/3	0/3	3/3	0/3	0/3
Organic, protein-containing	milk powder	3/3	3/3A	3/3	0/3	3/3	3/3	3/3	0/3	3/3
	infant formula	3/3	3/3A	3/3	0/3	3/3	0/3	0/3	0/3	0/3
	white flour	3/3	3/3N	3/3	2/3 (1/3 I)	3/3	3/3	0/3	0/3	3/3
Organic, no protein	coffee creamer	0/3	3/3N	0/3	0/3	3/3	0/3	1/3	0/3	0/3
	instant pectin	0/3	3/3N	0/3	0/3	2/3	0/3	0/3	0/3	0/3
	acetaminophen	3/3	3/3N	3/3	0/3	0/3	0/3	0/3	0/3	0/3
	powdered sugar	0/3	3/3N	0/3	0/3	0/3	0/3	0/3	0/3	0/3
	cornstarch	0/3	3/3N	0/3	3/3	0/3	0/3	0/3	0/3	0/3
	polyethylene glycol 3300	0/3	3/3N	0/3	0/3	0/3	0/3	1/3	0/3	0/3
Inorganic	toothpaste powder	0/3	3/3B	0/3	0/3	3/3	0/3	2/3	0/3	0/3
	baking powder	0/3	3/3B	0/3	0/3	3/3	1/3	0/3	0/3	0/3
	antacid	0/3	1/3N 2/3B	0/3	0/3	3/3	0/3	0/3	0/3	0/3
	baking soda	0/3	3/3B	0/3	0/3	3/3	0/3	0/3	0/3	0/3
	Epsom salt	2/3	1/3N 2/3A	0/3	0/3	0/3	0/3	1/3	0/3	0/3
	gym chalk	1/3	3/3B	0/3	0/3	3/3	0/3	2/3	0/3	0/3
	Borax	0/3	3/3B	0/3	0/3	0/3	0/3	0/3	0/3	0/3
	talc	0/3	3/3N	0/3	0/3	0/3	0/3	1/3	0/3	0/3
	road dust	0/3	3/3N	0/3	0/3	0/3	0/3	1/3	0/3	0/3
	kaolin	0/3	3/3N	0/3	0/3	0/3	0/3	0/3	0/3	0/3
	popcorn salt	0/3	3/3N	0/3	0/3	0/3	0/3	0/3	0/3	0/3
Combined results		18/66	—	15/66	—	32/66	10/66	18/66	3/66	9/66

A = acid; B = base; N = neutral; I = indeterminate.

Suspicious Powder Testing: Immunoassays

No false-positive results were generated for any of the 8 single-plex B. anthracis immunoassay products, which all achieved 0.95 LCB/95% CL (Table 11A). Of the 8 single-plex ricin immunoassay products that were tested, 6 had no false-positives and achieved 0.95 LCB/95% CL. The ENVI ricin immunoassay generated 1 false-positive for instant pectin (only for the optical reader, not for manually reading the same assay), so 13 additional powder samples were tested at random. Because there was only 1 false-positive in 79 samples, the ENVI met the 0.95 LCB/95% CL. The BioDetect^™ immunoassay generated false-positive results for 3 of 3 kaolin test sample replicates with both the optical reader and manually reading the result, and additional samples were analyzed for a total of 79. With 3 false-positives out of 79 test samples, the BioDetect^™ achieved 0.91 LCB/95% CL per the values listed in Table 1. The single-plex ricin immunoassay data are shown in Table 11B. Of the 4 multiplex immunoassay products, only 1 generated any false-positive results (Table 11C). The IMASS^™ produced a false-positive result for 3 of 3 kaolin test sample replicates. With additional sample testing (79 total), the IMASS achieved 0.91 LCB/95% CL.

Table 11

(A-C). Immunoassay testing of commonly encountered suspicious powders using B. anthracis immunoassays (A), ricin immunoassays (B), and multiplex assays (C). Unshaded cells met the 0.95 LCB/95% CL criteria. Shaded cells achieved a 0.91 LCB/95% CL. Data are reported as number of positive results/total tests performed.

A.
	BADD^™	BioDetect^™ (Reader and Manual)	BioThreat Alert^® (Reader and Manual)	CANARY^® Zephyr	ENVI (Reader and Manual)	NIDS^® 3 Ba (Reader and Manual)	RAMP^®	Smart^™ II
Sample Type	Ba	Ba	Ba	Ba	Ba	Ba	Ba	Ba
Powders (0.1 mg/mL)	0/66	0/66 (R, M)	0/66 (R, M)	0/66	0/66 (R, M)	0/66 (R, M)	0/66	0/66
R = reader; M = manual

B.
	BADD^™	BioThreat Alert^® (Reader and Manual)	NIDS^® 4 Ricin (Reader and Manual)	RAMP^®	Smart^™ II	Toxin Screen	ENVI (Reader and Manual)	BioDetect^™ (Reader and Manual)
Sample Type	Ricin	Ricin	Ricin	Ricin	Ricin	Ricin	Ricin	Ricin
Powders (0.1 mg/mL)	0/66	0/66 (R, M)	0/66 (R, M)	0/66	0/66	0/66	1/79 (R)	3/79 (R, M)^a
							0/79 (M)
R = reader; M = manual
^a3/3 kaolin test sample replicates resulted in false-positive results (reader and manual).

C.
	Pro Strips^™	RAID^™ 5	RAID^™ 8	IMASS^™
Sample Type	Multiplex	Multiplex	Multiplex	Multiplex
Powders(0.1 mg/mL)	0/66	0/66	0/66	3/79^a

^a3/3 kaolin test sample replicates resulted in false-positive results for ricin.

Summary

Our estimated limits of detection for spores and pure ricin toxin for biological indicator products were higher than manufacturer claims. None of the immunoassay product's claimed limits of detection were achieved with our spore testing, although our ricin immunoassay results were comparable to other published studies.^19,21 As demonstrated by our preliminary studies on B. anthracis growth media and strain type, the differences between PNNL B. anthracis limit of detection estimates, manufacturer claims, and published data could be due to variations in sample preparation or spore strain type. For products that had optional optical readers, we found that the use of an optical reader does not always improve sensitivity.

B. anthracis spores and ricin were detected with a 0.95 LCB/95% CL by several products, including general biological indicators. Two of the protein tests (BioCheck^® and INDIPRO) met these test criteria for B. anthracis and ricin. The PROFILE^® 1 ATP test met these criteria for B. anthracis. All other biological indicator test kits fell below 0.85 LCB/80% CL for both B. anthracis and pure ricin. Initial screening of 3 crude ricin preparations showed promise for protein tests and 1 of the ATP tests, but not the DNA test. Additional testing of these sample types is needed.

In general, the immunoassays performed slightly better for ricin detection than for B. anthracis detection. Seven out of 12 immunoassays met our most stringent criteria for B. anthracis detection, while 9 of 12 met our most stringent test criteria for ricin detection. Five out of 12 B. anthracis immunoassays and 3 of 12 ricin immunoassays failed to meet the minimum criteria of 0.85 LCB/80% CL. All of the immunoassays detected all forms of crude ricin in 6 of 6 replicates except the RAID^™ 8 (5 of 6) and Toxin Screen (4 of 6). Additional testing would be needed to determine if any of these technologies can achieve 0.95 LCB/95% CL for detection of ricin in crude castor seed preparations.

It is important to note that as more equipment and assays become commercially available, it will be necessary to quickly and cost-effectively compare their performance using an objective testing framework that measures performance relative to a standard. Likewise, as manufacturers continue to modify their existing instruments and assays, these new products will need to be reevaluated to determine if they still meet acceptable performance criteria. Our confidence interval–based testing approach enables efficient instrument performance testing to defined confidence levels, and our testing results allow users to make better informed decisions regarding the purchase and use of biodetection equipment.

Footnotes

Acknowledgments

The Department of Homeland Security, Science and Technology Directorate, provided funding for this research through contracts HSHQDC-08-X00843/6 and HSHQPM-12-X-00169/6 to Pacific Northwest National Laboratory (PNNL). PNNL is operated by Battelle Memorial Institute for the US Department of Energy under contract DE-AC06-76RLO. This evaluation does not indicate an endorsement or adoption of any of the technologies by PNNL or the US Department of Homeland Security. The authors declare no conflicts of interest related to this work.

References

Gessler

, Pagel-Wieder

, Avondet

, Boehnel

. Evaluation of lateral flow assays for the detection of botulinum neurotoxin type A and their application in laboratory diagnosis of botulism. Diagn Microbiol Infect Dis, 2007; 57(3):243-249.

Ozanich

, Bartholomew

, Bruckner-Lea

, Hess

, Arce

, Cardamone

. Biodetection Technologies for First Responders: 2015 Edition. Richland, WA: Pacific Northwest National Laboratory; 2015. http://biodetectionresource.pnnl.gov. Accessed December 5, 2016.

AOAC SMPR 2010.003. Standard method performance requirements for polymerase chain reaction (PCR) methods for detection of Bacillus anthracis in aerosol collection filters and/or liquids. J AOAC Int, 2011; 94(4):1347-1351.

AOAC SMPR 2010.004. Standard method performance requirements for immunological-based handheld assays (HHAs) for detection of Bacillus anthracis spores in visible powders. J AOAC Int, 2011; 94(4):1352-1355.

AOAC SMPR 2010.001. Standard method performance requirements for polymerase chain reaction (PCR) methods for detection of Francisella tularensis in aerosol collection filters and/or liquids. J AOAC Int, 2011; 94(4):1338-1341.

AOAC SMPR 2010.002. Standard method performance requirements for polymerase chain reaction (PCR) methods for detection of Yersinia pestis in aerosol collection filters and/or liquids. J AOAC Int, 2011; 94(4):1342-1346.

AOAC SMPR 2010.005. Standard method performance requirements for immunological-based handheld assays (HHAs) for detection of ricin in visible powders. J AOAC Int, 2011; 94(4):1356-1358.

ASTM. E2885-13: standard specification for handheld point chemical vapor detectors (HPCVD) for homeland security applications. In: E2885-13. West Conshohocken, PA: ASTM International; 2013.

Ozanich

, Colburn

, Victry

, et al. Evaluation of PCR systems for field screening of Bacillus anthracis. Health Secur, 2017; 15(1).

10.

Buhr

, McPherson

, Gutting

. Analysis of broth-cultured Bacillus atrophaeus and Bacillus cereus spores. J Appl Microbiol, 2008; 105(5):1604-1613.

11.

Wunschel

, Melville

, Ehrhardt

, et al. Integration of gas chromatography mass spectrometry methods for differentiating ricin preparations. Analyst, 2012; 137(9):2077-2085.

12.

Armbruster

, Pry

. Limit of blank, limit of detection and limit of quantitation. Clin Biochem Rev, 2008; 29(Suppl 1):S49-S52.

13.

Clinical and Laboratory Standards Institute. CLSI Evaluation of Detection Capability for Clinical Laboratory Measurement Procedures; Approved Guidelines. CLSI document EP17-A2. 2012.

14.

ASTM. E28000-11: Standard Practice for Characterization of Bacillus Spore Suspensions for Reference Materials. West Conshohocken, PA: ASTM International; 2011.

15.

King

, Luna

, Cannons

, Cattani

, Amuso

. Performance assessment of three commercial assays for direct detection of Bacillus anthracis spores. J Clin Microbiol, 2003; 41(7):3454-3455.

16.

Poore

, Clark

, Emanuel

. An evaluation of suspicious powder screening tools for first responders. J Hazard Mater, 2009; 172(2-3):559-565.

17.

Zasada

, Forminska

, Zacharczuk

, Jacob

, Grunow

. Comparison of eleven commercially available rapid tests for detection of Bacillus anthracis, Francisella tularensis and Yersinia pestis. Lett Appl Microbiol, 2015; 60(5):409-413.

18.

Peckham

, Hew

, Waller

, Holdaway

, Jen

. Spores: a portable, low-cost approach to the ELISA. Int J Electrochem, 2013; 2013(1):1-6.

19.

Slotved

, Sparding

, Tanassi

, Steenhard

, Heegaard

. Evaluating 6 ricin field detection assays. Biosecur Bioterror, 2014; 12(4):186-189.

20.

Hoile

, Yuen

, James

, Gilbert

. Evaluation of the rapid analyte measurement platform (RAMP) for the detection of Bacillus anthracis at a crime scene. Forensic Sci Int, 2007; 171(1):1-4.

21.

Hodge

, Prentice

, Ramage

, et al. Comprehensive laboratory evaluation of a highly specific lateral flow assay for the presumptive identification of ricin in suspicious white powders and environmental samples. Biosecur Bioterror, 2013; 11(4):237-250.

22.

ASTM. E2458-10: Standard Practices for Bulk Sample Collection and Swab Sample Collection of Visible Powders Suspected of Being Biothreat Agents from Nonporous Surfaces. West Conshohocken, PA: ASTM International; 2010.

23.

Amant

, Carey

, Guelta

. An improved method for the production of Bacillus subtilis var. niger spores for use as a simulant for biological warfare agents—quality analysis. Aberdeen Proving Ground, MD: Edgewood Chemical Biological Center; 2003. Accession No.: ADA483822.