Quantitative Validation and Comparison of Multiplex Cytokine Kits

Introduction

The goal of biomarker studies is to develop simple, noninvasive tests that identify disease states. Molecular biomarkers are routinely used in the detection and classification of some diseases, including prostate-specific antigen (PSA) in prostate cancer ¹ and HER2 in breast cancer. ² The multifactorial nature of many complex diseases, however, causes difficulties in the traditional approach to biomarker studies, focused on identification of individual molecular biomarkers as indicators of disease. Similarly, due to inherent variation of some plasma proteins, it is often difficult to determine if a disease is present or if a given measurement may represent high “normal” levels of a protein. Thus, despite intensive research, disease-defining biomarkers for many multivariant complex diseases remain elusive.

In view of this, focus is beginning to shift toward the identification of patterns of biomolecules for disease identification and characterization. Each disease possesses a unique molecular fingerprint that could be used diagnostically to differentiate it from diseases with closely related phenotypes. Examination of a molecular pattern should provide a more comprehensive indication of pathophysiological conditions and a better understanding of the molecular processes underlying diseases than individual biomarkers. ³ This novel concept, although still in its infancy, is being applied to several complex diseases, including cancer diagnosis, ⁴ Alzheimer’s disease, ⁵ polyarticular juvenile idiopathic arthritis, ⁶ psychiatric disorders, ⁷ Sjögren’s syndrome, ⁸ and chronic obstructive pulmonary disease (COPD). ^9,10

Multiplexing technologies are crucial to the extremely complex task of deciphering these disease-specific biomolecular patterns as they provide the opportunities for a more holistic approach, not previously possible using conventional technologies such as the enzyme-linked immunosorbent assay (ELISA), to be taken. However, prior to being employed in routine laboratory practice, it is essential that these technologies are investigated for reproducibility, precision, and accuracy. It is also vital that cytokine measurements obtained using multiplexing technologies are comparable to those obtained using single-point measurements (e.g., ELISA). Two commonly used multiplexing technologies are the Cytokine Bead Array (CBA) ¹¹ and Luminex ¹² systems, which are bead-based technologies based on the principles behind ELISAs. As with sandwich ELISAs, bead-based multiplexing kits employ immobilized antibodies to capture soluble ligands from within samples. The captured ligand is subsequently identified by a secondary antibody-based detection system. Although the general principles behind ELISAs and bead-based multiparametric technologies are the same, differences occur within the specific methodologies. First, bead-based assays employ fluorescence systems to detect binding of the secondary antibody to the soluble ligand, whereas ELISAs generally use enzymatic amplification of a colorimetric substrate. Second, for bead-based experiments, the capture antibody is immobilized onto spherical beads in suspension rather than immobilized on the flat surface used in ELISAs. It is the fluorescent intensities of these spherical beads that provide the multiplexing capacity of such technologies.

Although ELISAs are a well-established technique, bead-based technologies are relatively new and require thorough validation. Several studies have previously been undertaken to compare CBA kits with ELISAs, ^13,14 Luminex kits with ELISAs, ^15,16 and Luminex kits from different vendors. ¹⁷ However, there has been no experimental comparison between CBA kits and Luminex kits. Here we examine the reproducibility and accuracy of data obtained using CBA and Luminex kits and compare concentration measurements to those obtained using sandwich ELISA. One disease that would benefit from the aforementioned approach to diagnostics is COPD. ⁹ Although the ideal target for such procedures would eventually be a blood sample, there are problems associated with analysis of the blood proteome ¹⁸ that could distort results obtained using commercial cytokine analysis kits. To avoid these potential difficulties, therefore, alternative clinical samples will be used during these validatory investigations. The lung is the representative system for respiratory disorders such as COPD, and clinical samples traditionally examined include induced sputum, bronchoalveolar lavage, lung tissue, and, more recently, exhaled breath condensate. ¹⁹ All of these sample types could potentially be analyzed for patterns of biomarkers, but the availability of lung tissue samples has led us to examine these in this study.

Materials and Methods

Multiplex cytokine detection kits

Three different premixed CBA kits (BD Biosciences Pharmingen, San Diego, CA)—Th1/Th2, Inflammation, and Chemokine—were examined according to manufacturer’s instructions, allowing detection of 14 different cytokines, 3 of which (interleukin-8 [IL-8], interleukin-10 [IL-10], and tumor necrosis factor [TNF]) were duplicated between kits. Flow cytometry was performed using an EPICS Altra Flow Cytometer (Beckman Coulter, Fullerton, CA) or a BD LSR II System (BD Biosciences). Data were analyzed using the BD Cytometric Bead Array software (BD Biosciences). Luminex bead kits (Biosource, Carlsbad, CA) for IL-8, IL-10, and TNF, corresponding to the duplicated cytokines contained within the CBA kits, were examined according to manufacturer’s instructions. Data were acquired on a Bio-Plex 200 system and analyzed with associated software (Bio-Rad, Hercules, CA). All multiplex assays were performed in duplicate. Standard curves for each cytokine were generated using the premixed lyophilized standards provided in the kits, and the cytokine concentrations in samples were determined from the appropriate standard curve. This workflow of the comparisons undertaken is summarized in Figure 1 .

OPEN IN VIEWER

Fig. 1.

The workflow of the comparisons between multiplex technologies undertaken in this study.

Results and Discussion

The accuracy and reproducibility of cytokine measurements made using commercial multiplexing kits is highly dependent on the availability of high-quality standard curves. In the course of this study, a variety of problems were encountered during the generation of standard curves contained within the CBA kits ( Fig. 2 ). Curves generated for individual cytokines from the same premixed lyophilized samples varied in accuracy. In this example, data for IL-6 ( Fig. 2A(i) ) were accurate and fitted well to the calculated curve. In contrast, the individual data points for IL-12p70, produced from the same diluted samples, were highly variable and resulted in a less accurate curve fit ( Fig. 2A(ii) ). Many curves were found to be flat, decreasing the accuracy of measurements made at lower concentrations ( Fig. 2B ). In some instances, the original flow cytometry data used to measure fluorescent intensity of the detection antibody were split into 2 distinct populations ( Fig. 2C(i) ) as opposed to the single population normally observed ( Fig. 2C(ii) ). Finally, in some examples, the bead populations failed to resolve properly along the y-axis ( Fig. 2D(i) ), causing overlapping between different cytokine groups that should not occur ( Fig. 2D(ii) ). These issues were not encountered during the generation of standard curves using Luminex technology ( Fig. 3A ).

OPEN IN VIEWER

Fig. 2.

Problems encountered during production of the standard curves contained within cytokine bead arrays: (A) inconsistencies within a premixed standard curve solution; (B) flat standard curves; (C) “tailing” of samples at certain dilutions of a standard curve; (D) indiscrete regions for individual cytokines within some standard curves. Standard curves were produced using the premixed lyophilized standards contained within Th1/Th2, Inflammation, and Chemokine cytokine bead array kits. Flow cytometry was performed using an EPICS Altra Flow Cytometer, and data were analyzed using the BD Cytometric Bead Array (BD Biosciences, San Diego, CA) software. Plots are representative of the different problems encountered during standard curve generation.

OPEN IN VIEWER

Fig. 3.

Interkit and intrakit variability of (i) interleukin-8 (IL-8), (ii) interleukin-10 (IL-10), and (iii) tumor necrosis factor (TNF) standard curves generated using (A) Luminex and (B) Cytokine Bead Array (CBA). Standard curves from the different kits were generated on 2 separate occasions, and data were acquired on an EPICS Altra Flow Cytometer (CBA) or a Bio-Plex 200 system (Luminex). (○) Kit 1, repeat 1; (●) kit 2, repeat 1; (□) kit 1, repeat 2; and (■) kit 2, repeat 2.

Such problems will affect the reliability of downstream cytokine measurements made using CBA kits. The majority of studies involving premixed CBA kits use only 1 kit for cytokine measurements, negating the need for validation of interkit variability. However, in cases where maximal characterization of the cytokine profile of a sample is required, it may be necessary to employ multiple kits in conjunction. In such cases, for example, when trying to identify disease-specific biomolecular patterns, it is imperative that cytokine levels determined by individual kits are comparable. In addition, if kits are to be used to characterize patient samples, it is vital that data are consistent on different days. Thus, levels of intrakit and interkit variability in standard curves must be minimal and can be assessed by examination of the cytokines duplicated between kits.

During this study, intrakit and interkit variability was found to be high for the IL-8 standard curves produced using CBA kits ( Fig. 3B(i) ). Intrakit variability was also high for IL-10 and TNF CBA kit standard curves, but in these cases, interkit variability was low ( Fig. 3B(ii),(iii) ). In contrast, standard curves produced using Luminex technology had low intrakit and interkit variability for IL-8, IL-10, and TNF ( Fig. 3A ). Only one previous study has compared the reliability of measurements made by CBA kits containing duplicated cytokines. Hodge and colleagues ²⁰ demonstrated low variation in IL-10 and TNF measurements made using the Th1/Th2 and Inflammation CBA kits, supporting the findings of this study. However, there is no evidence of a similar study comparing the Inflammation and Chemokine CBA kits having been undertaken previously, and thus the poor reliability of IL-8 standard curves has not been reported previously. It is of interest that the number of data points for generating standard curves also varies between the different multiplex kits (n = 7 for Luminex and n = 9 for CBA). Both technologies do, however, exceed the minimum number of data points required for accurate logistic regression analysis. ²¹

One advantage of the CBA kits over the Luminex system is the possibility of collecting data on general flow cytometers already installed in many laboratories ²² rather than using a specialized detection system. The manufacturer of the CBA kits recommends the use of a BD flow cytometer for data collection. To ensure that the problems encountered during the generation of the CBA standard curves were not due to the flow cytometer used, we ran samples in parallel on a BD LSR II System flow cytometer and an EPICS Altra flow cytometer. The variations observed in inter- and intrakit variation occurred regardless of the machine used ( Fig. 4 ). Using a BD LSR II System flow cytometer in place of an EPICS Altra flow cytometer did, however, increase the sensitivity of readings at lower dilutions of the standard curve ( Fig. 4 ).

OPEN IN VIEWER

Fig. 4.

Interkit variation in standard curves generated for (A) interleukin-8 (IL-8), (B) tumor necrosis factor (TNF), and (C) interleukin-10 (IL-10) on 2 different flow cytometers. (○) Kit 1, EPICS Altra flow cytometer; (●) kit 2, EPICS Altra flow cytometer; (□) kit 1, BD LSR II flow cytometer; and (■) kit 2, BD LSR II flow cytometer.

In addition to investigating intra- and interkit reliability, this study also examined intertechnology reliability. Concentrations of IL-8, IL-10, and TNF in lung biopsy supernatants were measured using the CBA and Luminex kits. Of the 3 cytokines, only IL-8 was detectable in the samples. IL-10 and TNF could not be detected (data not shown) due either to the absence of the cytokines in the samples or to the cytokine concentration falling below the detection limit of the CBA (IL-10: 2.8 pg/mL; TNF: 2.8 pg/mL) and Luminex (IL-10: <5 pg/mL; TNF: <10 pg/mL) technologies. IL-8 was subsequently measured using ELISA for comparison.

The level of IL-8 measured within any 1 sample varied greatly depending on the technique employed in making the measurement, with data obtained for 1 patient’s sample ranging from 288 to 914 pg/mL ( Fig. 5A ). Such highly variable IL-8 measurements may be due in part to the different antibodies used by different companies for their different technologies. ¹⁷ However, although absolute concentrations were highly dependent on the detection kit, correlations with ELISA, the current gold standard for cytokine measurements, were good for all technologies ( Fig. 5B ). These values are consistent with or higher than those previously obtained during comparisons of ELISA with Luminex or CBA kits. ^14-16,23

OPEN IN VIEWER

Fig. 5.

Variations in interleukin-8 (IL-8) measurements in lung tissue supernatants as detected by (▲) enzyme-linked immunosorbent assay (ELISA); (●) Cytokine Bead Array (CBA) Inflammation kit; (○) CBA Chemokine kit; (■) Luminex kit 1; and (□) Luminex kit 2. (A) The IL-8 concentrations observed; (B) correlation between the IL-8 concentrations obtained using ELISA, CBA, and Luminex.

To conclude, cytokine levels determined within a sample are highly dependent on the technology chosen for analysis. When deciding upon the technique to be employed for identification of patterns of biomolecules in biomarker studies, the data obtained during this and previous studies ¹⁴ should be considered. Duplication of IL-8, IL-10, and TNF enabled validation of measurements made for these cytokines within the different CBA kits. The problems encountered during CBA standard curve generation were not, however, restricted to these cytokines and affected multiple cytokines, including IL-12p70, IP-10, and MIG. These difficulties, coupled with the variation observed in IL-8 measurements, raise questions about the reliability of concentrations obtained for other cytokines that are not duplicated. In contrast, the Luminex technology was found to be highly reproducible. Despite the aforementioned difficulties, the high correlation between concentrations obtained using the different technologies indicates that biomolecular patterns (i.e., our biomarker panel) should be identifiable using any of the technologies investigated. It is critical, however, that once a technology has been selected, it is used consistently if comparisons are to be made between different data sets. Furthermore, this study highlights the importance of using a control sample of known concentrations to normalize between runs undertaken on different days for quality control purposes.