How Do Postgenomic Innovations Emerge? Building Legitimacy by Proteomics Standards and Informing the Next-Generation Technology Policy

Broader context for the study

Various ways, in which data become accepted as scientific facts or knowledge, have been discussed in the STS literature (Felt et al., 2017; Latour, 1988). We combine these insights with literature on legitimacy, to examine the role of policy or political tools, as a specific kind of actor or nonhuman actor. These tools function as a mechanism to legitimize a scientific field (and its contribution to knowledge).

A group's legitimacy is associated with its social identity, providing a “classification scheme” that members and outsiders use to identify with and assign membership to (King and Whetten, 2008). People make assumptions about how a group should behave based on that categorization (Brown, 2008; King and Whetton, 2008). Whether chosen or assigned, a social identity subjects a group to the normative standards of that category; if it does not comply with these expectations, a “legitimacy discount” is experienced by that group (King and Whetton, 2008).

Proteomics, the large-scale high-throughput study of proteins, aims at the collective characterization of biological molecules (proteins) that make up an organism. Proteomics was emergent (Boenink, 2010) at the time of our study (2011–2015) in the sense that (1) the techniques which allowed the mass spectrometer to analyze proteins were only developed in the late 1980s, and the term for the field was not coined until 1994 (Ma et al., 2018); and (2) the complexities of protein analysis (compared to the relatively straightforward analysis of DNA) combined with the importance of genomics to the public's and funders' priorities meant that the field was still establishing itself and its relevance during this time period, particularly compared to genomics (Holmes et al., 2016a, 2016b; Ma et al., 2018; McNalley and Glasner, 2007).

To attract funding and interdisciplinary collaborations necessary for the continuation of their work, scientists were obliged to establish proteomics' legitimacy as a distinct field essential to the scientific understanding of biological functioning and human disease processes. For an emergent science such as proteomics, claiming legitimacy was essential to reduce the perceived risk of its new and untested procedures and products (Rose and Rose, 2013). With no established social categories to cement a community of practice beyond its own acolytes, it might be seen as suffering from a “legitimacy discount.” As Battilana et al. (2017, 142) note “Acquiring legitimacy in the eyes of important external constituents thus presents a considerable challenge … [a lack of legitimacy] making it hard to obtain vital resources.”

Legitimacy and standards: background concepts

Legitimacy means that organizations or groups are perceived to be “appropriate and accepted actors, whose activities can be justified in terms of the values, norms, laws and expectations of their social contexts” (Brown, 2008, 2; King and Whetten, 2008). Establishing or enhancing legitimacy may prove ephemeral as it can be eroded over time and depends on the perceptions of multiple audiences internal and external to that group or organization (Brown, 2008).

Legitimacy may be easier to establish facing inwards than outwards. Widely recognized as an important component of legitimacy (Hough et al., 2010), trust involves “a willingness to accept vulnerability” built on expectations of “technically competent role performance from those involved with us in social relations and systems” (McEvily et al., 2003, 92–93). Huising and Silbey (2017, 799) note, “scientists and scientific work still depend on trust,” even if that trust is harder to build in a world with more scientists, some of whom cross disciplinary boundaries. Legitimacy, then, is a social construction (Black, 2008) essential for an organization to achieve its desired operational or functional ends.

Legitimacy can be claimed on a number of bases depending on the organization's particular context or identity, including demonstrated performance and expertise; compliance with regulations; association with other legitimate actors, organizations or groups; fit with cognitive expectations; and political representation (Black, 2008; Brown, 2008). Groups may need to actively construct legitimacy arguments to serve the strategic and political ends of that organization (Brown, 2008).

Socially identified as an emerging science, proteomics offered an opportunity to explore the reasons and means behind the legitimation of a new field. Hackett et al. (2017) note that,

Subgroups within the scientific community attempt to develop and win acceptance for research programs that challenge the current state of scientific knowledge, or supplant established scientific techniques … Such movements can succeed when they mobilize key material and emotional resources and high-status intellectuals and recruitment centers, and when they frame their research in ways that resonate with others in the field.

But emerging sciences must also show that they meet the basic expectations inherent in the scientific identity. Merton's model of scientific norms (1973) describes ideals about the conduct of competent scientists and, by extension, competent science. However, norms are context and discipline specific, often contested, and change over time (Huising and Silbey, 2017). Merton's norm of organized skepticism (or the testing of all knowledge claims) demands critical evaluation of science's findings to meet its “institutional need for certifiably reliable knowledge” (Jasanoff, 2015, 1738). Another Mertonian norm is universalism. The institution of standards may mean that the scientific work is more likely to be separated from any sociopolitical valuation of the personal or institutional status associated with that work.

Standards are one mechanism to address reproducibility in the critical evaluation of science. Standards are instruments that create and affirm knowledge (Castel, 2009; Hogle, 2009), enable “systematic recourse to the collective production of evidence” (regulatory objectivity) (Cambrosio et al., 2006) and serve a collaborative or coordinative function (Cambrosio et al., 2006). They are “the sine qua non [indispensable condition] of modern science” (Rogers and Cambrosio, 2017, 166), providing rules, guidelines, and characteristics to encourage processes in different laboratories to be similar.

Subject to local resistance (Huising and Silbey, 2017), standards support, but do not guarantee the production and validation of reliable research findings (Timmermans and Epstein, 2010). As Brown (1993, 156) notes: “scientists strive for standardization in rendering their somewhat ad hoc activities in the laboratory into replicable and reputable public accounts,” aiming to lift research results to the level of facts.

Standard setting processes assist organizations to manage their reputations and make their claims credible. While standard-setting can be motivated by issues of quality and efficiency (Timmermans and Epstein, 2010), it can also reflect a strategy to, for example, become a market leader or, we would argue, to assist with the legitimization of a scientific field. Standards are an important method of demonstrating legitimacy in science as they may increase the perception (externally and internally) of the robustness, validity, and usefulness of the field of science by affirming that the processes through which results are obtained are reliable and consistent (Timmermans and Epstein, 2010).

In other areas of scientific practice, notably the biomedical sector, standards are used as strategic and political resources to improve a health profession or organization's position (Castel, 2009) and to legitimize that profession or organization. As such, engagement with standards as a legitimacy tool is a mechanism to leverage power and position, in this context, attracting grants, funding, position, and influence within the omics sciences. It also assists to garner power for a given form of science over other types (e.g., genomics vs. proteomics).

This suggests that an emerging scientific field must engage with a fundamental aspect of scientific identity—that is, to be seen to critically evaluate its research findings (using new technologies and methods in the proteomics case), to ensure that the field is regarded as legitimate and hence trusted. Creating standards is one mechanism to remedy a “legitimacy discount.”

Research methodology

This case study of standards making activity in the proteomics community between 2011 and 2015 occurred when the field, as we argue below, was working to establish its legitimacy. Data were collected using ethnographic methods (Emerson et al., 2011), including fieldnotes from participant observation, qualitative interviews with proteomics scientists, and analysis of the proteomics literature. Interview transcripts and fieldnotes were analyzed thematically by C.H. using Atlas-Ti qualitative analysis software.

We used participant observation to follow processes of standardization, the debates and engagement with standards in real time, at international HUPO conferences from 2011 to 2015 (by C.H., F.M., and M.J.), and at the 2012 and 2013 APS conferences (by F.M.). Established in 2001 (Kaiser, 2002), HUPO is an international scientific consortium to “promote proteomics through international cooperation and collaborations” (https://hupo.org/about-hupo) with a number of affiliated regional or national groups, such as the APS (HUPO, 2010).

Globalization and the global nature of contemporary research have made conferences central to observing the interaction of scientific communities and networks, such as proteomics. Conferences may be the only “local” site at which those involved regularly congregate (Hannerz, 2003; Holmes et al., 2016a, 2016b). The political aspects of proteomics standards setting are openly discussed and performed at these public conferences. Open ended, qualitative interviews (Brinkmann and Kvale, 2014) (N = 36) were conducted with scientists from a variety of different countries, working in proteomics. Our interviews focused on the role of standards and knowledge translation in the scientific process. The names of participants in interviews are confidential pursuant to the terms of research ethics approvals, although those who presented publicly at conferences are named.

The project received approval from Dalhousie University and Queensland University of Technology's Human Research Ethics Committees. This project was independent academic social science research to understand the sociopolitical emergence of proteomics data intensive science and standards and was not linked to any proteomics scientific research, organization, nor to any commercialization initiatives.

A legitimacy problem for proteomics science

Huising and Silbey (2017) argue that trust among scientists, their laboratory spaces, and their publics is crucial in cementing expectations of technical competency. They suggest that trust provided by the relative standardization of the laboratory's construction and composition has allowed the laboratory to “disappear as an epistemological marker” (799). Rapid technological advances in the field, coupled with incidents that have questioned the reliability of research results, however, had undermined this “relative standardization” for proteomics, specifically, and in omics more widely. A perceived lack of trustworthiness projected upon a group (Adjekum et al., 2017) can create a legitimacy discount. Our interviews with proteomics scientists, observation at HUPO/APS, and review of the literature reveal three attributes of this legitimacy discount.

First, the legitimacy of Omics science has plagued the wider family of Omics at the time of this study, of which proteomics is a member. The recent rise of multiomics and integrative analyses across several omics fields have led to new insights on common complex and rare human diseases (Pirih and Kunej, 2017). However, for many countries worldwide, multiomics science is relatively new. While individual omics fields do exist in various countries, they are often implemented in a patchy or piecemeal manner. In other words, focusing on a narrow technology vision or a singular omics platform can limit the broader perspectives available from systems biology at a whole organism scale.

The most prominent omics science, genomics, made large anticipatory promises of social benefit shaped, in part, by scientists and a host of assigned or self-appointed regulatory actors, followed by significant disappointment when it underdelivered, something that is not uncommon with many emerging technologies or their governance frames (Balmer et al., 2015; Boycott et al., 2013; Fortun, 2008; Holmes et al., 2016a; Lopez and Lunau, 2012; Tutton, 2012). This may have created a more cautious reception for newer omics sciences and postgenomic technologies (Rose and Rose, 2013), especially when some genomics research published in prominent journals was found to have important scientific limitations, compounded with the growing issue of nonreproducibility in science. In the late 2000's, these scientific limitations have led to law suits and reviews of genetic clinical testing procedures (Couzin-Frankel, 2011; Hayden, 2012; Kolata, 2011).

Second, proteomics, as a new science, had the challenge of establishing its legitimacy at the time of the present study (Rose and Rose, 2013). Any emerging science will find this challenging simply because it is new, untried, and untested (Rose and Rose, 2013).

Indeed, as one interviewee noted, proteomics was an “emerging field, so it is on Moore's law at the moment” (Proteomic Scientist [PS1]), reflecting the rapid technological changes in the field. The emerging science issue was exacerbated by the fact that proteomics is a great deal more complex than genomics (Holmes et al., 2016a; Ma et al., 2018; McNally and Glasner, 2007). New techniques and technologies, such as MS, were received with caution.

Presenters at the HUPO and APS conferences commented that some journal reviewers had requested further confirmatory experiments using more established protein detection methods, such as western blots or enzyme-linked immunosorbent assay, before articles reporting MS data were accepted. At a Human Proteome Project session at HUPO 2012, one scientist suggested that skepticism about the MS (an expensive technology) hindered funding and research:

It is not the price. It is the price to data relationship. If a Dean invests in gene sequencing, it is clear what comes out. If they do this in mass-spec, it is not clear what comes out. The price tag is not the issue—it is the price versus the output.

For society to continue to fund expensive new technologies and not abandon them for cheaper alternatives within the research spectrum, it requires trust in the reliability of the field and data emergent from new technologies such as proteomics.

At the 2013 HUPO conference, some scientists expressed hope that “the day would soon come” when confirmation of MS findings with other methods would no longer be necessary. The prominent proteomics scientist Ruedi Abersold argued at HUPO 2012 that the MS approach would replace older protein identification methods. This was illustrated in his presentation's title: ‘Can the HPP [Human Proteome Project] help move research beyond the Western Blot?’ Prominent proteomics scientists at the 2013, 2014, and 2015 HUPO conferences stated that the technology had “matured” enough that MS was now a legitimate and reputable technique.

Third, problems with the quality, validity, and reproducibility of proteomics research data became apparent in the late 2000s. Mann (2009) suggested that “the potential of mass spectrometry-based proteomics to advance biology and biomedicine is nearly unlimited but so is its potential for generating bad data.” One interviewee described much early MS research as only generating “data cemeteries” (PS2).

A HUPO working group, concerned about research quality, sent test samples to 27 proteomics laboratories around the world (Bell et al., 2009). Only seven were initially able to characterize the proteins correctly, although most generated MS data that were sufficient to identify them with additional analysis. The study identified generic problems with databases (database matching and curation) as a primary issue, but also showed problems with missed identifications and environmental contamination of samples, indicating the need for data and database standards and standards for the quality control of laboratory work (i.e., for sample preparation and the liquid chromatography and MS stages). The researcher, skeptical about “data cemeteries,” further noted in reference to the Bell study:

You want to be sure you can at least characterize the proteins, right? So [they] sent these [samples] around the world to groups that were getting in millions of dollars in grants per year. And most of them couldn't do this. I mean, come on! A Formula 1 driver has to be able to drive the car around the circuit and not crash into the first pole, right? (PS2)

The laboratory equivalent of being able to “drive the car around the circuit” typifies the normative standards required for legitimization (King and Whetton, 2008) as a viable scientific field. Failure to “get around the track” would see that field sustains a legitimacy discount which it would need to overcome.

The Bell et al. (2009) article did not merely garner attention within the field of proteomics but also in the broader scientific community. It should be noted, however, that the Bell study attests to the early emerging phases of proteomics, and long before the current era of multiomics integrative research that generates higher quality Big Data (Pirih and Kunej, 2017).

At the 2013 HUPO conference, scientists from the National Cancer Institutes (U.S.) (NCI) noted having to address concerns raised by the Bell et al. (2009) article about the validity and reproducibility of proteomics science before the NCI would agree to supplement its genomics program with a proteomics arm. By the 2015 HUPO conference, an NCI representative announced they had sufficiently addressed some of these concerns to do a comparative study: typing cancer tumor cells using both genomic and proteomic methods and then comparing the results.

Interviewees and editorials also noted examples where proteomics research overreached identifying new biomarkers that could not be replicated by other laboratories (Albar and Canals, 2013; Baggerly et al., 2005; Diamandis, 2010). Albar and Canals (2013) suggested that:

The problems and drawbacks of these studies [candidate biomarkers], may be due to different causes such as poor study design, insufficient power and/or the lack of appropriate validation … the majority of proteome-centric studies on human serum and plasma have been hampered by poor reproducibility, inadequate significance, and a lack of robustness.

Ye et al. (2009) reported that proteins approved as clinical biomarkers by the U.S. Food and Drug Administration (FDA) were small in number compared to the larger potential of proteomics research. This was restated at HUPO Clinic Days and panels on translational proteomics research in 2015 and echoed in Steffen et al. (2016). For example, as discussed at HUPO 2013, in 2005, 14 proteomics biomarkers were submitted to an initiative of the NCI for consideration. Of the top five, only two biomarkers passed clinical prevalidation, gaining FDA approval in 2012. The NCI's Centre for Strategic Initiatives, in its discussion of NCI-CPTAC DREAM Proteogenomics Computational Challenge (a combined field of genomics, transcriptomics, and proteomics) on February 6, 2018, commented that:

Characterization of alterations in the proteome has the promise to shed new light into cancer development and may be used for development of biomarkers and therapeutics. However, measuring the proteome is still challenging even with the recent rapid technology developments in mass spectrometry that are enabling deep proteomics analysis, and it would be much cheaper and easier if it was possible to simply measure for example mRNA levels and predict the protein levels with high accuracy. (National Cancer Institute, 2018, para. 2)

Therefore, translation to clinical application was a challenge for the field, while it labored under decade old criticisms about the quality of its data.

Standards as a tool to improve the legitimacy of an emerging science

A “field-level consensus, or … level of agreement among researchers in a field” (Hackett et al., 2017, 741) grew among proteomic scientists that standards were an important tool to remedy proteomics' performance legitimacy discount. Several scientists recognized that standards were of “crucial importance” and “required” for the future development of the field (Albar and Canals, 2013; Bielow et al., 2016; Chiva et al., 2018; Malm et al., 2013). Albar and Canals (2013) note:

However, for proteomics to reach its technological maturity and being able to successfully deliver all its potential to the scientific community, there is an unavoidable need for quality control procedures that cover all the steps involved in proteomic analysis.

Bielow et al. (2016, 777) went further alluding to the link between scientific norms, or what it means to do good science, and standards for quality control: “Data quality is the cornerstone of solid research, demanding repeatability and reproducibility.”

Clearly, quality control standards for developing technologies are a prerequisite to legitimacy for an emerging science within the scientific community. Another interviewee noted: “Yes, standards are a critical thing. You know, it is probably the most important issue for HUPO.” (PS3)

The importance of data quality was made clear by a presenter in a High Quality Disease Proteomics panel at HUPO 2013, “It [data] can't just look like gold,” he commented “it must be gold.” An interviewee further noted:

I think in the early days, they [proteomics scientists] thought it was just a simple matter of “this pattern is different.” From that ovarian cancer study they thought, “wow, we could use this to predict incidence of ovarian cancer.” And they found that as soon as another lab did this with their equipment and their handling methods, it was completely different. And so, large organizations and institutes like the National Cancer Institute in the States are taking the lead and saying, okay, we have to come to standard operating procedures on what is the technology and the training that you need. (PS4)

An NCI representative at the New Technologies and Standardization seminar at the 2013 HUPO conference identified standards as necessary to reassure other scientific fields that proteomics was a legitimate field of science. He acknowledged the relevance of considering the expectations of other communities of practice when developing standards and suggested including journal reviewers, biologists, omics scientists, clinical chemists, and statisticians. By not including end users such as patient groups, or instrument manufacturers, funders, regulators, or clinicians, this scientist suggests that legitimacy within the broader scientific community is the first priority.

The comment, however, arose in the context of a discussion about how a funding agency was persuaded to significantly invest in proteomics research despite concerns about validity, indicating implicit acknowledgement of the broader political considerations for standards to legitimize the emerging scientific field.

Standards, as a mechanism to provide assurance that the processes used to generate results are both valid and clinically significant, are also important to generate funding to translate proteomics research from the bench to the bedside and in downstream regulatory practices for pharmaceutical or biomedical products to be applied clinically.

Gu and Yu (2014) note: “… establishing quality control standards for proteomics data generation and evaluation will help regulatory agencies meet obligations to utilize proteomics data in conjunction with drug review and biomarker qualification processes.” Ye et al., (2009) and Poste (2011) suggest that validation methods were pragmatically necessary to confirm clinical utility for regulatory agencies.

Remedying the legitimacy discount

The process of and discussion about standards within proteomics was uneven due to the number of different kinds of potential standards possible for proteomics research. For example, the Proteomics Standards Initiative (PSI), an official and early initiative of HUPO, created and implemented data standards, with a mechanism for compliance enforced by scientific journals (discussed in the next section). In contrast, laboratory standards were championed by different organizations and individuals and therefore were more unevenly implemented.

Furthermore, some proteomics scientists put more emphasis on laboratory standards than others, depending on their research goals (e.g., clinical translation vs. technological innovation) (Holmes et al., in preparation). Despite this, there was significant focus within proteomics on addressing its performance legitimacy discount through the design and implementation of standards for many different types of activities within proteomics science.

Timmermans and Berg's (2003) typology of standards (Fig. 1) categorize various types of standards which are possible in clinical and scientific settings, among others. As Figure 1 illustrates, it is possible for some standards to overlap into more than one category. Applying this typology of standards to proteomics during the period of our case study illustrates that proteomics is addressing all potential categories of standards and, therefore, all aspects of its performance legitimacy discount.

OPEN IN VIEWER

FIG. 1.

Typology of Standards (Timmermans and Berg, 2003). (1) Design standards that relate to the properties and features of tools and products (i.e., technological standards); (2) terminological standards that define terms and vocabulary; (3) performance standards that set outcome specifications; and (4) procedural standards that specify how processes are to be performed.

Timmermans and Berg (2003) draw on a range of possible settings for standards, and therefore the meaning of their categories may be different from the labels scientists use when they discuss standards.

Procedural standards, which include steps that must be taken when certain conditions are met such as clinical practice guidelines, are often the most difficult to achieve and the most contested type of standards since they prescribe behaviors (Timmermans and Berg, 2003). This category can include proteomics laboratory sample preparation protocols and trial testing as part of NCI initiatives and Association of Biomolecular Resource Facilities (ABRF) testing. When to develop (at which developmental stage of the science) and how to implement such standards in proteomics were a matter of some debate during the time period of the study.

Terminological standards, to ensure that terms held the same meaning over different locations, were well established during this period. They were spearheaded through the PSI and then required by many proteomics journals.

Design standards are structural, explicitly defining the specifications of individual components (Timmermans and Berg, 2003) so that elements of a system can work together. They include things such as the size of injection needles so that they fit syringes, and relate to properties and features of proteomics work, such as MS technical requirements. These are less discussed by scientists and more often the province of scientific instrument companies in respect to proteomics.

Performance standards specify the required outcome, or what the result of an action/set of actions should be. An example is a minimum score on an examination. This could include standards such as the minimum requirements for what must be discussed in a proteomics journal article when reporting on a proteomic experiment (MIAPE). (However, many of the actions reported, including the design of the study itself, would draw on procedural standards). As with terminological standards, this kind of standard was well established within the time period of the study and was required by many proteomics journals.

All of these types of standards represent points at which the scientific norms of reproducibility and validity of data are at risk. If accepted scientific norms are at risk, then so too is the perceived legitimacy, especially in relationship to performance, of an emerging scientific field.

Performance standards setting processes in proteomics

What is the relationship, then between proteomics' performance related problems, which created a performance legitimacy deficit (Brown, 2008), and the range of standards created for proteomics science? This section explores how proteomics moved from data-focused standards to the broader range of performance-focused standards during our study.

The importance of establishing sound standards was recognized at the outset by proteomics scientists. One interviewee noted: “Standards have been a big part of proteomics and HUPO. As a matter of fact, it [the PSI] has been one of the most active initiative groups” (PS3). In 2002, 1 year after HUPO was founded, it established the PSI with the mandate to establish data standards for presenting and storing MS and protein–protein interaction data (Hermjakob, 2006; Kaiser, 2002). This process arose from internal needs for standard data formats and terminology in proteomics to enable different research groups to compare large data sets (Orchard and Hermjakob, 2011).

The PSI, a highly specialized expert group with a focus on bioinformatics, database curation, terminology, and addressing instrumental data sharing, has created guidelines in multiple areas (www.psidev.info). While the PSI focuses on instituting standards to improve the quality of databases and hence bioinformatic analysis and reporting standards, its website is clear that “we do not address issues of quality” in the context of creating standards for laboratory processes (www.psidev.info/about). It was not surprising that the initial focus of an emerging science such as proteomics, using rapidly evolving high-throughput and Big Data technology, for example the MS, initially focused on functional issues, such as data curation standards, which were desperately needed by the science itself.

While Bell et al. (2009) indicated a reproducibility problem in proteomics research that was partially attributable to database and curation issues (that continue to be addressed by the PSI), the broader issues related to the quality of laboratory processes raised by Bell et al. (2009) remained unaddressed by the PSI or by any other coordinated focus on quality standards by the early 2010's. At the HUPO conferences in 2011, the focus was on data standards, and it was not until 2012 that we saw the emergence of increasing discussion of quality standards and quality control. Indeed, at HUPO 2012 in a Standardization workshop, Ruedi Abersold, acknowledged that “despite rapid technological developments and successful application in many biological studies, the reproducibility of MS-based proteomics has been called into question.”

While the issue of interlaboratory reliability and quality processes did not appear to attract large numbers of attendees at the conferences we observed, work was being done behind the scenes in individual laboratories, within research collaborations, and with organizations such as HUPO and its regional groups such as EuPA (European Proteomics Association) and the HUPO Plasma Proteomics Group, as well as the ABRF, the NCI, and ProteoRed (the Spanish Proteomics Network) to begin to address these issues (Albar and Canals, 2013; Cottingham, 2009; Jackson and Bramwell, 2013; Rai et al., 2005).

Some of these initiatives had more longevity and translation into working practice within the proteomics community than others. For example, at the HUPO 2012 Standardization workshop, there was discussion of a proposed “Proteomicum,” an online resource compendium of reference methods and materials, based on the model of the European Union organic chemistry guide. It was intended as a tool to self-assess platforms and methods to be created as a community driven effort without preventing future developments in the field. It was to be community driven, involving submission of protocols that would then be tested within the community, followed by discussion with journal editors about minimum guidelines required for publication on quality standards. While there was a follow-up presentation at HUPO 2013 for a planned website, as of 2019, the website had not been launched.

In contrast, another approach to the problem of performance legitimacy saw the Proteomics Research Group (PRG) of the ABRF run quality control studies (ABRF, 2017). Within ABRF, there was a Proteomics Standards Research Group (sPRG) that was identifying and implementing technical standards since 2006 (ABRF, 2018), in which laboratories volunteer to participate in surveys that examine issues such as quality control between or within laboratories over an extended period of time, as well as other topics. Data from these studies are then made public. Laboratories have an anonymous identifier to see how they perform compared to other laboratories. The importance of such an initiative was explained by an interviewee as follows:

You are asking about standards, so I think this is a perfect example of that. So they [ABRF] started actually providing SOPs [standard operating procedures] together with the samples, telling how the thing should be done, but still they were getting different answers from the labs. But I think over the years, it actually improved a lot and now most of the labs are doing it well. The data are available on the website—the ABRF website—so you can see how the field was progressing over time.

You can still see that there are labs that are doing the best, especially the big facilities, while the small labs—because that data is available, you know. It is all anonymous, but you know whether the lab consisted of two people and was new in the field, or not. You can still see that these people are outliers, but there is a big progress over the years in providing better and better data. And I think every lab, at least in the life of the lab should at least participate once in such a study to see how good they are, because that would—many labs just think they can do things, but once they participate in that study and they see they are not where they should be in comparison to the other labs, but they are clearly outliers. If they are outliers on the best side, that's good, but if they are outliers on the wrong end, then that would at least let them know that there are things that need to be fixed. (PS5)

The variety of initiatives aimed at laboratory/protocol standards emerging from different groups within the wider proteomics community showed both an awareness of the legitimacy discount in the field and an attempt to address it. The Bell et al. (2009) study has been repeated on several occasions (e.g., Boja et al., 2011; Colomée et al., 2012; Wang et al., 2014), and the results indicated a significant improvement in reliability. It is likely that this improvement was a response to the perceived lack of “performance legitimacy” of the science of proteomics.

Further mechanisms to address this, discussed at the 2014 and 2015 HUPO conferences and the 2013 APS conference, were the development of standardization kits, and associated instructional materials, which a variety of groups in Europe, Canada, and the United States created and which were close to being marketed. (Examples discussed in Percy et al., 2015a; 2015b). This would enable laboratories to test for interoperator reliability and focus on training if deficits are identified or as one speaker at APS 2013 puts it, if laboratory staff were not “up to snuff.”