Methodology discourses as boundary work in the construction of engineering education

Abstract

Engineering education research is a new field that emerged in the social sciences over the past 10 years. This analysis of engineering education research demonstrates that methodology discourses have played a central role in the construction and development of the field of engineering education, and that they have done so primarily through boundary work. This article thus contributes to science and technology studies literature by examining the role of methodology discourses in an emerging social science field. I begin with an overview of engineering education research before situating the case within relevant bodies of literature on methodology discourses and boundary work. I then identify two methodology discourses – rigor and methodological diversity – and discuss how they contribute to the construction and development of engineering education research. The article concludes with a discussion of how the findings relate to prior research on methodology discourses and boundary work and implications for future research.

Keywords

academic capital boundary work disciplinary development engineering education research methodology discourses rigor

Introduction

In June 2013, a group of leading engineering education researchers from Virginia Tech, Purdue, Clemson, and the University of Florida gathered for the Qualifying Qualitative Research Quality (Q³) Workshop at the University of Georgia. The workshop’s web site explained,

The Q³ Workshop series seeks to initiate a sustained conversation and build capacity around issues of research quality across a range of interpretive methods currently adopted by the engineering education research community. Through the workshops we hope to develop a shared understanding and common language within the community with the aim of promoting the adoption, acceptance and impact of qualitative methods. (Walther, 2013)

In many ways, this workshop, which was funded by the National Science Foundation (NSF) (Walther, 2012), is an outcome of events that began approximately 10 years ago. Around 2003, engineering education began to emerge as a distinct field of research with academic departments, graduate degrees, and journals dedicated to empirical and systematic engineering education research (EER). In the field’s formative years, methodology discourses in EER emphasized rigor and trustworthiness. Perhaps quality – the focus of the Q³ Workshop – will become the newest keyword in the shifting landscape of methodology discourses constituting the field.

The ways in which themes emerge, persist, and fade in methodology discourses are relevant to a wide range of issues. As Richards and Schuster (1989) suggested, methodological or epistemological discourses are implicated in ‘the history of science as a social institution in its wider context’ (p. 698). Over the past several decades, critical examination of methodology discourses in various scientific contexts, together with studies of boundary work, has explored how scientific methods are used in attempts to differentiate one field from another and science from nonscience. This article contributes to that body of work by addressing the following questions: What methodology discourses can be identified, and what purposes have they served in the context of the emerging field of EER?

Early methodology discourse research was largely concerned with demonstrating the mythic character of the scientific method within the biological and physical sciences. This study of EER contributes to an emerging literature that examines methodology discourses in the social sciences. Following Schuster and Yeo (1986), I examine ‘the ways in which appeals to method are significant in debates surrounding the constitution of disciplines’ (p. xxvi). I identify how two leading discourses – rigor and methodological diversity – have shaped EER via boundary work. In that regard, this analysis differs from discourse work with epistemological and semiotic concerns that examines scientists’ accounts of their practices and beliefs (e.g. Gilbert and Mulkay, 1984) or is concerned with demonstrating the rhetorical nature of science more generally (e.g. Feyerabend, 1993).

In the interest of transparency, I should note my involvement in engineering education. I write this as a critical participant (Downey, 2009) in engineering education, in which I have worked since 2008. I experience the field as both an insider and an outsider, and I participate in the discourses described in this article. Moreover, I have a stake in the development of the field. This analysis is an attempt to step back and examine the field as an object of social inquiry in order to contribute to science and technology studies (STS) conversations on methodology discourses and boundary work.

The article begins with an overview of the emerging field of EER, needed to contextualize the following analysis because EER is a new field and has not received much attention within STS. I then situate the article within relevant bodies of literature. I identify two methodology discourses and demonstrate that they contributed to the construction and ongoing development of EER as a field. It should be emphasized that I am referring to rigor in EER, and not engineering education more broadly, where ideologies of rigor have not undergone similar changes (Riley, 2013). The article concludes with a discussion of the methodology discourse and boundary work claims that can be made from this case and implications for future research.

EER as an emerging field

In the United States, Europe, and Australasia, EER is increasingly promoted and studied as a new field of scholarship or discipline, distinct from the engineering education of old, which has a long history of journals, conferences, and professional societies. About 10 years ago, the field began shifting away from the traditional focus on description and toward research (Borrego and Bernhard, 2011; Borrego et al., 2007, 2008, 2009; Jesiek et al., 2009, 2011).¹ For example, while the Journal of Engineering Education (JEE) dates back a century, in 2003, it was refashioned as a journal that would only publish empirical research.² The shift in activity in EER is evident in a variety of forums, such as journal articles on the status and emergence of the field, large-scale research and advocacy projects by professional societies and their journals, research networks and groups, the refashioning of journals to focus on empirical research, new conference series, and university departments, centers and degrees (Benson et al., 2010; Borrego and Bernhard, 2011; Jamieson and Lohmann, 2009). In 2014, the first Cambridge Handbook of Engineering Education Research will be published (Johri and Olds, in press). With respect to older formal engineering institutions, the newer EER institutions largely exist as distinct spaces within preexisting engineering institutions. For instance, the Center for the Advancement of Scholarship on Engineering Education was created in 2002 within the National Academy of Engineering. The Research in Engineering Education NSF grant program was established within the Division of Engineering Education and Centers, and the engineering education PhD programs at Purdue and Virginia Tech were established within the engineering colleges. In 2009, Jesiek et al. (2009) summarized the emergence of the field:

Claims about the establishment or ‘birth’ of a new discipline are surfacing with increasing regularity …, largely in tandem with the formation of dedicated engineering education departments and graduate degree programs at Purdue in 2004 … and Virginia Tech in 2005. And through three NSF-supported National Engineering Education Research Colloquies (EERC) in 2005 and 2006 … In light of these trends, it has been argued that ‘research’ is replacing ‘reform’ as a primary organizing paradigm for the field of engineering education. (p. 41)³

Tracing the history of EER further back, Jamieson and Lohmann (2009) related recent research trends to US national interests dating back to the 1980s:

The importance of engineering education research began to surface in the United States in the mid-1980s, when the National Science Board issued its report Undergraduate Science, Mathematics, and Engineering Education …, in which it stated: ‘The recommendations of this report make renewed demands on the academic community – especially that its best scholarship [emphasis added] be applied to the manifold activities needed to strengthen undergraduate science, engineering, and mathematics education in the United States’ … These and other efforts paved the way for the assembly of a small community of scholars in engineering education by the beginning of this century. (p. 9)

They note that the growth of the EER community was accelerated by several factors: the repositioning of JEE as an education research journal in 2003, the 2007 launch of the Research in Engineering Education Symposium conference series, and continued NSF funding. Additionally, several large-scale research projects were undertaken by professional societies and journal editors to advance the field. One such project was Advancing Global Capacity for Engineering Education Research, organized by JEE and the European Journal of Engineering Education, which entailed special sessions at 10 conferences around the world in order to collect information about EER in various countries and to foster relationships between researchers (Jesiek et al., 2010; Lohmann, 2008). This was followed Creating a Culture for Scholarly and Systematic Innovation in Engineering Education, a project undertaken by the American Society for Engineering Education (Jamieson and Lohmann, 2009). This project, which sought to enroll engineering faculty in processes of systematic research and innovation, took place in the United States, but the growth of EER is not limited to the United States. In Australia, for example, the first ‘winter school’ for engineering education was held in June 2011 (Australasian Association for Engineering Education [AAEE], 2010), and the professional society, the AAEE, has an Educational Research Methods division. In Europe, there are now degrees offered in engineering education. The European Society for Engineering Education has a working group on EER (SEFI, 2013), and there are also regional research networks, such as the Nordic Network in Engineering Education Research (NNEER, 2013). These activities are being supported by cyber-infrastructure, such as the development of databases for collecting and searching EER publications (Madhavan et al., 2010), online bibliographies (AAEE, 2009; Jesiek and Beddoes, 2013), websites to collect and share knowledge gained from NSF-funded projects (CLEERHUB, 2009), and websites to facilitate networks of researchers (REEN, 2013).

Borrego and Bernhard (2011) provide a more detailed international retrospective in which they describe who does EER, what motivates it, where it is published, what infrastructures support it, and differences between EER in the United States and Europe (see also Beddoes, in press). As they point out, despite the fact that the majority of those involved in EER are still engineers, it is an interdisciplinary field that draws from and collaborates with fields such as education, psychology, sociology, and women’s studies. However, the extent to which it draws on these fields varies significantly, and quantitative and positivist contributions are more readily accepted than are critical qualitative approaches. However, there is no consensus regarding what qualifies as EER or how the field should develop. The field is not epistemologically homogeneous and the discourses identified have not been universally adopted within the community. ‘As both an emerging and interdisciplinary field, EER does not have widely agreed upon expectations for what constitutes quality work’ (Borrego and Bernhard, 2011: 24). Moreover, many of those interested in and supportive of EER question its sustainability, while engineering colleagues question its necessity and value (Jesiek et al., 2010, 2011).

As seen above, the emergence of EER has been linked to institutional structures, national interests, and the activities of professional societies. In contrast, I aim to show that methodology discourses have played an integral role in constructing the new field. Examining the field through this lens allows for new perspectives on the development of the field while contributing to STS scholarship on methodology discourses and boundary work. I turn now to an overview of those traditions that inform my analysis.

Methodology, discourse analysis, and boundary work

STS scholars use discourse analysis to analyze the roles and functions that methodology discourses have played across time and disciplines. As noted above, most research on methodology discourses has focused on the scientific method and the biological and physical sciences, and much of the early work was concerned with demonstrating that scientific methods were normative rather than descriptive of reality, thus asking what role they play if not describing actual practices (Mercer, 2002; Richards and Schuster, 1989; Schuster, 1984; Schuster and Yeo, 1986). For instance, in their study comparing traditional to so-called feminine methods, Richards and Schuster (1989) concluded that both types of methods are myths in the sense that they do not function in the ways claimed, and are instead ‘flexible rhetorical resources’ (p. 697). In addition to identifying the roles played by methodology discourses, other questions that have been raised are as follows: Why are method discourses convincing in the face of evidence that they are not literally true? How is their plausibility created and sustained? (Schuster, 1984).

Schuster and Yeo (1986: xi) identified three different levels on which methodology discourses have operated: internal technical debates over knowledge claims, ‘institutional and disciplinary organization and politics’, and the public level involving nonscientist stakeholders. The second level is where methodology discourses play the biggest role in EER:

[on the second level,] one finds essentially political phenomena, such as the formulation and enforcement of institutional ideologies within scientific societies; the demarcation (and internal consolidation) of socio-cognitive groupings within or across disciplines; and the production and advancement of vested interests in particular disciplinary approaches, tools or techniques. These phenomena can be directly relevant to the ‘internal’ evolution of fields and to the construction of knowledge within them. Such institutional and disciplinary politics are often conducted in part through attempts to impose views of proper methodology in order to account for alleged weaknesses in theories, and to advance preferred research programmes. (Schuster and Yeo, 1986: xiii)

This description aptly applies to the history of EER. Because most of the methodology discourse analysis has focused on biological and physical sciences, however, it is also important to recognize the limitations, given differences such as less consensus and greater variety of methods, of applying that work to the social sciences (Biglan, 1973; Lodahl and Gordon, 1972). Method research on social sciences has tended to trace the development of specific methods or types of methods, rather than the purposes methodology discourses serve. For example, the development of qualitative methods and feminist methods within a field or discipline has often been studied (Beddoes, 2012; Harding, 1987; Hart, 2006; MacNabb et al., 2001; Riley, 1999; Stacey and Thorne, 1985; Stanley, 1997), as has the fate of positivism across fields (Steinmetz, 2005).

Gieryn (1999) suggests that boundary work is interesting because of variations and changes across episodes; here, I trace the ‘contingencies of the moment’ (p. 5) within EER. Kinchy and Kleinman (2003) explore boundary work in ecology aimed at differentiating science from politics, focusing on why certain discourses are used and why they are successful or not. In contrast to Gieryn (1999), who saw boundary work as strategic and instrumental, Kinchy and Kleinman (2003) contend that ecology’s boundary work makes use of ‘historically resonant discourses’ in ways that are not consciously strategic. Nonetheless, most boundary work is concerned with credibility in some way (Evans, 2005; Gieryn, 1999). In another study, Burri (2008) explores how ‘doing distinctions’ in radiology is used to ‘improve both the prestige of the profession and actors’ individual status within the scientific community’ (p. 35). Technology in the form of new medical imaging meant that radiologists had to renegotiate their identities. As in my case, Burri explores boundary work among actors within a scientific community, rather than between science and politics or between disciplines. The case of radiology is also similar to EER in that boundary work and doing distinctions improved both the prestige of the entire field and the positions of individuals within that field. Finally, of particular significance to this analysis is Vincenti and Bloor’s (2003) work on the role of rigor in demarcating pure from applied mathematics and creating disciplinary boundaries in the field of aerodynamics. I will revisit their interactional model of rigor below.

Constructing engineering education through methodology discourses

Broadly speaking, the history of EER can be traced back over two distinct but intersecting lines of discourse integral to its development. The first is a discourse of rigor, which later faded, giving way to a discourse of methodological diversity. In both cases, methodology discourses constituted a type of boundary work. In the case of rigor, the boundaries were drawn between scholarly, rigorous research on the one hand, and, on the other, purely descriptive or ‘show and tell’ publications, which were the dominant mode of engineering education publications prior to 2003. In contrast, the methodological diversity discourse was mobilized to expand the methodological boundaries of what was considered EER, while still engaging a modified discourse of rigor and trustworthiness.

The evidence presented below is drawn from relevant engineering education journals and conferences. There are currently only four journals devoted specifically to engineering education: Journal of Engineering Education (JEE), European Journal of Engineering Education, International Journal of Engineering Education, and Advances in Engineering Education, which was founded in 2007. During the time covered in this analysis, aside from those identified below, there were no counter discourses or resistance mobilized in these formal publications. Resistance that did occur is discussed below, but was informal in nature and hard to capture as evidence. The scholars cited in relation to the discourse of rigor are primarily from the United States, reflecting the fact that that is where the discipline-building efforts were originally based. Scholars cited in the methodological diversity section represent a more international group, coming from Europe, Australia, and South Africa.

Rigor

Beginning around 2003, rigor increasingly became a boundary word meant to demarcate and legitimize the new field of EER. As Jesiek et al. (2009) wrote,

Beginning in 2003, editor Jack Lohmann contributed to this movement by positioning the JEE as an ‘archival record of research in engineering education’ … The journal’s goals now include advancing the rigor of engineering education scholarship and building a vibrant ‘discipline’ or ‘community’ of dedicated engineering education researchers. (p. 41)

Over the next several years, the word ‘rigor’ and its derivatives were increasingly seen in the titles of publications and NSF grants. Given the nature of the data for this analysis, I use the term ‘rigor discourse’ to refer to those literal uses of the term ‘rigor’, but the push for rigor has larger implications beyond the use of the term. Examples include ‘Rigorous research in engineering education: Developing a community of practice’ (Streveler et al., 2005); ‘Conducting rigorous research in engineering education’ (Streveler and Smith, 2006); and ‘Development of engineering education as a rigorous discipline: A study of the publication patterns of four coalitions’ (Borrego, 2007). Several of these publications were funded by an NSF grant entitled ‘Rigorous research in engineering education: Creating a community of practice’ (Huband et al., 2004), which aimed to teach engineering educators how to become rigorous researchers. The point I wish to make here is not simply that publication and grant titles contained the word ‘rigor’, but rather that the content of those publications explicitly linked the construction of the new field with rigorous research, differentiating rigorous research from prior engineering education scholarship. The demarcation and legitimation purposes of this discourse are evident in assertions such as:

The field of engineering education is in the process of reinventing itself and the January 2005 special issue of the Journal of Engineering Education was a milestone event in this transition … The four most recent guest editorials have documented this reinvention and have suggested shifts that are needed to establish engineering education as a serious and rigorous research-based discipline … (Streveler and Smith, 2006: 103)

And:

Calls for increased rigor can be understood in terms of the on-going development of engineering education as a new discipline … It might be said that engineering education now has the infrastructure but not the research consensus to be called a distinct discipline. In this case, calls for rigor would be an appropriate next step to developing the field of engineering education. (Borrego, 2007: 5, 6)

And JEE (2005) aims to ‘catalyze a global community of scholars and practitioners dedicated to advancing rigorous scholarship in engineering education’ (p. 283).

It should be asked here, What did these engineering education researchers mean by rigorous? Several of the leading scholars who first mobilized the rigor discourse explained rigor with reference to two frameworks. One framework was the US National Research Council’s guidelines for Scientific Research in Education. For example, Streveler and Smith (2006) defined rigorous research by

using the guidelines provided by the National Research Council (NRC) in the work, Scientific Research in Education … According to this NRC report, scientific or rigorous research in education (including engineering education) should: 1. Pose significant questions that can be answered empirically. 2. Link research to relevant theory. 3. Use methods that permit direct investigation of the question. 4. Provide a coherent and explicit chain of reasoning. 5. Replicate and generalize across studies. 6. Disclose research to encourage professional scrutiny and critique. (p. 103)

They then noted that ‘These guidelines parallel the criteria for rigorous research in engineering and science and thus are familiar to engineering educators’ (Streveler and Smith, 2006: 103).

Second, these authors also laid out a hierarchy of ‘levels of inquiry’, which was later taken up by others. In this framework, there are four levels of inquiry. Level 1 is Excellent teaching. Level 2 is Scholarly Teaching. Level 3 is the Scholarship of Teaching, and level 4 is Rigorous Research in Engineering Education (Streveler et al., 2005, 2007). Level 4 is uniquely characterized as

(1) Begin with a research question not an assessment question. Assessment questions often deal with the ‘what’ or ‘how much’ of learning, while research questions more often focus on the ‘why’ or ‘how’ of learning … (2) Tying the question to learning, pedagogical, or social theory and interpreting the results of the research in light of theory. This will allow for the research to build theory and can increase the significance of the findings … (3) Paying careful attention to design of the study and the methods used. This will enable the study to hold up to scrutiny by a broad audience, again creating a potential for greater impact of results. (Borrego, 2007: 7)

This framework was adapted from Boyer’s (1990) report Scholarship Reconsidered, in which he delineated four levels of education scholarship: discovery, integration, application, and teaching.

The focus on rigor can be traced back to the fact that the vast majority of the engineering education community are engineers and the intended audience largely engineering professors. They have been trained to equate rigor with quantitative results and traditional notions of objectivity. As Borrego and Bernhard (2011) stated, ‘Although manifested in a variety of ways, the legitimacy of EER is a widespread concern’. In order to be seen as legitimate, then, engineering education researchers needed to prove that their research was scientific and rigorous, as those terms were understood among engineers (Beddoes, 2012; Riley, 2013). The push for rigor ‘can be understood in terms of development of a new field by engineers who have been trained in well-established technical fields to expect more clearly defined standards of rigor’ (Borrego, 2007: 6). Further contextualizing the issue in 2010, Borrego et al. (2010) explained,

Experienced STEM education researchers know that disseminating their innovations, together with compelling assessment results, is necessary – but not sufficient – to stimulate faculty to change their teaching practices … Within U.S. engineering education, this realization was translated into a call for more rigorous – and therefore more convincing – research. (p. 185)

As we will see, this need for legitimacy among engineers influenced the later methodological diversity discourse, even as calls for rigor faded.

The new insistence on rigor was not well received by all, and created tensions between United States and European and Australian communities (Borrego and Bernhard, 2011). Some saw it as exclusionary and US-centric. Reporting on several international conference sessions designed to advance EER, Jesiek et al. (2010) stated that ‘One Global Colloquium group characterized EER as “stratified from local to rigorous”, and they expressed concerns about the field being overly focused on the latter’ (p. 113). Additionally,

The influence of the concept of ‘rigorous research’ is reflected in many of these goals, although precise standards or definitions of rigor were not developed. Conversations about specific goals and criteria for research were often low consensus and adversarial in character, with some participants defensive of specific research subjects or approaches. Specifically, we observed that tensions between quantitative and qualitative research methods contributed to some of these disagreements. (Jesiek et al., 2009: 47)

Due in part to such international resistance, most of which occurred ‘behind the scenes’, so to speak, most leaders in the rigor discourse stopped using the term and made efforts to recognize national differences. Evidence of this trend can be found in Borrego and Bernhard (2011), who discuss United States and European EER, stressing difference rather than inferiority. That article marked a significant shift from publications seen just 5 years earlier, as Borrego was a prominent early mobilizer of ‘rigor’. Additionally, in her keynote address at the 2012 AAEE conference, Streveler (2012), a prominent early mobilizer of the rigor discourse, spoke about her contribution to that discourse with mild regret, explaining that she regretted the hierarchy it had set up between different types of engineering education scholarship, thus recognizing the boundary work that had been done, without using that terminology. Notably, in contrast to her work from 2005 and 2006, instead of using the word ‘rigor’, the title of the talk was ‘High quality engineering education research: Key elements and persistent misconceptions’ (Streveler, 2012). Most recently, in June 2013 at the American Society for Engineering Education Annual Conference, Riley (2013) gave a lecture critiquing the ways in which exclusionary engineering standards of rigor have influenced engineering education and EER. In sum, those scholars who first and most prominently mobilized the rigor discourse are no longer doing so. They no longer use the term ‘rigor’ in their publications, or publish articles on the need for rigorous research, and instead, they, and others, have started to advocate broader notions of quality.

Despite the tensions and international variations, however, the influence of the rigor discourse was widespread. It allowed early EER leaders to differentiate, at least rhetorically, a new field with a new identity. The discourse was also manifested in statements by qualitative researchers needing to assert that their work was rigorous. For example, Paretti (2008) asserted, ‘[q]ualitative case studies are well-established, frequently used methods of rigorous research for exploring communication learning’ (p. 492). Similarly, Leydens et al. (2004) argued that

[c]riticisms of qualitative research methods as unsystematic or lacking in rigor may stem from a misunderstanding … Since such research often entails many variables and few cases … its procedures for establishing trustworthiness are fundamentally different than those used to establish rigor in quantitative research. (p. 65)

This point will be revisited below when we see that concerns over trustworthiness persist in the methodological diversity discourse.

Methodological diversity

The rigor discourse has not disappeared. For instance, Purdue’s School of Engineering Education still proclaims on its web site that it is ‘the world’s first engineering education doctoral program, for students who wish to pursue rigorous research in how engineering is best taught, learned, and practiced’ (School of Engineering Education, Purdue University, 2013). Yet, the rigor discourse has faded, and another discourse has grown up alongside it with increasing popularity: methodological diversity. Rather than being caused by any significant change of audience or professional standing, the methodological diversity discourse seems to be a reaction to the rigor discourse in two ways. At the same time that researchers resisted some of the rigor discourse’s implications, they also used the rigor discourse’s claims about EER as a new field to justify why methodological diversity was needed (for the good of the field).

As early as 2004, engineering education scholars were examining the state of qualitative methods, noting the dominance of quantitative and positivist work (e.g. Leydens et al., 2004). Beginning in about 2008, a series of articles explicitly advocating a wider range of methods and theories emerged. Like rigor, methodological diversity played a role in constructing the field because methodological diversity was advocated specifically in terms of improving and sustaining the field it was simultaneously helping to construct. This discourse was mobilized with the aim of broadening the methodological boundaries of the field in order to improve it and ensure its success. For example, Koro-Ljungberg and Douglas (2008) stated that

We believe that it is this growing diversity of approaches and perspectives that marks the field of engineering education as vibrant and strong. The use of qualitative methods in these articles provides important insights that would not have been possible through quantitative approaches. (p. 172)

In 2009, Jawitz and Case (2009) argued that making one’s theoretical perspective (in which they included methodology) explicit would ‘assist in building a community of engineering education researchers with a common language and an understanding and respect for the different approaches that exist in educational research’ (p. 154). Borrego et al. (2009) published an article entitled ‘Quantitative, qualitative, and mixed methods research in engineering education’. Douglas et al. (2010) published ‘Challenges and promises of overcoming epistemological and methodological partiality: Advancing engineering education through acceptance of diverse ways of knowing’, summarizing the methodological changes and central tensions engineering education had experienced, and ultimately arguing that epistemological and methodological diversity would benefit and advance the field in many different ways. Then, in 2011, Case and Light wrote an article entitled ‘Emerging methodologies in engineering education research’, in which they identified the following as emerging methodologies: case study, grounded theory, ethnography, action research, phenomenography, discourse analysis, and narrative analysis. They argued that

a compelling case has been presented for the significance of engineering education research, as well as the ongoing need for development of this scholarly field. This article focuses on methodology as a crucial area with which researchers need to grapple in order for the quality and scope of research to continue to develop. It is argued that … researchers need to consider a broad range of methodological options … We suggest that a further deep engagement with issues of methodology is likely to yield a dramatic growth in the range of research findings that can be generated in the field. (Case and Light, 2011: 186, 207)

Most recently, I built on Waller’s (2005) conference paper on ‘Feminist methodologies and engineering education research’, arguing that ‘feminist methodologies have much to offer engineering education researchers (even those who do not identify as feminists) interested in advancing the field’s scholarship in ways that challenge inequalities built into social science processes’ (Beddoes, 2013: 115), and that ‘[b]y extending conversations about methodology, this analysis demonstrates what could be gained through greater engagement with feminist theories and methodologies that challenge commonly held beliefs about knowledge and social research’ (p. 108).

The methodological diversity discourse aimed to broaden the methodological boundaries of EER while still upholding an ideal of good, systematic, sound methods. Boundary work, even when about expanding boundaries, is, as Gieryn (1999) suggests, about ‘objective methods and trust-worthy experts’ (p. 10). Each of the articles quoted above emphasized that the new methodological diversity they were advocating was just as legitimate and ‘trustworthy’ as quantitative, positivist research. For example, Koro-Ljungberg and Douglas (2008) stated,

While quantitative research requires use of statistical methods which can provide an aura of trustworthiness, qualitative research can appear at first glance as if it simply involves interviewing a few people and then writing up a summary … In fact, qualitative research can be just as difficult to conceptualize, and be as methodologically and theoretically challenging, if not more challenging, than quantitative research. It is important for qualitative researchers to strive for high standards of rigor … Continuous and systematic exposure to the methodological tools available to study complex problems and socio-cultural phenomena … would assist researchers interested in qualitative research questions and methods to conduct rigorous studies. (p. 172–73; see also Walden and Foor, 2008)

In a similar fashion, Case and Light (2011) explained that

The means for assuring research rigor in these perspectives tend to rest on a different epistemological foundation. Instead of the traditional tests for validity, the researcher has a range of means of assuring the integrity of results, what Lincoln and Guba refer to as ‘trustworthiness’. The focus shifts from traditional measures of objectivity towards ‘confirmability’, the assurance that research findings are rooted in contexts and persons apart from the researcher, and that they did not merely arise in the researcher’s imagination. (p. 188)

Douglas et al. (2010) similarly addressed trust issues when justifying their push for methodological diversity by explaining that ‘when scholarly work is discursively and methodologically diverse and is built on methodological congruency (interconnectedness between research questions, purposes and methods) the trustworthiness of qualitative research could be improved and randomness in qualitative inquiry could be avoided’ (p. 253). Likewise, I addressed concerns over legitimacy: ‘When the limitations and biases of traditional research are recognized, it becomes clear that feminist methodologies offer alternative paths for improving EER that are no less valid or scholarly’ (Beddoes, 2013: 115–16). Thus, the notion of rigor continued to come into play in the methodological diversity discourse, even as it has increasingly been replaced with terms such as ‘trustworthy’ and ‘quality’.

Discussion and conclusion

In both cases, methodology discourses were mobilized to improve, expand, or advance the field of EER. Initially, a discourse of rigor served to legitimate the field as scientific, construct it as a new field, and distinguish it from earlier efforts. More recently that discourse has given way to diversifying and expanding methodologies, with the discourse of rigor falling out of use by many of those who originally mobilized it. The history presented here supports the assertion that in cases where a field lacks consensus over a ‘right’ methodology, method discourse as resource and topic will be ubiquitous (Schuster, 1984: 27).

Those within engineering education who have commented on EER as a new field or discipline have attributed its emergence to national interests and cited as evidence of its emergence typical markers of a field, such as academic departments, degree programs, and publishing outlets. The boundary work done by methodology discourses in the actual construction and development of the field has thus far not been identified or explored. Yet, as this article has demonstrated, methodology discourses have been central to the construction of EER.

In order to distinguish EER from earlier descriptive traditions, boundary work in the form of a discourse of rigorous research was mobilized. Vincenti and Bloor (2003) put forth an interactional model of rigor, arguing for a social definition of rigor (as opposed to one based in mathematical reality), one that recognizes rigor as a community-defined concept. Moreover, they contend that critical discourses play a ‘constitutive role’ in that definition, and we cannot assume that rigor means the same thing in different contexts:

Rigor … is seen as arising from critical arguments conducted by persons with specific goals and aims. A line of reasoning is called ‘rigorous’ if it passes the critical scrutiny of an interacting community of accepted experts … [In] the interaction model [rigor] is relative to the norms sustained by the relevant community. (Vincenti and Bloor, 2003: 491)

In the case of EER, rigor was explained with reference to two different frameworks from national research guidelines and higher education literature. It was intended to create a boundary between the new research and earlier scholarship characterized as more descriptive and less systematic.

The push for rigorous research as the boundary for the newly emerging field should be understood in the context of a primarily engineering audience, who may have dismissed social science research as less scientific and objective than traditional engineering research. Because the target audience was ostensibly engineering faculty members trained as scientists and engineers, EER needed to be seen as legitimate, credible, scientific research by engineers with positivist notions of objectivity and generalizability. Although traditional notions of objectivity and belief in the greater objectivity of quantitative research have long been criticized for being inaccurate, normative, gendered, and racialized (Harding, 1987, 2004, 2006; Harding and Norberg, 2005; Longino, 1990; Sprague, 2005), such notions and beliefs persist among engineers (Riley, 2008, 2013; Slaton, 2010). Thus, EER followed in the footsteps of fields such as psychology and epidemiology, which emulated the methodologies of the ‘hard sciences’ as a means to legitimacy (Amsterdamska, 2005; Sherif, 1987). In engineering education, these struggles have shaped the kinds of scholarship that can get published and funded. And yet despite those efforts, EER still struggles for respect and legitimacy among traditional engineering colleagues (Borrego and Bernhard, 2011; Jesiek et al., 2010, 2011).

In the past few years, the discourse of rigor faded and one of methodological diversity emerged. Scholars mobilizing this latter discourse argued for expanding methodological boundaries of the field, contending that its success depends on increasing the range of theories and methods that are trusted and valued within the engineering education community. However, the methodological diversity discourse continues to emphasize that broadening to include new theories and methods maintains the field’s scientific legitimacy or trustworthiness. Thus, the boundary work being done by this discourse is twofold: it attempts to expand the range of the field while keeping the field within a credible scientific space. The continued emphasis on ‘trustworthy’ research, centered on employing the ‘right’ methods, calls to mind Law’s (2004) critiques of the quest for proper, ‘hygienic’ methods (p. 9).

Identifying particular methodology discourses and their functioning in a given case is but a first step. Further analysis is needed to better understand why those discourses were deployed and why they were successful (Kinchy and Kleinman, 2003). In this case, we should first ask in what ways have these discourses been successful? The fading of the rigor discourse hides the early role it played in establishing and bounding the field, as well as its relation to a more microlevel, individual view of success. While methodology discourses were integral to constructing and developing the new field of EER, they also contributed to individuals’ career successes.

Engineering education researchers acquired academic capital (Bourdieu, 1984; Eddy, 2006) by mobilizing methodology discourses. Engineering education researchers deploying methodology discourses have received funding from the NSF, published numerous articles, and led related events such as workshops. In addition to the numerous articles discussed above, engineering education scholars have also been able to publish papers with the sole aim of introducing one specific method to an engineering education audience (e.g. Banerjee and Pawley, 2010). For individuals, then, methodology discourses have been a means to academic capital because they were a way to differentiate, legitimate, and establish the superiority of their work over prior work, leading to grants and publications.

Parallels can then be drawn between EER and other fields. As noted, in radiology new imaging technology and the images it produced served as ‘epistemic resources deployed in practices of boundary work and distinction’ (Burri, 2008: 35). As epistemic resources, the images contributed to radiologists’ social capital in the medical arena. Methodology discourses have served a similar purpose in EER where they are rhetorical resources, or ‘discursive structures and strategies used to render arguments persuasive in given situations’ (Schuster and Yeo, 1986: xii). In sum, methodology discourses have been epistemic and rhetorical resources mobilized by individuals (and teams of individuals) engaged in boundary work. That boundary work served to advance their own positions within the emerging community by contributing to funding, publications, and reputations.

Of course, the success of those scholars (and by extension the field) as measured by their purported goal of influencing actual engineering education practices is another question, one that is beyond the scope of this article, but has begun to be explored by other scholars (Riley et al., 2013). Likewise, other open questions include the extent to which the rigor discourse actually worked to convince engineering faculty of the legitimacy of EER, the effects that the shift to the methodological diversity discourse has on perceptions of EER among engineering faculty, and the epistemological and identity issues implicated in shifts toward greater methodological diversity.

The key point is that it was through the boundary work done by methodology discourses that these individual successes were linked to the emergence of the field itself. Methodology discourses were a resource available to individual actors, and those actors used them to help construct a new field by differentiating it from prior scholarship, and later, to broaden the methodological boundaries of the field. This case thus seems more in line with Gieryn’s original conception of boundary work as strategic and instrumental, rather than with later analyses suggesting that in some cases ‘boundary-work as self-conscious and strategic often does not provide an adequate understanding of how the credibility of science is maintained’ (Kinchy and Kleinman, 2003: 871); Kinchy and Kleinman observed that in ecology, scientists simply engaged with ‘historically resonant discourses’. While an appeal to historically resonant discourses may explain the rigor discourse in EER, it holds less explanatory power for the methodological diversity discourse. These contrasts would seem to warrant further consideration of variation in levels of strategy in boundary work across different fields and contexts.

The central purpose of this analysis is to identify ways in which methodological discourses helped construct a new field; other aspects of the story have necessarily remained underexplored. For instance, parallels can also be drawn to work on rhetorics of rigor and objectivity in other fields, such as Porter’s (1995) work on quantification as a ‘technology of trust’ in public decision making and engineering, and analyses of evidence-based medicine (Timmermans, 2010; Timmermans and Berg, 2003). Furthermore, the case also presents interesting interplay between science-in-general and science-in-particular (Michael, 1992). The rigor discourse (and corresponding boundary work) appealed to a sense of science-in-general; yet, some examples within the methodological diversity discourse (and corresponding boundary work) may represent more immediate and specific engineering education concerns, which would be understood as science-in-particular. For example, Riley’s (2013) critique of rigor was immediately concerned with diversity in engineering. Nonetheless, I would argue that in this case, both the appeals to science-in-general and science-in-particular have been equally strategic, albeit to different ends. Of course, I cannot make any definitive claims regarding authors’ intentions or level of reflexivity from their texts (Fuhrman and Oehler, 1986). Potential tensions between concerns over the image of the field and appeals to science-in-general, and particular concerns (e.g. diversity) could prove an interesting story to follow.

The history I have traced in this article is quite recent, spanning the past 10 years, and is ongoing. Clearly, EER is continuing to develop and change. Questions remain about its ultimate success and its status relative to other academic engineering disciplines; nonetheless, the field currently presents a valuable opportunity to examine the emergence of an academic field as it unfolds. Continuing to trace the roles methodology discourses play in its future successes (or decline) can offer STS scholars further insight into the types of work – boundary and otherwise – that methodology discourses do.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Notes

Author biography

Kacey Beddoes is a Postdoctoral Researcher in the School of Engineering Education at Purdue University, USA. She received her PhD in Science and Technology Studies from Virginia Tech in 2011. Her current research interests include interdisciplinary engineering education, gender in engineering education research, research methodologies, and peer review. She serves as Managing Editor of Engineering Studies and Assistant Editor of the Global Engineering Series at Morgan and Claypool publishers.

References

Amsterdamska

(2005) Demarcating epidemiology. Science, Technology, & Human Values 30(1): 17–51.

Australasian Association for Engineering Education (AAEE) (2009) Bibliography. Available at: http://aaee-scholar.pbworks.com/w/page/1177020/Bibliography (accessed 19 August 2013).

Australasian Association for Engineering Education (AAEE) (2010) AAEE Newsletter, November.

Banerjee

Pawley

(2010) Institutional ethnography: A research method to investigate the work-life experiences of women faculty members in STEM disciplines. Pesented at The American Society for engineering education annual conference, Louisville, KY, 20–23 June.

Beddoes

(2012) Feminist scholarship in engineering education: Challenges and tensions. Engineering Studies 4(3): 205–232.

Beddoes

(2013) Feminist methodologies and engineering education research. European Journal of Engineering Education 38(1): 107–118.

Beddoes

(in press) Using peer reviews to examine the micropolitics and disciplinary development of engineering education: A case study. Discourse: Studies in the Cultural Politics of Education 35(3).

Benson

Becker

Cooper

Griffin

Smith

(2010) Engineering education: Departments, degrees and directions. International Journal of Engineering Education 26(5): 1042–1048.

Biglan

(1973) The characteristics of subject matter in different academic areas. Journal of Applied Psychology 57(3): 195–203.

10.

Borrego

(2007) Development of engineering education as a rigorous discipline: A study of the publication patterns of four coalitions. Journal of Engineering Education 96(1): 5–18.

11.

Borrego

Bernhard

(2011) The emergence of engineering education research as an internationally connected field of inquiry. Journal of Engineering Education 100(1): 14–47.

12.

Borrego

Douglas

Amelink

(2009) Quantitative, qualitative, and mixed research methods in engineering education. Journal of Engineering Education 98(1): 53–66.

13.

Borrego

Froyd

Knight

(2007) Accelerating emergence of engineering education via the International Conference on Research in Engineering Education (ICREE). Journal of Engineering Education 96(4): 281–282.

14.

Borrego

Froyd

Hall

(2010) Diffusion of engineering education innovations: A survey of awareness and adoption rates in U.S. engineering departments. Journal of Engineering Education 99(3): 185–207.

15.

Borrego

Streveler

Miller

Smith

(2008) A new paradigm for a new field: Communicating representations of engineering education research. Journal of Engineering Education 97(2): 147–162.

16.

Bourdieu

(1984) Distinction: A Social Critique of the Judgment of Taste (trans. Nice

). Cambridge, MA: Harvard University Press.

17.

Boyer

(1990) Scholarship Reconsidered: Priorities of the Professoriate. Princeton, NJ: The Carnegie Foundation for the Advancement of Teaching.

18.

Burri

(2008) Doing distinctions: Boundary work and symbolic capital in radiology. Social Studies of Science 38(1): 35–62.

19.

Case

Light

(2011) Emerging methodologies in engineering education research. Journal of Engineering Education 100(1): 186–210.

20.

CLEERHUB (2009) Collaboratory for engineering education research. Available at: http://cleerhub.org/ (accessed 19 August 2013).

21.

Douglas

Koro-Ljungberg

Borrego

(2010) Challenges and promises of overcoming epistemological and methodological partiality: Advancing engineering education through acceptance of diverse ways of knowing. European Journal of Engineering Education 35(3): 247–257.

22.

Downey

(2009) What is engineering studies for? Dominant practices and scalable scholarship. Engineering Studies 1(1): 55–76.

23.

Eddy

(2006) Academic capital, postgraduate research and British universities: A Bourdieu inspired reflection. Discourse: Learning and Teaching in Philosophical and Religious Studies 6(1): 211–223.

24.

Evans

(2005) Introduction: Demarcation socialized: Constructing boundaries and recognizing difference. Science, Technology, & Human Values 30(1): 3–16.

25.

Feyerabend

(1993) Against Method: Outline of an Anarchistic Theory of Knowledge. New York: Verso.

26.

Fuhrman

Oehler

(1986) Discourse analysis and reflexivity. Social Studies of Science 16(2): 293–307.

27.

Gieryn

(1999) Cultural Boundaries of Science: Credibility on the Line. Chicago, IL: The University of Chicago Press.

28.

Gilbert

Mulkay

(1984) Opening Pandora’s Box: A Sociological Analysis of Scientists’ Discourse. New York: Cambridge University Press.

29.

Harding

(ed.) (1987) Feminism and Methodology. Bloomington, IN: Indiana University Press.

30.

Harding

(2004) Rethinking standpoint epistemology: What is ‘strong objectivity’? In: Hesse-Biber

Yaiser

(eds) Feminist Perspectives on Social Research. New York: Oxford University Press, pp. 39–64.

31.

Harding

(2006) Science and Social Inequality: Feminist and Postcolonial Issues. Champaign, IL: University of Illinois Press.

32.

Harding

Norberg

(2005) New feminist approaches to social science methodologies: An introduction. Signs 30(4): 2009–2015.

33.

Hart

(2006) Women and feminism in higher education scholarship: An analysis of three core journals. Journal of Higher Education 77(1): 40–61.

34.

Huband

Smith

Streveler

(2004) Rigorous research in engineering education: Creating a community of practice. NSF DUE Grant #0341127. Arlington, VA: National Science Foundation.

35.

Jamieson

Lohmann

(2009) Creating a Culture for Scholarly and Systematic Innovation in Engineering Education. Washington, DC: American Society for Engineering Education. Available at: http://www.asee.org/about-us/the-organization/advisory-committees/CCSSIE/CCSSIEE_Phase1Report_June2009.pdf (accessed 19 August 2013).

36.

Jawitz

Case

(2009) Communicating your findings in engineering education: The value of making your theoretical perspective explicit. European Journal of Engineering Education 34(2): 149–154.

37.

Jesiek

Beddoes

(2013) Diversity in engineering (DinE) bibliography. Available at: http://inesweb.org/dine/ (accessed 19 August 2013).

38.

Jesiek

Borrego

Beddoes

(2010) Advancing Global Capacity for Engineering Education Research (AGCEER): Relating research to practice, policy, and industry. Journal of Engineering Education 99(2): 107–119.

39.

Jesiek

Borrego

Beddoes

Hurtado

Rajendran

Sangam

(2011) Mapping global trends in engineering education research, 2005–2008. International Journal of Engineering Education 27(1): 77–90.

40.

Jesiek

Newswander

Borrego

(2009) Engineering education research: Discipline, Community, or Field? Journal of Engineering Education 98(1): 39–52.

41.

Johri

Olds

(eds) (in press) The Cambridge Handbook of Engineering Education Research. New York: Cambridge University Press.

42.

Journal of Engineering Education (JEE) (2005) JEE strategic plan, 2005–2010: Advancing engineering education research worldwide. Journal of Engineering Education 94(3): 283–284.

43.

Kinchy

Kleinman

(2003) Organizing credibility: Discursive and organizational orthodoxy on the borders of ecology and politics. Social Studies of Science 33(6): 869–896.

44.

Koro-Ljungberg

Douglas

(2008) State of qualitative research in engineering education: Meta-analysis of JEE articles, 2005–2006. Journal of Engineering Education 97(2): 163–175.

45.

Law

(2004) After Method: Mess in Social Science Research. London: Routledge.

46.

Leydens

Moskal

Pavelich

(2004) Qualitative methods used in the assessment of engineering education. Journal of Engineering Education 93(1): 65–72.

47.

Lodahl

Gordon

(1972) The structure of scientific fields and the functioning of university graduate departments. American Sociological Review 37(1): 57–72.

48.

Lohmann

(2008) Advancing the Global Capacity for Engineering Education Research (AGCEER): A year of international dialogue. Presented at The American Society for Engineering Education annual conference, Pittsburgh, PA, 22–25 June.

49.

Longino

(1990) Science as Social Knowledge: Values and Objectivity in Scientific Inquiry. Princeton, NJ: Princeton University Press.

50.

MacNabb

Cherry

Popham

Prys

(eds) (2001) Transforming the Disciplines: A Women’s Studies Primer. Binghamton, NY: The Haworth Press.

51.

Madhavan

Xian

Vorvoreanu

Jesiek

Johri

Wang

. (2010) Interactive Knowledge Networks for Engineering Education Research (iKNEER). Available at: http://www.ikneer.org (accessed 19 August 2013).

52.

Mercer

(2002) Scientific method discourses in the construction of ‘EMF Science’: Interests, resources and rhetoric in submissions to a public inquiry. Social Studies of Science 32(2): 205–233.

53.

Michael

(1992) Lay discourses of science: Science-in-general, science-in-particular, and self. Science, Technology, & Human Values 17(3): 313–333.

54.

NNEER (2013) Nordic network in engineering education research. Available at: http://www.didaktik.itn.liu.se/nneer.html (accessed 19 August 2013).

55.

Paretti

(2008) Teaching communication in capstone design: The role of the instructor in situated learning. Journal of Engineering Education 97(4): 491–503.

56.

Porter

(1995) Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton, NJ: Princeton University Press.

57.

REEN (2013) Research in engineering education network. Available at: http://grou.ps/reen/home (accessed 19 August 2013).

58.

Richards

Schuster

(1989) The feminine method as myth and accounting resource: A challenge to gender studies and social studies of science. Social Studies of Science 19(4): 697–720.

59.

Riley

(2008) Engineering and Social Justice. San Rafael, CA: Morgan & Claypool Publishers.

60.

Riley

(2013) Rigor/us: Merit standards and diversity in engineering education. Distinguished Lecture. Presented at The American Society for Engineering Education annual conference, Atlanta, GA, 23–26 June.

61.

Riley

Henry

Leighton

(2013) What makes faculty adopt or resist change in engineering education? Presented at The American Society for Engineering Education annual conference, Atlanta, GA, 23–26 June.

62.

Riley

(1999) Challenging demography: Contributions from feminist theory. Sociological Forum 14(3): 369–397.

63.

School of Engineering Education, Purdue University (2013) Graduate program. Available at: https://engineering.purdue.edu/ENE/Academics/Graduate (accessed 19 August 2013).

64.

Schuster

(1984) Methodologies as mythic structures: A preface to the future historiography of method. Metascience 1(2): 15–36.

65.

Schuster

Yeo

(1986) Introduction. In: Schuster

Yeo

(eds) The Politics and Rhetoric of Scientific Method. Dordrecht: D. Reidel Publishing Company, pp. ix–xxxvii.

66.

SEFI (2013) Working group on engineering education research. Available at: http://www.sefi.be/?page_id=1192 (accessed 19 August 2013).

67.

Sherif

(1987) Bias in psychology. In: Harding

(ed.) Feminism and Methodology. Bloomington, IN: Indiana University Press, pp. 37–56.

68.

Slaton

(2010) Race, Rigor, and Selectivity in U.S. Engineering: The History of an Occupational Color Line. Cambridge, MA: Harvard University Press.

69.

Sprague

(2005) Feminist Methodologies for Critical Researchers: Bridging Differences. Walnut Creek, CA: AltaMira Press.

70.

Stacey

Thorne

(1985) The missing feminist revolution in sociology. Social Problems 32(4): 301–316.

71.

Stanley

(1997) Knowing Feminisms: On Academic Borders, Territories and Tribes. Thousand Oaks, CA: SAGE.

72.

Steinmetz

(ed.) (2005) The Politics of Method in the Human Sciences: Positivism and its Epistemological Others. Durham, NC: Duke University Press.

73.

Streveler

(2012) High quality engineering education research (EER): Key elements and persistent misconceptions. Presented at The Australasian Association for Engineering Education annual conference, Melbourne, VIC, Australia, 3–5 December. Available at: https://www.engineersaustralia.org.au/sites/default/files/shado/aaee-2012-keynote-streveler.pdf (accessed 19 August 2013).

74.

Streveler

Smith

(2006) Guest editorial: Conducting rigorous research in engineering education. Journal of Engineering Education 95(2): 103–105.

75.

Streveler

Borrego

Smith

(2007) Moving from the scholarship of teaching and learning to educational research: An example from engineering. In: Robertson

Nilson

(eds) To Improve the Academy. Boston, MA: Anker Publishing, pp. 139–149.

76.

Streveler

Smith

Miller

(2005) Rigorous research in engineering education: Developing a community of practice. Presented at The American Society for Engineering Education annual conference, Portland, OR, 12–15 June.

77.

Timmermans

(2010) Evidence-based medicine: Sociological explorations. In: Bird

Conrad

Fremont

Timmermans

(eds) Handbook of Medical Sociology. Nashville, TN: Vanderbilt University Press, pp. 309–323.

78.

Timmermans

Berg

(2003) The Gold Standard: The Challenge of Evidence-Based Medicine and Standardization in Health Care. Philadelphia, PA: Temple University Press.

79.

Vincenti

Bloor

(2003) Boundaries, contingencies and rigor: Thoughts on mathematics prompted by a case study in transonic aerodynamics. Social Studies of Science 33(4): 469–507.

80.

Walden

Foor

(2008) ‘What’s to keep you from dropping out?’ Student immigration into and within engineering. Journal of Engineering Education 97(2): 191–205.

81.

Waller

(2005) Work in progress – Feminist research methodologies: Why, what, and how. Presented at The Frontiers in Education (FIE) annual conference, Indianapolis, IN, 19–22 October.

82.

Walther

(2012) CAREER: A quality framework for interpretive engineering education research. NSF EEC Grant. #1150668, 14 September. Arlington, VA: National Science Foundation.

83.

Walther

(2013) The 2013 Qualifying Qualitative Research Quality (Q³) workshop. Available at: http://www.georgiacenter.uga.edu/uga-hotel/conferences-events/register/q3-workshop (accessed 19 August 2013).