Open Science: A Candid Conversation

Smith

Matt, you’ve been on the front lines of the replication crisis and ensuing open science reform efforts from the beginning. How did that start for you?

Makel

I love origin stories! Alas, mine isn’t quite as dramatic as being bitten by a radioactive spider. As we describe in the introduction of our book (Makel & Plucker, 2017), my entry into this world was in 2007 when Jonathan dropped off a news article talking about the low rate of replication in marketing research.

A year later, I was working on my dissertation which was a conceptual replication of some of Carol Dweck’s work. I want to be clear, I wasn’t trying to replicate her work because I doubted her results, but because I believed them. I expected to be able to replicate the work in a different domain.

I failed to replicate the findings and viewed it as what is now called a “constraint on generality” (Simons, Shoda, & Lindsay, 2017; although I hadn’t heard of the term at the time) and a limitation of my own ability as a researcher. I look back at that younger, naive version of myself with a mixture of nostalgia and embarrassment. But that is also how I feel about all portions of my younger self.

Those are the two specific experiences that brought these issues to my attention. At some point I began to wonder,

if findings can’t always be replicated and results don’t always generalize to other situations/contexts, then how can the field of gifted education rely on findings from general education? . . . And how can gifted education programs that use one identification or service model rely on research based on a different identification or service model?

That’s my open science origin story. What’s yours?

Smith

It was the summer after my second year as a doctoral student and I was trying to write up the results of the first quantitative study I’d taken the lead on. I had a dozen variations of the data set and had run every analysis I halfway knew how to do on each of them, in every combination I could think of—different covariates, different predictors, different exclusion criteria, different variable transformations, different subscales and aggregations, and on and on.

I could check the boxes for statistical significance and a “medium” effect size, but the only things I was reasonably confident of were: the design was not good, the measurement was worse, and I didn’t understand half of my output. There was a conference deadline looming however, and my CV was not yet evincing “an established research trajectory with clear prospects for external funding.” I also needed to present at the conference to get funding to attend said conference. So I mimicked the language and citations of articles that justified similar data analytic decisions, crafted some interesting implications, and submitted it.

Shortly thereafter, I stumbled across a draft/preprint of an article you coauthored that explained the fundamental problems with this approach to research and called for open science reforms in our field (McBee, Makel, Peters, & Matthews, 2018). It opened the door to a whole new world of understanding about what empirical research is, how hard it is to do well, how much harder it is to do well in applied fields like gifted education, and how easy our current system makes it to fool ourselves into thinking we are doing it well when we aren’t.

Reading that preprint was like being bitten by a radioactive spider.

Makel

Well that is very flattering! I am glad we caught your attention. What happened next for you? For me, once I noticed one problem, I began to see problems everywhere. Despite a reputation for being self-correcting, there seemed to be no formalized mechanism in the standard research process to systematically correct or verify findings. Schmidt’s (2009) paper on how replication is neglected in the social sciences really moved me. If a field doesn’t systematically check to see whether results replicate, how can its results be trusted, much less serve as the foundation for educational practice? That paper served as the groundwork for a string of papers establishing estimates of how often replication studies are conducted in various fields. Conducting replications are not about trying to say “Gotcha!” to other researchers. If done appropriately, conducting replications is about verification of findings or testing their generalizability, both of which are hugely important to any field that calls itself a science.

Smith

Next for me was the realization that I didn’t have a well-formed idea of what “science” was to begin with. I switched into this field from law, and while I could have recited the steps in the scientific method, I had little sense of what they meant in education, or even the social sciences more broadly. So I went on a learning binge.

The rest of that summer was an exhilarating roller coaster ride through the history of the replication crisis, Paul Meehl’s (1989) philosophical psychology lectures (I highly recommend!), disturbing findings from the growing field of meta-research (e.g., Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2016), and discussions about how to fix things.

Makel

Meehl’s lectures are available online (http://meehl.umn.edu/unpublished-material/philosophical-psychology-1989). I think they are must-see videos.

I had heard the phrase “If you torture the data long enough, it will confess” (attributed to economist Ronald Coase) before, but I was still aghast when I read the Simmons, Nelson, and Simonsohn (2011) paper demonstrating how much flexibility researchers have when analyzing data. Using seemingly innocuous methods, they showed how researchers could conjure dubious (and even impossible) findings.

The more I looked, the less rigorous accepted established research practices appeared. Even worse, I saw that researchers had been pointing out these problems for decades. Fortunately, at about the time I was hitting rock bottom in my trust in academic research, several fortuitous things happened. I joined twitter, had the great fortune to engage with the incomparable Bobbie Spellman, and found a community of peers who shared similar concerns and were looking to improve practices. That community has since formed the Society for the Improvement of Psychological Science (SIPS).

Smith

Yes, three cheers for academic Twitter and SIPS.

Makel

I believe that open science embodies the values of access and inclusion. The traditional academic research process hides research articles behind expensive paywalls, thus making them only accessible to those with resources or an affiliation to an organization with resources. Similarly, important research materials such as surveys, analysis code, and even data have also often historically been kept by those who developed them only to be shared with friends or sold for profit. Under open science models, such products are freely shared (by default, not upon request) whenever possible. There are obvious exceptions where not everything can be shared. I worked on a paper that reported adulthood educational, occupational, and creative accomplishments of some of Duke TIP’s highest scoring students. Sharing those data is not possible because identity would not remain anonymous. Regardless, doing what you can, when you can, to be open with all phases of the research process will help make research more transparent and verifiable while also making subsequent research easier to conduct, thus lowering the barrier to entry. If publications, data, and materials are all freely available, then fewer resources are needed to consume (and participate in) research, all of which make research practices more inclusive and research products more accessible to more people.

Smith

Transparency as the default also incentivizes researchers to be their best selves. Making my data and syntax/code open motivates me to double-check their accuracy, and preregistration is a way of committing my future self to a particular course of action—kind of like not keeping cookies in the house or moving your alarm just out of reach.

Makel

I like your cookie example. It reminds me of Simine Vazire’s phrase, “science means never having to say trust me.” I was inspired by Vazire’s (2018) paper framing many open science behaviors as a credibility revolution. Using the shared goal of accumulating knowledge as a launching pad to argue that self-correction is a core value of scientists, Vazire used Mertonian norms (universalism, communalism, disinterestedness, and organized skepticism) as guiding principles to better align our behaviors with our espoused values. This is a mind-set I would love to see adopted widely. It’s about what behaviors are accepted and expected when conducting research.

Smith

I wonder how much more productive we would be as a community with a core set of agreed-upon norms and expectations for what doing different types of research should look like. This is especially important for education research because the field is so methodologically and epistemologically diverse. That plurality can be a strength, but only if there’s some shared foundation that allows us to integrate and make sense of our different approaches. When I’m reading an article in our field, oftentimes it’s not only hard to independently evaluate the knowledge claims in it but also to determine what criteria the authors used for claiming something as knowledge to begin with.

Makel

I find myself with similar struggles. One way that I’ve grown to look at this is that, like all fields, in the gifted and talented field, there are many widely held beliefs, but beliefs are not facts. This style of assertion-as-fact is like a punch-in-the-mind to me. To equate assertions with established conclusions dangerously skips a great many necessary steps such as evidence accumulation and interrogation. I find this often happens when we (at our great risk) conflate what we wish with what we know. It is my hope that, when adopted at scale, open science can help differentiate facts from beliefs. Such differentiation can help guide researchers toward what future research needs to be conducted and guide practitioners toward what to implement.

Smith

I have a Meehl reference for everything these days, but that reminds me of a point he made in one of his lectures about the need to distinguish the scientific “game” from other games like politics or journalism. Science isn’t the only game in town, but different games have different rules. We have to be clear about which one we’re playing.

Makel

That’s a useful framing. Calls for open science are declarations that the rules of the game need to be changed.

Not to malign your previous profession, but an analogy that I like asks whether researchers are more like detectives or lawyers (Johnson, 2013), where detectives follow evidence wherever it may lead, whereas lawyers present evidence supporting their predetermined position. My scientific worldview falls much more under the detective role, but I know that researchers (especially education researchers) often wear advocacy hats too. Given this, I think applying open science helps us draw brighter lines of distinction between when we live up to Merton’s scientific norms and when we do not.

Smith

I’ve been contemplating the legal analogy a lot lately. It’s apt for illustrating your point about distinguishing advocacy from the aims of research. I think it’s also a useful comparison for the related point about needing to then align our norms and practices with that shared goal. From that perspective it’s striking how many open science reforms actually mirror things that are already standard in the legal community. Transparency is the default rather than the exception in most legal proceedings; there are codes of professional conduct and strict rules about conflicts of interest and payment; lawyers are required to give the other side access to relevant evidence; there are different evidential burdens for different legal situations, with stricter standards when the cost of getting it wrong is greatest; and there is a formalized self-correction mechanism in the appeals process. Overall, there’s a fairly robust (if imperfect) network operating at multiple levels that balance individual actions toward the foundational system goals.

I see open science reforms as a collective effort to build a similarly robust network and optimize our chances of accomplishing our own shared knowledge-building goal.

Makel

I hope your point about open science serving as a set of norms for the field resonates with the education research community. I am heartened by so many people from other fields making similar points. Katie Corker’s (2018) closing presidential remarks last year at the SIPS conference proposing that we frame open science as a set of behaviors and not an identity was a poignant moment for me. I think making open science about what we do and not who we are will have many benefits toward greater implementation of open science practices.

Smith

We talked a bit about our rationales for supporting open science practices and their potential benefits, but have you experienced any of these benefits in your own work?

Makel

I am in the middle of two registered reports projects and will struggle going back to the old way of publishing because registered reports are so wonderful. As an author, the registered reports format provides me with suggestions from experts at a time when they are most useful (i.e., before I collect data). For me, it shifted the review process from necessary burden to shockingly helpful advice. I think early scholars may be better suited here; they don’t have to unlearn old habits and registered reports follow the general pattern of a thesis or dissertation. Of course, registered reports only work in journals that accept them. But there are two in gifted education (including this one) and 156 overall (as of January 2019).

Smith

I do find it odd that most of our conferences have in-progress research sessions to submit proposals and get feedback, but only for grad students. That process is beneficial as a student, but I’m not clear on why it would suddenly stop being so upon graduating.

Makel

Well, receiving feedback and changing plans may require a bit of intellectual humility on the part of us more experienced folks! But it will also require unlearning some things we were trained to do.

I remember raising the potential of doing a registered report with a group of collaborators. Karen Rambo-Hernandez was the one who would be conducting the statistical analyses for the project and was not familiar with registered reports; she had some initial concerns. However, at that time, she was in the midst of revising analyses for the third time for a manuscript—due to reviewers’ suggestions for alternative ways to run the models. I think skeptical excitement would best describe the reaction I got when I said that a registered report would mean the analyses would only have to be conducted once. If it meant she wouldn’t have to rerun analyses multiple times at the request of reviewers, she was on board. Doing a registered report not only shifted our work but also saved us (mostly Karen, and also the reviewers and editors) a ton of time.

Have you seen benefits in your work?

Smith

Open access publishing has provided immense benefit to me. Your preprint is a great example. I read it over a year before the published version appeared online. I cringe to think about all the things I learned and scholars I met (like you) in that year that I wouldn’t have if I’d had to wait for it to make it through the traditional publication process.

There is a universe of intellectual activity happening via preprints, podcasts, blog posts, web videos, Twitter threads, Facebook groups, Slack channels, and whatever medium might be invented tomorrow. It’s difficult to overstate how valuable it is to be a part of that conversation, even though it can sometimes be overwhelming.

As one example, when I joined Twitter in the summer of 2017, the “Redefine Statistical Significance” preprint had just been posted (Benjamin et al., 2017), in which 72 scholars argued that the default alpha value in statistical significance testing should be changed from .05 to .005. Within 8 weeks, 88 additional scholars had drafted a response preprint arguing that researchers should instead “Justify their Alpha” (Lakens et al., 2017), followed by a third group suggesting they should altogether “Abandon Statistical Significance” (Blakeley et al., 2017). Not only did I benefit from reading those papers well before they were published (as of this writing the third still hasn’t been published), but my understanding of the issues deepened immeasurably by following the live debate that happened online in that intervening 2 months.

Makel

I also learn a ton from Twitter while trying my best not to get swallowed by all the content. I learned through Twitter that openness is not limited to research! For example, many faculty have started sharing their syllabi. Morton Gernsbacher (https://online225.psych.wisc.edu/), Sanjay Srivastava (https://osf.io/2yvsc/), and Neil Lewis (https://osf.io/kj9hq/) have all shared really cool syllabi for their research methods courses that build on open science practices.

Smith

Oh yes, open access education resources are amazing. I started teaching research methods this year and have pulled a lot from those open syllabi. Many also post course materials and even recorded lectures or screencasts. I’ve found those incredibly helpful, particularly for boosting my statistics expertise and learning to use open source software like R. I’m in the middle of Richard McElreath’s course on Bayesian statistics (http://xcelab.net/rm/statistical-rethinking/), and Daniël Lakens has a great introductory one on statistical inference (https://www.coursera.org/learn/statistical-inferences).

In the spirit of Mertonian skepticism, are there any open science practices that you’re critical of or don’t think would translate well in our field? What’s the best critique of open science practices you’ve heard?

Makel

I may be one of the worst people to ask this question because I’m in too deep. This isn’t unique to our field, but implementing open sciences practices will require deviation from established behaviors. Such changes will likely have unexpected and unintended consequences. For example, how should hiring or promotion committees recognize data sharing? Who is the more appealing candidate, one who collects data and published three papers with it or one who collects data, publishes two papers with it, but then shares it and four other papers are published with it? Personally, I think the latter candidate has created more value to the field, but I’m not sure we’ve created recognition/reward mechanisms to recognize that value yet. Problems will arise if one area advances too far without bringing other areas along with it.

Similarly, there is huge societal value in uncovering false findings. But the current system cultivates large personal and political burdens and barriers against doing so. Gelman proposed a thought experiment of a time-reversal heuristic (Gelman, 2016). Why should a study published first automatically garner more trust than a subsequent study (particularly if the subsequent study has merits such as a much larger sample)? Given publication biases for significant effects as well as the flexibility and incentives researchers have, we need to exercise caution against giving priority to results just because they were published first. But I’m not sure I am really answering your question.

How about you? What are the weaknesses of open science? Where will it be hardest to implement?

Smith

It’s more of a cautionary note than realized weakness, but I think there’s much wisdom in Bastion’s (2018) reminder that we can’t ignore science in our efforts to fix science (see also Gelman, 2018). We’re changing things because empirical evidence indicates what we’ve been doing is not good, but whatever new practices we implement need to be subjected to the same empirical criticism. Change is hard. Making sure the changes remain open to future changes is next level.

I imagine education can claim the corner as the field where most open science practices will be hardest to implement, although we probably have the most to gain from them. The political and personal barriers you mentioned are most frustrating because they block solutions.

Where I think we still have the most work to do in creating solutions is data. People get hung up on privacy concerns, which aren’t trivial, but are again more beset by political challenges than anything else. What we need to figure out is how to use data efficiently. Where do we store it? How do we store it? What are the standards for storing it? How do we combine it? Open data are different than usable open data.

Makel

That is a really good point. I would like to amend my previous comment to say that one of the biggest weaknesses of open science is that most of us were not trained to be open and thus don’t really know how to do it (well) yet, although I am putting a lot on that last word.

Smith

That’s another reason I’ve benefited from some of these online communities. Participating in open science practices can be intimidating for people at my career stage who are first learning how to do research. When I think back to that first study I did, no way would I have wanted to make my mess of SPSS and Excel files public. Yet without that expectation, there’s little motivation to ever get better at it. Openness raises the bar, and I think that’s a good thing. But it also raises it for everyone. There are so many senior scholars changing their practices and learning right along with me, even admitting loss of confidence in their prior findings (https://lossofconfidence.com/about/).

Makel

I think raising such bars would be quite beneficial. We may even raise them high enough to meet the expectations that I think the public holds for what we do.

Another issue that often comes up in conversation is a concern that preregistering hypotheses will stifle creativity. Preregistration is a plan, not a prison. If, after seeing the data, researchers want to conduct a whole suite of new analyses, they can. The only stipulation is that the preplanned analyses and the post hoc analyses are clearly demarcated. Similarly, researchers can preregister with an embargo so that no one can see the preregistration (or only a chosen few can) until researchers choose to share it. Worries of scooping because of preregistration are needless.

What issues do you find regularly coming up when you talk about open science?

Smith

As an early career researcher, there is a lot of discussion about how supporting open science practices might affect career prospects. I’ve certainly had some of those thoughts. At the end of the day though, I’m persuaded by Yarkoni’s (2018) argument that the incentives created by the publish or perish culture aren’t a justifiable excuse, especially now that we have so much evidence of the problems with traditional practices.

Of course, I also know I might easily talk myself out of that position tomorrow or when that next conference deadline rolls around. That’s why the cookies can’t be in the house.

Makel

I’ve heard concerns about not being able to afford to adopt open science practices, that they take too much time, effort, and aren’t worth the costs. I take the opposite view. I think knowing (not just believing) with confidence and precision what interventions are effective, when they replicate, when they generalize, and to whom they generalize is too important to ignore. Education should not be an annual half trillion dollar placebo.

Smith

That’s a powerful statement.

Makel

One of the important roles of research in education is to tell us what works (there’s a whole clearinghouse based on this premise). And one of the goals of open science is to make research more accurate and transparent.

That said, it is relatively easy for me to say things like that because I’m not on the job market. I do worry about value alignment between job candidates and hiring institutions. Moreover, I also worry about all the potential candidates who grow disillusioned with the research process from not being able to replicate previously published findings. The field loses a lot of potential when strong early career researchers leave the field. And it’s very easy for these losses to be invisible.

Smith

If you could wave a magic wand and change only one thing/implement one open science practice, what would it be?

Makel

Obviously, if I had a magic wand to change anything, I would grant myself the ability to change more than one thing! That’s the classic default magical wish. But more seriously, because so much education/gifted education research is conducted without funding, I’ll skip what funders could do to increase open science practices. Instead, I’ll go to journal editors and reviewers. They are the primary gatekeepers of the publication process. If the default shifted from “no open science” to “some open science” or maybe even “open science unless you have a reason,” then I think we’d be on the right track. Once we do this, I think we will better understand what we know as a field. Once we know what we know, we will have a stronger platform to speak with policy makers, practitioners, and parents about real-world applications.

But this is a means/ends question. Changing the norms of what are accepted research practices is the means to an end. The end that I think these means will help us accomplish is greater understanding of how to identify and serve gifted and talented students.

What’s your wish?

Smith

Well said.

I think there’d be worlds of improvement if we just enforced some of the things that are nominally required right now. Consistent reporting of descriptive statistics would be lovely. Otherwise, I’d go back to the one you mentioned at the beginning of our conversation-constraints on generality. The difference between evidence reported in results sections and the verbal gymnastics that happen in discussions, abstracts, and titles of articles are astounding. Communicating our research is an essential part of what we do, and storytelling is the most powerful way to do it. Yet I think we should use that power to highlight rather than hide the complexity and uncertainty in our findings, especially in a field where the empirical and normative are constantly bumping up against each other.

Makel

Growing up, my dad often joked that one should “never let the facts get in the way of a good story” (I learned later that my dad did not invent this phrase, but mentioning this likely doesn’t abide by the dictum). Obviously, that is fun for parties, but I believe that our duty as researchers is not to let a good story get in the way of the facts. We also need to not confuse what we believe to be a good story with what we know to be fact. Infusing open sciences practices more into our research practices can play a large part in making this happen.

I am optimistic about our future, especially when I speak with early career researchers like you. I think 20 years (maybe even 10) from now we’ll look back and wonder how/why we ever did things differently. The question we face now is whether we as individuals or as a field want to help lead the charge or hang back. I hope we choose to lead.