Demystifying Literature Reviews: What I Have Learned From an Expert?

Jia Wang (JW): There are so many terminologies out there to describe different types of literature reviews. How would you categorize them?

Margaret Foster (MF): Well, there are probably 20 to 25 different names that have been indexed regarding literature reviews; people are getting down to semantics. It really doesn’t matter what you call it, just pick one term and set standards up for it. It’s when everything is so open-ended, and people don’t know what to do, you get into a problem. These are the issues I am more concerned about than what you call it. Generally speaking, they really just fall into three basic categories: narrative reviews, systematic reviews, and scoping reviews.

Narrative review is the one all of us feel comfortable with because we all have to write them for school assignments, for the beginning of primary studies, and for other things. This is where you are given a topic and have limited space to write about it. Depending on how much time and space you have, you are going to find what you can. Therefore, narrative review is very subjective depending on what you select, how you choose to use what you have found, and how you choose to frame it. It’s all subjective. Normally, researchers don’t tell their readers how they found and selected literature; as a result, this type of review leaves your readers wondering what else is out there. It is designed for sharing expert opinion about a topic; so when you look at the quality of a narrative review, a lot of times who wrote it comes into play. A challenge for narrative review is people are going to ask if the authors give both sides of the view. Some people could write a more balanced narrative review, and some just report what they believe or see.

Systematic review is on the other end of the spectrum, which is very different from narrative review. People also call it “critical review,” “best practice,” ‘evidenced synthesis,” “qualitative synthesis,” “structured review,” and “meta-analysis.” Systematic review is a research methodology. It requires you to have a research question that can be answered by studying a group of studies that all are trying to answer the same question. You are going to collect, sort, and code a group of studies that are all very similar. For that, the question has to be very narrow and all the studies need to be answering the same question. Not every topic could be examined through systematic review. If you think about in medicine and in education, researchers often study the same interventions or the same policies although with different populations. The value of doing systematic review is to bring all the populations with the same intervention together. This is just one type of systematic review. You can also do a review where you look at what theories or frameworks have been used. And, you can do another type of review where you try to build a framework or a theory. So reviews can go anywhere from testing a hypothesis to developing a framework. What makes systematic review different is that you study a set of studies that are all the same and they all are going to be treated the same. Another feature of systematic reviews is that you should always include a critical appraisal and articulation of risk bias. This is where we see any issues in the collection of evidence that will make you wonder about the synthesis of it. We don’t see a lot of this in narrative review.

Scoping review falls between narrative and systematic review. Some people also call it “mapping.” Right now, Campbell is using the term “evidence map.” This is where you have a broader question, and you want to be able to tell your readers every step of the way what you did, so that your readers can understand enough of what you did to evaluate and decide for themselves whether what you did is appropriate or not. The idea of doing a scoping review is you have a question and you set a boundary. There are set standards for scoping reviews in medicine and social sciences, and the most popular article to look at for social sciences is by Arksey and O’Malley (2005). The basic process is to set your scope, to collect what you found, and to describe what you found. What’s different is that not everything is the same, and you really can’t synthesize information the way we do with systematic reviews. So scoping and narrative reviews can only be descriptive—we can only describe what we found without much synthesis. Scoping reviews help lay out the overall picture within a framework. For example, if you really want to do a study to show there is not much out there on the topic of your interest, a scoping review can help you accomplish this goal.

JW: In HRDR, we publish integrative literature review. Where do you put this within your three categories?

MF: According to what I read in HRDR, I would say your integrative reviews are more like scoping reviews. Right now, nobody has come up with a set of standards for integrative reviews. Until you set standards to show this is what you are looking for, you can’t really hang your hat on it. The case is the same for systematic reviews. Many disciplines have adopted this term without really defining what it means in their discipline. For example, I’ve seen a lot of systematic reviews in engineering. But when I read them, I think they are more like scoping reviews, even just biometric analysis, because they just tell me these many articles published in this year in this journal. You really have to define it. Each field has to define the meaning of reviews for themselves. So if you are going to use different terms, you need to set standards and tell your readers what they mean.

JW: Now that you differentiate three basic types of review, can we use career women in engineering as an example? How would you develop a research question that requires each type of review?

MF: Ok, let me try. Using your topic, a narrative review question could be: What does the new push for diversity in human resource development (HRD) mean for the future of women in engineering careers? You see, the best applications for narrative reviews are for very subjective questions, such as, what does this mean? What is the future? You have someone who understands the historical contexts of this topic, so you are looking for future trending. That’s the quintessential narrative review.

A question for a scoping review could be: How have the careers of women in engineering been studied? In this type of review, you have questions that are broad and descriptive. You might limit your search to the past five years. It’s usually like a sampling idea and it won’t end up being everything. You will provide people information such as this is the context, this is what’s out there, and I am going to walk you through a framework and show you what I did.

For a systematic review, here is one question: What is the effect of having a male mentor versus a female mentor on the career development of women in engineering?

To sum up the differences, let me give you an analogy: A systematic review is looking for all apples; a meta-analysis is looking for all apples of the same color and of the same variety; a scoping review is looking for fruit salads; and a narrative review is looking for everything—whatever you want to bring! So if I have only red Gala apples, I can take a closer look at each apple and its unique characteristics. If I have a fruit salad, I can identify the types I have—a banana, an apple, a pineapple, etc. So in a systematic review, I may spend several hours really delving into each study (one Gala apple); while in a scoping review, I can only look from a higher up (variety of fruits). In a narrative review, I can go any direction and do anything, not even telling you why I did what I did.

JW: As a researcher, how do I decide which type of literature review to conduct?

MF: I have a series of steps for a researcher to go through to figure out what type of review is going to make sense. The first step is just listening to the question. You can take any question and refine it and turn it into any type of review, depending on what makes most sense to you. As a researcher, you have to know your purpose for doing this review and where it is going to make the decision.

After you listen to the question, you can find out what has already been done. I will take the concept and do a review of a title, and look at multiple databases for the topic. You may also want to see how the studies on that topic have been done. Sometimes people have great questions, but there are no studies done on the topics. If this is the case, you will not be able to do a systematic review but you can definitely still do a scoping review. Sometimes people just create a protocol, which outlines all the steps to follow to conduct a systematic review without actually conducting it. If we look at the most important database of systematic reviews in clinical medicine (Cochrane), as about 25% of the protocols are there, there are absolutely no studies. This is where scoping reviews can fill in.

The next step is to think about sorting your preliminary searches to figure out the feasibility—not only if these search results apply but also if you have time to review all the results. Sorting helps you determine if you need to change the question or make it narrower. Depending on your timeframe and how many people you have on your research team, and your resources, you really need to narrow the question down. You are not trying to save the world. Even when people do a narrative review, they tend to look out for too much, because for the most part, researchers become “hoarders.” They just collect and collect. The problem may have been: you didn’t really know what you were looking for when you started to do the collecting. As humans, collecting is fun and easy. That’s one reason why I like systematic literature review—you set up a filter for exactly what you want, so when you find a paper, you know whether to say Yes or No.

JW: How do you know exactly what you want before you start searching?

MF: You can’t. You have to be in the discovery mode for a while. When people are still window shopping for review ideas, I suggest just looking at review articles, because review articles already sum up everything in the area. Often at the end of the review articles, there will be unanswered questions and what needs to be done. This gives you a chance to window shop a little bit, because everything has already been pulled together. Then if there is a topic that you really like, you can start to read a little about it, because you’ve got to have some context in there. No matter what type of review you are working on, the context is half as important as everything else. For example, last year, I was asked to write a systematic review for a journal on critical care nursing. I am not a critical care nurse but I can do a systematic review. So I pulled out articles, I did all the searches, and I got tables together. But when it came to the time to write about what it means, I can’t tell you what it means. All I can give you is dry facts. It’s like I did the autopsy, now I need someone to write the eulogy. Autopsy is very dry, very technical, but what are implications? Now that you know this, as a nurse, what will you do? That part takes time. That’s experience; that’s reading; that’s education. You can’t short change that. So when you do a systematic review, that’s where it really helps to have a team of people.

JW: How do we know that researchers did a good job with narrative reviews?

MF: That’s tricky if you don’t have anything on which to evaluate them. That’s where the scoping review can be helpful, because at least it has a framework to tell your readers what you selected. If you look at how systematic reviews evolved in medicine, they started out not telling you what researchers did; then journal editors and reviewers would ask what you did—they want to know more. So over time, these standards evolved—they are still evolving in medicine. It’s all getting down to, as a reader, having to understand what you did; otherwise, I have nothing to evaluate. It’s really down to journal editors saying, “This is what we want to see,” because they are the gatekeepers to get papers published. In medicine, over time, we have established structures and standards to tell researchers what to report. The problem is we don’t have that in other areas such as education as we have in medicine. Until you have structure in place, you are going to continue with the question: What did you do?

JW: Is it too much to ask researchers to tell you the details about the literature search process?

MF: I don’t think it’s too much to ask people where they searched to make sure they had looked at all the right places, especially considering issues of publication bias, where people are only looking at the peer-review publications, or they are only looking at a set of journals. All these are going to lead to publication bias. Publication bias is when due to the nature of the findings, things get published or do not get published. So unless you have this big, exciting finding, maybe your work doesn’t go any further than a conference paper or a dissertation. If you are trying to decide if this intervention is helpful or not, all you have is just five studies to say “this is the most wonderful thing that ever happened.” You don’t know the other 20 studies in the trash can that showed the intervention didn’t work. So if I don’t know what you did, then I have to take your word that you looked at all the right places. We see this all the time. I see studies that look at mental health, but the authors didn’t search the psychology database. Or, the researchers are looking at an educational topic, but they didn’t search ERIC, the most important education database.

JW: Related to the issue of literature search, let me give you a scenario. A doctoral student is interested in studying women’s career experience in science, technology, engineering, and mathematics (STEM) fields, from the HRD perspective. She is curious about what HRD scholars have done. So she decided to search all four Academy of Human Resource Development (AHRD)-sponsored journals and found only five or 10 publications. By design, she decided not to go beyond the HRD field to include STEM literature. What do you think of this approach?

MF: There is nothing wrong with this approach but it is super narrow if you only look at certain journals. HRD is much bigger than that. So instead of focusing on four journals, you should at least open up the HRD database. We have human resources abstracts through EBSCO. One thing we do in medicine, because we are trying to study how an intervention worked across a large group of people, we don’t limit our search by peer reviewed, by language, or by anything else. We want to find all relevant studies. You can only say, “For these journals, this is what it is.” It does not tell you what it is for all HRD. People likely publish outside of their mainstream journals, so I look at concepts, not at any one journal, to think about where this topic might have been published. Once you start limiting down to journals, you just added a lot of biases.

JW: Often times, we use peer review as a quality indicator. That’s why we include only peer-reviewed publications. What’s your take on that?

MF: What we say is: Don’t judge on where it is published or its scientific merit alone. Just because something is published in a top journal doesn’t make it perfect or doesn’t mean it has no issues. We need to look at everything with a critical eye. Of course, you can’t make the search too big either. So you do have to narrow it down at some point, but you can’t always use peer review as the quality indicator. It’s an indicator that some groups of people thought a paper that deserves to be published got published. This is helpful but depending on your field, I would argue that conference papers are just as important to review as peer-reviewed articles. As a field where you have a lot of practitioners, they have a lot of great things to say. By looking at only peer-reviewed papers, you could be leaving practitioners’ voices out completely, because they are not worried about writing a peer-reviewed paper like we do in academia. Practitioners are out there, they are doing it, and they are talking to each other at conferences, but you are missing it.

JW: Now let’s talk about some specific techniques in the search process. How do we decide keywords? How do we use the best keywords for effective searching?

MF: The main thing is to think about what are the concepts you are looking for in this article and what are some different ways people express a specific concept? This is because everybody has a different way of expressing—some concepts are easier to express than others. For example, “manage” is a really hard one, because it is also a verb. So think about all the ways people express that, whether it’s people from different disciplines or different countries. You can have one concept but everybody uses a slightly different phase. And then, you have to think about: How am I going to combine these different concepts depending on what I am looking for? I designed a Venn diagram for myself—just three little circles. If you are not finding something, it could be you have too many criteria for your search, or it could be that you are not looking at the best place. So it’s about training yourself to look at many places, not just one.

JW: How do you differentiate inclusion and exclusion criteria?

MF: I don’t really say inclusion and exclusion criteria; I just call them all eligibility criteria. Regarding exclusion criteria, here is an example: I am going to look into adult education interventions about social media applications. I am excluding certain groups of people. Also, I am thinking more about concepts—where you might have big concepts; you are using exclusion to further define it. It’s your way of relating to your readers what you did.

JW: Speaking of concepts, let me use my earlier example of STEM women careers for illustration. There are a lot of studies focusing on career attitude or career drive, but I am interested in career experience. Do you consider these are three different concepts?

MF: It’s very complicated, because in one survey, I can ask people all kinds of questions, such as motivation and attitude. And then I can ask the question you are looking for. That’s where it gets really tricky in social sciences because there are too many other ways to describe what you are saying. So all you can do is to pull all the studies on the career of women in STEM and the only way you can sort them is by reading the paper. You can’t do it by keywords themselves. Another problem is that people often write terrible abstracts. They don’t think about how people are going to find my paper. So if you are looking to get yourself cited better, it’s really important that you put the five most important words in your abstract. Lots of databases don’t include author keywords in the abstract. So don’t try to use fancy language in the abstract; use the most common terms to make sure people can find your work.

JW: How can I rigorously define the search boundary?

MF: One big problem is that we don’t know how much is out there; we are never going to know that. So what you need is a set of standards, which is why I have to know where you searched and how you searched, so that I know if you meet the standards. It’s about the context. I have to know enough about what databases are out there to know you searched the most important ones. In medicine, they start to call on librarians as expert, to evaluate searches in systematic reviews. When I’m looking, I am looking to see if you searched all the right places, if you searched them in the best way, and if you looked at references and citing references. That’s going to give me the whole picture about where you looked. I can’t do that if you don’t tell me.

JW: At what point, do I stop searching?

MF: It depends on what you are doing. When you are in the discovery mode, you are just shopping around. Once you find what it is you want to focus on, you want to stop searching and develop your protocol. Whether you are going to do a narrative, scoping, or systematic review, you’ve got to have a filter. You’ve got to set your boundaries; otherwise, you just stay in the hoarding mode. You’ve got to stop collecting and start mapping. If you don’t know where you are going, you’ll never get there. After you figure out where you want to go, then design your protocols: What is your question? What are your criteria? Where are you going to look? That’s when you come to talk to a librarian who can help you map out three or four places for a search. Then you pull all publications and sort them out based on relevancy. That’s how you will go forward. If you stay in the collecting mode, you are just spinning the wheels. I have one student who literally just pulled the wagon, and piled up all the documents in my office and said, “Here is everything I collected; I need to cite it.” You need to decide from the beginning what you are going to do, not at the end. That was a very painful process for both of us.

JW: What do you think of reviewing literature published only in English?

MF: This is something I’ve been thinking about as a librarian: How can I help people expand their search? If we really want to be global in our thinking, and if we really want to include all the different voices, we can’t limit our search to English only. There could be whole a lot happening somewhere else that we don’t know because we are not even bothering to look at what they are doing. This is another way to look at publication bias. In medicine, only articles that have huge, exciting findings get published in English; others get published in their own language in their own country. People are missing out on trends without reading publications in languages other than English. An intervention could have happened for the past five years somewhere in the world before we know about it. So when I help people on systematic reviews, I encouraged them to find someone who speaks another language to help code the article, instead of having the article translated, which can be very costly. In this way, you can have that voice included, and know that’s where we need to go. It’s wonderful when international students want to include articles published in their own languages and in their own countries; that’s their strength.

JW: What is your advice about doing systematic reviews in a country where little research on the topic has been published?

MF: I had this experience with several students. I remember one student from Thailand. We were trying to look at some health topics in Thailand and we were struggling to find publications. So we started with identifying some professional organizations in that country that may be writing about the topic. We looked at their conference proceedings. This approach helped us identify some papers and it also helped us identify some authors for the student to contact. We wrote our eligibility criteria and the student emailed to some authors we found. We got articles back from them. At the end of this project, the student had an internship lined up back home. He ended up getting a job from them. So I think a lot of times, people are almost afraid of reaching out. Sometimes you just need to go in and figure out how to talk to those key players.

In the countries where publications are limited, you can still do a literature review, but it may not be on peer-reviewed articles. It could be conference papers and practitioners’ work. You could still report studies they are doing. You could still evaluate the quality of their work, just not where it was published. In medicine, we have a lot of Centers for Disease Control and Prevention (CDC) reports. CDC is not worried about the peer-reviewed aspect; they are trying to get information out. We have a lot of these types of groups publishing white papers—they are just as qualified to write and talk about your topic. Sometimes we get a little too hung up on peer review.

JW: Does the number of publications matter in systematic reviews?

MF: No. It’s very arbitrary to say there is any right number. It’s more about the research question you have, the criteria you set, and whether you looked at all the right places and did the right things. I’ve seen systematic reviews published where authors claimed they looked at 24,000 citations and then included only two. I’ve also seen the inclusion of 150 articles. So there is no right number; it’s about whether you took the appropriate steps to find and include what you should have. We have no idea about how many publications are out there; this is where context comes into play. Lots of first systematic reviews pointed out the poor quality of studies reviewed and consequently suggested areas for improvement. Just because there is a small number of studies available does not mean you don’t have something to say.

JW: How do we use Google Scholar to aid our search?

MF: One problem with Google Scholar is that they don’t tell us from where they are pulling articles. As a librarian, if I look at a whole bunch of other databases, they tell me what journals they are pulling from and when/how they are pulling from them. I have a lot of information once I go search in these databases. Google Scholar tells us nothing; so it would be the last on my list. I would use it in a very strategic way—I will wait until I search all other places and then put the identified articles in Google Scholar to see who is citing them and whom they cited. Also, while Google Scholar says “Scholar,” they are pulling from what I call “grey literature,” for example, conference papers and reports. Google Scholar searches in a different way and as a result, they get into full texts. It’s great because in a second, you can get millions of hits. However, what are you going to do with those hits? Another problem with Google Scholar is that we don’t know if they have latest research because they don’t tell you. I would use Scopus which indexes more journal titles than many other databases and keeps track of who cites whom. Like the human resource database, it was designed for human resource research so it speaks the language and has the limiters that make sense for you all.

JW: What are some possible ways of synthesizing literature?

MF: There is a special type of systematic literature review we haven’t talked about—meta-analysis. It is the most rare type because the studies have to be so close that you could treat all the participants as they were in one big study. And you get down to “one effect” size. It’s rare because it has to make logical and mathematical sense to combine everything together. All we can do is synthesize descriptively. But you don’t want to just to talk about this study did this, and that study did that. You want to provide a higher-level synthesis where you can say, for example, “The sample size ranges from A to D.” You can also give examples of unique or representative studies. One of my personal heroes is Dr. Andrew Booth who published a book Systematic Approaches to Successful Literature Review (Booth, Papaioannou, & Sutton, 2016). In his book, Dr. Booth detailed how to do a qualitative synthesis and a quantitative synthesis. He provide over 10 different techniques you can try.

JW: As a systematic literature review expert and a librarian, you have assisted many researchers on their review projects. Based on your experience, what would you say are some common challenges they face?

MF: The number one challenge is having a good question to start with. An average time to take a review is one year to a year and a half. I’ve worked on a review that took me seven years, and another one that took only 11 weeks. If you have only one semester to do a systematic review, you have to have your question figured out in Week 1 and it has to be really narrow. Many researchers have much broader questions. Use your example of women’s career in engineering. What aspects are you looking for—entry, retention? You have to be specific. The hardest part of a review is to figure out the right question to ask. Also, I think for students, a big problem is that many of them haven’t had a research methods class required of them, so they don’t even know what risk bias means. That’s a big problem no matter what field you are in. You should know what are the issues and biases. Even in your own systematic literature review, you could easily introduce potential biases. For example, you only limited to peer-reviewed—it’s a publication bias. Furthermore, if you were the only one who selected all the articles, then you need to talk about potential selection bias. You can also have selective outcome bias—where you only looked at certain outcomes because they matched what you wanted to say. Finally, you could set up your question to be biased and here is one example: what are the benefits of having a male mentor for a female engineer, instead of asking what are the differences between having a male mentor versus having a female mentor for a female engineer? You see, there are all kinds of biases we can bring to a review.

So, you need to look at your own literature with a critical eye and making your method transparent to your readers by recognizing biases in studies and in your own practices. It will move any field forward if we reflect on what we are doing well and not doing well, what we should be doing, and where we should go from there.

Another issue is related to criteria. If you have criteria to start with, you have something to fall back on. If somebody asks why you didn’t include this, you can say, “It doesn’t match my criteria.” If you have no idea where you looked, you just have to take their word that you should include this and that. It may be that you have wrong criteria, or you didn’t search in the right places. That’s another problem. That’s where librarians can offer big help.