Comparative research

Abstract

The teaching of research methods has been at the core of public administration education for almost 30 years. But since 1990, this journal has published only two articles on the teaching of research methods. Given the increasing emphasis on data driven decision-making, greater insight is needed into the best practices for teaching public administration research methods. This research note attempts to build on these previous articles to offer a new approach to the teaching of undergraduate research methods within a department of political science and public administration. The approach combines traditional approaches to experiential education with a focus on comparison among different methodological tools. Grounded in both learning theory and developmental psychology, the article outlines a research method assignment that encourages comparison and uses focus groups and pre-/post-tests of substantive knowledge to demonstrate learning. Implications and advice for teaching are provided.

Keywords

Research methods experiential education comparative pedagogy problem-based learning cognitive reasoning

Introduction

Courses in research methods have become the norm in public administration education. An early study of the public administration curriculum (Roeder and Whitaker, 1993) and a recent informal survey of the US News and World Report (2016) which ranked Master of Public Administration programs, found required research methods courses to be the norm in public affairs education. Within public administration’s parent discipline, political science, there has been an expanded focus on research methods at both the undergraduate and graduate level (Thies and Hogan, 2005), which is not surprising given the behavioral shift of the discipline and the importance of research method skills for today’s economy. With the explosion of information in our society, graduates need to be able to read and evaluate claims and to generate knowledge. Employers place a high value on these types of analytical skills (Anderson and Harsell, 2005; Fitzpatrick, 2000; US Department of Labor, 2003).

This is particularly true in a policy-making context in which valuing objective findings is heavily influenced by normative factors. The creation of research questions and the selection of appropriate methods are grounded in the political domain in which policy analysts operate (Hanney et al., 2003). Consequently, students, as future consumers and creators of policy research, need to have sufficient technical skills in order to better understand how policy findings are shaped by their normative context. Without sufficient ability to judge the quality and origin of research, they will be ill-equipped to decipher the normative influences (Sharp and Richardson, 2001).

That said many undergraduate students find research methods classes irrelevant and difficult (Bernstein and Allen, 2013; Hubbell, 1994). Irrelevant because few students will undertake academic research upon graduation and difficult because most research methods classes utilize the analytical tools of the social science disciplines rather than memorization techniques found in the study of public institutions. While a lot of progress has been made in the teaching of public administration research methods with the increasing popularity of problem-based or experiential approaches (Kramer and Schechter, 2011), the subject remains conceptually difficult for many students. This research note introduces a pedagogical approach to teaching undergraduate research methods based on comparison that attempts to increase learning and cognitive development with the goal of better understanding and application of research methods. Students in the class complete group projects around a common research problem but utilizing different methodological approaches (e.g. surveys, textual analysis and experiments). They are then forced to reflect on and compare the results of these methods. This method fosters student learning and cognitive development by forcing students to understand the trade-offs associated with a multiple range of methods.

The paper begins with an introduction to the theory underlying the teaching method including how the method uses experience and comparison to foster learning and development. The theory is then applied to a required undergraduate course in the department of political science and public administration at a midsize public university. Pre- and post-tests of substantive knowledge and focus group feedback are used to demonstrate the approaches’ strengths and weaknesses, and implications for future teaching are discussed.

Learning and development theory in the teaching of research methods

Teaching of research methods lacks from a unified approach and clear goals. Fostering information literacy (Marfleet and Dille 2005), creativity (Hubbell, 1994) and analytical and technical capacity (Andersen and Harsell, 2005; Kramer and Schechter, 2011) are among the many goals sought. Yet, this journal and others have highlighted approaches to the teaching of research methods which demonstrate effectiveness in achieving student learning (Fenwick, 1992; Harris and Nikitenko, 2014). While no definitive approach exists, past scholars have identified several best practices that can be used in developing a research methods curriculum including focusing on consumption of information, encouraging active engagement, addressing a wide range of methods, and showing the relevance of the practice for professional advancement (Hubbell, 1994).

The approach to teaching research methods used in this article builds on these best practices but increases the focus on comparative research methods,¹ and to this end, it is grounded in two course design principles and their underlying theory: (1) problem-based learning with a basis in experiential learning theory; and (2) comparative reasoning with a basis in cognitive developmental psychology.

Problem-based learning

The use of problem-based approaches to learning research methods is not new and has been shown to have positive educational outcomes including getting students to learn in ways that reflect real research, increasing research skills and greater knowledge about the research process (Kramer and Schechter, 2011; Spronken-Smith, 2005). The commonality among what Kramer and Schechter (2011) call the applied-research seminar is that students in the research methods classes are responsible for devising, pretesting, administering, analyzing and presenting original research. To this end students use a real world topic to apply the academic principles historically taught in research methods classes. This has perhaps become the norm in research methods courses in the social sciences as part of a shift toward student-centered approaches to learning and has been shown to be effective within public administration (Aguado, 2009; Whitaker and Berner, 2004).

The theoretical underpinnings of problem-based learning are most often associated with the work of David Kolb (1984). At the most basic level, Kolb’s work, and those of his predecessors including Lewin and Dewey, suggests that experience is the primary driver of learning and that traditional educational techniques that encourage information acquisition, manipulation and recall often fail to produce lasting changes in knowledge and ability. Kolb argues that the process of learning requires the resolution of conflicts. This is best achieved in an educational environment where individuals transact with their environment to test ideas and assumptions. In this way, “learning is the process of creating knowledge” where individuals use their environment to test the hypotheses to which they have been exposed (Kolb, 1984: 36). To this end, learners move through a repeating cycle of concrete experience, reflection on that experience, the development of abstract concepts and active experimentation of their newfound knowledge (Kolb, 1984). For the teaching of research methods, experiential learning gives the students the opportunity to begin with preconceived notions about the research process, experience research first hand and then reflect and conceptualize new notions of research based on their own problem-driven experience.

The application of Kolb’s model to research methods has been shown in recent years in both management and public administration education. For example, a study of undergraduate business students in the United Kingdom found that learners who utilized an experiential approach to learning research methods were better prepared for subsequent research and demonstrated better research content knowledge (Benson and Blackman, 2003). A recent study in the journal Politics, found that student’s understanding of research methods was particularly enhanced when exposed to the real world challenges of the research process (Ryan et al., 2014). This learning is fostered when student investigate research problems that are grounded within their discipline and specific interests, and in which they are able to work through the problem from the beginning to the end of the research process (Leston-Bandeira, 2013).

Comparative reasoning

In addition to the focus on learning found in Kolb’s work and in the research of other scholars who study experiential and problem-based learning, the course design builds on theories of cognitive development. Traditional learning theory focuses on the “horizontal growth” of individuals as they acquire knowledge, skills and behaviors. In this way, learning does not necessarily require increasing cognitive complexity but rather new knowledge and ability within a particular stage of complexity (Cook-Greuter, 2004). In contrast, development involves “vertical transformation” where cognitive ability becomes more sophisticated. An individual could undergo no new learning (that is, possess the same knowledge and skills), but could be able to understand those knowledge and skills in more complex ways (Cook-Greuter, 2004).

Many theories of development are based on the work of Piaget (2013) and suggest that individuals move through stages of thought (sensorimotor cognition, preoperational thought, concrete operational thought and formal operational thought) characterized by increasing levels of complexity. Piaget suggests that as individuals experience the world around them, they are simultaneously balancing the tensions between assimilation and accommodation. Assimilation is the process by which individuals take new information and categorize it into their existing schema of the world. Accommodation requires the altering of one’s schema in light of new information gained (Piaget, 2013). To this end, comparative approaches to learning serve a particularly valuable role for development because they force students to confront the potential similarities and conflicts between their view of the topic and new information that is gathered through learning by comparison.

The documented advantages of comparison are widespread with evidence demonstrating that the comparative process is an effective mechanism for individuals to learn about language (Graham et al., 2010; Wang and Baillargeon, 2008), spatial orientation (Casasola, 2005), mathematical concepts (Rittle-Johnson and Star, 2007) and computational ability (Rittle-Johnson and Star, 2009). Likewise, this is true for political science and public administration. Bernstein and Allen’s (2013) recent study found that when students build their understanding of quantitative methods on a basis of qualitative methods, then greater learning occurs. The approach discussed in this article starts with experiments as the basis for comparison, but nonetheless finds similar comparative advantages. These benefits of comparison arise from the ability to focus on some aspects of a situation while constraining others. This facilitates logical reasoning and the potential for counterfactual logic. This effect is particularly profound when comparing solution methods as is done in research methods (Rittle-Johnson and Star, 2009). Lastly, comparative research methods have the added practical advantage of allowing students to overcome the constraints of time introduced by experiential learning. As each student group uses different methods to solve the same problem, they gain broader knowledge while nesting their learning within a particular experience.

Integrating approaches

The comparative research assignment is designed to bring about the best of both experiential and comparative approaches. A fuller discussion of the project is discussed in the next section and in Appendix 2, but it is worth briefly exploring how the project brings these philosophical approaches together. Over the course of the first half of the semester, students are exposed to multiple methods of inquiry (e.g. experiments, surveys and interviews). During this period, they practice each method with a short field-based assignment. For example, they might interview a classmate utilizing the techniques explored in the class or conduct a short survey experiment on campus. This gives the students at least basic knowledge of each method. Half way through the semester they select groups that pursue one of the covered methods in depth. This does pose some risk of specialization, which is somewhat offset by the comparative seminar, a two hour long exploration of their research findings and an extensive discussion of the strengths and weaknesses of each method relative to the question under investigation. This enables the students to gain broader knowledge while overcoming the challenge of time-intensity that experiential learning poses.

Course details

This method was developed for use in a midsize public university with moderate admissions criteria. Students enrolled in the class represent a diversity of majors with the majority of students coming from the department of political science and public administration. The course size averages around 20 students and is required for several social science degrees.

In order to train experts who can understand and utilize a wide range of methodological approaches, the students work with both qualitative and quantitative approaches to research, but the course does not include data analysis techniques drawn from statistics that are covered in a subsequent class. An outline of the class is featured in Appendix 1. The course covers general issues of research methods such as causality, measurement, sampling, reliability and validity, but also focuses intently on the nuances of different methodological approaches including survey research, interviews, textual analysis, focus groups and experiments. The course objectives are that students will be able to:

Translate political or administrative observations into testable hypotheses

Critique research found in the scholarly or mass audience

Evaluate scholarly sources and conduct a scholarly literature review using primary and secondary texts

Evaluate sampling techniques and choose the best process based on constraints of time and resources

Design and implement a public administration research project

The class writing is intensive with five papers addressing different aspects of research methods, but the culmination of the class and the focus of this article is a group project in which students are given a common research problem which changes annually and are asked to execute a study on that problem using their assigned method (Appendix 2). Each group has a distinct method through which they work starting with a literature review and proceeding through study design, data collection and analysis to a class presentation. The presentations allow the students the opportunity to understand how a common problem can be answered using different approaches, but also how each approach produces different results and/or emphases. The students engage in reflection and discussion of the presentations to internalize learning. Consequently, these presentations supplement the experiential learning as they highlight the advantages, disadvantages and the methodological contributions of each approach.

In their review of past research on comparative learning, Rittle-Johnson and Star (2009) have identified several characteristics that maximize the potential for comparison to increase cognitive development. These include the findings that: (1) two examples are better than one; (2) two examples presented together are better than two examples presented separately; and (3) instructional support increases the learning associated with comparison. The conference format in which students present their research to each other helps achieve these ends by comparing five or six groups at one time with support and commentary from the faculty expert.

The results in this paper come from two semesters of the class. Following the first semester in which the class was taught with this format, focus groups were conducted with the students to learn about which aspects of the assignment were helpful and what challenges remained in the pedagogy. During the second semester that the course was taught, a pre- and post-test was conducted to demonstrate learning. The test measured a wide range of concepts from research methods including questions about each of the covered methodological designs. The questions on the pre- and post-test included 28 objective multiple choice questions. A copy of the instrument is available upon request but sample questions included, “The major report on ethics in research is called: A. The Warren report, B. The Kinsey report, C. The Belmont report, or D. the Tuskegee report” or “Reliability refers to: A. measure consistently producing the same results, B. a measure measuring what it is supposed to, or C. a measure that provides a lot of detail”. The pre-test was administered during the first course period. The post-test was administered during the final exam period in week 16 of the semester.

While pre- and post-test designs without control groups lack inferential strength, the rotation of the class in the schedule limits comparisons. To address this limitation, the author has attempted to triangulate the pre- and post-test results with student work and the results of the focus group. To this end, the results described below are supporting evidence as part of a larger argument about student engagement and learning. Additionally, the students did not have access to the pre- and post-test questions during the semester in order to decrease the likelihood of “learning for the test.”

Results and discussion

The pre- and post-tests do show a significant amount of learning on the course-related topics (Figure 1). While the level of mastered knowledge is still low, there is a 48% increase (a change of 18% points) in the number of questions answered correctly over the course of 15 weeks. However, one should interpret these results with caution. If the class were broken in half by scores on the post-test, the learning overwhelmingly accumulates to the most talented students. Those in the top half of the class on the pre-test experience showed more than a 72% increase (a 27% point change) in their scores while those in the bottom half only experience a 25% increase (a 9% point change). This suggests that any increases in cognitive complexity are unevenly distributed and accrue mostly to the most talented students.

Figure 1.

Pre- and post-test scores for the course and student subgroups.

The focus group evidence, nonetheless, supports the finding that the comparative approach is important for the learning experience. When prompted about whether or not they would want to investigate their own unique research questions, students responded that “since we all had the same question, we were more confident. With different questions, we would all be more doubtful of ourselves” and more importantly that “I thought learning all the methods was really interesting. Now I have a better insight on what every method does. Before, I never would’ve thought about doing textual analysis. Now you can see where it has its strengths and weaknesses. For every different method I can see where I would use it.”²

Additional feedback from the focus group suggests that in addition to the ability to carry out research students feel more comfortable “being able to criticize/analyze other research,” are more aware of “shock value” research (i.e. how research is misused in mass media) and the difficulty of doing good research. As several students discussed: “It (research) is a lot more difficult than I thought it was” and “Any research I did in high school was so easy. Then I got to this class and it was like a tornado in my head. It’s just so much more difficult than anything I’ve ever done.” These types of comments, including those from the student who went from the “tornado” to being one of best undergraduate researchers, suggest that students are grappling with the complexity of research methods and finding that good research requires more than just memorization. In many ways, comparison is at the heart of the scientific process so it seems logical to integrate it into the study of the process (Caiden, 1989). As one student reported, “I just catch myself asking more questions. Anytime we talk about something in class, I just keep asking questions, or how they got to that conclusion. I think it taught me how to ask questions.”

Implications and limitations

Without a strong comparison group, it is difficult to prove that this approach to the teaching of research methods is best. However, the combination of experiential-based learning and comparative research methods holds promise. The results from two semesters show significant increases in learning and the focus group results reflect a consensus that students valued these methods of inquiry and were pushed to expand their understanding of research methods through the comparative process. While historically, students have found research methods to be “irrelevant,” this approach to the teaching of research methods highlights how methodological choices shape research findings.

The method was not without its own drawbacks. Among the biggest concerns of the students is time management. Introduction of the various methods required approximately nine weeks before students could be assigned their topics for independent analysis. As a result, they only had 5–6 weeks to develop and execute a full research project. As one student commented “I think the hardest part was just the time in general. I know most of us didn’t start on it until the last three or four weeks before it was due… So it was mind-boggling and so stressful to put it together.” This was further compounded by the characteristics of the academic calendar. The project worked better in the spring semester than in the fall because the timing of spring break did not interfere in the same way that the Thanksgiving holiday interrupted data collection in the fall. One request from students was for more incremental deliverables to help hold them accountable to making progress on the research. This change was made for the next section.

From a project design standpoint, there are several other important considerations. First, this is a project which required significant social and academic support. Students found the project personally stressful and academically challenging. The faculty members involved served as consultants and needed to regularly challenge students to make the most of the experience. Those student groups who got regular feedback and support produced research projects that tremendously surpassed those from the groups who operated more independently.

Second, because learning did not accrue evenly across the class, some consideration should be paid to group formation. Students were allowed to choose their own groups. In most instances, students who scored high on the pre-test grouped with other students who scored high on the pre-test. In the few instances where low performers grouped with high performers, the scores of low performers rose. This was not true for groups made up only of low performers who experienced much more moderate increases on post-test scores. Improving group work and collaborative learning is beyond the scope of this paper, but Wolfe (2012) presents a nice overview of strategies.

Third, as we have continued to teach this course, we have shortened the length of the presentations and reduced the number of groups (when class size makes this possible). The debriefing of the presentations has proven to be a particularly valuable learning tool and helps reduce some of the limitations discussed in this section. During a two hour final exam period, we now try to limit the presentations to half of the time leaving one hour for discussion and debriefing.

A comparative experience-based approach to research methods also presents a number of more philosophical risks. Tashakkori and Teddlie (2003) have warned about the dangers of increasing specialization among research methodologists. Specialization comes with the risks of loss of general knowledge and not becoming widely trained practitioners. Allowing the students to “specialize” in a research method may limit their interest and ability to practice other methods. Similarly, a comparative approach to research methods continues to treat methods as distinct as opposed to methods that should and can be integrated together. As students develop as researchers, they need to be aware of how and when methods can be complemented and that methods are not equally valid substitutes. This point was not completely lost on the students as one student commented, “Every different method does provide a different result. [The student went on to compare the results of the textual analysis and interview groups] So there’s just so many differences between each method.” Through debriefing, this complementary and relative merit of the methods can be explored. Likewise, another student who was in the survey group explained “I think it’s really difficult for people to be able to say whether there’s a real difference opposed to what they see to be different.” This comment facilitated a discussion of how experiments can be used to confirm self-reported data.

This last point is a major concern with the assignment design. Given that students use differing methods to address the same research problem, there remains a threat that this pedagogy reinforces the relativism that is occurring in research methods today (Tashakkori and Teddlie, 2003). Good research design flows from the problem to the methodology. Forcing multiple methodologies on the same research problem encounters the risk that the students will find each method substitutable, rather than uniquely situated for different types of problems. In order to help rectify this risk, two steps were taken. First, during the 11th week of the semester class time was dedicated to exploring the appropriateness of different methods to different types of questions. Second, the presentations were debriefed in terms of how they matched the research problem discussed. Special attention was paid to the strengths and weaknesses of the methods relative to the question under investigation.

The last important issue for discussion is students’ understanding of the use of policy research. There is a robust history of scholarship on how policy research does or does not get utilized dating back to the 1970s (Lester, 1993; Weiss, 1979). There is significant evidence that use of social science research in policy decisions is not based on the methodology or the quality of the research, but rather on more normative considerations such as the scholars’ academic affiliation or professional network (Landry et al., 2003). In this sense, the design of the class does not prepare students for the political elements of policy research. This should be rectified early in the semester. The two sessions of the course that focus on the context and types of policy research could be used to explore its utilization.

Conclusion

Experiential-based approaches to the teaching of research methods have come a long way and hold a lot of potential for the future. This article presents a twist on many problem-based approaches by encouraging students to think comparatively: the addition of a comparative research project that asks different student groups to use different methods to tackle the same problem results in learning and potentially increasing complexity. Faculty members too often teach research methods as distinct tools in a toolkit rather than a common set of tools that needs to be thought of as different ways to examine different aspects of a common problem. Through integrating comparison into experiential-based approaches, this approach seeks to highlight the relative advantages and disadvantages of the tools while increasing the cognitive complexity necessary for true development.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Appendix 1

Appendix 2

References

Aguado

(2009) Teaching research methods: Learning by doing. Journal of Public Affairs Education 15(2): 251–260.

Andersen

Harsell

(2005) Assessing the impact of a quantitative skills course for undergraduates. Journal of Political Science Education 1(1): 17–27.

Benson

Blackman

(2003) Can research methods ever be interesting? Active Learning in Higher Education 4(1): 39–55.

Bernstein

Allen

(2013) Overcoming methods anxiety: Qualitative first, quantitative next, frequent feedback along the way. Journal of Political Science Education 9(1): 1–15.

Caiden

(1989) The value of comparative analysis. International Journal of Public Administration 12(3): 459–475.

Casasola

(2005) When less is more: How infants learn to form an abstract categorical representation of support. Child Development 76(1): 279–290.

Cook-Greuter

(2004) Making the case for a developmental perspective. Industrial and Commercial Training 36(7): 275–281.

Fenwick

(1992) Some problems of teaching research method in public policy. Teaching Public Administration 12(2): 77–81.

Fitzpatrick

(2000) What are our goals in teaching research methods to public administrators? Journal of Public Affairs Education 6(3): 173–181.

10.

Graham

Namy

Gentner

. (2010) The role of comparison in preschoolers’ novel object categorization. Journal of Experimental Child Psychology 107(3): 280–290.

11.

Hanney

Gonzalez-Block

Buxton

. (2003) The utilisation of health research in policy-making: Concepts, examples and methods of assessment. Health Research Policy and Systems 1(1): 1–28.

12.

Harris

Nikitenko

(2014) Comparing online with brick and mortar course learning outcomes: An analysis of quantitative methods curriculum in public administration. Teaching Public Administration 32(1): 95–107.

13.

Hubbell

(1994) Teaching research methods: An experiential and heterodoxical approach. PS: Political Science and Politics 27(1): 60–64.

14.

Kolb

(1984) Experiential Learning: Experience as the Source of Learning and Development. Englewood Cliffs, NJ: Prentice-Hall.

15.

Kramer

Schechter

(2011) Shock and awe: Rapid-fire theory, some surprising survey results, and triage statistics in an applied freshman research seminar. Journal of Political Science Education 7(3): 329–347.

16.

Landry

Lamari

Amara

(2003) The extent and determinants of the utilization of university research in government agencies. Public Administration Review 63(2): 192–205.

17.

Lester

(1993) The utilization of policy analysis by state agency officials. Science Communication 14(3): 267–290.

18.

Leston-Bandeira

(2013) Methods teaching through a discipline research-oriented approach. Politics 33(3): 207–219.

19.

Marfleet

Dille

(2005) Information literacy and the undergraduate research methods curriculum. Journal of Political Science Education 1(2): 175–190.

20.

Piaget

(2013) The Construction of Reality in the Child. New York, NY: Routledge.

21.

Rittle-Johnson

Star

(2007) Does comparing solution methods facilitate conceptual and procedural knowledge? An experimental study on learning to solve equations. Journal of Educational Psychology 99(3): 561–574.

22.

Rittle-Johnson

Star

(2009) Compared with what? The effects of different comparisons on conceptual knowledge and procedural flexibility for equation solving. Journal of Educational Psychology 101(3): 529–544.

23.

Roeder

Whitaker

(1993) Education for the public service: Policy analysis and administration in the MPA core curriculum. Administration and Society 24(4): 512–540.

24.

Ryan

Saunders

Rainsford

. (2014) Improving research methods teaching and learning in politics and international relations: a ‘reality show’ approach. Politics 34(1): 85–97.

25.

Sharp

Richardson

(2001) Reflections on Foucauldian discourse analysis in planning and environmental policy research. Journal of Environmental Policy and Planning 3(3): 193–209.

26.

Spronken-Smith

(2005) Implementing a problem-based learning approach for teaching research methods in geography. Journal of Geography in Higher Education 29(2): 203–221.

27.

Tashakkori

Teddlie

(2003) Issues and dilemmas in teaching research methods courses in social and behavioural sciences: US perspective. International Journal of Social Research Methodology 6(1): 61–77.

28.

Thies

Hogan

(2005) The state of undergraduate research methods training in political science. PS: Political Science and Politics 38(2): 293–297.

29.

US Department of Labor (2003) Computer and internet or e-mail use at work by occupational group, September 2001 (percent). Available at: http://www.bls.gov/careeroutlook/2003/spring/oochart.pdf (accessed 21 March 2016).

30.

US News and World Report (2016, 4 15) Best public affairs programs. Available at: http://grad-schools.usnews.rankingsandreviews.com/best-graduate-schools/top-public-affairs-schools/public-affairs-rankings.

31.

Wang

Baillargeon

(2008) Detecting impossible changes in infancy: A three-system account. Trends in Cognitive Sciences 12(1): 17–23.

32.

Weiss

(1979) The many meanings of research utilization. Public Administration Review 39(5): 426–431.

33.

Whitaker

Berner

(2004) Learning through action: How MPA public service team projects help students learn research and management skills. Journal of Public Affairs Education 10(4): 279–294.

34.

Wolfe

(2012) Implementing collaborative learning methods in the political science classroom. Journal of Political Science Education 8(4): 420–432.