Exploiting a natural experiment in assessing student learning outcomes in public and nonprofit administration: A demonstration

Abstract

This study exploits a natural experiment to establish the equivalence and/or difference in student learning outcomes between online and face-to-face public and nonprofit administration courses. Its main contribution is thus methodological. We will reexamine the online v. classroom debate - the face-to-face lecture is still the most effective method to deliver course content to students - using a new dataset and estimation approach. Our research addresses this causal inference question: Does the format of course delivery impact student learning? The most robust empirical strategy to rule out alternative explanations in causal studies is the experimental approach. In this study, we did not employ the experimental research design or any standard techniques, for example, regression analysis, available to the program evaluator or policy analyst. Instead, we exploited a naturally occurring phenomenon in a classroom environment to approximate statistical equivalence in the characteristics of students in the online and classroom formats and satisfy the exogeneity assumption of the treatment variable. Its more practical contribution is the use of learning theory and new research in online pedagogy to discuss the study’s conclusions and implications for online programming, instruction, and program coordination. We developed the feedback as teaching philosophy or approach to close the gap between the learning outcomes of completely asynchronous online and entirely face-to-face classes in public administration.

Keywords

Natural experiment online teaching assessment of student learning outcomes task-related feedback process-related feedback self-referenced feedback public affairs education program evaluation quasi-experiment as-if-random assignment endogeneity treatment effects analysis

Introduction

Online learning in higher education continues to expand (Allen and Seaman, 2017) with advances in information and communications technology (ICT) as its primary platform.¹ ICT is indeed disruptive (Christensen, 1997); its speed, 24/7 convenience, and widespread diffusion have fundamentally changed the market for post-secondary education. Internet technology, mobile computing, and social media have provided an efficient alternative to the traditional, brick-and-mortar delivery of post-secondary academic credentialing, effectively expanding the market reach of private, nonprofit, and public universities and colleges.²

Higher education institutions (HEIs) welcome online learning because of its potential to “bend the cost curve” (Deming et al., 2015). For example, Georgia Institute of Technology offers its pioneering online master’s degree in computer science at an “ultra-low” tuition of about $7,000, which is only about a sixth of the cost of an on-campus degree (McKenzie, 2018).³ More importantly, online programming attracts non-traditional and underserved students who require some flexibility in their academic journeys (Woldeab et al., 2020). It effectively expands tuition revenues and manages the risks and variations in enrollments by appealing to a segment of the student population who would not otherwise have pursued higher education because of the time, geographic, and other physical constraints of attending face-to-face classes. Online learning is thus a cost-saving and market-expanding product in higher education.

However, despite its vast potential to expand access to higher education, the completely online format could be difficult. Its quality cannot be guaranteed. Online education’s ability to meet students’ academic needs may depend on the convergence of at least three “success” factors - technology, student quality, and faculty acceptance of their new roles in a virtual academic environment (Meine and Dunn, 2017).

First, HEIs entering the online market must invest in a learning management system and associated software applications that facilitate online course design, development, delivery, and evaluation of student learning outcomes. These online learning technologies must be user-friendly to students and instructors alike and sufficiently robust to facilitate more frequent and profound student-to-student and student-to-instructor interactions. They must also safeguard academic integrity; otherwise, the economic role of academic degrees as a signal or indicator of ability to employers (Spence, 1973) breaks down.

Second, students who register for online courses should be comfortable with technology, less face-to-face interactions, and more self-paced learning. They must be able to balance the demands of work, family, and school independently. Absent the satisfaction of these conditions, students will be frustrated with the online format of instruction. The synchronous or asynchronous mode of e-learning is not for everybody.

Finally, faculty should embrace and value the potential of online instruction. Like students, they should also be comfortable with technology. More fundamentally, online instructors should accept that their traditional role as purveyors of information and demonstrators of skills is diminished significantly in the virtual classroom. In asynchronous e-learning, which is the more popular format in online education because of its flexibility, instructors are not required to lecture for at least three hours a week in real time. Instead, coursework is delivered via the web, email, and discussion boards. Accordingly, online instructors act more as learning facilitators and virtual forum moderators than lecturers. Contrary to popular belief, teaching an online class is more time-intensive as instructors are expected to engage with each of their students more frequently and establish their presence in the virtual classroom continuously.⁴

For a good reason, graduate programs in public administration (PA) and nonprofit management (NPM) have also joined the online bandwagon (Ginn and Hammond, 2012; Ni, 2013). However, given the technological, student selection, and instructional hurdles of online programming, leaders of these graduate programs should pause and reconsider the implications of the rapid expansion of online teaching and learning. While they are responsible for recruiting and attracting students by developing and offering new online certificate and degree programs and contributing to the financial bottom line of their respective colleges/universities, their ultimate goal, as curriculum leaders, is ensuring the quality of online instruction. The achievement of student learning outcomes should be independent of the instructor, section, and format of delivery of their graduate courses. The continuing oversight provided by the university administration and accrediting institutions like the Network of Schools of Public Policy, Affairs, and Administration (NASPAA), the Nonprofit Academic Centers Council (NACC), and the Higher Learning Commission (HLC) also applies the requisite pressure to ensure that the quality of instruction is not instructor- or format-specific.

This paper exploits a “natural experiment” in a previous academic year to establish if online graduate courses in public and nonprofit administration are at least on par with their traditional, face-to-face counterparts. Its main contribution is thus methodological. We will reexamine the online v. classroom debate - the face-to-face lecture is still the most effective method to deliver course content to students - using a new dataset and estimation approach. Its more practical contribution is the use of learning theory and new research in online pedagogy to discuss the study’s conclusions and their implications for online programming, instruction, and program coordination at the graduate level.

The next section reviews the theoretical and empirical literature on online teaching and learning. The third section discusses the data and method we used to establish the equivalence or non-equivalence of the learning outcomes of the two modalities of instruction, while the fourth section presents the empirical results. We review in more detail the natural experimental approach to guide other evaluators and analysts and program and curriculum coordinators, especially those in charge of assessing student learning outcomes, on how to identify situations in which natural experiments occur and exploit the same to answer causal research questions in their respective fields. The final section concludes and derives implications for managing, delivering, monitoring, and evaluating online certificate and degree programs in PA and NPM.

Theoretical and methodological motivation

Our motivation for this research centers around three inter-related issues: (1) the advantages and disadvantages of the online and physical classroom methods of delivering course content, including those of graduate public and nonprofit administration (PNA) courses, (2) the compatibility of the virtual academic environment with more technical and analytical courses in PNA graduate programs, for example, public finance and budgeting, research methods and statistics, and (3) the methodological limitations of extant studies that compare the learning outcomes of online and face-to-face courses. This section briefly discusses these three areas of concern.

Is the net advantage of the face-to-face format over its online counterpart not different from zero?

Traditional instructors and their students will agree that the classroom lecture is the more effective method of delivering course content. They will most likely cite social connectedness as the underlying reason for the learning outcome advantage of the face-to-face modality of instruction. The knowledge and skills of the academic professor, who usually has a PhD in her teaching area, and the professor of practice, who typically has a master’s degree and a significant professional experience in her field or subfield, for example, nonprofit fundraising, are maximized in a lecture format where she can summarize what she knows about the subject in front of her students who, in turn, can react and ask questions instantaneously. As classroom instructors introduce topics and sub-topics in real-time, their face-to-face students are more likely to ask more technical, specific, and profound questions that further advance their understanding of the course material.

The face-to-face instructor also benefits from continuous interaction. Their ability to feel the “pulse” of the class allows classroom lecturers to adjust their instruction and necessary interventions faster. Remedial instruction can be implemented in real-time in the traditional classroom environment, especially for seasoned academicians.

Social connectedness may also be associated with student persistence. Regular face-to-face interaction with instructors and classmates produces students who are more engaged and more committed to performing well in class and completing the course. The observability of their academic effort in the classroom is an endogenous pressure on students to complete the required readings and apply what they learned in course evaluations, for example, quizzes, exams, in-class discussions, and paper assignments.

Finally, some aspects of a course may be best transmitted face to face. In more technical courses, demonstrating to students (1) complex procedures like testing a research hypothesis using statistical software and (2) the associated perseverance that these analytical methods require may be easier in a face-to-face academic environment in which the instructor can use all the tools at her disposal, for example, PowerPoint, whiteboard and markers, Excel spreadsheets, to facilitate and deepen student understanding.

On the other hand, the operative terms in e-learning are “democracy” and “anonymity.” Concannon et al. (2005) believe online education is more democratic. Anonymity in a virtual class environment provides a safer space to participate, share ideas, be more creative, and even experiment and take risks. Experimentation and risk-taking are positively correlated with learning and innovation. Mozer, (2016) also showed that minority students were less likely to be discriminated against in an online academic environment: while she found that White students significantly outperformed their Black, Hispanic, and multiracial counterparts in the traditional classroom environment, this learning outcome disadvantage of minority students disappeared in the virtual class environment.

The other part of the argument for online learning is the inherent absence of a requirement to show up and prepare for an intensive 3-hour-a-week class or an all-day class on a Saturday (Ebdon, 1999). The online convenience of a self-determined academic schedule and self-paced learning provides students with more time to think and reflect on their learning and effectively sidesteps the sometimes-unproductive effort to complete a hundred pages of required readings to prepare for a weekly class in which every student is expected to participate in discussions and be tested on their understanding of the current as well as previous course topics. Not everybody who attends a 6:00–9:30 p.m. class on a Tuesday is prepared for classroom work. On the other hand, the online student who blocks out 10 hours a week on any day or days he wants, depending on the demands of his work, family, and other social obligations, will be much more ready to engage with the material, his fellow students, and the instructor.

Evidently, both formats have their own strengths and weaknesses in delivering course content. Thus, the question – Is the net advantage (or disadvantage) of one format over another not significantly different from zero? – begs to be raised. For those of us who manage graduate programs in public and nonprofit administration, this question is a legitimate one. It is a question that cannot be answered sufficiently by simply appealing to learning theory, one’s philosophy of education, and one’s feelings toward teaching traditions and the role of educational technologies that are beginning to supplant these traditions.

Are technical public administration and nonprofit management graduate courses and the virtual academic environment compatible?

The second part of the motivation for this study is the question of the compatibility of online instruction with more technical and analytical courses in PA and NPM graduate programs (Harris and Nikitenko, 2014; Ni, 2013). This issue extends the online-classroom format debate.

Because public and nonprofit professionals, especially supervisors and managers, are expected to contribute to the collection and analysis of empirical evidence to support public decisions, the standard curriculum in master’s programs in PA, public policy, and NPM and leadership includes courses in public finance and budgeting, nonprofit accounting and financial management, economic analysis, public policy analysis, program evaluation, and applied research methods and statistics. These courses are inherently more “scientific” and analytical than other graduate courses and rely to a greater extent on the theoretical and empirical literature and established techniques of a specific discipline or subdiscipline, for example, economics, finance, policy studies, and statistics. These quantitative public and NPM courses are thus comparable with science, mathematics, and engineering courses, which to education scholars, are more challenging to comprehend.

Quantitative techniques, for example, utility maximization and cost minimization using elementary differential calculus, and discounting of future revenues and costs, might also be involved. Technical steps like calculating a statistic, for example, a t-ratio that determines when a null hypothesis can be rejected, interpreting a financial performance metric, and identifying and valuing all known benefits and costs of a proposed policy or program are included in these analytical courses.

Given the inherent difficulty of teaching these technical and analytical public and NPM courses, e-learning may not be the ideal format. Adjunct faculty members, especially if they do not have a graduate degree in these more technical disciplines or subdisciplines, may not be the ideal instructor either. Moreover, a resident faculty with a doctorate in PA or philanthropic studies may not be maximized because her acquired knowledge and skill, as well as her research in the field, may not be transmitted well in the online academic environment. The procedures and methods necessary to handle these technical analyses may not be easily communicated and taught to online students either. In the final analysis, the successful transmission of technical knowledge and skills in the online academic environment, for example, the ability to conduct a cost-benefit analysis of a non-profit program, will most likely depend on what we earlier described as the triad of (1) faculty acceptance and understanding of their role as online instructors, (2) student comfort with technology and prior preparation to understand technical steps using online texts and videos, and (3) an adequate investment by the university in technology and software applications that facilitate the transfer of technical knowledge.

Internal validity of current comparative online-classroom studies

Finally, we know from a brief review of the literature that studies comparing the outcomes of online and face-to-face courses have methodological limitations. In outcome evaluations, the most critical condition to satisfy the validity of a causal argument⁵ is the No Rival Hypotheses requirement. If comparative online and face-to-face studies did not consider, for example, the differences in baseline characteristics of online and classroom students, they did not rule out alternative explanations or rival hypotheses. Accordingly, defending their internal validity will be more challenging.

Empirical strategy

Our causal research question is: Does course format delivery (X) impact student learning? Our key independent variable is course format, which can be completely face-to-face, entirely online, or a mix of the two modalities. Our outcome variable of interest is learning outcomes, more specifically, learning outcomes in applied research methods and statistics.

The identification problem

Establishing that variations in course format delivery (X) cause variations in student learning (Y) is methodologically challenging. A simple difference in expected outcomes between the completely face-to-face and the entirely online groups cannot identify the causal effect because the treatment variable is endogenous. Students who gravitate towards the online format may have reasons to do so: they may be more mature, more experienced, and more confident that they can thrive in the convenience and flexibility of the online format.⁶ Suppose every new graduate student optimizes (Cunningham, 2021) and chooses the format that maximizes her chance of success in a course. When that happens, we have what the methodology literature calls an “identification problem” (Manski, 1999). To solve the identification problem, we have to find a way to disentangle or isolate the effect of course format (X₁) on student learning outcomes (Y) from the effects of student-level characteristics like experience, motivation, maturity, gender, race and ethnicity, and socio-economic status (X₂ – X_k). In short, we have to make the treatment variable exogenous. See Figure 1 below.

Figure 1.

Main antecedents of the causal relationship between course format and student learning outcomes.

Achieving exogeneity: A brief review of research design and methods

To identify the causal effect of an independent variable (X₁) like a new program, policy, intervention, method, or approach, X₁ must be exogenous, or its variation must happen completely outside the factors that affect the outcome variables (Remler and Van Ryzin, 2015). Thus, an exogenous treatment variable (X₁) implies that its causal relationship with a dependent variable (Y) is not confounded by other factors, say, X₂ to X_k, that impact Y.

Y = β_{0} + β_{1} (t r e a t m e n t) + β_{2} X_{2} + β_{3} X_{3} + β_{4} X_{4} + β_{5} X_{5} + \dots + β_{k} X_{k}

(1)

In equation (1) above, if all factors that affect Y are observed, measured, and included in the model, the treatment variable (X₁) is assumed exogenous, and its regression coefficient can have a causal interpretation (Barnow et al., 1980). The regression coefficient of X₁ is interpreted as the expected or predicted effect of the treatment or intervention, for example, a drug, a policy innovation, a new social program, or an alternative way of delivering course content, on the outcome variable (Y), after removing the independent effects of rival explanations X₂-X_k on Y. This familiar ceteris paribus (all things being equal) assumption in OLS regression analysis is similar to the all-factors-being-held-constant-except-for-the-key-independent-variable assumption in scientific experiments. In laboratory settings, the independent variable of interest, let us call it X₁ for consistency, is manipulated or changed, while other factors that may cause the outcome variable are experimentally controlled or physically held constant (Remler and Van Ryzin, 2015) to easily identify or isolate the effect of X₁ on Y. The results of these scientific experiments provide credible evidence of causation from X to Y or the lack thereof, and such empirical results are usually called “internally valid” in the methodology literature.

While a standard approach in the natural sciences, like physics, laboratory experiments cannot be applied in the social sciences. Experimentally controlling for or holding constant all relevant variables outside of the treatment variable is not feasible. The approach closest to a laboratory experiment is the randomized experiment in which two groups - an experimental group and a control group - are exposed to or share the same factors, except for the treatment or intervention. In the medical sciences, randomized experiments are more commonly referred to as Randomized Controlled Trials (RCTs), where cases are randomly assigned to either the treatment/experimental or the control group.

Because the assignment or cases to either (1) the experimental/treatment group or (2) the control group is random, the treatment variable (X₁), by design, is not correlated with other factors, X₂-X_k, that also affect the outcome. For example, more intelligent and more motivated individuals are equally likely to be in the experimental or the control group as a result of random assignment. As long as the number of subjects is sufficiently large, random assignment more or less guarantees that the characteristics, observable and unobservable, of the treated and untreated groups, are statistically equivalent. The treatment variable is thus exogenous or statistically independent of rival or alternative explanations, X₂ to X_k. Refer to Figure 2 below. The subsequent data analysis is also less complicated, as a simple difference in mean outcomes estimates the treatment effect instead of relying on parametric estimation with independent variables X₁ to X_k as regressors.

Figure 2.

The exogeneity of the treatment variable in randomized experiments.

To summarize, answering the attribution question from the treatment to the outcome variable is easier in experimental studies, the gold standard in outcome or impact evaluation studies. Because of random assignment, the experimental and control groups are different only in the treatment: one group gets the treatment while the other group does not.⁷ Thus, the evaluator can more credibly argue that the remaining differences in outcomes between the treatment and control groups can be attributed to the treatment and the treatment alone and not to some internal or external factors (Singleton and Straits, 2004).⁸

A randomized experiment in the social sciences is often not politically acceptable and technically feasible. In PA and NPM, the random assignment of customers, clients, or constituents to welfare and anti-poverty programs, labor training programs, and other government and nonprofit programs is not ethically defensible.⁹ Accordingly, political leaders and nonprofit stakeholders, for example, donors and the board of directors, usually allow the target population to self-select themselves into federal, state, or local programs.¹⁰

The methodology literature has produced sophisticated approaches or methods that can approximate the rigor of randomized experiments. A middle ground between (1) experimental studies that use random assignment and (2) statistical controls in non-experimental data includes a family of quasi-experiments.¹¹

Most quasi-experimental approaches, including instrumental variable estimation, propensity score matching,¹² regression discontinuity design, and natural experiments, attempt to create a comparison group¹³ that resembles the treatment group in relevant respects. They mimic or imitate the random assignment process in experimental studies to achieve the same exogeneity of the treatment variable (X₁). For example, the Regression Discontinuity Design (RDD), which is considered the silver standard in policy and program evaluation, uses a program eligibility cut-off or threshold, for example, an individual becomes eligible to participate in the program if his or her GPA is at least 3.0 and ineligible if it is below that threshold, to create comparable treated and untreated groups. The argument that those just above the cutoff, for example, their GPAs are between 3.0 and 3.1, and those just below the cutoff, for example, their GPAs are between 2.90 and 2.99, are not systematically different from each other is plausible. Instead of a cut-off, the natural experimental (NE) approach uses a naturally occurring phenomenon to produce at least two groups that are as-if randomly assigned to either the treated or untreated group. We will explore this defining characteristic of natural experiments (as-if random assignment) in the next subsection.

Natural experiments

Natural experiments are events not under the control of the researcher or evaluator but can divide a population into exposed (treated) and unexposed (untreated) groups without regard to the characteristics of the members of this population (Dunning, 2012; Remler and Van Ryzin, 2015). First, these natural events could be new laws, policies, economic opportunities, or even disasters like famine caused by a war that affect one group but not another. Second, the occurrence of these events that defines which subjects or cases go into the treated or untreated group is not and cannot be controlled by the researcher. In true, randomized experiments, how the treatment, program, or policy is administered and implemented, including decisions about its dosage or level of intervention, is decided by the researcher. Since the researcher does not manipulate natural experiments, they are not planned or implemented like true experiments. Instead, natural experiments are discovered and later exploited to answer causal research questions. Third, the natural event is not intended to affect the outcome variable of interest, which further differentiates the natural experimental approach from randomized experiments. In true experiments, a new drug, policy, program, or method, for example, a reskilling program, or a reduction in class sizes, is administered to members of the treatment/experimental group with the objective of affecting or influencing the outcome variable of interest, for example, wage levels, academic achievement, positively. Events in natural experiments, in contrast, are not in any way related to the outcomes of a population. Finally, natural experiments result in two groups that are not systematically different from each other; the members of the population are divided into the treated or the untreated group as if by random assignment. This characteristic of natural experiments could be construed as either a similarity or a slight difference between natural experiments and randomized experiments. In the literature, as-if random assignment to the exposed or unexposed group is the defining feature of natural experiments.

Let us look at one example from the empirical literature to illustrate the characteristics or features of natural experiments. An interesting causal research question involves the examination of the impact of poverty or income level (X₁) on the mental health of children (Y). The treatment or key explanatory variable, which is poverty or income level, is endogenous because of the presence of causal variables that affect both treatment and outcome variables. Individuals with higher abilities and motivation or better work ethic are more likely to have higher incomes (or be out of poverty) and raise mentally healthy children. The endogeneity of the treatment variable is thus caused by common causes or rival explanations. To identify the causal effect of poverty on children’s mental health, the researcher has to isolate the pure effect of the treatment variable from the independent effects of rival explanations such as ability and motivation. If randomized experimentation is technically feasible and politically and ethically acceptable, the researcher will randomly assign subjects to either treatment or control group and assign members of each group a certain level of income. Random assignment will remove the correlation between income (X₁) and other explanatory variables (X₂-X_k) that impact the outcome, that is, more motivated and more intelligent individuals are equally likely to be in the poverty group or the out-of-poverty group. Since the two groups are similar in both observable and unobservable characteristics as a result of random assignment, the researcher can attribute whatever differences in children’s mental health outcomes between the two groups to the treatment variable (income) and not to rival explanations X₂-X_k. However, implementing a social experiment such as this is not easy. So, researchers turn to more feasible approaches. Costello et al. (2003) discovered a natural experiment that would allow them to tease out the causal effect of poverty on children’s mental health. The event was the opening of a casino on an Indian reservation in western North Carolina. The opening of a casino, which provided profit-sharing payments to tribal members on the reservation, was an exogenous shock to the treatment variable, which is income. Income was “manipulated” or “boosted” exogenously or outside of the common causes of variations in income levels and mental health outcomes. Those exposed to the treatment (more income due to the casino) were not individuals with higher abilities and motivation or better work ethic. The treatment was thus not related to any individual-level characteristic: everybody on the reservation received the treatment regardless of their observable or unobservable characteristics. The opening of the casino on an Indian reservation divided low-income families or households into exposed (those who received extra income) and unexposed (those who did not receive casino income because they were outside of the reservation). These families were mainly poor prior to the casino opening, but one group, as if by random luck, got extra income from casino profits, while the other group was not that lucky. This as-if random assignment makes this study a natural experiment. Moreover, how the treatment was manipulated was clearly outside the researcher’s control. The event is also not related to the outcome variable of interest since the opening of the casino is not a program that is intentionally implemented to improve the mental health outcomes of children.

Our natural experimental data

Like Costello and her colleagues, we also discovered a natural experiment to disentangle the pure effect of course format (X₁) on student learning outcomes (Y) from the independent effects of student-level characteristics (X₂-X_k).

This natural experimental approach presented itself when a graduate course in research methods for PA and NPM was taught for the first time in academic 2016–2017. Previously, public and nonprofit administration master’s students were combined with MBA students to take a business-oriented course in research methods and statistics.

MPNA 600 was offered for the first time in the fall of 2016. All newly admitted MPNA, MPA, and MNLM graduate students are required to register for MPNA 600 in the first semester of their respective graduate programs. This requirement is communicated to new students at most thrice within the month of their admission: in their respective admission letters from the graduate program director, the follow-up email, phone call, and/or face-to-face meeting with their respective faculty advisors, and the semi-annual new graduate student orientations that are usually held a week before the start of the fall and spring semesters.

In the 2016–2017 academic year, the Dean also assigned the teaching of both (1) completely online and (2) completely face-to-face formats of MPNA 600 to only one instructor. Since the new faculty was hired to start in the fall 2016 semester, he was also provided sufficient latitude to decide the schedule of the course’s online and face-to-face sections. The online section was scheduled for the fall semester and the classroom section for the spring semester.

The requirement imposed on newly admitted students to take MPNA 600 in their semester of graduate studies created a natural experiment in the 2016–2017 academic year, a research opportunity that can be exploited by evaluators interested in the outcomes of public affairs and NPM education. It thus divided newly admitted students for that academic year into exposed (those admitted in the fall and required to take the online section of MPNA 600) and unexposed (those admitted in the spring and required to take the face-to-face section instead) groups, without regard to student-level characteristics. We do not have compelling theoretical reason to believe that students who were admitted in the fall and required to take the completely online MPNA 600 were more intelligent, more motivated, more experienced, more technologically savvy, and more likely to be of a specific gender, race and ethnicity, and socio-economic status, than their counterparts who were admitted in the spring and registered for the entirely face-to-face section. Thus, the as-if random assignment assumption is plausible in this evaluation research. Treatment assignment, that is, whether a new graduate student took the online or face-to-face section of the same course, was not in any way correlated with variables X₂-X_k that also affect student learning outcomes (Y). We forced the exogeneity of the treatment variable by exploiting a policy or guideline that manipulates variations in the treatment variable outside of student-level characteristics.

That only one instructor taught both formats, completely online and completely face-to-face, made the natural experimental design even stronger: we can also rule out the confounding effect of instructor quality. We can reasonably assume that the same instructor applied the same general approach, philosophy, and skill in teaching basic statistics and applied research methods in both online and face-to-face formats. In fact, the same instructor used the same syllabus, required readings, lecture notes and slides, discussion questions, and assignments for both formats of the course.¹⁴

If the two groups, the completely online and face-to-face classes, are more or less homogenous by “natural” design, then their difference in mean outcomes as the average treatment effect is internally valid. Refer to Figure 3 below. We can thus reasonably attribute any outcome differences to the change in the format and not to some confounding factors like instructor quality and student characteristics. In the estimation of effect sizes, our natural experimental study can use the same uncomplicated difference-in-means estimator employed in true experimental research. See equation (2) below.

{A T E}_{NATURALEXPERIMENT} = E (Y_{F a c e - t o - F a c e}) - E (Y_{O n l i n e})

(2)

Figure 3.

The exogeneity of the treatment variable as a result of the natural experiment.

Empirical results

This section discusses the difference in learning outcomes between the completely face-to-face and completely online sections of MPNA 600, a required first course in the PA and NPM master’s programs Table 1.

Table 1.

The difference in student learning outcomes by format.

Outcome variables	Completely face-to-face (n₁ = 19 students)	Completely online (n₂ = 17 students)	Difference in outcomes	p-value
Developing public and nonprofit management-relevant research questions and testable research hypotheses	94.8	95.3	−0.5	0.7547
Justifying research hypotheses using theory and related studies	92.5	92.3	0.2	0.9168
Final paper: Research proposal	90.4	94.0	−3.6	0.0990^a
Discussion posts and peer-reviews	92.1	92.2	−0.1	0.9820
Statistical data analyses	93.6	89.0	4.6	0.0350^b
Final grade, 0–100	93.0	95.2	−2.2	0.1980
Final grade of A, binary	0.41	0.71	−0.30	0.0800^a
Course completion	100.0	90.0	0.10	0.1687

^aSignificant at the 10% level.

^bSignificant at the 5% level.

Seventeen (17) newly admitted graduate students took the completely online section of MPNA 600 in the fall of 2016. Master’s students do not have much choice because a course has only one section and format in any semester. Nineteen (19) students registered for the completely face-to-face MPNA 600, which is the only section and format offered in the spring of 2017.¹⁵ Self-selection is thus unlikely; students normally take the format available in their first semester of graduate study.

MPNA 600 requires students to develop a research question relevant to PA, public policy analysis, NPM, and social advocacy. After obtaining feedback from their peers and the instructor, students will either continue with their original research question or change it to one that better fits the policy and management relevance criterion. As soon as the research question is approved, students will develop a hypothesis, which is simply a tentative answer to their research question.

Secondly, students will look more closely at their hypothesis, for example, smaller classes lead to better K-12 academic outcomes, develop a theoretical explanation of why certain relationships in their respective hypotheses exist, and synthesize the relevant empirical literature that either supports or rejects these hypothesized relationships. The second assignment is, therefore, an exercise of using both theory and empirics to support a research hypothesis in public and in PA and NPM.

The final paper is a 10–12-page applied research proposal that also includes a section on data and methods of analysis that graduate students propose to use to answer their respective research questions. The last section of the proposal discusses the implementation timetable for the applied research.

The two formats are statistically equivalent in the student learning outcome, the ability to develop a policy- and/or management-relevant research question and hypothesis in PA or NPM. The same can be said of the learning outcome on supporting hypotheses using theory and the relevant empirical literature. While there are observed differences between the two, these differences are not significantly different from zero. There is no advantage of one format over another.

However, we did observe a learning outcome advantage of the online format when we considered the quality of the research proposal. See Supplementary Appendix 1 for details about the criteria used in evaluating the final paper in both formats. The mean grade of online students in the final paper was 94.0, compared to only 90.4 among face-to-face students. This outcome advantage of online teaching and learning is almost four percentage points, which is (marginally) significant at the 10% level.¹⁶

Interestingly, we saw a reversal of outcome advantages in statistical analysis, which is the acknowledged more technical part of the applied research methods course. In MPNA 600, assignments on constructing frequency distributions and cross-tabulations and calculating measures of central tendency (e.g., means and proportions) and dispersion (e.g., ranges and variances), differences in sample means and proportions, and inferential statistics like the t-ratio and the chi-square statistic are standard. The interpretation of these descriptive and inferential statistics and the derivation of policy and management implications that can be acted upon by relevant not-for-profit decision-makers are the central focus of the course after students learn how to derive these numbers by hand or through the use of the software. The inferential analysis part of the course is crucial in building the confidence of graduate students in reading the relevant literature of more advanced courses like public policy analysis, program evaluation, public finance, nonprofit fundraising, and nonprofit financial management. The ability to read the empirical literature will also be useful when students write their master’s Capstone projects, which is the final requirement before they can obtain their respective graduate degrees. To reiterate, our results showed that face-to-face students performed better than their online counterparts in statistical data analyses. Face-to-face students obtained a mean grade of 93.7, compared to only 89.0 for online students.¹⁷ This learning outcome advantage of the face-to-face format of about 4.6 percentage points is statistically significant at the standard five percent level.¹⁸

In terms of final grades, the previous advantages and disadvantages of the two formats in research paper writing and statistical data analyses may have canceled each other out. The completely online format had about 2.2 percentage point advantage over the completely face-to-face format, but this difference is not significantly different from zero (p > .15). However, if we look at the binary measure of the final grade, which was coded 1 for students who earned an A and 0 otherwise, a larger percentage of online students obtained an A. This 30% point difference is significant at the 10% level.

Finally, while more students did not complete the online course (two students in the completely online course compared to none in the classroom format), this 10% point difference is not statistically significant (p > .15).

Discussion and conclusion

Our results showed that the online teaching and learning modality is on par with the face-to-face format in the less technical content areas of a graduate-level introductory course in research methods and statistics. Teaching how to write a rigorous applied research proposal that includes a policy- and management-relevant research question, testable hypotheses, theoretical and empirical support for these hypotheses, and the data and methods of analysis to test them can be delivered effectively in either a completely face-to-face, classroom experience or an entirely online methodology. This finding is consistent with a recent and comprehensive systematic meta-analytic review, which compared face-to-face and online education and concluded that there was no significant difference between these two modalities (Woldeab et al., 2020) and an earlier study that compared online and brick-and-mortar quantitative methods courses in PA (Harris and Nikitenko, 2014).

Difference in learning outcomes between course delivery formats

We saw some learning outcome advantages of online learning when we considered the overall quality of the research proposals and the proportion of students who obtained a final grade of A. While these two outcome advantages of the virtual classroom are only marginally statistically significant at the 10% level, most likely due to a relatively small sample size, their practical or economic significance is not negligible. Seventy-one (71) percent of online students obtained an A compared to only 41% of their counterparts in the traditional lecture format. It was thus plausible that the absence of a rigid three-hour-a-week schedule allowed online graduate students to spend more time reading and understanding the course material; thinking creatively about their research questions, hypotheses, and methods; and writing and rewriting their final paper.

However, in statistical data analyses, the face-to-face lecture appears to be the more effective platform to deliver this technical content to PA and NPM graduate students. The almost 5%-point advantage of in-class learning over its online counterpart was statistically significant at the 5% level. In a point-and-click culture that puts a premium on convenience, teaching the required perseverance to undertake complex technical procedures and analytical techniques might be more difficult in an online academic environment. If this is the case, online instructors teaching technical courses like economic analysis, public finance, nonprofit financial management, and applied research methods to current and future public administrators and nonprofit managers will have to do more than posting online texts and moderating virtual discussion forums.

Online instructors may have to frontload more and introduce technical topics more extensively. Since online students are more likely to disconnect from the course than their face-to-face counterparts, faculty need to motivate their students continuously by introducing each technical topic in more detail and discussing why these topics, for example, interpreting inferential statistics, discounting future costs and benefits, matter in their public and nonprofit careers.¹⁹ Online instructors must prepare their students sufficiently before requiring them to undertake complex, technical analyses. Course design, including the alignment between technology and pedagogy is crucial in the online academic environment (Smith, 2014; Woldeab, et al., 2020).

How information is presented online significantly impacts student learning (Mayer, 2010). Online instructors should use more videos, graphics, and animations, including narrated PowerPoints, to motivate students to engage with and understand the course material. Links to external instructional videos that discuss techniques and procedures relevant to analytical courses like applied data analysis should be included on the course website. Software that allows online instructors to produce their own instructional videos and other interactive teaching and learning aids also looks promising.²⁰

At a more fundamental level, online instructors should be more familiar with their students before or at the beginning of the term. Entering graduate students have different levels of prior knowledge, may lack pre-requisite component skills, and have varied academic and professional experience in technical analyses (Ambrose et al., 2010). The classroom instructor has a built-in initial advantage because she can interact with her students for at least three hours at the beginning of the first week of class. This student-to-instructor interaction at the beginning of the term allows the classroom instructor to identify prior knowledge (or lack thereof) that might impede understanding technical topics like calculating and interpreting descriptive and inferential statistics. The online instructor needs the same information about prior knowledge but must have strategies to obtain it without classroom interaction. For example, requiring online students to prepare a text, audio, or video self-introduction to discuss, among other things, their academic background, professional experience conducting data analyses and writing technical research reports, and their comfort level with statistics and quantitative analysis will help the online instructor gauge students’ prior knowledge.²¹ The online instructor can then introduce supplemental material²² at the start of the term and, more importantly, modify and strengthen course design to address any technical deficiencies.

In both environments, students benefit from practice and feedback (Gagne, 1985; Martin et al., 2004). In an online environment, practice and feedback may come from self-check exercises, computer-graded exercises, group discussions, simulations, one-on-one student-to-teacher interactions, and worked examples. Online instructors must identify appropriate strategies that maximize opportunities for students to practice technical procedures and find the needed resources. Repeated practice supports learning. Spaced repetitions are more effective than non-spaced repetitions, and both presentations of learning material and retrieval practice opportunities produce benefits when utilized as spaced repetitions (Thalheimer, 2006). In applied research methods courses, the next problem set may include a similar problem from the last assignment to allow students a critical, second opportunity to apply the same technical concepts and procedures. Allowing second chances to students who performed poorly in a previous assignment to resubmit their work for credit might also help ensure that everybody will have a firm handle on the technical content of the course.²³

Feedback as teaching

Providing quality feedback to students is a lever that instructors can push to close the gap between the learning outcomes of completely online and entirely face-to-face classes. We will briefly argue in this concluding section that preparing qualitative comments on students’ work is an essential part of teaching and critical to alleviating the weaknesses of a particular course delivery format. For lack of a better term, the “feedback as teaching” philosophy elevates instruction as a commitment to assisting every student in meeting the course’s learning objectives. Educative or formative assessment monitors student learning and provides feedback that can potentially close the gap between the student’s current and expected level of performance, and thus, is superior to auditive assessment, which only monitors whether the student succeeded in absorbing the material they studied (Fink, 2003). Providing feedback reconsiders every task the instructor had already completed, from setting the learning targets to assembling the required readings, deciding on pedagogical strategies to deliver course content, and developing assessment tools to determine how students meet learning expectations. Providing qualitative feedback (an educative assessment) over and above giving a numeric score (an auditive assessment) is thus teaching all over again, and not every faculty has the wherewithal to do it consistently.

Strengthening the overall feedback mechanism in online classes can potentially tip the scale toward improving student learning in statistical data analysis. Following Brookhart (2008), the instructor’s feedback should be task- and process-related.²⁴ Students must first be informed if they produced the appropriate statistical tables and graphs and, more importantly, if they correctly interpreted the statistics they calculated. Knowledge about the quality of the work, which includes the accuracy or precision in calculating the correct statistics, is task-related feedback. More importantly, students should receive feedback on the suitability of the approaches used to produce the analytical work. Informing students that they did not produce the right frequency table, cross-tab, or correlation matrix, or did not calculate the precise statistic or statistics is simply insufficient. Attaching the answer key or solution to the problem set is an attractive and easy alternative to writing individualized feedbacks, but this feedback mechanism may not lead the student to identify gaps in his approach or process efficiently. Instructors, especially online instructors, have to examine the student’s work, for example, the accompanying Microsoft Excel spreadsheet, or in more advanced research methods courses, lines of commands in statistical software like Stata, and identify specific steps that the student did not do or incorrectly implemented, for example, reversing the order of the dependent and independent variables in a cross-tab. This process-related feedback will help the student identify and implement the correct approach to the same problem, especially if the statistical skill is tested again in subsequent assignments or exams. The specific guidance about the right process increases the student’s confidence to self-correct, adopt the correct approach, and ultimately learn the skill targeted by the course.²⁵

A potential mechanism that can explain the learning outcome advantage of online classes in writing the final paper (in this study, an applied research proposal in PA, NPM, policy analysis, or social advocacy) is the focus on online discussion boards as a teaching and learning methodology. Since we started teaching online research methods classes, we have required students to upload each section they write for the final paper (including sections on the policy relevance of the research question and the plausibility of their hypotheses) onto the discussion boards that are accessible to every member of the class.²⁶ An online student can practically download and read all submissions. This ability to read the work of others and the requirement to review and comment on their strengths and weaknesses relative to a set of criteria, for example, the research question is relevant to public/nonprofit management and policy, the research hypotheses are supported by theory and related studies, provides online students self-reference feedback, which is distinct from the task- and process-related feedback. Online students will be able to compare their work with their peers,’ identify the class exemplars, and decide how they will raise their effort to meet or exceed the course’s learning targets. This comparative mental exercise may give online students who are making slow progress the extra push to engage with the readings, interact with their online classmates, and think critically about their proposed research questions a little bit more (Weimer, 2013).

A robust feedback system that includes task- and process-related feedback from the instructor and interactive, dynamic discussion boards requiring students to read and comment on each other’s work and acquire self-reference feedback addresses the declining student participation and interaction observed by Rawat et al. (2023) in online public administration courses.

In a face-to-face class, most students, especially learners with full-time jobs and family obligations, rely heavily on classroom lectures and discussions to learn the material. They drive to our campuses after work, instead of registering for the online section of the course, because of their confidence they will learn better in the classroom. In this study, learning how to write a research paper relevant to public and nonprofit decision-making was not a strength of face-to-face classes. This weakness can be mitigated by the “feedback as teaching” pedagogy. Self-reference feedback can be integrated into a traditional classroom format to improve learning outcomes in research paper writing. Classroom instructors can request students who performed well in a segment of the paper to share their policy-relevant research question and the process they used to develop it. In the latter part of the course, other class exemplars may also discuss how they developed theoretical justifications for their hypotheses and identified related studies that empirically support them.

Instructor assistance to students struggling with the course material should also be improved and strengthened in the online academic environment. In the classroom, if students run into trouble following a statistical procedure in Excel or SPSS or Stata, they can look over each other’s shoulders and ask questions of one another or the instructor. In the 1930s, Vygotsky wrote about the zone of proximal development, which addresses the nudge that students can get with a bit of help from more capable peers or the instructor. Help in an online environment could be hours away – from either the instructor or peers. Online instructors have solved these challenges, even in massively open online courses, by promoting self-organized study groups, allowing drop-in virtual office hours, and publishing answers to common questions in messaging applications like Slack that can effectively reduce back-and-forth emails about the same topic. Regularly encouraging online students to use Voice Over Internet Technology (VOIP) like Zoom, Skype, and Teams to contact and seek help from their instructor will also make a difference in alleviating the weaknesses of the completely online format in delivering technical content areas like data analytics.²⁷

Management of online learning

For coordinators of graduate programs in PA and NPM, the online v. face-to-face debate has become irrelevant. Online programs are here to stay; the demand for online courses has become more robust (Scutelnicu et al., 2019), especially after the COVID-19 pandemic. From our experience teaching in a public university operating in a large metropolitan area in midwestern United States, it is now easier to fill an online class than a face-to-face class. The primary reason is obvious: online learning is convenient enough to attract more nontraditional and underserved students. Accordingly, the continuing responsibility of program leaders and department chairs is ensuring their online courses’ quality. They should continuously advocate for a more robust learning management system, for example, Canvas and D2L Brightspace, and other software applications that support online learning. They have to prepare prospective and new graduate students for online learning. Faculty advisors, curriculum coordinators, program directors, and department chairs must strengthen new student orientations and advisor-student meetings to remind students to take online courses if they are comfortable with technology, less face-to-face interactions, and self-paced learning (Harris and Nikitenko, 2014). The university should also ensure that students taking online courses are prepared for e-learning by providing introductory online training. Finally, we must convince our faculty colleagues to embrace online teaching and their new, more complex role in the online academic environment (Martin et al., 2019) as subject matter expert, instructional designer, web content and IT manager, and forum moderator all rolled into one. They have to strengthen their skills as learning facilitators by taking advantage of training opportunities in online teaching technologies, instructional design, and communication strategies so they can engage with their students continuously and effectively and alleviate the weaknesses of online teaching and learning.²⁸ Convincing resident and adjunct faculty and university officials to accept that the online format is the more time-consuming format, that successful online courses are a product of the collaboration of a dedicated faculty member and a skilled instructional designer (Scutelnicu et al., 2019), and that excellent classroom lecturers are not necessarily effective online instructors will be a great place to start.

Supplemental Material

Supplemental Material - Exploiting a Natural Experiment in Assessing Student Learning Outcomes in Public and Nonprofit Administration: A Demonstration

Supplemental Material for Exploiting a Natural Experiment in Assessing Student Learning Outcomes in Public and Nonprofit Administration: A Demonstration by Katelyn M. Sileo, Robert Bilyk, and Daniel Woldeab

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Reynold Galope

Supplemental Material

Supplemental material for this article is available online.

Notes

References

Allen

Seaman

(2017) Digital learning compass: distance education enrollment report 2017. Babson Survey Research Group. Available at: https://onlinelearningconsortium.org/read/digital-learning-compass-distance-education-enrollment-report-2017/.

Ambrose

Bridges

DiPietro

, et al. (2010) How Learning Works: Seven Research-Based Principles for Smart Teaching. San Franciso: Jossey-Bass.

Barnow

Cain

Goldberger

(1980) Issues in the analysis of selectivity bias. Evaluation Studies Review Annual 5: 43–59.

Brookhart

(2008) How to Give Effective Feedback to Your Students. Alexandria: Association for Supervision and Curriculum Development.

Christensen

(1997) The Innovator’s Dilemma: When New Technologies Cause Great Firms to Fail. Boston: Harvard Business School Press.

Concannon

Flynn

Campbell

(2005) What campus-based students think about the quality and benefits of e-learning. British Journal of Educational Technology 36(3): 501–512. DOI: 10.1111/j.1467-8535.2005.00482.x.

Costello

Compton

Keeler

, et al. (2003) Relationship between poverty and psychopathology: a natural experiment. Journal of the American Medical Association 290(15): 2023–2028.

Cunningham

(2021) Causal Inference: The Mixtape. New Haven: Yale Univerity Press.

Deming

Goldin

Katz

, et al. (2015) Can online learning bend the higher education cost curve? The American Economic Review 105(5): 496–501. DOI: 10.1257/aer.p20151024.

10.

Dunning

(2012) Natural Experiments in the Social Sciences: A Design-Based Approach. Cambridge: Cambridge University Press.

11.

Ebdon

(1999) Teaching public finance administration online: a case study. Journal of Public Affairs Education 5(3): 237–246. Available at: https://www.jstor.org/stable/40215443.

12.

Fink

(2003) Creating Significant Learning Experiences. San Francisco: Jossey-Bass.

13.

Gagne

(1985) The Conditions of Learning. New York: Holt, Rinehart & Winston.

14.

Galope

(2016) A different certification effect of the small business innovation research (SBIR) program: evidence from the Kauffman firm Survey. Economic Development Quarterly 30(4): 371–383. DOI: 10.1177/0891242416658346.

15.

Ginn

Hammond

(2012) Online education in public affairs: current state and emerging issues. Journal of Public Affairs Education 18(2): 247–270. DOI: 10.1080/15236803.2012.12001683.

16.

Harris

Nikitenko

(2014) Comparing online with brick and mortar course learning outcomes: an analysis of quantitative methods curriculum in public administration. Teaching Public Administration 32(1): 95–107. DOI: 10.1177/0144739414523284.

17.

Johnson

Hewapathirana

Bowen

(2019) Faculty development for online teaching. In: Keengwe

(ed) Handbook of Research on Virtual Training and Mentoring of Online Instructors. Hershey: IGI Global, 40–55.

18.

Manski

(1999) Identification Problems in the Social Sciences. Cambridge: Harvard Business School Press. Available at: https://doi.org/10.4159/9780674265790.

19.

Martin

Budhrani

Kumar

, et al. (2019) Award-winning faculty online teaching practices: roles and competencies. Online Learning 23(1): 184–205.

20.

Martin

Klein

Sullivan

(2004) Effects of Instructional Events in Computer-Based Instruction. Bloomington: Association for Educational Communications and Technology. Available at: https://eric.ed.gov/?id=ED484984.

21.

Mayer

(2010) Multimedia Learning. Cambridge: Cambridge University Press.

22.

McKenzie

(2018) Online, cheap -- and elite. Inside Higher. 20. Available at: https://www.insidehighered.com/digital-learning/article/2018/03/20/analysis-shows-georgia-techs-online-masters-computer-science.

23.

Meine

Dunn

(2017) MPA programs and internet education: validation of quality and acceptance despite challenges surrounding online delivery. Journal of Public Affairs Education 23(2): 665–676. DOI: 10.1080/15236803.2017.12002277.

24.

Mozer

(2016) A comparison of online and face-to-face achievement in physical science. EdD. USA: Dissertation, Kennesaw State University.

25.

(2013) Comparing the effectiveness of classroom and online learning: teaching research methods. Journal of Public Affairs Education 19(2): 199–215. DOI: 10.1080/15236803.2013.12001730.

26.

Rawat

Yan

, et al. (2023) When public administration education switches online:student perceptions during COVID-19. Teaching Public Administration 41(1): 122–142. DOI: 10.1177/01447394221119092.

27.

Remler

Van Ryzin

(2015) Research Methods in Practice: Strategies for Description and Causation. Thousand Oaks: Sage.

28.

Research and Markets (2022) E-learning - global market trajectory & analytics. Available at: https://www.researchandmarkets.com/reports/2832322/e-learning-global-market-trajectory-and

29.

Rovai

Ponton

Wighting

, et al. (2007) A comparative analysis of student motivation in traditional classroom and e-learning courses. International Journal on E-Learning 6(3): 413–432.

30.

Scutelnicu

Tekula

Gordon

, et al. (2019) Consistency is key in online learning:evaluating student and instructor perceptions of a collaborative online-course template. Teaching Public Administration 37(3): 274–292. DOI: 10.1177/0144739419852759.

31.

Singleton

Straits

(2004) Approaches to Social Research. New York: Oxford University Press.

32.

Smith

(2014) Conquering the Content: A Blueprint for Online Course Design and Development. San Francisco: Jossey-Bass.

33.

Spence

(1973) Job market signaling. Quarterly Journal of Economics 87(3): 355–374. DOI: 10.2307/1882010.

34.

Thalheimer

(2006) Spacing Learning Events over Time: What the Research Says. Somerville, MA: Work-Learning Research, Inc. Available at: https://www.work-learning.com/catalog.

35.

Weimer

(2013) Learner-Centered Teaching: Five Key Changes to Practice. San Francisco: Jossey-Bass.

36.

Woldeab

Yawson

Osafo

(2020) A systematic meta-analytic review of thinking beyond the comparison of online versus traditional learning. E-Journal of Business Education & Scholarship of Teaching 14(1): 1–24.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.41 MB