Measuring What Students Know: SNAP’s Guidelines and Suggestions for Assessing Goal 1 Content in Psychology

Abstract

Although many psychology departments and instructors are aware of the American Psychological Association Guidelines for the Undergraduate Psychology Major Version 2.0, they are often less aware of the means by which to assess student mastery of the recommended goals. Our purpose is to discuss general principles for assessment, offer a psychology learner taxonomy that aligns with Goal 1 of the Guidelines 2.0, and present a rubric for reviewing assessments. Goal 1 of the Guidelines 2.0 is based on content knowledge in psychology. Whereas most assessments allow for the measure of the mastery of content to different extents, the results of those assessments can be invalid due to the design or inappropriate use of the rubric. The working group at the Summit on National Assessment of Psychology addressed these issues and curated evidence-informed assessment exemplars designed to measure content knowledge in psychology.

Keywords

assessment student learning outcomes content taxonomy guidelines

The pressure from internal administrators and external accrediting bodies to produce incontrovertible assessment data is at an all-time high (Gaston, 2018). Institutions use the data to market programs, ensure programmatic excellence, and make budgeting decisions. Accreditors use the data to judge whether or not the program meets their guidelines and fulfills the promises put forth by their institutions. As a result, program chairs and faculty are under intense scrutiny to produce these data as evidence to further support their programs. However, chairs and faculty are sometimes naïve when it comes to effective assessment practices and principles and often do not know where to start (Hutchings, 2010). Helping these individuals and programs was the guiding principle for the American Psychological Association’s (APA) Summit on National Assessment of Psychology (SNAP).

The Backstory

When the task force that developed APAs Guidelines for the Undergraduate Major (American Psychological Association, 2007) published their recommendations for the first set of student learning outcomes (SLOs) about the content and skills psychology students should know at graduation, they were surprised by the questions the document generated for related assessment strategies. In response to faculty and departmental demand for more guidance on assessment, the task force went beyond their original charge to create the Assessment Cyberguide (Pusateri et al., 2009) that was a detailed inventory of existing assessments that could be used to assess the outcomes. Unfortunately, at the time, several of the recommended assessments were cost-prohibitive for many programs and did not provide supportive evidence for the claims made, nor did they necessarily align with the design of the departmental programs themselves.

When the APA Guidelines 2.0 was officially revised in 2013, the new task force differentiated outcomes that were considered “foundational” (i.e., curriculum for lower level curriculum) versus “baccalaureate” (i.e., upper level curriculum) indicators. Guidelines 2.0 actually addressed assessing curricular goals within the guidelines themselves, yet the task force did not endorse any specific assessments. Most psychology departments teach courses in the core content areas of introductory, social, abnormal, developmental, personality, and research methods (Norcross et al., 2016), and many departments have program outcomes that are aligned with APA Guidelines 2.0. Yet many departments continue to struggle with the best ways to assess the recommended learning goals.

To this end, in 2016, the APAs newly minted Committee on Associate and Baccalaureate Education (CABE) issued a call for academic faculty and staff with a vested interest in the assessment of psychology to participate in a weeklong summit devoted to the curation and creation of evidence-informed assessment to support Guidelines 2.0. The SNAP invited assessment enthusiasts to meet at the University of Wisconsin—Green Bay in response to questions that CABE received related to the use of the Guidelines. Although most psychology departments use the Guidelines in part or in their entirety (Norcross et al., 2016), there is little consistency in the way that the psychology programs assess those goals. SNAP brought like minds together to construct recommendations for assessments aligned with Guidelines 2.0.

Our working group’s SNAP task was to focus on assessments related to Goal 1: Knowledge Base in Psychology. Thus, our organizing questions were as follows: How should psychology content be assessed? How do we, as instructors of psychology, develop these assessments? What strategies can we use to share successful assessment resources? And most importantly, how do assessment results inform our practice?

Early on in our discussions, we discovered a paradox; not only did it appear that we had the “easiest” goal to address, but in many ways, we also had the most insurmountable goal to address. It was easy because almost every assessment in psychology addresses psychological content knowledge to one extent or another. It was insurmountable because the scope of possible assessments would make it impossible to scrutinize all possible content assessments in psychology. Consequently, we focused on making recommendations for the development and use of effective content assessment practices (i.e., guiding principles of assessment for instructors and administrators), curating a rubric to evaluate those assignments (i.e., psychology assessment evaluation rubric), and establishing a framework into which those assessments might align with Guidelines 2.0 (i.e., the psychology learner taxonomy). We designed the latter framework to capture the nuances of learning psychology content, to categorize student learning objectives, and to focus psychology education on having our students not only understand and comprehend psychology content but to be consumers and producers of psychology. We conclude the article with an examination of whether a nationally normed instrument could solve faculty assessment problems.

Assessment of Goal 1: Content Knowledge in Psychology

As shown in Table 1, Guidelines 2.0 Goal 1 includes three subgoals in which students should be able to do the following:

describe key concepts, principles, and overarching themes in psychology (1.1);

develop a working knowledge of psychology’s content domains (1.2); and

describe applications of psychology (1.3).

Table 1.

Goal 1 Outcomes and Indicators.

OutcomesStudents Will:	Foundation IndicatorsStudents Will:	Baccalaureate IndicatorsStudents Will:
1.1 Describe key concepts, principles, and overarching themes in psychology	1.1a Use basic psychological terminology, concepts, and theories in psychology to explain behavior and mental processes	1.1A Use and evaluate theories to explain and predict behavior, including advantages and limitations in the selected frameworks
	1.1b Explain why psychology is a science with the primary objectives of describing, understanding, predicting, and controlling behavior and mental processes	1.1B Describe the complexity of the persistent questions that occupy psychologists’ attention
	1.1c Interpret behavior and mental processes at an appropriate level of complexity	1.1C Analyze the variability and continuity of behavior and mental processes within and across animal species
	1.1d Recognize the power of the context in shaping conclusions about individual behavior	1.1D Examine the sociocultural and international contexts that influence individual differences (e.g., personality traits, abilities) and address applicability of research findings across societal and cultural groups
	1.1e Identify fields other than psychology that addresses behavioral concerns	1.1E Compare and contrast the nature of psychology with other disciplines (e.g., biology, economics, political science), including identifying the potential contribution of psychology to interdisciplinary collaboration
1.2 Develop a working knowledge of psychology’s content domains	1.2a Identify key characteristics of major content domains in psychology (e.g., cognition and learning, developmental, biological, and sociocultural)	1.2A Compare and contrast psychology’s major subdisciplines
	1.2b Identify principal methods and types of questions that emerge in specific content domains	1.2B Speculate about why content domains differ in the kinds of questions asked and the methods used to explore them
	1.2c Recognize major historical events, theoretical perspectives, and figures in psychology and their link to trends in contemporary research	1.2C Summarize important aspects of history of psychology, including key figures, central concerns, methods used, and theoretical conflicts
	1.2d Provide examples of unique contributions of content domain to the understanding of complex behavioral issues	1.2D Explain complex behavior by integrating concepts developed from different content domains
	1.2e Recognize content domains as having distinctive sociocultural origins and development	1.2E Predict how sociocultural and international factors influence how scientists think about behavioral and mental processes
1.3 Describe applications of psychology	1.3a Describe examples of relevant and practical applications of psychological principles to everyday life	1.3A Articulate how psychological principles can be used to explain social issues, address pressing societal needs, and inform public policy
	1.3b Summarize psychological factors that can influence the pursuit of a healthy lifestyle	1.3B Evaluate how the mind and body interact to influence psychological and physical health
	1.3c Correctly identify antecedents and consequences of behavior and mental processes	1.3C Propose and justify appropriate psychology-based interventions in applied settings (e.g., clinical, school, community, or industrial settings)
	1.3d Predict how individual differences influence beliefs, values, and interactions with others, including the potential for prejudicial and discriminatory behavior in oneself and others	1.3D Explain how psychological constructs can be used to understand and resolve interpersonal and intercultural conflicts

Source. APA (2013).

As is the case with all of the APA Goals, Goal 1 is divided into foundation- and baccalaureate-level indicators. Foundation indicators specify what students should be able to know or do after only a few courses, upon receiving an associate degree, or after achieving a minor in psychology. As students progress from foundation-level indicators to baccalaureate-level indicators, their understanding of the content and ability to apply psychological concepts and ideas increases. In contrast, baccalaureate level indicators establish performance expectations for students upon the completion of a bachelor’s degree.

Given the dual levels of Guidelines 2.0 (e.g., foundation vs. baccalaureate indicators), corresponding assessments might be crafted to measure students’ mastery at different levels as well. With this idea in mind, the working group began with the premise that there might be different levels of performance required by different assessments and that students should be able to demonstrate multiple levels of mastery. This philosophy led to the creation of principles of assessment to guide instructors and departments in their assessment efforts and a multilevel taxonomy of learners to reshape the priority of how and what we want students to learn in psychology.

The Guiding Principles of Assessment

At its best, assessment is a process intended to support reflection and guide responsive instruction to improve student learning. At its worst, assessment is a rigid process used to punish both faculty and students (Kramer, 2006). Psychology educators are in a unique place—at the intersection of disciplinary research, rich and engaging pedagogical content, and the inquiry of student learning. For some psychology educators, implementing assessment practices will be a natural extension of their scholarly curiosity, but for others, who are less inclined to see assessment as a natural outcome of their work, the learning curve may be steep. As Chew et al. (2018) stated, change is never easy and usually daunting.

The modern student is already sitting in our classrooms or taking courses online, grappling with wicked problems, and begging for a meaningful, progressively challenging learning experience that develops skills and allows them room for exploration. It is imperative that faculty and departments continue to advance teaching and learning practices to engage students and leverage emerging knowledge of evidence-based teaching and learning strategies. To assist in surmounting this obstacle, we created four guiding principles of assessment in psychology.

Principle 1

Align curriculum, instruction, and assessment to achieve the best learning results

By alignment, we mean that instruction, curriculum, and assessment are working in concert and not at cross-purposes (Chudowsky et al., 2004). The main function of an assessment is to improve teaching and learning. To fulfill that promise successfully, assessment efforts must be faculty-led (Hutchings, 2010), and the relationship between assessments, teaching methodology, and learner outcomes should be communicated and reflected upon regularly. Often when faculty discuss alignment, their focus is on curriculum mapping, that is, matching what is being taught with how it is assessed (English, 1980; Jankowski & Marshall, 2017). Curriculum mapping and course alignment matrices are important components of the process but alignment activities should not stop there. Moving beyond conceptual alignment encourages viewing assessment as multifaceted and layered paradigm for learning. Alignment extends into every aspect of the learning environment.

We recommend using backward design to align curriculum, instruction, and assessment (McTighe & Wiggins, 2012). This approach begins with the end result of learning (i.e., the intended learning objective or Student Learning Outcome [SLO]) to set the context for any assessment, learning, teaching, or pedagogical approach. Psychology educators should be explicit about their end goals and constantly highlight relevant connections. This method is an alternative approach to the more traditional content coverage model. In this model, the content drives what the students learn, which may or may not lead to student learning objectives. Unlike the content coverage model, the backward design approach is purposeful and focuses on organizing and aligning a meaningful learning sequence, selecting teaching and learning activities in service of the SLO to ensure the SLO will be achieved.

For those who are serious about backward design, several resources exist for developing SLOs that support curricular objectives. Advice about writing effective SLOs can be found in several resources (Diamond, 2009; Fink, 2003; Gronlund & Gronlund, 2004; Nilson, 2016; Suskie, 2018; Wiggins & McTighe, 2005). Articulating SLOs takes time. But this process is not contrary to teaching nor does it divert from its primary purpose. Instead, articulating SLOs and aligning them with assessment rubrics and methodologies strengthens teaching and learning. Diamond (2009) highlights this idea:

A quality education does not happen by chance; it requires careful planning, skilled teaching, and an overall structure that ensures that every student can reach the goals of the program in which he or she is enrolled. A quality education requires a level of orchestration seldom found at colleges and universities and the active involvement of a faculty that is paying a great deal of attention to structure, content, and process. It requires hard work. (p. 49)

Once instructors identify SLOs and map curricular alignments, selection of pedagogical and assessment strategies completes the alignment. The result of learning should drive the assessment methodology, not the other way around, and the chosen pedagogical approach should enhance student learning, not define it. In this framework, the content knowledge is not the end of learning; instead, the learner’s manipulation, synthesis, and application of the content knowledge is the focus. Additionally, sound assessment practice suggests looking at an SLO across several learning experiences, such as exit tickets, exams, and applied projects, to promote formative assessment in which students receive feedback to develop their abilities. Will the student experience a variety of learning objects and multiple ways to demonstrate learning and receive feedback before the instructor renders a summative judgment, such as a course grade? If not, the intended learning sequence may present gaps in the student’s knowledge or skills that are left unexplored.

When applicable, instructors should identify additional alignments, such as to discipline standards, program-level, or institution-level learning outcomes. For example, map course-based student learning objectives with discipline-level learning using the APA Guidelines 2.0. For several psychology courses, student learning may extend beyond the discipline or program and link to general education learning outcomes and liberal arts curriculum. If applicable, alignment should be identified to institution-level learning outcomes as well.

The point is not to assess everything, to make all assessment high-stakes, or even to align every aspect of the learning environment. Instead, an instructor’s focus should be on the relationships between the components of the learning environment and strengthen connections where needed. Wang et al. (2013) predicted that students will adjust their learning behaviors to the learning environment, so it is worth our time to align for learning and to investigate those connections.

Principle 2

Use empirical inquiry to select scientifically sound assessment strategies

We suggest that both instructors and administrators should consider empirical inquiry, leaning on the scientist-educator model (Bernstein et al., 2010) to select evidence-based measures and determine a strategy to systematically collect evidence of learning. There are no perfect assessment measures; each assessment comes with advantages and disadvantages. For instance, performance-based assessment measures allow students to display higher order thinking skills but they come at a cost. They take considerably more time to grade and more investment for learners to complete, but they do provide rich details about student learning and may produce more enduring learning. Objective measures are less time-consuming but also provide less information about the depth of student learning. Moreover, instructors should pay special attention to issues of equity in assessment practices, particularly when assessing nonmajority learners. Our choices in assessment matter; select strategies that provide a deep examination of what students learn or solve problems about learning gaps.

Principle 3

Collaborate to reduce assessment burdens

Leverage existing expertise and partnerships to create and evaluate assessments of psychological content. Every institution employs experts, either assessment officers who are intentionally hired to address that need or faculty who develop expertise through experience with best practices, who can assist with the quest to improve student learning. Depending on institutional type, key personnel and other resources may be located in institutional effectiveness, institutional research, assessment, and testing, or center for teaching and learning offices. Additionally, disciplinary experts are another place to look. Disciplines such as architecture and film use critics and juried assessments, respectively; within these disciplines’ assessment methodology is embedded in their disciplinary thinking about student learning. Partnerships and collaboration in the discipline and across divisions can provide a wealth of information and support.

Principle 4

Disseminate assessment results to build assessment culture

What is learned through any assessment effort potentially can add knowledge and value at the levels of the course, department, and national interests. Instructors should take every opportunity to share their findings. Start sharing locally and then scale up. When sharing, focus on the data—What were the results? Where did students perform well, and where did learners underperform? Lead with findings, placing data at the center whether the data are qualitative or quantitative. The intent is on gathering information about student learning that can be used to make improvements and address some of the difficult questions about learning. Assessment results can shed light on questions such as:

Are obstacles to learning this content related to prerequisite material?

Was the cognitive load on this threshold concept considered when selecting the activity/assignment?

Is learning being impacted by misconceptions, inequitable practices, or competing for prior knowledge?

Are nonmajority students learning at equal rates?

Discuss what was discovered, what could be changed or how efforts could be scaled, and what questions still need to be explored. Remember, tell the learning story around the specific learning objective/SLO.

The teaching and learning process is complex and turning that process into a set of data points or a single number may oversimplify all that teachers do to engage students in their learning. Of course, fears about assessment are not unwarranted (Kramer, 2006). Many valid reasons justify shying away from sharing results. Generally, instructors do not want their numbers to be viewed as a political liability or constitute a threat to a successful teaching career. But letting those reasons crowd out the desire to share only reinforces the defensive, unproductive pockets of higher education. Faculty who share their data are taking an important step in demonstrating that student learning is important to them. Well-informed departments will value the assessment process over the results and encourage faculty to take the time to use the results to improve their pedagogical practices as needed. In addition, sharing the data with others can lead to broader discussions of resources and needed support services and can help other faculty to improve their classroom practices as well.

The context matters. Meaningful systemic learning improvements will not happen if results are not shared and discussed. Although there are many principles that can inform assessment, those mentioned above represent a starting point for psychology educators. Next, we turn attention to different roles students can play in assessment contexts.

The Psychology Learner Taxonomy

To make suggestions about exemplary assessments when discussing categories and taxonomy of different assessments, the SNAP working group noticed a few issues within the literature on assessment in teaching of psychology. First, we investigated the cognitive taxonomies of Bloom et al. (1956) and the update of Bloom’s Taxonomy by Anderson and Krathwohl (2001). However, the working group found Bloom’s approach to be too complicated. The entirety of Bloom’s taxonomy includes three domains: the cognitive domain (i.e., knowledge), the affective domain (i.e., self, attitudes), and the psychomotor domain (i.e., abilities, skills). However, when educators discuss Bloom’s taxonomy, they typically limit their discussion to the levels of the cognitive domain. With the Anderson and Krathwohl (2001) revision, the dimensions of the cognitive domain were somewhat rearranged and nouns representing the levels of complexity were changed to verbs; matrix can be formed of processes compared against levels of knowledge (Clark, 2015; Clark & Chopeta, 2004); yet the new Anderson and Krathwohl model still failed to specifically apply to psychology content. Second, we struggled with applying Bloom’s taxonomy to assess the growing trend in psychology of both knowledge and skills-based learning. Bloom’s and subsequent taxonomies did not address the need for psychology students to be assessed on their ability to not just understand and know psychological concepts, but to be able to be savvy consumers of this information, and—most importantly—to be able to use these concepts to produce and solve real-world problems. Thus, the SNAP working group created the psychology learner taxonomy in which we conceptualized the learner interaction with psychology content into three hierarchical categories (i.e., students as scholars, students as consumers, and students as producers).

Table 2 is recreated from Clark (2015), nicely combining the new cognitive dimension from the Anderson and Krathwohl (2001) with typical knowledge-based cognitive tasks (for their precise definitions, see Clark, 2015). Our model—students as scholars, students as consumers, and students as producers—simplifies this complex picture. Rather than have 30 separate functions to try and differentially assess, psychology educators can think in terms of three broad categories of assessments:

Table 2.

New and Old Cognitive Dimension.

Knowledge Dimension	Remember	Understand	Apply	Analyze	Evaluate	Create
Facts	List	Paraphrase	Classify	Outline	Rank	Categorize
Concepts	Recall	Explains	Show	Contrast	Criticize	Modify
Processes	Outline	Estimate	Produce	Diagram	Defend	Design
Procedures	Reproduce	Example	Relate	Identify	Critique	Plan
Principles	State	Converts	Solve	Differentiates	Conclude	Revise
Metacognitive	Proper use	Interpret	Discover	Infer	Predict	Actualize

Source. Clark (2015).

students as scholars—when students are merely absorbing information and performing minimal information processing tasks, making basic decisions about factual correctness;

students as consumers—when students must now understand conceptual relations between facts and begin to understand underlying complexities and assumptions underlying knowledge and its tenuous nature; and

students as producers—where students understand the information presented to the extent that they can take it and manipulate that knowledge into a new form and communicate it back to an external audience, that is, an accurate information-based product/item is produced.

To justify substituting the psychology learner taxonomy for Bloom’s taxonomy, we demonstrated the alignment of the two approaches (see Table 3 for a comparison). Next, we explore each of the levels of complexity and provide recommendations from the compilation of assessments in project assessment.

Table 3.

A Comparison Between Anderson and Krathwohl’s (2001) Cognitive Taxonomy and the Psychology Learner Taxonomy Attributes.

Knowledge Dimension	Remember	Understand	Apply	Analyze	Evaluate	Create
Facts	Scholar	Scholar	Scholar	Scholar	Consumer	Consumer
Concepts	Scholar	Scholar	Scholar	Consumer	Criticize	Consumer
Processes	Scholar	Scholar	Consumer	Consumer	Producer	Producer
Procedures	Scholar	Scholar	Consumer	Consumer	Producer	Producer
Principles	Scholar	Scholar	Consumer	Consumer	Producer	Producer
Metacognitive	Scholar	Consumer	Consumer	Consumer	Producer	Producer

Students as Scholars

Characteristics

This category of assessments is the beginning of understanding for psychology students. Students as scholars refers to a student’s ability to memorize and recall facts and basic knowledge, without much questioning of conceptual validity. Students within the area of their educational development in a subject area should be acquiring a strong factual or declarative knowledge of psychological concepts such as what would be hoped for in a well-designed introductory psychology course.

Comparison to Bloom’s taxonomy

Much like Krathwohl’s (2002) revised taxonomy, students as scholars should have knowledge of psychological terminology and specific elements to theories and should be able to recognize and recall these concepts. In relation to Bloom’s original taxonomy and Anderson and Krathwohl’s revised taxonomy, this category of assessment equates to remembering and understanding with assessments focusing on recognition and recall concepts, terms, and theories at a fundamental level (Anderson and Krathwohl, 2001). Unlike the original cognitive taxonomy, students as scholars also crosses over into apply and analyze (see Table 3). It is our belief that students as scholars requires more than just remembering and understanding facts; instead, students need to be able to apply and analyze these facts.

Project assessment examples of students as scholars

Through the work of SNAP, Project Assessment has several examples of how to assess students’ ability to perform as scholars. For example, in the assessment titled “email a friend” (APA, 2017), students email friends to use the basic psychological concepts centered on why their friends are doing so poorly on their exams (e.g., the importance of sleep, the benefits of distributed practice). In this students as scholars example, students demonstrate a basic understanding of psychological concepts (see Table 4 for additional examples of students as scholars assessments in PASS).

Table 4.

Psychology Learner Taxonomy and Project Assessment Examples.

Assessment Category	Project Assessment Examples	American Psychological Association Goals
Students as scholars	Distinguishing correlational vs. experimental (Vita, n.d.)	2.2, 2.4
	The superstitious pigeon: Summarizing, interpreting, and paraphrasing psychological literature (Dunn et al., n.d.-b)	1.1, 1.2, 1.3, 2.2, 2.4
	Find the format flaws in an American Psychological Association style paper (Taylor & Chew, n.d.)	4.1
Students as consumers	Experiential learning activity: The metacognitive interview (Richmond & Richmond, n.d.)	1.1
	Analyzing potential sociocultural challenges in scientific inquiry (Ronquillo-Adachi, n.d.)	2.5
	Finding flaws in claims: Multiple choice assessment (Levine, n.d.)	2.4
	Ethical decision making (Frantz et al., n.d.)	3.1
Students as producers	The good life (Halonen, n.d.)	3.3
	Career preparation Self-Efficacy Scale (Rudman & Tucker, n.d.)	5.1
	Child-raising simulation: Lessons learned (Bradley, n.d.)	1.3

Students as Consumers

Characteristics

Sternberg (1999) originally discussed students as “consumers” in the context of consumer psychology research. In this context, students must now understand conceptual relations between facts and begin to understand underlying complexities and assumptions underlying knowledge and its tenuous nature.

Comparison to Bloom’s taxonomy

This level of cognitive skill involves applying, analyzing, and evaluating information from the environment much like the traditional levels of the cognitive domain from Anderson and Krathwohl (2001; see Tables 2 and 3). This middle ground takes information beyond the mere consumption stage—judgments are made, relations are approximated—but the work is not taken so far as to remanipulate the information/data into a new form and demonstrate knowledge/skill/competence in that manner.

Project assessment examples of students as consumers

Project assessment has several quality examples of students as consumers (see Table 4 for additional examples). For example, Dunn et al. (n.d.-a) described an assessment titled, Applying Psychological Principles to Improving Behavioral Health that places students in the role of the “behavioral assistant” using the knowledge of psychological terms, such as framing, availability heuristic, cognitive dissonance, collectivist cultures, and operant condition to help their hypothetical patients. In this assessment, not only are students demonstrating their scholar level but they are also demonstrating their students as consumers by applying and analyzing their knowledge of these psychological concepts.

Another exemplar of students as consumers evaluates sociocultural factors in scientific inquiry (Ronquillo-Adachi, n.d.). This strategy assesses the student’s ability to recognize and apply when a researcher’s social and cultural values and biases may influence the research process by how they interpret results or how they design a study or how they may discuss the implications of a study (see Table 4 for additional examples).

Students as Producers

Characteristics

In the final category of assessment, students as producers refers to students being able to understand the information presented to them to the extent that they can take it and manipulate that knowledge into a new form and communicate it back to an external audience in such a way that an accurate information-based product/item is produced (Keegan & Bell, 2011; Kotze & Du Plessis, 2003; Lee et al., 2006, Void et al., 2016). Students as producers equates to applying and creating with assessments focusing on the generation of novel ideas, products, or ways of viewing psychological concepts, terms, and theories (Buff, n.d.).

Comparison to Bloom’s taxonomy

This level of cognitive skill involves primarily Anderson and Krathwohl’s (2001) cognitive domains of evaluating and creating and within the knowledge dimension of processes, procedures, principles, and metacognition (see Tables 2 and 3). Moving from consumption of psychological knowledge and skills to assessment and evaluation of psychological knowledge and skills, at this level of cognitive skill, students are assessed on their ability to combine the previous cognitive skills (i.e., scholars and consumers) to create processes or principles or to evaluate procedures.

Project assessment examples of students as producers

Timeline of My Life (Julian, n.d.) requires students to create a scrapbook that chronicles significant events in their lives that demonstrate their knowledge, application, synthesis, and evaluation of different development theories and periods. Students act as producers in this assessment by creating a product that visually shows integration of multiple theories of development across the life span.

In another creative assessment, Jones (n.d.) has students create and produce a Restaurant Menu for Zombies. In this assessment, Jones has students create a menu of seven different items that demonstrates their understanding of the structure of the brain, including definition and function. In this assessment, students are producers because they are not only understanding the structure and function of the brain they are also creating a product to demonstrate their understanding (see Table 4 for additional examples of students as producers assessments in PASS).

We hope adoption of the psychology learner taxonomy will make identifying and designing appropriate assessments easier to accomplish. Ultimately, we believe that the field of psychology would be better served if we created a culture of learning that focused on students as scholars, consumers, and producers. Next, we address a systematic process for determining whether any given assessment is useful or appropriate.

Psychology Assessment Evaluation Rubric

From the myriad psychology assessments available to instructors of psychology, how do we discern which assessments will fit our needs and be effective? How do we know when and where to use these assessments? Do these assessments map on to APA Guidelines 2.0 to ensure that we are incompliance with our department’s commitment to this process? To answer these questions, our group envisioned a rubric that was dynamic and applicable to many different areas (see Appendix Table A1 for complete description of the rubric).

The psychology assessment evaluation rubric focused primarily on content, but we also wanted an instrument that would align to other outcomes in the Guidelines 2.0 (Bernstein et al., 2010; Fulks, 2004; Landrum et al., 2010; Richmond et al., 2016). The working group wanted the rubric to reflect universal designs for learning (CAST, 2018); in that, it had to allow for the evaluation of various types of student engagement, mode of response, and student abilities. Based on the works of Fulks (2004), Hess (2018), and CAST (2018), the group developed a rubric that could be used to evaluate assessments to designate as “exemplars” for all. The rubric provides objective criteria for evaluating assessments on several different variables and may be particularly useful in making choices when evaluating the merits among multiple assessments.

The psychology assessment evaluation rubric includes the following elements:

assessment profile (e.g., method, level, formative vs. summative, cost, time for implementation, cognitive taxonomy);

supporting information (e.g., teacher directions, student directions);

delivery method (e.g., whole group, individual, electronic);

mapping to APA Guidelines (i.e., Fully, Partially, N/A);

evaluation of the scoring guide to be used with the assessment;

adherence to universal design for learning; and

recommendations and overall appraisal of the strengths and weaknesses of the assessment.

We envision both instructors of psychology and administrators to use this rubric to create, curate, and review assessments that will be both valid and valuable to the outcomes they are trying to measure.

On the Need for a National Assessment Exam

In this final section, we take on the elephant in the room—should we advocate for a national assessment in psychology to address the assessment of content? Many educators believe that a national assessment, perhaps sponsored by APA or some other psychology-focused enterprise, would make an enormous contribution due to the benefits that would accrue, including ease of administration, standardization of essential content, and facilitation of a database that would allow interinstitutional comparisons, among other justifications.

Three commercially available assessments already exist. The Graduate Record Exam subject test in psychology, the Educational Testing Service major field test in psychology, and the psychology area concentration test. They vary in terms of intended purpose, sophistication, and cost, but many departments have adopted one of these measures to keep an eye on trends over time in what content students take away from their courses measured just prior to graduation. Programs that are interested in a comparison with peer institutions may do well to investigate any one of the previously mentioned commercial assessment products.

At great length, the working group discussed the possibility of constructing or endorsing a standardized comprehensive national exam for the psychology major but ultimately declined to do so for multiple reasons. The variation of resources available at different institutions, the myriad delivery methods of courses and distribution of students, and the expense involved in development and implementation of such an assessment with many institutions having barriers to participation made this approach untenable. Rejecting this option allowed the group to focus on smaller assessments that could be implemented over a wider range of settings and institutions. Our attention was on the different areas of Goal 1 and the creation of APAs Project Assessment (a digital repository of peer-reviewed assessments organized by content area) from which instructors of psychology could download to assess their content—available at pass.apa.org after account and password creation. We think this is a more reasonable response to generating assessment data than advocating for a national, potentially expensive fix that would be hard to maintain to address assessment mandates.

Conclusion

In the end, assessment is never complete. It is a constantly evolving and continual process. As such, we discussed that the difficulties surrounding the creation of a national assessment, we have outlined guiding principles for assessments in psychology, provided a rubric that can be used to assess assessments and created a way in which we can reconceptualize the goals and objectives how and what we want students to learn. Although our suggestions for assessment of content in psychology are by no means an “assessment panacea,” we hope that they will further the discussion and subsequent action of assessment practices in psychology.

Footnotes

Appendix

Table A1.

Psychology Assessment Evaluation Rubric.

Assessment Profile
Method (I = indirect or D = direct)
Level of assessment (SaS = students as scholars, SaC = students as consumers, SaP = students as producers)
Cognitive taxonomy (R = remembering, U = understanding, A = applying, AN = analyzing, E = evaluating, C = creating)
Usage (F = formative or S = summative)
Constructed response (essay, multistep response with explanation and/or rationale required for tasks)
Product (research paper, editorial, log, journal, play, poem, model, multimedia, art products, script, musical score, portfolio pieces, etc.)
Performance (demonstration, presentation, science lab, dance or music, athletics, debate, etc.)
Short answer (short constructed response, fill in a graphic organizer or diagram, explain your thinking or solution, make and complete a table, etc.)
Selected response (multiple choice, multiple select, evidence-based selected response, true–false, matching, etc.)
Cost of instrument
Estimated time for implementation
Supporting Information on Assessment			Yes/No
Teacher directions (may include prerequisites/description of instruction before giving the assessment; e.g., this assessment should be given after students have learned…)
Scoring guide/rubric
Sample evidence for student performance
Student materials
Student directions
Assessment task/prompts
Delivery Method of Assessment			Yes/No
Whole group
Small group
Individual
Paper and pencil
Electronic
Universal design for learning dimensions of assessment			Yes/No
Provide multiple means of accessing the assessment (e.g., allow students to access information in ways that do not require them to read standard print)
Provide multiple means of responding to the assessment (e.g., allow students to complete activities, assignments, and assessments in different ways or solve/organize problems using some type of assistive device or organizer)
APA Guidelines 2.0 Evaluation	Fully	Partially			N/A
1. Knowledge base in psychology
2. Scientific inquiry and critical thinking
3. Ethical and social responsibility in a diverse world
4. Communication
5. Professional development
Describe the content knowledge/concepts assessed:
Describe the skills/performance assessed:
Scoring Guide to be Used With the Assessment			Yes/No
Generalized rubric (e.g., for writing an argument, for all science labs)
Task-specific rubric (only used for the particular task)
Scoring guidelines (e.g., checklist with score points for each part)
Answer key, scoring template, computerized, or machine scored
Anchor papers (student samples at each score point)
Are the score categories clearly defined and coherent across performance levels?If no, please explain.
Do the rubric/scoring criteria address all of the demands within the task or item?If no, please explain.
Are directions for the items or tasks presented in as straightforward a manner as possible for a range of learners? If no, identify problematic items/tasks and provide suggestions for improvement.
Is the vocabulary and context(s) presented free from cultural or other unintended bias? If no, identify problematic items/tasks and provide suggestions for improvement.
Recommendations for this Assessment
What are the advantages and disadvantages of this assessment?What are the strengths and weaknesses of this assessment?
Assessment Decision				Yes/No
This assessment can be used without revisions (explain):
This assessment can be used with minor revisions (explain):
This assessment can be used with significant revisions (explain):
This assessment should not be used (explain):

Source. The form was combined/modified from Fulks (2004), Hess (2018), CAST (2018), and Rhode Island Department of Education (n.d.).

Authors’ Note

The authors are grateful for the American Psychological Association’s (APAs) Education Directorate and Committee on Associate and Baccalaureate Education (CABE) sponsorship of Summit on National Assessment of Psychology.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research and/or authorship of this article: Additional resources were provided by grants from the National Science Foundation (Division of Undergraduate Education, award number 1622982), the University of Wisconsin—Green Bay, APAs Board of Educational Affairs, Society for the Teaching of Psychology, Psi Chi, and Psi Beta.

References

American Psychological Association. (2007). APA guidelines for the undergraduate major. Author.

American Psychological Association. (2013). APA guidelines for the undergraduate psychology major: Version 2.0. http://www.apa.org/ed/precollege/undergrad/index.aspx

American Psychological Association. (2017). Email a friend. American Psychological Association, Project Assessment. http://pass.apa.org/docmail-to-a-friend/http://pass.apa.org/docs/email-to-a-friend

Anderson

L. W.

Krathwohl

D. R.

(Eds.) (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Longman.

Bernstein

D. J.

Addison

Altman

Hollister

Komarraju

Prieto

Rocheleau

C. A.

Shore

(2010). Toward a scientist–educator model of teaching psychology. In Halpern

D. F.

(Ed.), Undergraduate education in psychology: A blueprint for the future of the discipline (pp. 29–45). American Psychological Association.

Bloom

B. S. (Ed.).

Engelhart

M. D.

Furst

E. J.

Hill

W. H.

Krathwohl

D. R.

(1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. David McKay.

Bradley

(n.d.). Child-raising simulation: Lessons learned. American Psychological Association, Project Assessment. http://pass.apa.org/docs/child-raising-simulation-lessons-learned

Buff

(n.d.). Students as producers: An introduction. Center for Teaching and Learning. Vanderbilt University. https://cft.vanderbilt.edu/2013/09/students-as-producers-an-introduction/https://cft.vanderbilt.edu/2013/09/students-as-producers-an-introduction

CAST. (2018). Universal Design for Learning Guidelines version 2.2. http://udlguidelines.cast.org

10.

Chew

S. L.

Halonen

J. S.

McCarthy

M. A.

Gurung

R. A. R.

Beers

M. J.

McEntarffer

Landrum

R. E.

(2018). Practice what we teach: Improving teaching and learning in psychology. Teaching of Psychology, 45, 239–245. https://doi.org/10.1177/0098628318779264

11.

Chudowsky

Glaser

Pellegrino

J. W.

(2004). Knowing what students know: The science and design of educational assessment. National Academies Press.

12.

Clark

(2015). Bloom’s taxonomy of learning domains. http://www.nwlink.com/∼donclark/hrd/bloom.html

13.

Clark

Chopeta

(2004). Graphics for learning: Proven guidelines for planning, designing, and evaluating visuals in training materials. Jossey Bass/Pfeiffer.

14.

Diamond

R. M.

(2009). Designing and assessing courses and curricula: A practical guide. Wiley.

15.

Dunn

D. S.

Feldman

Franks

Gonzalez

Gurung

R. A. R.

Halonen

Julien

McEtarffer

Vita

(n.d.-a). Applying psychological principles to improve behavioral health. American Psychological Association, Project Assessment. http://pass.apa.org/docs/applying-psychological-principles-to-improve-behavioral-health

16.

Dunn

D. S.

Feldman

Franks

Gonzalez

Gurung

R. A. R.

Halonen

Julien

McEntarffer

Vita

(n.d.-b). The superstitious pigeon: Summarizing, interpreting, and paraphrasing psychological literature. American Psychological Association, Project Assessment. http://pass.apa.org/docs/the-superstitious-pigeon-summarizing-interpreting-and-paraphrasing-psychological-literature

17.

English

F. W.

(1980). Curriculum mapping. Educational Leadership, 37(7), 558–559.

18.

Fink

L. D.

(2003). Creating significant learning experiences: An integrated approach to designing college courses. Jossey-Bass.

19.

Frantz

Halonen

Hoss

McCarthy

Nolan

Pusateri

Wickes

(n.d.). Ethical decision making. American Psychological Association, Project Assessment. http://pass.apa.org/docs/ethical-decision-making

20.

Fulks

(2004). Assessing student learning in community colleges. http://www2.bakersfieldcollege.edu/courseassessment/Default.htmhttp://www2.bakersfieldcollege.edu/courseassessment/Default.htm

21.

Gaston

(2018, April). Assessment and accreditation: An imperiled symbiosis [Occasional Paper No. 33]. University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).

22.

Gronlund

N. E.

Gronlund

N. E.

(2004). Writing instructional objectives for teaching and assessment. Pearson/Merrill/Prentice Hall.

23.

Halonen

(n.d.). The good life. American Psychological Association, Project Assessment. http://pass.apa.org/docs/the-good-life/

24.

Hess

(2018). A local assessment toolkit to promote deeper learning: Transforming research into practice. Corwin.

25.

Hutchings

(April, 2010). Opening doors to faculty involvement in assessment. National Institute for Learning Outcome Assessment. http://www.learningoutcomeassessment.org/

26.

Jankowski

N. A.

Marshall

D. W.

(2017). Degrees that matter moving higher education to a learning systems paradigm. Stylus.

27.

Jones

(n.d.). Restaurant menu for zombies. American Psychological Association, Project Assessment. http://pass.apa.org/docs/restaurant-menu-for-zombies

28.

Julian

(n.d.). Timeline of my life. American Psychological Association, Project Assessment. http://pass.apa.org/docs/timeline-of-my-life/

29.

Keegan

Bell

(2011). YouTube as a repository: The creative practice of students as producers of Open Educational Resources. European Journal of Open and Distance E-Learning, 1–12. http://usir.salford.ac.uk/id/eprint/19282/2/Keegan_Bell_Creativity_and_OER.docx.pdf

30.

Kotze

T. G.

Du Plessis

P. J.

(2003). Students as “co-producers” of education: A proposed model of student socialisation and participation at tertiary institutions. Quality Assurance in Education, 11(4), 186–201.

31.

Kramer

P. I.

(2006). Assessment and the fear of punishment: How the protection of anonymity positively influenced the design and outcomes of postsecondary assessment. Assessment & Evaluation in Higher Education, 31(5), 597–609.

32.

Krathwohl

D. R.

(2002). A revision of Bloom’s taxonomy: An overview. Theory Into Practice, 41(4), 212–218.

33.

Landrum

R. E.

Beins

B. C.

Bhalla

Brakke

Briihl

D. S.

Curl-Langager

R. M.

Pusateri

T. P.

Van Kirk

J. J.

(2010). Desired outcomes of an undergraduate education in psychology from departmental, student, and societal perspectives. In Halpern

D. F.

(Ed.), Undergraduate education in psychology: A blueprint for the future of the discipline (pp. 145–160). American Psychological Association.

34.

Lee

M. J.

Chan

McLoughlin

(2006, July). Students as producers: Second year students’ experiences as podcasters of content for first year undergraduates. In 7th International Conference on Information Technology Based Higher Education and Training (pp. 832–841). IEEE. https://ieeexplore.ieee.org/document/4141716

35.

Levine

G. M.

(n.d.). Finding flaws in claims: Multiple choice. American Psychological Association, Project Assessment. http://pass.apa.org/docs/finding-flaws-in-claims-multiple-choice-assessment

36.

McTighe

Wiggins

(2012). Understanding by design framework. Association for Supervision and Curriculum Development.

37.

Nilson

L. B.

(2016). Teaching at its best: A research-based resource for college instructors. Jossey-Bass.

38.

Norcross

J. C.

Hailstorks

Aiken

L. S.

Pfund

R. A.

Stamm

K. E.

Christidis

(2016). Undergraduate study in psychology: Curriculum and assessment. American Psychologist, 71(2), 89–101. http://doi.org/10.1037/a0040095

39.

Pusateri

Halonen

J. S.

Hill

Mccarthy

(2009). The assessment cyberguide for learning goals and outcomes. https://teachpsych.org/page-1670227

40.

Rhode Island Department of Education. (n.d.). Assessment review tool: A part of the assessment toolkit. Rhode Island Department of Education and the National Center for the Improvement of Educational Assessment Inc. https://www.ride.ri.gov/Portals/0/Uploads/Documents/Teachers-and-Administrators-Excellent-Educators/Educator-Evaluation/Online-Modules/Assessment-Review-Tool.pdf

41.

Richmond

A. A.

Richmond

A. S.

(n.d.). Experiential learning activity: The metacognitive interview. American Psychological Association, Project Assessment. http://pass.apa.org/docs/experiential-learning-activity-the-motivation-metacognitive-interview/

42.

Richmond

A. S.

Boysen

G. A.

Gurung

R. A. R.

(2016). An evidence-based guide to college and university teaching: Developing the model teacher. Routledge.

43.

Ronquillo-Adachi

(n.d.). Analyzing potential sociocultural challenges in scientific inquiry. American Psychological Association, Project Assessment. http://pass.apa.org/docs/analyzing-potential-sociocultural-challenges-in-scientific-inquiry

44.

Rudman

Tucker

(n.d.). Career Preparation Self-Efficacy Scale. American Psychological Association, Project Assessment. http://pass.apa.org/docs/career-preparation-self-efficacy-scale

45.

Sternberg

R. J.

(1999). Teaching psychology students to be savvy consumers and producers of research questions. Teaching of Psychology, 26, 211–213.

46.

Suskie

L. A.

(2018). Assessing student learning: A common sense guide. Jossey-Bass.

47.

Taylor

Chew

(n.d.). Find the format flaws in an APA style paper. American Psychological Association, Project Assessment. http://pass.apa.org/docs/find-the-format-flaws-in-an-apa-style-paper

48.

Vita

(n.d.). Distinguishing correlational vs. experimental research. American Psychological Association, Project Assessment. http://pass.apa.org/docs/distinguishing-correlational-vs-experimental-research

49.

Void

Braun

Lundesgaard

(2016, September). Flipped classroom—Students as producers. In 15th International Conference on Information Technology Based Higher Education and Training (ITHET) (pp. 1–6). IEEE. https://ieeexplore.ieee.org/document/7760747

50.

Wang

Cheung

Wong

Kwong

(2013). An exploration of Biggs’ constructive alignment in course design and its impact on students’ learning approaches. Assessment & Evaluation in Higher Education, 38(4), 477–491. https://doi.org/10.1080/02602938.2012.658018

51.

Wiggins

McTighe

(2005). Understanding by design (2nd ed.). Association for Supervision and Curriculum Development.