Research Protocol: A Transdisciplinary Multi-Case Study Research Design Using Mixed Methods to Evaluate the Long-Term Impact of Holocaust Museum Education in Australia

Abstract

Holocaust museums around the world are distinctive in their emphasis on educational missions premised on the idea of “never again”, a deep belief that increasing public awareness of the history and contemporary significance of the Holocaust – the mass murder of six million European Jews by the Nazi regime during World War Two – can prevent the future recurrence of such events. In the Australian context, tens of thousands of school students visit Holocaust museums every year to participate in learning programs designed to impart powerful historical and moral lessons about the Holocaust, its antecedent conditions, and its ongoing relevance. The aspirations attached to these programs and the scale at which they are delivered stand in stark contrast to the lack of empirical evidence to support their efficacy. This research protocol describes an extensive four-year research study designed to evaluate the presumed linear relationship between Holocaust education programs and ethical, moral, and affective learning that leads to durable attitudinal and behavioural change. Funded through the Australian Research Council’s Linkage Projects scheme, the study will assess the impact of Holocaust museums’ education programs on high school students from diverse demographics, including the extent to which participation influences the views and actions of young people in relation to racial and identity-based discrimination. The research design is longitudinal and transdisciplinary, integrating qualitative, quantitative, and participatory techniques across three case studies. By strategically employing rapid ethnographic assessment procedures, archival research, analysis of museum representational practices, surveys, and visual research methods, we will examine both the educational intent behind Holocaust education programs and their actual impacts on a large sample of Australian Year 10 students. We expect the results of the research to be used by our project partners to develop Holocaust education programs that effectively engage students on cognitive and affective levels.

Keywords

holocaust genocide studies museum education transformative learning ethics longitudinal research rapid ethnographic assessment visual research methods surveys interviews

Background and Study Justification

Witnessing the ever-increasing number of students attending the Sydney Jewish Museum’s (SJM) Holocaust education programs, a long-time volunteer guide was prompted to ask, “But how do we know these programs are working? Are they really changing attitudes and actions?” Her observations capture the fundamental question underpinning our research project, titled Learning from the Past? Evaluating the Impact of Holocaust Museum Education in Australia (LFTP): does learning about difficult pasts contribute to the development of ethical and/or moral behaviour in individuals and broader contemporary society?

Holocaust education programs (HEPs) in museum settings incorporate not only learning about historical events, but also an explicit injunction to act on that learning. Support for Holocaust museums and Holocaust education initiatives continues to increase in Australia and overseas. In February 2019, with the backing of the Department of Foreign Affairs and Trade (DFAT), Australia became a full member of the International Holocaust Remembrance Alliance (IHRA). Membership denotes official acknowledgement of the salience of Holocaust education to key civic values in Australian culture, committing the government to implementing Holocaust education across Australia’s secondary education sector. Subsequently, at the 2021 Malmo International Forum on Holocaust Remembrance and Combatting Antisemitism, the Australian Federal government pledged to work toward the establishment of Holocaust museums or education centres in each jurisdiction and recognised the need for continuing action to combat racism in contemporary society.

The growing support that institutions like the SJM receive for their educational efforts remains predicated upon the assumption that visitors will act upon the moral lessons that learning about genocide are believed to impart. Holocaust museums internationally and in Australia have placed increasing emphasis on educating school students (Alba, 2020), devoting significant spatial, financial and staff resources to student focused HEPs. One expectation of both private philanthropic and public funding bodies is that these institutions provide detailed historical content while also conveying lessons on civics and citizenship, highlighting contemporary instances of racism and injustice, and advocating for human rights (Gross & Doyle Stevick, 2015; Eckmann et al., 2017). Based on the affective potential of engaging with survivor testimony (Simon, 2005; Witcomb, 2013), the impact of HEPs on both student and general visitors’ beliefs and behaviours is often taken for granted, but rarely scrutinised or rigorously evaluated.

Although academics and practitioners have argued that museums “may have been designed to inculcate social values” (Sandell, 2007, p. 10 – emphasis in original), there is a lack of evidence as to how this happens and whether museums are, again in Sandell’s terms, effective “social agents”, or capable of prompting individuals to act differently when confronted by injustice in the course of their everyday lives (Levinson, 1997). While the literature on visitor studies has a long history (see inter alia Falk & Dierking, 2000, 2011; Kelly, 2010), the evidence remains elusive, especially if new understandings stressing the influence of performative and affective elements of the relationship between visitors and the museum are considered (Cooke & Frieze, 2016; Witcomb, 2013). Further, the process through which engagement with museums may facilitate long-term attitudinal change on the part of the visitor is a matter of intense professional and academic debate (Cooke & Frieze, 2016).

While studies exploring the impact of HEPs are beginning to emerge internationally (EU Agency for Fundamental Rights, 2010; UNESCO, 2017), they remain inconclusive. Australia’s first comprehensive national survey of Holocaust knowledge and awareness (Cooke et al., 2022) found that higher levels of knowledge and awareness are evident in student cohorts who have participated in HEPs delivered in schools and Holocaust museums. However, the same study confirmed that, without research into specific pedagogical approaches, the educational impact and ability for these institutions to catalyse effective long-term learning will remain limited (ibid., p. 45). With the rise of antisemitism and other forms of racism in Australia and abroad prompting policy initiatives in which cultural institutions are positioned as agents of social change (Carbone, 2019), there is new interest in understanding and maximising the long-term educational benefits of these programs.

Research Aims

Focusing on high school students in Year 10 who visit Holocaust museums in connection with school curriculum covering topics related to the Holocaust, our research will investigate and assess the long-term educational impact of HEPs through our core research question: Can and do Holocaust museums create effective conditions for ethical, moral, and affective learning, leading to long-term attitudinal and behavioural change?

Developed in partnership with three of Australia’s Holocaust museums – the Sydney Jewish Museum (SJM), the Melbourne Holocaust Museum (MHM), and the Adelaide Holocaust Museum and Steiner Education Centre (AHMSEC) – our innovative methodology will enable a comprehensive understanding and evaluation of how young people respond to HEPs, including the potential of these programs to raise historical awareness of the Holocaust, shape moral reasoning, and encourage ethical behaviour. The outcomes of the project will have implications for the development of more targeted and effective programs that engage school students on cognitive and affective levels.

In collaboration with our museum partners, scholars from the University of Sydney, Deakin University, and the University of Technology Sydney (UTS) will:

1. Examine and assess the presumed linear relationship between experiencing Holocaust museums as educational environments (comprising exhibition spaces, educational programs, and personal, audio-visual, written, and material forms of testimony) and understanding the history of the Holocaust and its contemporary significance.

2. Analyse and evaluate the impact of HEPS on students from diverse backgrounds and educational contexts, including the extent to which participation in HEPs empowers young people take informed action against racial and identity-based discrimination.

These aims are further broken down into the following four objectives, which directly guide the different phases of the research:

i. To document and assess the intended student learning outcomes of Holocaust education programs (how do Holocaust museums blend historical studies and values education to position themselves as sites of memory and/or agents of change?).

ii. To evaluate the Holocaust knowledge, attitudes, and values of students (and their teachers) before participating in the Holocaust education program.

iii. To examine the emergent experiences of students during the Holocaust education program, including affective and cognitive dimensions.

iv. To assess the influence of Holocaust education programs on students’ thinking about the Holocaust and its contemporary relevance (including developments in attitudes and behaviours) at six weeks and six months post-participation.

Research Context and Pilot Study

The impetus for LFTP came from the partner organisations and their interest in understanding their local impact, acknowledging the paucity of evidence internationally to corroborate the claim that HEPs can meaningfully intervene against the emergence of identity-based victimisation. Collectively, the three museums host over fifty thousand school students annually. Through their material spaces and related engagement strategies, these sites have been consciously configured as places “within which the lessons of the past would be taught” (SJM, 2019). All three organisations see themselves as bulwarks against the dissipation of national, communal, and individual awareness of the impact of the Holocaust and human rights abuses. Yet, none have comprehensive evidence to enable them to evaluate the extent to which they are succeeding in these objectives or, indeed, if HEPs sometimes produce unintended negative responses in students.

As a subset of the broader research approach, the project utilises and further develops Visual Research Methodologies (VRM) as a novel approach to capturing the educational and affective experience of HEPs. VRM was successfully piloted by Cooke and Frieze at the MHM in 2016. Unlike traditional participant observations, in which students are often filmed by a third-party observer, VRM allows for the subjective experience of the participant to be recorded and then contextualised within the overall visit and program. The VRM to be used in this study is based on hands-free audio-visual recording glasses (“smart glasses”) worn by participants. These glasses are less visible and less intrusive than other recording devices, capturing what the student sees and hears during their visit. A full explanation of our utilisation of this technique can be found in the following sections of this protocol.

Research Approach and Methods

Theoretical and Epistemological Framing

Transdisciplinary Design

LFTP employs a transdisciplinary research design that is theoretically informed by intersecting concepts from Holocaust and genocide studies, ethics, historiography, museology, and education. Transdisciplinary research is distinctive in its deliberate integration of scholarly methods with practice-based insights (Bergmann et al., 2012). This linkage is achieved by combining scholarly and non-academic (practitioner or stakeholder) expertise on research teams and by employing participatory research processes – including co-design of research questions, methodology, and joint interpretation of research findings – so that diverse perspectives can recursively inform, enrich, or identify the need for course-correction in the research program. By adopting this orientation, our project seeks to achieve holistic understanding of the HEP experience as a complex, multidimensional, real-world phenomenon and enable better decision-making about the future design of these educational programs.

Our integrative framework moves beyond existing approaches to understanding sense-making processes that occur in museums, such as comparing the intent of museums and students’ learning expectations (Trofanenko & Segall, 2012), dissecting museums’ representational practices (Jaeger, 2020), or evaluating how participants articulate their immediate response to museum programs dealing with difficult pasts (Smith, 2016). The project’s methodology will enable us to examine long held assumptions, particularly regarding the educational efficacy of affective learning that hinges on the ability of students to make intellectual and empathic connections between past and present without the diminishment or distortion of either.

Pragmatic Constructionist Epistemological Orientation

To accommodate findings from both qualitative and quantitative techniques, the study adopts the philosophical orientation of pragmatic constructionism – also termed critical pluralism (Repko et al., 2017) – whereby epistemological incongruence between research methods is de-emphasised in favour of gaining holistic knowledge of a research subject and insights that have practical value in real-world settings (Boix Mansilla, 2017).

Pragmatic constructionism does not demand the reconciliation of competing epistemological claims about criteria for legitimate knowledge acquisition. Instead, this orientation foregrounds the need to continuously adapt understandings of real-world phenomena – such as the transformative learning that may result from the interaction between school curricula, culturally-informed notions of ethical and/or moral citizenship, museum education programs, and the individual context of each participant – by acknowledging the complexity of such phenomena and the inescapably provisional status of related knowledge. Being “pragmatic” also signals an emphasis on the applied purpose of inquiry, where accumulating understanding of a phenomenon – acquired from plural sources of evidence and both scholarly and stakeholder expertise – is held in “reflective equilibrium” and remains open to revision (Boix Mansilla, 2017, p. 267). Therefore, pragmatic constructionism creates scope in our study to weave together strands of interpretivism (investigating the HEP experience as the interplay of multiple learning goals and interpretations), phenomenological inquiry (attentiveness to the embodied and emotional dimensions of the HEP as significant to student sense-making), and post-positivist research (using a scalable survey instrument to measure attitudinal changes over time).

Mixed Methods Case Study Approach

Intentionally conceived as an applied evaluative research study (Merriam, 2009, pp. 3-4), LFTP will systematically and empirically investigate the efficacy and impact of museum HEPs to inform the ongoing redesign of these programs at our project partner museums in Australia. By leveraging strong international networks of Holocaust scholars and museum professionals, we anticipate that the research findings will resonate globally, influencing museum-based Holocaust education and, potentially, the design of school curricula to which the visitation of Holocaust museums is typically linked.

LFTP adopts a mixed methods approach, where the advantages of both qualitative and quantitative techniques are strategically combined to address a real-world research problem, with the key feature of the research design being the triangulation of information gathered from diverse sources (Teddlie & Tashakkori, 2003, p. 11). Using mixed methods reduces the limitations of relying exclusively on either qualitative research (which has disadvantages such as small sample size, restricted generalisability, researcher bias, etc.) or quantitative techniques, with their limited explanatory power (Cresswell et al., 2003; Merriam, 2009). In our project, the complementary use of qualitative and quantitative techniques will allow us to not only interrogate the hypothesis that HEPs induce attitudinal and behavioural change in participants, but also to explore how Holocaust museums and their education programs may (or may not) produce these outcomes (Teddlie & Tashakkori, 2003; Yin, 2014). A mixed methods approach to data collection and analysis is also consistent with the project’s transdisciplinary orientation and epistemological framing in pragmatic constructionism.

As noted by Merriam (2009) and Yin (2014), case study design is well-suited to research seeking a holistic, deep understanding of a real-world phenomenon, site, or community that qualifies as a bounded system (a self-contained entity) and where “the researcher aims to uncover the interaction of significant factors characteristic of the phenomenon” (Merriam, 2009, pp. 42-43). As each of our case studies – the SJM, MHM, and AHMSEC – is representative of Holocaust museums as a class of cultural institution, we will be able to triangulate the results from our mixed methods research within and across these cases to distinguish generalisable relationships between the structure of HEPs and the impacts they generate for participants (Yin, 2014).

Consistent with case study research process (Yin, 2014, pp. 3-25; Baskarada, 2014; Sangaramoorthy & Kroeger, 2020), iterative progress reviews and sense-making with project partners and stakeholders will assist in verifying the accuracy of the case study and validating the analysis of findings through the introduction of diverse perspectives. An annual multi-day workshop will bring together all project partners to jointly review emerging findings and fine-tune the methodology where needed. Additionally, biannual reviews will be held with stakeholders at each individual case study museum. This dimension of the research design aligns with the project’s transdisciplinary orientation and epistemological framing in pragmatic constructionism by deliberately building co-design with stakeholders into the process of research planning, execution, and analysis of findings.

In the following sections of the protocol, we outline the progression of the research across three main phases, each aligned to a specific research objective linked to the central research question. We describe and justify our selection of individual methods for data collection and analysis, as well as relevant ethical considerations and data management strategies. The indicated time horizon for the research phases is contingent on, among other things, the timing of formal research ethics approvals for the project.

Primary Research Phase 1a: Understanding the Educational Environment of the Holocaust Museum (February – December 2024)

Objective: To document and assess the intended student learning outcomes of Holocaust education programs (how do Holocaust museums blend historical studies and values education to position themselves as sites of memory and/or agents of change?).

First, the research will delineate the existing educational goals and strategies designed to position the case study institutions as agents of social change. While falling into the broad category of “Holocaust museum”, each case study is characterised by unique variations, based on its history, location, professional and stakeholder community, organisational size and structure, and other formative factors. As such, it is vital to understand each museum as a system of factors that combine to produce a specific learning environment in which school students experience Holocaust education. Establishing this holistic understanding of each case requires the triangulation of multiple sources of evidence.

Rapid Ethnographic Assessment Procedures

Phase 1a seeks to understand how the educational objectives of each institution are coordinated and articulated through social practices, spaces, artefacts, and both official and unofficial texts (Tummons, 2017, pp. 148-149). Accordingly, we will use Rapid Ethnographic Assessment Procedures (REAP) to document how a range of informants, such as museum volunteers and guides, curators and collection managers, and museum educators, conceptualise the educational role of the museum and contribute to its fulfilment. REAP incorporates diverse methods for gathering and analysing data, such as direct observations of behaviour, semi-structured interviews, focus groups, and participatory mapping (Bernard, 2017; Sangaramoorthy & Kroeger, 2020).

Interviews

We will conduct semi-structured key informant interviews with each museum’s CEO and Board members, curators, public program and education staff, as well as volunteers (including Holocaust survivors working as guides). The aim of this maximum variation sampling approach within each museum’s internal stakeholder community (Seidman, 2006) is to explore the ways in which the museum’s educational mission, and the expected educational objectives and impacts of HEPs, are conceptualised, shaped, and communicated within the institution by different stakeholders.

We will conduct a small number of pilot interviews to test the interview design, including the logistics of accessing and communicating with museum staff (Seidman, 2006, p. 39). The pilot interviews will also help us assess the effectiveness of the semi-structured interview questions.

Participatory Mapping

Participatory mapping via walk-throughs of the museum space with floor staff will assist us to map sites they consider to be important to the museum’s educational mission. Ethnographic mapping is a common research technique that enables researchers to record participants’ associations with place using maps, drawings, and photographs. As described by Bernard (2017) and Sangaramoorthy and Kroeger (2020), participatory mapping uses participants’ insider knowledge of places, and activities that occur within them, to uncover patterns of behaviour and locate sites of social or cultural significance. In our project, participatory mapping via walk-throughs of the museum space will enable participants to communicate the areas and objects that they consider to be educationally impactful on both cognitive and affective levels. We will seek participation from floor staff and guides who are at the frontline of visitor engagement and who conduct the educational tours.

The researchers will write field notes during interviews and participatory mapping to supplement audio recordings with details of the physical setting, key phrases, mood, and the researcher’s emergent reflections and memos (Sangaramoorthy & Kroeger, 2020) for the purposes of subsequent data analysis (Saldana, 2009).

Ethical Considerations for Phase 1a

Representatives from our partner museums have indicated that their staff and volunteers are accustomed to people observing how they interact with visitors (including what they say and how they use the museum space), so the presence of researchers is unlikely to cause discomfort. However, to build trust and reduce reactivity – the tendency for participants to change their behaviour when they know that they are being observed (Bernard, 2017) – researchers will spend time informally in the museums and get to know each community of museum workers and stakeholders before commencing any formal observations or interviews.

Despite the existing institutional support provided for the project, it is important that participants understand that their involvement is completely voluntary. Our Participant Information Statement and Consent Form makes clear that there is no obligation to participate in the study, that participants are free to withdraw at any time, and that the decision to participate in the research will not influence participants’ relationship with the partner museums, or the researchers (and their universities) in any way. As a default, all participant data will be deidentified. We have developed a qualitative data de-identification protocol based on data management guidelines described by Myers et al. (2020), which outlines a process for establishing and maintaining the anonymity of participants. This includes de-identification of data files (removal of all direct and indirect identifiers) both within the file and in its file name prior to storage and analysis. Only the named research team (chief investigators) will have access to the identifiable files.

Details of the interview and participatory mapping guides can be found in Online Appendixes 1 and 2.

Complementary Sources: Secondary Data and Analysis of Museum Representational Strategies

Consistent with the case study strategy of converging evidence to build the robustness of findings (Johnson & Turner, 2003; Yin, 2014), the REAP will be combined with information retrieved from official museum documents and archival records (e.g., official policies, strategic plans, online content, publications, reports, etc.). These investigations will also include a comprehensive review of the museums’ educational programs and materials. We will seek to obtain broad and comparable coverage of documentation across the three case studies, using internet searches, the museums’ own libraries and archives, and media coverage relating to the institutions’ education goals and HEPs.

We will also conduct a detailed documentation of each museum’s representational strategies (Jaeger, 2020), including analysis of significant spaces and physical artefacts associated with HEPs, exhibition and public space design, narrative construction using curation of artefacts, survivor testimonies, and other communication techniques. Referring to war museums (including Holocaust museums), Jaeger (2020, p. 9) notes that these museums function as “a composite multisensory medium” where visitors “can either learn about the past, develop their own war memories, or be steered toward preconceived narratives that comprise master narratives and cultural memory politics”. Accordingly, when considering the museum as an educational environment, it is vital that our research incorporates analysis of the ways in which interpretive texts, objects, imagery, audio-visual and digital content, spatial design, and other elements of museum communication are orchestrated to communicate historical narratives and encourage specific kinds of learning.

The combination of data collected through the REAP, analysis of secondary materials, and assessment of each museum’s representational practices will result in a rich understanding of the desired outcomes of HEPs, forming a baseline against which actual learning outcomes can be evaluated.

Primary Research Phase 1b: Understanding the Pre-HEP Knowledge and Attitudes of Students and their Teachers (August 2024 – July 2025)

Objective: To evaluate the Holocaust knowledge, attitudes, and values of students (and their teachers) before participating in the Holocaust education program.

Complementing our research into the educational aims of the case study institutions, we will establish a baseline for evaluating the impact of HEPs by examining the pre-visit context of participating students and teachers. Visitors do not enter museums as a “blank slate” primed to receive the messages of the institution. Rather, each visitor “uses the raw material provided by [the institution] to construct a new experience that is both unique to the individual and potentially satisfying within the identity-construct that has already shaped him or her” (Falk, 2006, p. 161). Furthermore, high school students encounter HEPs in the context of wider standardised curriculum units and specific classroom experiences. An important aspect of our project is, therefore, to understand the entrance narratives of HEP participants in relation to the key themes addressed by Holocaust museums, including knowledge of the Holocaust as a historical event as well as its contemporary relevance. Specifically, we seek to understand how the existing knowledge, expectations, background, and social context of individual participants affect the process through which they make sense of the museum experience.

Sampling and recruitment of school students for Phases 1b-3

This phase of the study focuses on student groups in Year 10 of high school (normally students 15-16 years of age) who attend our Partner museums as part of organised school tours. Year 10 students comprise the majority of school group visits to the museums, owing to direct links between the primary themes of the institutions and core elements of the Year 10 history curriculum in New South Wales, Victoria, and South Australia, with related topics including World War Two in the wider context of 20th century history, genocide in the 20th century, ethics and social responsibility, citizenship, and human rights.

The size and demographic diversity of the available cohort will enable us to achieve maximum variation sampling, leading to more robust and reliable interpretation of student responses to HEPs (Merriam, 2009, p. 77). Our sample size allows for the inclusion of a cross section of private/public, faith-based, socio-economic, and ethnically diverse school populations, and we will benchmark the sample against the Australian Bureau of Statistics data for school type, gender, and age to ensure representativeness. The schools will be selected from the existing large databases of the partner museums. We will obtain the required permissions from schools and relevant educational authorities (in addition to university human research ethics approval) before commencing the research.

The student-based research phases will occur across two rounds at each partner museum. For each round, ten schools will be selected per museum, with two groups of approximately twenty students participating from each school (HEPs conducted by the partner museums typically consist of around twenty students per facilitator). The total sample equates to 400 students per round per museum (800 per museum for two rounds; 2400 total across the three museums).

Analysis of Curriculum and Teaching Materials

Once the participating school groups for each museum have been identified, we will examine curriculum and preparatory teaching materials to analyse the ways in which students are exposed to information about Holocaust and other genocides before and after their museum visit. Standard curriculum documentation will be obtained from education authorities in each State, while each school and/or individual teachers will be asked to provide details about the specific curriculum components taught in their classrooms and any other supplemental materials that students engage with prior to the museum visit.

Survey

A vital component of Phase 1b is the first administration of a survey to assess student (and teacher) beliefs, values, attitudes, and prior knowledge of the Holocaust before their HEP experience. The survey will be administered online to all the Year 10 HEP participants in the sample, with each participant receiving a unique identifier to enable longitudinal matching across subsequent completions (see Phase 3). As detailed above, the sample amounts to approximately 2400 individual respondents. Ethical considerations regarding the consent process for Phases 1b to 3 can be found in a forthcoming dedicated section of this protocol.

The survey questionnaire (currently in development) is informed by the Gandel Holocaust Knowledge and Awareness in Australia Survey (2021); Cooke & Buckley, 2021) and several other international surveys designed to measure religious tolerance, attitudes relating to antisemitism, and extent of transformative learning. Building on the design of these studies will enable us to compare our findings to existing data sets. The results will provide a statistically significant, quantitative baseline against which to measure changes in Holocaust awareness, attitudinal change, and transformative learning over time at two intervals after the HEP experience (see Phase 3 described below).

Primary Research Phase 2: The HEP in Real Time – Utilising Innovative Methods to Capture Affective Responses and Learning (September 2024 – August 2025)

Objective: To examine the emergent experiences of students during the Holocaust education program, including affective and cognitive dimensions.

In its second phase, the research will examine HEPs as an emergent encounter between students, teachers and educators within the specific temporal, cultural, informational, and sensory environment of the museum. Via field observations, the research will explore the potential for affective disruption and learning from narratives (including testimony), and other communication and social devices, used by educators and guides in the delivery of HEPs. This phase also includes the introduction of VRM to capture the immediate student responses (including affective responses and peer-to-peer interactions).

Field Observation

During the HEPs, field observations will focus on both the students and the museum staff (educators, guides, volunteers) involved in delivering the program. Directly observing the behaviour and responses of students in the natural setting of the HEP will enable us to systematically collect group-level data about the HEP experience and contextualise individual students’ VRM footage (Merriam, 2009). Additionally, by observing museum guides and floor staff, we will document how educational messages are reinforced through significant moments in museum tours (for example, recording where guides dwell in the exhibition space, or which exhibits they encourage visitors to pay attention to, and forms of non-verbal communication such as tone, pace, and volume of speech).

We will conduct the observations overtly as nonparticipants (Bailey, 2007). For each round of observation, researchers will unobtrusively accompany the school group from their entry into the museum and across their HEP experience, including their guided museum tour and/or workshop sessions and facilitated discussions. To enable consistent and comparable data collection, observations will be systematised through a set of prompts (see Online Appendix 3).

VRM

The VRM to be used in this study is based on hands-free audio-visual recording glasses (“smart glasses”) which are less visible and hence less intrusive than other recording devices. These glasses record what the student sees and hears as they move through the exhibitions, capturing the interplay between exhibition narratives and the subjective perceptual field of the participant as an embodied encounter (Cooke & Constantinidis, 2020; Rose & Tolia-Kelly, 2012). In each research round, one student in each of the two groups from each school will wear the smart glasses. We are developing field procedures to ensure that the VRM is implemented consistently for school groups across all three case studies.

The videos will be watched and analysed by the research team, noting where the students looked, how they participated, the group dynamics, and engagement with different forms of testimony. VRM also gives participants the opportunity to critically reflect on their experiences (Rose, 2014), as they review their visit with researchers or peers to generate findings that might be obscured in talk-only interviews or standard filming techniques (see Phase 3 below). Use of VRM technology was successfully trialled in a pilot project by Cooke and Frieze at the Melbourne Holocaust Museum in 2016. The trial found that students were better able to reflect candidly on their reactions to the HEPs, including to the testimony of survivors, than through traditional methodologies (Cooke & Frieze, 2017). Accordingly, each of the students who wore the glasses will participate in paired peer reflections at six-week and then six-month intervals (see Phase 3). Most importantly, by removing the researcher from these peer conversations, VRM reduces the problem of social desirability bias by “reversing the gaze” and enabling students to assess their experience through their own subjective perspective (Cooke & Frieze, 2017).

Primary Research Phase 3: Evaluating the Longitudinal Impact of HEPs on Attitudes and Actions (October 2024 – March 2026)

Objective: To assess the influence of HEPs on students’ thinking about the Holocaust and its contemporary relevance (including developments in attitudes and behaviours) at six weeks and six months post-participation.

Crucially, and to our knowledge unprecedented in the Australian setting, the research will assess the short and longer-term impacts of HEPs. By assessing learning outcomes over an extended post-visit timeframe, we will build new knowledge of how (and which) educational messages are ultimately interpreted, internalised, and operationalised by students.

We will use the same selection and sequence of data collection techniques at both stages of Phase 3 (at six weeks, and then six months post-HEP) to enable evaluation of the impact of HEPs according to indicators of deep learning (Ramsden, 2003). The combination of quantitative and qualitative techniques during this phase are designed to gauge whether the HEP experience induced enough “disequilibrium” to provoke personal re-evaluation of students’ attitudes and values, and whether students can meaningfully integrate their learnings into modified thought processes and concrete actions (Wiley et al., 2021). We will also examine the motivations, expectations, experiences, and post-visit reflections of teachers who bring their students to the partner organisations.

Survey

We will repeat the whole-of-cohort survey at both the six-week and six-month intervals to track shifts in student (and teacher) beliefs, values, and knowledge of the Holocaust following their HEP experience, and assess whether the HEP experience settles over time into durable attitudinal and behavioural change compared to the pre-visit results. Acknowledging that a range of events and experiences can influence developments in social attitudes and behaviour, the survey will be designed to control for confounding factors such as certain demographic factors (e.g., belonging to a specific minority group), the level of pre-existing historical knowledge, or previous visits to Holocaust museums.

Reflective Conversations Using VRM

In addition to interview and documentary data, our research design will capture the affective and social resonance of the HEP experience through observation of peer-to-peer student conversations using the VRM recordings as stimulus (Cooke & Buckley, 2021; Cooke & Frieze, 2017). The students who wore the AV recording glasses during their initial museum visit will watch their individual recording with one of their peers, allowing the students to provide a narrative of their experiences without feeling judged, thus reducing social desirability bias by removing the researcher from the immediate discussions (the Reflective Conversations guide can be found in Online Appendix 4). Student dialogues will be audio-recorded, transcribed, and analysed by the research team.

Alongside capturing how students narrate the information, sensory experience and feelings associated with the HEP, the VRM will provide the opportunity to learn whether and how students who participated in the HEP relate historical events to contemporary social issues. It will reveal which forms of testimony they engage with, how they reflect on the use of testimony in the exhibition, and how they articulate its transformative potential (Simon, 2005).

Teacher Interviews

As part of the data collected during the two post-visit stages, we will interview teachers to gain their perspective on how students have responded to the HEP, as well as how the teacher’s own knowledge and awareness of the Holocaust and interpretation of its contemporary significance has been influenced by their HEP experience (for the semi-structured interview questionnaire, see Online Appendix 5). In line with the museum interviews in Phase 1a, we will perform a small number of pilot teacher interviews to test and refine the interview questions and process.

Ethical considerations for Phases 1b-3

Phases 1b-3 of the project include participants under 18 years of age and engagement with potentially distressing content relating to the Holocaust and World War Two. Our intent is to pursue two separate Human Research Ethics approvals to allow us to progress with the lower-risk components of the research (Phase 1a) while gaining approval for higher risk phases that involve children as participants (Phases 1b-3).

In Australia (see NHMRC et al., 2023, p. 67–69), conducting research with children and young people under the age of eighteen invokes additional ethical considerations concerning informed consent and perceived coercion via unequal power relationships. Given that our sample will be school students around fifteen years of age (developing maturity), our consent process will include gaining consent from both the students and their parents. We will seek standing consent from parents (with student consultation) for the full span of the project, meaning that parents will not need to renew their consent with each phase of the research but will be notified at each new stage, with a clear reminder that they may withdraw consent at any time. The consent will cover the survey, observations of students during HEPs, and the potential for specific students to participate in the VRM (since selection of students for this part of the research will only happen onsite on the day of the HEP). Additionally, we will be seeking permission from relevant school administrations (e.g., state-level education departments, or Catholic and independent schools’ peak bodies, individual schools’ executive) as part of the consent process.

It should be noted that our research participant consent is in addition to the parental consent that the partner museums regularly obtain for students participating in HEPs, which includes obtaining permission for students to learn about the Holocaust, genocide, and other potentially distressing themes. The museums already have procedures in place to support student (and teacher) psychological wellbeing during the museum visit; our research does not specifically or additionally probe these themes, focussing instead on elements of the museum experience.

To enable longitudinal matching of the survey results, a unique identifier will be provided for each student, which they will use across all 3 survey completions. Individuals can technically be identified, and we are currently developing a quantitative data de-identification protocol as part of a rigorous data management policy that complies with the guidelines set out by our universities. The data management plan will specify how long the de-identification codes will be held, details of file encryption and secure storage, and who in the named research team will have access to the file.

Data Management and Analysis Across all Phases

All qualitative, quantitative, and complementary data (see Phase 1a) collected during the project will be systematically stored according to the project’s research data management plan, following guidelines specified by The University of Sydney. Meta-data related to the research (e.g., date, location, researcher identification, data type, name, and location of original data files, etc.) will be documented using a spreadsheet maintained by the chief investigators.

Qualitative Data Analysis

Analysis of the qualitative data (REAP, school group observations, teacher interviews, VRM reflective conversations) will be done with the help of NVivo, a qualitative data analysis software which can accommodate data in various formats, including transcripts, images, and video. The process of analysis and interpretation of the de-identified data will be team-based and iterative, commencing in tandem with data collection with researcher reflections in the form of analytic notes and memos (aided by regular debriefings). Progressing to the data reduction phase of analysis, we will use the constant comparative method (Charmaz, 2006; Merriam, 2009; Saldana, 2009) to collaboratively interpret and sort data into categories (codes) that are meaningful in the context of our research question. This process involves recursive cross-examination within and between data sources, types, participants, times, as well as the developing codes, allowing for constant refinement and differentiation of significant themes.

We plan to work collaboratively in the early stages to establish an initial set of codes that can be applied to the wider dataset, subject to ongoing team-based review in collaboration with our non-academic stakeholders. The data analysis and interpretation will culminate in the articulation of wider patterns and connections across the coded data, first within each dataset, and then transversally to discern higher-order relationships (Braun & Clarke, 2022; Charmaz, 2006; Sangaramoorthy & Kroeger, 2020). We will use these insights to conceptualise how various stakeholders and institutional intentions interact to shape Holocaust museums as learning environments (Phase 1a), and the longer-term impact of HEPs on students (Phase 1b-3).

Quantitative Data Analysis

As outlined earlier, the survey is designed to evaluate the impact of HEPs on participants’ values and attitudes, along with their knowledge and awareness of the Holocaust. Initially, descriptive statistics will summarise the principal characteristics of the data. This will be followed by advanced statistical analyses to explain the findings.

We will use statistical tests to compare responses before and after visits, aiming to identify trends in changes concerning Holocaust awareness and knowledge. Multivariate analysis, mainly logistic regression, will be used to examine the influence of demographic and other factors on these outcomes. Additionally, latent class analysis will be used to categorise distinct visitor profiles based on their responses, providing deeper insights into the diverse experiences and outcomes of their visits.

Supplemental Material

Supplemental Material - Research Protocol: A Transdisciplinary Multi-Case Study Research Design Using Mixed Methods to Evaluate the Long-Term Impact of Holocaust Museum Education in Australia

Supplemental Material for Research Protocol: A Transdisciplinary Multi-Case Study Research Design Using Mixed Methods to Evaluate the Long-Term Impact of Holocaust Museum Education in Australia by Helena Robinson, Avril Alba, Donna-Lee Frieze, Steven Cooke, and Andrew Singleton in International Journal of Qualitative Methods

Supplemental Material

Supplemental Material - Research Protocol: A Transdisciplinary Multi-Case Study Research Design Using Mixed Methods to Evaluate the Long-Term Impact of Holocaust Museum Education in Australia

Supplemental Material

Supplemental Material - Research Protocol: A Transdisciplinary Multi-Case Study Research Design Using Mixed Methods to Evaluate the Long-Term Impact of Holocaust Museum Education in Australia

Supplemental Material

Supplemental Material - Research Protocol: A Transdisciplinary Multi-Case Study Research Design Using Mixed Methods to Evaluate the Long-Term Impact of Holocaust Museum Education in Australia

Supplemental Material

Supplemental Material - Research Protocol: A Transdisciplinary Multi-Case Study Research Design Using Mixed Methods to Evaluate the Long-Term Impact of Holocaust Museum Education in Australia

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: CI Alba has had a standing consultancy with the Sydney Jewish Museum approved under the External Earnings policy of the University of Sydney. CI’s Alba and Cooke, and Senior Research Fellow, Frieze, are expert delegates to the International Holocaust Remembrance Alliance (IHRA), which is officially under the auspices of the Australian Government’s Department of Foreign Affairs and Trade.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Australian Research Council Linkage Projects scheme [grant number LP220100410].

ORCID iDs

Helena Robinson

Avril Alba

Donna-Lee Frieze

Steven Cooke

Andrew Singleton

Supplemental Material

Supplemental material for this article is available online.

References

Alba

(2020). Lessons from history? The future of holocaust education. In Gigliotti

Earl

(Eds.), A companion to the Holocaust (pp. 599–617). Wiley Blackwell. https://doi.org/10.1002/9781118970492.ch33

Bailey

C. A.

(2007). A guide to qualitative field research (2nd ed.). Sage. https://doi.org/10.4135/9781412983204

Baškarada

(2014). Qualitative case studies guidelines. Qualitative Report, 19(40), 1–25. https://doi.org/10.46743/2160-3715/2014.1008

Bergmann

Jahn

Knobloch

Pohl

T. W.

Schramm

(2012). Methods for transdisciplinary research: A primer for practice. Frankfurt-on-Main: Campus.

Bernard

(2017). Research methods in anthropology: qualitative and quantitative approaches (6th ed.). Rowman & Littlefield.

Boix Mansilla

(2017). Interdisciplinary learning: A cognitive-epistemological foundation. In Frodeman

Thompson Klein

Pacheco

R. C. S.

(Eds.), The oxford handbook of interdisciplinarity (pp. 261–275). Oxford University Press. https://doi.org/10.1093/oxfordhb/9780198733522.013.22

Braun

Clarke

(2022). Conceptual and design thinking for thematic analysis. Qualitative Psychology, 9(1), 3–26. https://doi.org/10.1037/qup0000196

Carbone

(2019). Post-multicultural challenges for cultural heritage managers and museums in the age of migrations. Museum Management and Curatorship, 34(1), 2–23. https://doi.org/10.1080/09647775.2018.1498298

Charmaz

(2006). Constructing grounded theory: A practical guide through qualitative analysis. Sage. https://doi.org/10.4135/9781412973588

10.

Cooke

Buckley

(2021). Visual research methods: Visual research methodologies and the heritage of ‘Everyday’ places. In Madgin

Lesh

(Eds.), People-centred methodologies for heritage conservation: Exploring emotional attachments to historic urban places. Routledge. https://doi-org.ezproxy.lib.uts.edu.au/10.4324/9780429345807

11.

Cooke

Constantinidis

(2020). Investigating ‘Ordinary’ Landscapes: Using visual research methods to understand heritage digital technologies and sense of place. In Lewi

Smith

Vom Lehn

Cooke

(Eds.), The Routledge international handbook of new digital practices in galleries, libraries, archives, museums and heritage sites. Routledge. (Chapter 40). https://doi.org/10.4324/9780429506765-41

12.

Cooke

Frieze

D.-L.

(2016). Affect and the politics of testimony in holocaust museums. In Tolia- Kelly

D. P.

Waterton

Watson

(Eds.), Heritage, affect and emotion: Politics, practices and infrastructures (pp. 75–92). Routledge.

13.

Cooke

Frieze

D.-L.

(2017). The social agency of museums in civic engagement: Investigating the impact of education programs at holocaust museums. In Reverberations: The Holocaust, Human Rights and the Museum. Sydney Jewish Museum.

14.

Cooke

Frieze

D.-L.

Singleton

Vergani

(2022). Gandel holocaust knowledge and awareness in Australia survey: Project report. Deakin University. https://gandelfoundation.org.au/wp-content/uploads/2022/01/Gandel-Holocaust-Report-FINAL-FOR-RELEASE-26_1_22.pdf

15.

Cresswell

J. W.

Plano Clark

V. L.

Gutmann

M. L.

Hanson

W. E.

(2003). Advanced mixed methods research designs. In Teddlie

Tashakkori

(Eds.), Handbook of mixed methods in social & behavioural research (pp. 209–240). Sage Publications.

16.

Eckmann

Stevick

Ambrosewicz-Jacobs

(Eds.), (2017). Research in teaching and teaching about the Holocaust: A dialogue beyond borders. Metropol Verlag & IHRA: EU Agency for Fundamental Rights. https://arodes.hes-so.ch/record/11745

17.

Falk

(2006). An identity-centered approach to understanding museum learning. Curator: The Museum Journal, 49(2), 151–166. https://doi.org/10.1111/j.2151-6952.2006.tb00209.x

18.

Falk

J. H.

Dierking

(2000). Learning from museums: Visitor experiences and the making of meaning. Altamira Press.

19.

Falk

J. H.

Dierking

(2011). The museum experience. Routledge. https://doi.org/10.4324/9781315417899

20.

Gross

Doyle Stevick

S. E.

(2015). As the witnesses fall silent: Holocaust education in curriculum, policy and practice. Springer International Publishing. https://doi.org/10.1007/978-3-319-15419-0

21.

Jaeger

(2020). The second world war in the twenty-first-century museum: From narrative, memory, and experience to experientially. De Gruyter. https://doi.org/10.1515/9783110664416

22.

Johnson

Turner

L. A.

(2003). Data collections strategies in mixed methods research. In Teddlie

Tashakkori

(Eds.), Handbook of mixed methods in social & behavioural research (pp. 297–319). Sage Publications.

23.

Kelly

(2010). Engaging museum visitors in difficult topics through socio-cultural learning and narrative. In Cameron

Kelly

(Eds.), Hot topics, public culture, museums (pp. 194–210). Cambridge Scholars Publishing.

24.

Levinson

(1997). Teaching in the midst of belatedness: The paradox of natality in Hannah Arendt’s education thought. Educational Theory, 47(4), 435–451. https://doi.org/10.1111/j.1741-5446.1997.00435.x

25.

Merriam

S. B.

(2009). Qualitative research: A guide to design and implementation. Jossey-Bass.

26.

Myers

C. A.

Long

S. E.

Polasek

F. O.

(2020). Protecting participant privacy while maintaining content and context: Challenges in qualitative data de-identification and sharing. Proceedings of the 83rd annual meeting of the association for information science & technology. Association for information science and technology. https://doi.org/10.1002/pra2.415

27.

National Health and Medical Research Council, Australian Research Council, and Universities Australia . (2023). National statement on ethical conduct in human research 2023. National Health and Medical Research Council.

28.

Ramsden

(2003). Learning to teach in higher education. Routledge.

29.

Repko

A. F.

Szostak

Buchberger

M. P.

(2017). Introduction to interdisciplinary studies (2nd ed.). Sage.

30.

Rose

(2014). On the relation between ‘Visual research methods’ and contemporary visual culture. The Sociological Review, 62(1), 24–46. https://doi.org/10.1111/1467-954x.12109

31.

Rose

Tolia-Kelly

D. P.

(2012). Visuality/materiality: Introducing a manifesto for practice. In Rose

Tolia-Kelly

D. P.

(Eds.), Visuality/materiality: Images, objects and practices (pp. 1–15). Ashgate. https://doi.org/10.4324/9781315547930-5

32.

Saldana

(2009). The coding manual for qualitative researchers. Sage.

33.

Sandell

(2007). Museums, prejudice and the reframing of difference. Routledge. https://doi.org/10.4324/9780203020036

34.

Sangaramoorthy

Kroeger

K. A.

(2020). Rapid ethnographic assessments: A practical approach and toolkit for collaborative community research (Vol. 1, 1st ed.). Routledge. https://doi.org/10.4324/9780429286650

35.

Seidman

(2006). Interviewing as qualitative research: A guide for researchers in education and the social sciences. Teachers College Press.

36.

Simon

R. I.

(2005). Introduction remembering otherwise: Civic life and the pedagogical promise of historical memory. In The touch of the past: Remembrance, learning, and ethics. Palgrave Macmillan. https://doi.org/10.1007/978-1-137-11524-9

37.

Smith

(2016). Changing views? Emotional intelligence, registers of engagement and the museum visit. In Gosselin

Livingstone

(Eds.), Museums and the past: Constructing historical consciousness (pp. 101–121). UBC Press. https://doi.org/10.59962/9780774830638-009

38.

Sydney Jewish Museum . (2019). About the museum. https://sydneyjewishmuseum.com.au/

39.

Teddlie

Tashakkori

(2003). Major issues and controversies in the use of mixed methods in the social and behavioural sciences. In Teddlie

Tashakkori

(Eds.), Handbook of mixed methods in social & behavioural research (pp. 3–50). Sage Publications.

40.

Trofanenko

Segall

(2012). Addressing the pedagogical purpose of indigenous displays: The case of the national museum of the American Indian. Diaspora, Indigenous, and Minority Education, 6(3), 141–156. https://doi.org/10.1080/15595692.2012.691134

41.

Tummons

(2017). Institutional ethnography, theory, methodology, and research: Some concerns and some comments. Studies in Qualitative Methodology, 15, 147–162. https://doi.org/10.1108/s1042-319220170000015003

42.

UNESCO . (2017). Education about the holocaust and preventing genocide: A policy guide. UNESCO. https://doi.org/10.54675/OSJZ5002

43.

Wiley

J. L.

Wiley

K. R.

Intolubbe-Chmil

Bhuyan

Acheson

(2021). A new, depth-based quantitative approach to assessing transformative learning. Journal of Transformative Education, 19(4), 400–420. https://doi-org.ezproxy.lib.uts.edu.au/10.1177/15413446211045164

44.

Witcomb

(2013). Understanding the role of affect in producing a critical pedagogy for history museums. Museum Management and Curatorship, 28(3), 255–271. https://doi.org/10.1080/09647775.2013.807998

45.

Yin

R. K.

(2014). Case study research: Design and methods (5th ed.). Sage Publications.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.54 MB

0.53 MB

0.52 MB

0.53 MB

0.58 MB