Measuring intangible outcomes can be problematic: The challenge of assessing learning during international short-term study experiences

Abstract

Short-term international study experiences, or study tours, aim to increase students’ global awareness, educate and empower them to be productive global citizens, and contribute to their future employability. Learning outcomes from study tours often include intangible personal characteristics or soft skills, as opposed to specific disciplinary skills and knowledge, and yet, these are not easily identified. Using an iterative Delphi process, this study aimed to understand the pedagogical experience of tour leaders (experts) and assist in future development of effective assessment. Findings include four key areas of agreement among the experts: (1) both discipline-based knowledge and the acquisition of intangible personal characteristics are important learning outcomes and are considered to be almost equally important, (2) assessment almost always occurs on tour, (3) formative assessment for learning is predominantly used (rather than learning for summative assessment to test the learning), and (4) facilitated reflection is an important way to encourage learning during a study tour. Yet, the findings also reveal uncertainty about assessing outcomes that include intangible personal characteristics and soft skills. The article concludes with pedagogical implications and recommendations for future research.

Keywords

assessment Delphi method employability intangible characteristics international learning learning soft skills study tour

International study tours and the links to employability

Short-term international study tours can offer a complex range of learning experiences for students and are currently the fastest growing format for international education (Tarrant and Lyons, 2012). These experiences aim to increase students’ global awareness, empower them and educate them to be productive global citizens (Lutterman-Aguilar and Gingerich, 2002). Many universities also recognize the need to internationalize their curricula and reflect the expansive values of globalism (Knight and de Wit, 1995; Rizvi and Walsh, 1998). In doing so, they promote intercultural experiences to prepare students for work in a global economy (Duffield, 2008). Research indicates that international experiences also help establish a student’s future employability (Archer and Davison, 2008). As a result, the number of students studying overseas has been steadily increasing, and international education programmes are expected to grow into the future as more students seek preparation for work in a globalized environment (Bøe and Hurley, 2015; Savicki, 2008; Scharoun, 2015).

Compared to longer term study abroad programmes, the short-term study tour experience is brief and intense and often follows the island model, whereby an academic tour leader(s) coordinates the activities and facilitates in situ lectures or tutorials, often assisted by local experts. This format and structure has led some to suggest that study tours can offer transformational learning through real-world experience and critical reflection (Perry et al., 2012). In addition, research from the United States found that short-term study abroad has a significant impact on increasing students’ functional knowledge of a foreign country that includes communicating with and understanding the cultural customs of local inhabitants, being able to explain aspects of foreign policy to a person from another country and understanding currency conversion rates (Kurt et al., 2013). Being able to function in a real-world setting and adjust to unfamiliar situations are important aspects of employment preparation.

Research on the impact of study abroad found that professional development benefits include understanding international career options, developing confidence in asking technical questions in a foreign language, practicing professional behaviour and gaining exposure and applied knowledge to increase marketability (Ingraham and Peterson, 2004). A study conducted through the Institute for the International Education of Students in the United States showed that studying abroad affected the career choices of nearly two-thirds of respondents (N = 3723), and half of the respondents developed careers with global aspects (Norris and Gillespie, 2009).

Understanding what employers require from new employees is also important. Scholars suggest that extra-curricular activities, which often include travel abroad, can provide opportunities for graduates to demonstrate skills and attributes that employers seek (Purcell et al., 2012). Most employers rated personality type and soft skills as more important than degree qualification (Archer and Davison, 2008; Foundation for Young Australians, 2015). Research with recruiters stressed the importance of personal characteristics in applicant selection beyond specific technical abilities (Clark et al., 2015: 7). Of the 10 most important skills rated by companies when hiring graduates, communications skills (86%), team-working skills (85%), integrity (83%), intellectual ability (81%), confidence (80%) and character and personality (75%) are all seen as important attributes. Furthermore, research suggests that employers believe qualities such as tolerance, open-mindedness, creativity, initiative, empathy and respect can promote the acquisition of positive employability skills (Malicki and Potts, 2013; Potts, 2015). Similarly, many employability models refer to the need for generic skills that include creativity, adaptability, flexibility, autonomy and independence, teamwork skills, time-management, sense of responsibility and the ability to plan, organize and coordinate projects or tasks (Pool and Sewell, 2007).

Intangible qualities and behavioural characteristics as learning outcomes

Barnett and Coate (2005: 63) state that curriculum development often focuses on preparing students for a changing world through the development of flexibility, adaptability, self-reliance and learning how to learn. Yet, these terms or concepts are admittedly ambiguous. The building blocks of curriculum they proposed are relevant in regards to these outcomes and include ‘knowing’ as a personal and positional act to engage with discipline-specific ideas, ‘acting’ which includes experiences of practice and engagement often connected to a particular form of knowing and ‘being’ which involves the students developing a sense of self and acquiring the capacity to flourish. In this regard, the intangible personal characteristics can be considered to align with the concept of being. However, as Barnett and Coate (2005) point out, there is considerable challenge in operationalizing this third building block within a curriculum. As this is suggestive of the outcomes often seen from study tours, it is important to investigate further.

Learning outcomes from short-term international study often include those that are considered to be intangible qualities and behavioural characteristics. These are sometimes referred to as soft skills. This has led to a recognition that study abroad programmes include learning that occurs outside the classroom and is not formally controlled or initiated by tour leaders or teaching staff (Tucker et al., 2011). Furthermore, Bennet (2008) has highlighted potential learning outcomes as ‘curiosity, initiative, risk taking, suspension of judgment, cognitive flexibility, tolerance of ambiguity, cultural humility, and resourcefulness’, and claimed that these qualities actually lay at the ‘heartset’ of international learning (p. 20).

Study abroad experiences also provide the opportunity for significant gains in intercultural understanding. A study by Deardorff (2006) found the three common elements of cross-cultural competence for North American universities: ‘awareness, valuing, and understanding of cultural differences; experiencing other cultures; and self-awareness of one’s own culture’ (p. 247). In addition, other research has found that students participating in a study abroad programme exhibited a sense of patience and humour when dealing with unexpected events or unfamiliar surroundings and demonstrated a change in perception of the world and one-self (Ingraham and Peterson, 2004). Moreover, according to intercultural scholars, specific components of intercultural competence include individual attributes such as curiosity, openness, respect for others and adaptability, as well as skills to listen, observe, analyse, interpret and relate (Deardorff, 2006). Traits associated with personal growth and intercultural awareness have also been shown to gradually increase as programme length increases (Ingraham and Peterson, 2004).

The challenge of assessment

Internationalizing a curriculum can be seen as a way to improve the learning outcomes of students (Leask, 2013). However, not all global experiences inherently provide quality educational outcomes or opportunities for learning (Lutterman-Aguilar and Gingerich, 2002). As there is a growing need for universities to demonstrate a tangible return on students’ investment in higher education (Barack, 2014; Robertson, 2010; Tomlinson, 2008), it is necessary to identify the learning that occurs during international study and develop the means to effectively assess it. Yet, the learning outcomes that arise during international study are not always easy to identify or provide evidence of, and there has been relatively little empirical research in this area (Tucker and Weaver, 2013). This is often not recognized or appreciated by institutions (Potts, 2015), and many have no formal assessment frameworks to identify or encourage such outcomes (Tucker and Weaver, 2013). Although critical reflection is often an essential aspect of transformative learning, as it enables students to challenge their preconceived beliefs (Mezirow, 1990), questions remain about how to evaluate the precise level of transformation that ensures students are benefitting from their learning.

Schmidt and Pardo (2017) promote the need for assessment of learning to ensure that study abroad helps students form human capital. Furthermore, Deardorff (2006) suggests that intercultural competence can, indeed, be measured, but must identify and assess the varying degrees of competence. However, it is also important to recognize that such learning is often measured over a period of time and justifies the use of multiple assessment methods. Despite these good intentions, there remains a difficulty of identifying the goals of intercultural competence and assessing whether students are achieving those goals (Deardorff, 2006).

There is, therefore, a need to discover what and if assessment should be carried out during international study tours and what form it takes. In addition, this study asks what the most important learning outcomes from international study tours are and how, if at all, assessment practices capture these learning outcomes.

Method

The Delphi method is a dialectical process used to reach a consensus about an issue for which there may not be initial agreement (Dick, 2002). It was conceived as a group technique to obtain a reliable consensus of opinion from a panel of experts through a series of intense questionnaires with controlled feedback (Landeta, 2006). As it includes multiple iterations of similar questions (typically three iterations) with immediate answer collation and distribution, it can be considered to be action research by which a panel of experts agree upon themes about a chosen issue. Once the first round of answers are collated and summarized, the experts then receive the answers given by others and can choose to either maintain their original answers or alter them based on the additional information provided to them. Follow-up questions are also asked if appropriate. This process has been shown to effectively lead to a convergence of themes and has the added benefit of mutual education, since experts, who are able to justify their initial answers based on their own experience, also learn from each other’s answers and experiences (Dick, 2002). In some ways, this process resembles the self-reflective and transformational learning process that students are often expected to go through as part of the assessment of global experiences.

An online Delphi process was deemed an appropriate method for this research. A potential limitation of the method is the multiple iterations requiring involvement, which can cause participation to drop off as experts are often busy, or for some other reason their commitment to the research may falter. Therefore, efforts were taken to minimize the burden on participants by keeping the number of questions to a minimum and the amount of time between each iteration relatively short.

To determine the panel of experts, the research team first conducted an online audit of international study tours around Australia, during which 117 study tours were found. For those 117 tours, 99 different study tour leaders (the experts) were identified and their email addresses recorded. It should be noted that some leaders coordinate more than one study tour. The 99 experts were approached via email to be part of the Delphi process. A total of 15 experts confirmed their participation, constituting a 15% response rate for the study. The participants included experts from 11 universities, from four of Australia’s eight states and territories. The sample size of 15 for the Delphi method is acceptable and is consistent with previous studies that were reviewed (Landeta, 2006).

To ease tracking and data entry, the questions were administered using Key Survey, which was controlled by the research team members. Prior to the administration of this survey, the study received university ethical approval for research on human subjects. The questions were open-ended and the participants were given approximately 2 weeks to respond to each round of questions. The four initial questions posed were the following:

As a leader of outbound short-term international study tours, how do you currently assess student learning?

Has your assessment changed while you have been a short-term international study tour leader? Why/Why not?

What do you understand as the most important learning outcomes from outbound short-term international study tours?

Does your current assessment capture these learning outcomes? If yes, please outline how. If no, what is preventing this process?

The research team received 15 responses from the first round of questions, and these were collated. The research team (including six researchers) then employed a thematic analysis process to determine and manually code the answers into a number of primary themes. This process included initial open coding to determine general themes, followed by a group discussion to develop consensus around these themes. A second round of coding was then completed by two of the researchers. Because the answers provided by the experts were generally brief and the sample size was relatively small, the second round of coding was done manually and resulted in a summary of the responses. This summary (with anonymous answers) was sent back to the 15 experts within 2 weeks for the next review and consideration. The team also asked four follow-up questions and asked for further explanation and reflection about their original answers. Nine participants responded to the second iteration of questions. The analysis process was repeated, and a third and final set of questions was then distributed. A total of 11 participants responded to the final iteration. A final report was then distributed to all participants.

Findings

The results are divided into five general topics identified through the thematic analyses: assessment format and timing, learning outcomes, assessment purpose, the effectiveness of assessment, and student reflection.

Assessment format and timing

The answers highlighted the use of varied assessment practices, including a record of attendance, creative projects, presentations, reflective journals, visual diaries, blogs and essays. More than half of the respondents indicated that they use a blog or journal as the assessment item for their study tour.

In regard to when the assessment occurs, the most common response was on tour, indicated by almost all participants (86%, 13 of 15). Just over half of the participants use post-tour assessment (53%, 8 of 15 participants) and just under half use pre-tour assessment (40%, 6 of 15 participants). See Figure 1 for a breakdown of the combinations for assessment timing.

Figure 1.

Assessment timing.

Just over half of the experts indicated that their assessment has not changed while they have been a short-term study tour leader (53%, eight participants). This was often because the tour had only been recently introduced and has not been offered long enough to justify changes. However, one-quarter of the participants (26%, 4 of 15) have added assessment to assist with student development in response to previous tours. One expert indicated,

I’ve added an extra component in the form of an end of day discussion by the group where they evaluate and share their responses to the day’s work. This was introduced after the first study tour when it became clear that the subject [ ] had an emotional impact that was best shared and aired by members of the group.

In addition, three experts indicated that their tour is quite site-specific and required changes based on each tour and their unique situations.

Learning outcomes

Two-thirds of the experts indicated that discipline-related knowledge is among the most important outcomes (66%, 10 of 15 participants). Typical responses described the learning outcomes as, ‘First hand subject knowledge gained through experience on the ground and interactions with professionals in the field’ and ‘discipline specific tasks and outcomes-theoretical’. These and similar responses indicate that international short-term study experiences offer an opportunity to reinforce and supplement discipline learning that is undertaken within more traditional contexts on campus. Figure 2 highlights the breakdown of learning outcomes considered most important by the experts.

Figure 2.

Perceived learning outcomes.

Two-thirds of the experts (66%, 10 of 15 participants) indicated that the acquisition of intangible personal qualities, or soft skills, were among the most important outcomes of their tour. Those mentioned include confidence, independence, respect, imagination, ambition, self-awareness, empathy and critical thinking. In addition, just over half of the respondents (53%, 8 of 15 participants) indicated that cultural competency and gaining a deeper cross-cultural understanding were important learning outcomes.

The opportunity for immersive, on-the-ground experience was seen as an important outcome, and often functioned to reinforce what students were learning in the classroom. Just over half of the experts (53%, 8 of 15) indicated that this was an important learning outcome for their study tour.

A total of 33% (5 of 15 participants) stated that their tours provided an opportunity for benchmarking Australia’s global position when comparing it to international standards. For example, one expert responded that their tour offers ‘the opportunity for staff and students to engage with an international audience and see that the work they are doing in Australia is of an international standard’. Finally, 33% (5 of 15 participants) indicated that professional practice skills, such as being on time, asking appropriate industry-related questions and being respectful, were an important learning outcome for their tour.

A follow-up question in Round 2, in which experts were asked to reflect on learning and content-based/discipline-related learning and whether their tour(s) addresses them, indicates that all but one expert (88%, 8 of 9 participants) addressed personal qualities and intangible personal characteristics, or soft skills, as outcomes during their tour. All the experts answered this question, and of the eight experts, seven indicated that their tour(s) addresses both intangible personal characteristics as outcomes and content-based learning outcomes, while one expert indicated that their tour only addresses intangible personal characteristic outcomes. This expert also responded that they will be addressing such outcomes in future tours.

Although few of the respondents highlighted specific intangible personal characteristics as outcomes, additional examples not already mentioned earlier in this article include sense of group identity and purpose and a sense that a worldview can be challenged and that this challenge is worth the effort.

Assessment purpose

A Round 2 question asked participants to consider whether the purpose of their assessment for the study tour was learning for assessment (that is, summative assessment that is separated and occurs after the learning process), or assessment for learning (that is, formative assessment that is used to facilitate learning tasks, with the outcome used to determine the level of learning).

A majority of participants indicated that their tour incorporated assessment for learning. Of the nine participants who answered this question, just over half (55%, 5 of 9) responded that their model would be framed only as assessment for learning. This was exemplified by responses such as

The assessment is definitely assessment for learning. Students gave feedback that the assessment had made their learning experience rich and several said they planned to approach all future travel in a similar way as they realised a benefit to their understanding and recall.

Another participant stated, ‘Very definitely the learning is achieved by completing the assessment and the outcome is used to assess the level of that learning’. Figure 3 shows the breakdown of the responses.

Figure 3.

Assessment purpose.

An additional two participants (22%, 2 of 9) responded that they would characterize their current assessment as both assessment for learning and learning for assessment and stated, ‘Both in that the outcome is separated and occurs after learning, but the nature of the assessment task, making a film using a theoretical approach, is an outcome determined by the level of learning’. ‘It is both – the reflective journal activity is a during-tour process, while the post tour major research essay occurs afterwards’. Only two participants (11%) responded that their assessment could be characterized as learning for assessment. One stated, ‘Learning happens through the three-week experience that makes up the study tour and this is then assessed through post-tour deliverables’.

The effectiveness of assessment

A question posed in the first iteration of questions asked whether the expert’s current assessment captures their indicated learning outcomes. A majority of the experts (80%, 12 out of 15) indicated that their current assessment captures the learning outcomes they have specified for their study tour, two experts (13%) indicated that their assessment does not capture the learning outcomes, and one expert (7%) indicated that their assessment captures only some of the learning outcomes. Figure 4 illustrates the breakdown of responses.

Figure 4.

Capturing the learning outcomes through existing assessment.

For those participants who responded positively to this question, seven indicated that an essay/presentation structure effectively captures an understanding of culture, three responded that current assessment (not specified) encourages a deeper understanding of their discipline, two responded that assessment includes immersion in another culture, and one participant indicated that their current assessment (not specified) encourages connections with industry partners.

Of the 10 participants who indicated that a key learning outcome of their tour is the acquisition of intangible personal characteristics or soft skills in an earlier question, 70% of participants (7 of 10) indicated that these were indeed captured and assessed through their current assessment practices.

In contrast, 30% of participants (3 of 10) indicated that intangible personal characteristics or soft skills as learning outcomes were not currently assessed. For example, one expert stated, ‘… respect for others or collaboration and teamwork are not formally assessed as they are considered desired generic skills, but not necessarily assessable items’. A second participant responded that the learning outcomes of the students were unknown prior to departure and assessment will be changed for future tours. Finally, a third expert responded,

Their greatest learning is not scholarly or traditionally academic, it comes through their involvement in an unforgettable experience and how much this expands their worlds. Unfortunately, academia requires graded outcomes, and measuring how much you have expanded somebody’s thinking and global knowledge of their profession is very difficult. So we seem to come back to more traditional deliverables because they are easier to use in the process of grading.

A follow-up question posed in Round 2 asked participants to reflect further on whether they still consider that their current assessment captures the learning outcomes of the tour and if they still agree with their previous answer. Almost half of the participants (44%, 4 of 9) responded that their current assessment framework captures both content-based and intangible personal characteristics as learning outcomes, but did not provide the criteria for doing so. One participant responded that their tour is not aligned with the learning outcomes of the unit. However, four participants responded that they do not currently assess intangible personal characteristics or soft skills as they are too difficult to assess.

Due to the growing divergence on this topic indicated by the experts’ answers, the research team included a follow-up question in Round 3. This asked whether the participants think it necessary to measure intangible personal characteristics or soft skills as learning outcomes and how they would actually do so. In addition, the team asked whether the participants’ assessment addresses these learning outcomes and what specific criteria they used to determine the grades.

This question elicited mixed responses. Just over half of the participants (55%, 6 of 11) indicated that they currently use assessment that focuses on intangible personal characteristics or soft skills, which includes reflective writing through journals or essays, role playing, staff observations of students during the tour, student reflection and questionnaires before and after the tour. One participant wrote,

We set out in an assessment rubric the specific behaviours that students were expected to demonstrate. These behaviours were commensurate with norms of international citizenship, but also of collaborative learning. This rubric was used as the basis for discussion of the behaviours. Methods to capture intangible learning include staff observation and student reflection.

However, most of those participants did not give examples of the specific criteria they employed, but did discuss the assessment tasks in general. For example, one respondent said, ‘The reflective journal is a way of observing the students’ shifts in thinking. The shift isn’t assessed, but is apparent’. Another participant wrote,

Cross-cultural awareness is evident to both the students and to their tutors and lecturers. We make an attempt to assess it by asking for a final reflection on their experiences while overseas. Those reflections do reveal evidence of self-awareness and a greater awareness of the culture on which they are reporting.

Many participants recognized the inherent challenge in assessing these outcomes. One participant indicated a level of uncertainty over how to determine whether assessed students had indeed gained intangible personal characteristics or soft skills:

The reflective blog assignment captures intangible learning to some extent. Grading criteria is a challenge, and I found I needed to be open to what students interpreted the assignment requirements to mean. But the questions in my head as I was marking the assignments was ‘has this student attempted to think about how their background and cultural norms affect their understanding of this place? Are they aware of cultural difference and are they demonstrating a capacity for empathy and open-mindedness? Have they used this as an opportunity to reach a deeper understanding of themselves and their country and society, as well as learning about [ ] and its people and culture’. There is a degree of subjectivism in this approach, but I’m not sure how else to measure this.

In addition, one participant questioned whether assessing intangible personal characteristics or soft skills during a study tour was possible:

If it were up to me, I wouldn’t assess my study tour but in order to make it a credit worthy unit, ‘measurement’ is required. I don’t feel the value of the tour is in the assessment, it’s in the experience and that a student’s involvement will inevitably expand their horizon. Measuring intangible learnings is problematic but I believe it would be possible if well thought through rubrics were created and offered to the students before the tour. This would subsequently steer the learning of the students who are academically ambitious, I say this because I believe a lot of the students who attend my study tour are not chasing grades but rather seeking experiences and growth.

Student reflection

During the second iteration of questions, the research team asked the experts to consider and discuss the potential value of facilitated reflection as a way to encourage students to examine their experiences to improve learning. Just over half of the experts responded to this prompt and all (5 of 9) indicated they viewed facilitated student reflection as an important element of the assessment process. One expert wrote, ‘I think facilitated reflection is a great tool and one that I used throughout the trip with students. It encouraged them to link their course learning with the experiences occurring during the trip’. Another expert indicated that students are required to engage in written reflection:

The facilitated reflection is very important – but we’ve changed what we’re asking for. So instead of the students writing about what they learned about the country, the topics of their stories, themselves etc, they are now being asked to reflect on a specific example of something that went particularly well or badly and explain how they responded to the circumstance, and how they learned from that experience.

Instead of written reflection, two participants indicated that the student reflection occurred through informal dialogue:

With the subject I have run for nine years, facilitated reflection has been a key element to encourage students to engage and examine their experiences and learning reflected in the end of day discussions.

My study tour is a 24/7 experience and I would argue that there is a degree of facilitated reflection. This does not happen on a formal level (doing so would feel constructed/forced) but there are often informal discussions which take place that evening, or in the following days, these seek to unpack and reflect on particular experiences/events. This is generally done socially i.e. while walking, over a meal or at evening drinks. Apart from seeming to be incidental conversations, they are often my opportunity to gauge what students have gained from an event/experience, to seek their feedback on how good the event was, and to encourage them to think more deeply about what they have experienced and what that might mean.

Discussion and conclusion

The study revealed four key areas of agreement among the experts: (1) both discipline-based knowledge and the acquisition of intangible personal characteristics or soft skills are important learning outcomes resulting from short-term international study experiences and are considered to be almost equally important; (2) assessment almost always occurs on tour, and just over half of the tours also include either pre- or post-tour assessment; (3) formative assessment for learning is predominantly used (rather than learning for summative assessment); and (4) facilitated reflection is considered an important way to encourage learning during a study tour. These areas of agreement demonstrate that there is a level of commonality in regard to the pedagogy used during study tours. Of particular note is the recognition that the acquisition of intangible personal characteristics or soft skills was a valuable learning outcome. Similar to previous research, the participating experts mentioned the following outcomes of short-term study tours: confidence, independence, respect, imagination, ambition, self-awareness, empathy, critical thinking, cultural understanding, a sense of group identity and purpose, and a sense that a worldview can be challenged and that challenge is worth the effort.

Yet, despite this recognition, the experts expressed increased divergence on the topic of assessing intangible personal characteristics or soft skills as learning outcomes during their tours throughout the Delphi process. Initially, 30% (3 of 10) indicated that their assessment does not address these learning outcomes. Yet, a follow-up question asked participants to reflect further on this. In response, 44% of the experts (4 of 10) responded that they do not currently assess intangible personal characteristics or soft skills as learning outcomes, as they are too difficult to assess. Furthermore, the research team asked the participants to describe the criteria used to assess these, specifically asking if they are able to capture and assess behaviour change, but none of the participants provided this information. Although this could be a limitation of the online Delphi process, it also indicates a level of uncertainty about how to assess this type of learning.

In addition, facilitated reflection was discussed as one way to encourage the acquisition of intangible personal characteristics or soft skills during a study tour. This is also highlighted by Lutterman-Aguilar and Gingerich (2002) who posit that any study abroad programme that does not ‘structure reflection and critical analysis of the international experience itself into the curriculum is not engaging in experiential education’ (p. 45). Yet, as they also point out, students do not know how to reflect automatically and must be taught how to do so, and this presents a topic for future exploration. In relation to employability, Pool and Sewell (2007) have identified the importance of providing opportunities for reflection and evaluation of the learning experiences to enable full consideration of how far students have developed their employability prospects and what they need to do in the future.

Mapping the types of assessment offered at one institution found that outcomes such as development, autonomy, knowledge and work-place practices are underused in assessment practice, suggesting that assessment tasks are not often fit for purpose (Gillett and Hammond, 2009). In addition, the research highlighted the need for assessment that fosters development and reflection. Furthermore, assessment methods need to be reassessed on an ongoing basis to stay current and to adapt to changing times in relation to the thought processes that build intercultural competence (Deardorff, 2006).

One limitation of this study includes the online facilitation of the Delphi process, rather than in-person. Due to the lack of face-to-face interaction between the researchers and participants, the questions were kept short and simple, and this was not conducive to the provision of questions for prompting. In one instance, the second part of a two-part question did not receive any response from the participants. Another limitation includes the relatively small response rate. Although acceptable for a Delphi process, the small number of participants did not lead to the acquisition of a large amount of data. Conducting further research with additional tour leaders to gain feedback on the findings presented here will be an important next step. In addition, the participants were only tour leaders and did not include administrators or students. It will be important to gain a deeper understanding of assessing intangible outputs by asking students about the personal characteristics they believe have been acquired. Another limitation of the study is the varied experience of the respondents. Although all were considered academic experts given their experience of running study tours, it is important to highlight that some respondents had run these international study experiences only once, while others had years of experience running a multitude of international study tours. This invariably impacts the amount of experience of different assessment practices that informed their answers.

A primary challenge for this study and for many other researchers examining the nature of learning on study tours is identifying, evaluating and understanding the nature of what has been referred to as intangible learning or the acquisition of soft skills (Bennet, 2008). There is certainly valuable learning that students acquire during their overseas experiences that are not catered for by traditional and/or conventional assessment modes. It is clear that personal growth in areas such as confidence, resilience and initiative exist and that they enhance students’ employability, but coming to an agreement as to where this type of learning is situated in relation to the many types of learning students experience is a complex matter. This study contributes to a better understanding of this phenomena, but recognizes the difficulties that exist in relation to the analysis.

Future research must first find a way to understand behaviour change that occurs as a result of international study more thoroughly and then identify forms of assessment that can encourage or elicit learning and legitimately capture it. Moreover, a question must be raised as to whether simply drawing attention to learning on a tour, either through assessment, facilitated reflection, or other means, can lead to transformational learning that influences students’ future behaviour. It therefore may be prudent to conduct action research and experiment with various modes of assessment and solicit immediate student feedback to better understand actual learning outcomes.

In addition, research is needed to determine the longitudinal impact of short-term international study experiences on students. Research highlights the fact that ‘expectations for substantial growth on such a complex learning outcome should be tempered by the reality that legitimate intercultural learning takes much more time than the length of even the longest study abroad experience’ (Lutterman-Aguilar and Gingerich, 2002). This also needs to be considered with the knowledge that assessing changes outside of a controlled environment will often include many unidentified variables.

As short-term international study tours are becoming more common, it is critical that research enables a deeper understanding of effective pedagogical practices during these experiences in relation to their value for students immediately and in the long term. More focus needs to be given to assessing intangible personal characteristics and soft skills to assist study tour leaders around the world. Although study tours almost always include on-tour assessment, there is no standard format or assessment method to ensure students are actually learning skills that are valuable for their future employment. This is especially critical as the demands for employment in a global market become more competitive. Only then we will be able to demonstrate how international study experiences offer students opportunities to enhance their employability and enable them to secure employment in uncertain and disruptive futures.

Footnotes

Acknowledgements

The authors would like to acknowledge the experts who participated in the Delphi process for their insight and consideration.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Office of Learning and Teaching (grant # SD15-5193).

Author biographies

Debra Flanders Cushing is a Senior Lecturer in landscape architecture and the Higher Degree Research Training Coordinator for the School of Design and the Design Lab at the Queensland University of Technology. Her research focuses on purposeful placemaking, intergenerational parks and child-friendly communities. Address: School of Design, Queensland University of Technology, 2 George Street, GPO Box 2434, Brisbane, QLD 4001, Australia. [email: debra.cushing@qut.edu.au]

Mark Pennings is a Senior Lecturer in Art History. He has taught study tours in New York and Tokyo since 1998. He researches the impact of immersive study tours on student learning, and the impact of tourism on global art, art institutions and experience-scapes. Address: School of Creative Practice, Queensland University of Technology, Australia. [email: m.pennings@qut.edu.au]

Dino Willox is the Director of the University of Queensland Student Employability Centre, which helps students to identify and engage with activities and experiences that enhance their studies and develop their employability. Dino is also a non-executive Board member of the Australian Collaborative Education Network (ACEN), which is the peak body for practitioners and researchers involved in Work Integrated Learning (WIL) in Australia. Address: Student Employability Centre, The University of Queensland, St Lucia, QLD 4072, Australia. [email: d.willox@uq.edu.au]

Rafael Gomez is a Lecturer in industrial design with an expertise in experience design. He is a researcher for the People and Systems Lab and co-leader of the Design Fabrication platform for the Design Lab, School of Design at the Queensland University of Technology. He has run several international study tours to Asia and has received support through the Australian Government’s New Colombo Plan. Address: School of Design, Queensland University of Technology, 2 George Street, GPO Box 2434, Brisbane, QLD 4001, Australia. [email: r.gomez@qut.edu.au]

Clare Dyson is a Senior Lecturer and the Program Manager of WIL within the Creative Industries Faculty at the Queensland University of Technology. She coordinates industry-based internships, creative projects and international study tours and is currently researching new curriculum models for WIL in the creative arts. Address: WIL, Queensland University of Technology, Australia. [email: c.dyson@qut.edu.au]

Courtney Coombs is a Brisbane-based artist, researcher and writer. Courtney teaches art history and studio practice at Queensland University of Technology and works in the area of equitable student engagement and learning. Address: School of Creative Practice, Queensland University of Technology, Australia. [email: Courtney.coombs@qut.edu.au]

References

Archer

Davison

(2008) Graduate Employability: What Do Employers Think and Want? London: Council for Industry and Higher Education.

Barack

(2014) Higher Education in the 21st Century: Meeting Real-world Demands. New York: The Economist Intelligence Unit Limited.

Barnett

Coate

(2005) Engaging the Curriculum in Higher Education. New York: Open University Press; McGraw-Hill Education.

Bennet

(2008) On becoming a global soul: A path to engagement during study abroad. In: Savicki

(ed.) Developing Intercultural Competence and Transformation: Theory, Research and Application in International Education. Sterling, VA: Stylus Publishing, pp. 13–31.

Bøe

Hurley

(2015) Gone international: Mobile students and their outcomes. Report on the 2012/13 Graduating Cohort, UK Higher Education International Unit, London, March 2015.

Clark

Marsden

Duncan

, et al. (2015) ‘It’s everything else you do…’: Alumni views on extracurricular activities and employability. Active Learning in Higher Education 16(2): 133–147.

Deardorff

(2006) Identification and assessment of intercultural competence as a student outcome of internationalization. Journal of Studies in International Education 10(3): 241–266.

Dick

(2002) Building Agreement from Disagreement: The Anatomy of Dialectical Processes. Chapel Hill, QLD, Australia: Interchange.

Duffield

(2008) Student reporting abroad: An international programme called journalism reporting field trips. Pacific Journalism Review 14(2): 102–122.

10.

Foundation for Young Australians (2015) The New Work Order Report: Ensuring Young Australians Have Skills and Experience for the Jobs of the Future, Not the Past. Melbourne, VIC, Australia: Foundation for Young Australians.

11.

Gillett

Hammond

(2009) Mapping the maze of assessment: An investigation into practice. Active Learning in Higher Education 10(2): 120–137.

12.

Ingraham

Peterson

(2004) Assessing the impact of study abroad on student learning at Michigan State University. Frontiers: The Interdisciplinary Journal of Study Abroad 10(4): 83–100.

13.

Knight

de Wit

(1995) Strategies for internationalisation of higher education: Historical and conceptual perspectives. In: De Wit (ed.) Strategies for Internationalisation of Higher Education: A Comparative Study of Australia, Canada, Europe and the United States of America. Amsterdam: European Association for International Education, pp. 5–32.

14.

Kurt

Olitsky

Geis

(2013) Assessing global awareness over short-term study abroad sequence: A factor analysis. Frontiers: The Interdisciplinary Journal of Study Abroad XXIII23: 22–41.

15.

Landeta

(2006) Current validity of the Delphi method in social sciences. Technological Forecasting and Social Change 73(5): 467–482.

16.

Leask

(2013) Internationalizing the curriculum in the disciplines-imagining new possibilities. Journal of Studies in International Education 17(2): 103–118.

17.

Lutterman-Aguilar

Gingerich

(2002) Experiential pedagogy for study abroad: Educating for global citizenship. Frontiers: The Interdisciplinary Journal of Study Abroad 8(2): 41–82.

18.

Malicki

Potts

(2013) The outcomes of outbound student mobility: A summary of academic literature. Report, AIM Overseas, Australia, August 2013.

19.

Mezirow

(1990) How critical reflection triggers transformative learning. In: Mezirow

(ed.) Fostering Critical Reflection in Adulthood. San Francisco, CA: Jossey-Bass Publishers, pp. 1–20.

20.

Norris

Gillespie

(2009) How study abroad shapes global careers: Evidence from the United States. Journal of Studies in International Education 13(3): 382–397.

21.

Perry

Stoner

Tarrant

(2012) More than a vacation: Short-term study abroad as a critically reflective, transformative learning experience. Creative Education 3(5): 679–683.

22.

Pool

Sewell

(2007) The key to employability: Developing a practical model of graduate employability. Education + Training 49(4): 277–289.

23.

Potts

(2015) Why learning abroad matters. Available at: http://www.universityworldnews.com (accessed 29 May 2015).

24.

Purcell

Elias

Atfield

, et al. (2012) Futuretrack 4: Transitions into Employment, Further Study and Other Outcomes. Warwick: Warwick Institute for Employment Research.

25.

Rizvi

Walsh

(1998) Difference, globalisation and the internationalisation of curriculum. Australian Universities’ Review 41(2): 7–11.

26.

Robertson

(2010) Challenges Facing Universities in a Globalising World. Bristol: Centre for Globalisation, Education and Societies.

27.

Savicki

(ed.) (2008) Developing Intercultural Competence and Transformation: Theory, Research and Application in International Education. Sterling, VA: Stylus Publishing.

28.

Scharoun

(2015) Short-term study tours as a driver for increasing domestic student mobility in order to generate global work-ready students and cultural exchange in Asia Pacific. Perspectives: Policy and Practice in Higher Education 20(2–3): 83–89.

29.

Schmidt

Pardo

(2017) The contribution of study abroad to human capital formation. The Journal of Higher Education 88(1): 135–157.

30.

Tarrant

Lyons

(2012) The effect of short-term educational travel programs on environmental citizenship. Environmental Education Research 18(3): 403–416.

31.

Tomlinson

(2008) ‘The degree is not enough’: Students’ perceptions of the role of higher education credentials for graduate work and employability. British Journal of Sociology of Education 29(1): 49–61.

32.

Tucker

Weaver

(2013) A longitudinal study of student outcomes from participation in an international study tour. Journal of University Teaching and Learning Practice 10(2): 1–14.

33.

Tucker

Gullekson

McCambridge

(2011) Assurance of learning in short-term, study abroad programs. Research in Higher Education Journal 14: 1–11.