Quantitative evaluation of T he I magineerium education project by students: Introducing the trowsdale index of confidence in experiential learning

Abstract

The Imagineerium is an arts and engineering based curriculum project designed to enhance student confidence in learning. This study reports on the development of the Trowsdale Index of Confidence in Experiential Learning, an instrument designed to conceptualise and operationalise a four-component model of confidence in experiential learning appropriate for upper primary school students, embracing confidence in creativity, confidence in competence, confidence in collaboration, and confidence in learning. Data provided by 140 9- to 10-year-old students both before and after participating in the 10-week programme, demonstrated a significant increase in scores on this measure at time two, although there was no increase in scores on a control variable hypothesised not to be influenced by the intervention.

Keywords

Imagineerium project quantitative evaluation primary school confidence in learning creativity collaboration competence

Introduction

The scientific evaluation of educational intervention programmes requires the development and testing of instruments properly calibrated to assess the intended outcome of the intervention (D’Agostino, 2005). Trowsdale et al. (2019) described an educational intervention programme (The Imagineerium), identified the intended outcome of this intervention (impacting student confidence as learners), and proposed a set of measures to operationalise the intended outcome (Trowsdale Indices of Confidence in Competence, Creativity and Learning). Drawing on data provided by a sample of 135 9- to 10-year-old students who completed the Trowsdale Indices of Confidence in Competence, Creativity and Learning, both before and after the 10-week intervention programme, Trowsdale et al. (2019) demonstrated increased scores on these indices at time two, although not on three control variables that were hypothesised as unrelated to the intended objective of the intervention. They interpreted this finding as supporting two conclusions. The first conclusion was that the data supported the construct validity of the Trowsdale Indices of Confidence in Competence, Creativity and Learning on the grounds that scores on these measures were impacted by the intervention. The second conclusion was that The Imagineerium was achieving its intended impact, on the grounds that scores on the measure increased following the intervention. Such circularity is inevitable in the slow process of developing and validating new measures.

The study reported by Trowsdale et al. (2019) was conceived as the first of a series of pilot studies during which both the intervention programme and the tools developed for the scientific evaluation of the intervention programme could be critiqued and developed. Building on the work reported by Trowsdale et al. (2019) the present study discusses the learning from the first pilot study and proposes the following three objectives of: (1) improving on the assessment of student confidence as learners by exploring a tighter conceptualisation of what this may mean; (2) developing a more compact instrument to assess student confidence as learners; (3) refining the use of the control variables. Each of these objectives will be discussed in turn, after offering a brief resume of The Imagineerium curriculum intervention project, as implemented between 2017 and 2019.

Introducing The Imagineerium

During The Imagineerium curriculum intervention project, 9- to 10-year-old students are ‘commissioned’ to work with professional ‘imagineers’ (artists, performers, designers, and engineers) and with their teachers, to imagine, to design, and to create partially working models of a mechanical, moving machine. In the project, children’s designs were inspired by a story from local history. Through an imaginative ‘mantle of the expert’ frame (see Heathcote and Bolton, 1995), adult imagineers induct the children into a ‘community of practice’ (Lave and Wenger, 1991), where they are encouraged to think and behave ‘like an imagineer’ through art-making activities (Trowsdale, 2020). In the curriculum suggested by The Imagineerium, children experience learning through a range of ‘subject lenses’ simultaneously. Different bodies of knowledge are developed and practised as necessary to the commission. As the commission is central, the processes involved in making in the arts (which frequently and eclectically draw from beyond the arts) frame children’s experience. Children act out ideas, imagine themselves into another character and situation. Physical theatre is also used to enable children to experience and understand principles of physical sciences in relation to forces and mechanisms (through construction and movement) using their bodies. They work practically, in groups, manipulating and testing how materials might combine to provide robust structures, effect movement and be aesthetically pleasing. Drawing on their imaginations, shaped by the possibilities of materials, they explore challenges in making, and come to understand what is involved in realising ideas and motion. The emphasis on ‘behaving like an imagineer’ (Imagineer, 2016; Trowsdale, 2020) encapsulated a number of desired dimensions of being creative such as ‘dreaming: letting your imagination fly’, ‘sharing and being open to new ideas’, ‘seeing the good things about “failing”’, ‘persevering’, ‘taking responsibility’ ‘negotiating’, and ‘supporting others’, many of which are noted in analyses of creativity in engineering (Lucas et al., 2014: 24–29). Idea generation, thinking and understanding are supported by the use of personal sketchbook/journals, by questioning and by talking.

The Imagineerium is designed to stimulate children’s aspirations, to channel their inventiveness, and to feed their appetite for the engineering sciences and arts as related to creative practices and professions. It has been recognised as a particular kind of STEAM education (Colucci-Gray et al., 2017) and reported on elsewhere as a particular, arts-rich, educational experience (Trowsdale, 2016) and ‘practice-based’ curriculum (Davies and Trowsdale, 2017), as well as being the potential impetus for the reconceptualization of multi-subject curricula (Davies and Trowsdale, 2021).

Assessing student confidence as experiential learners

Alongside the development of The Imagineerium curriculum intervention project, the research team conducted a series of evaluation projects, employing qualitative methods to assess the effects of the intervention on student confidence as learners. Data were generated through interviews, questionnaires, and journal entries, completed by teachers as well as by students (see Trowsdale, 2014, 2016, 2020). These qualitative data generated insight into the various aspects of their experience as learners that students considered had been impacted by participation in The Imagineerium. Overall students identified themselves as having been engaged with and caught up in an experiential learning environment. Experiential learning is conceived of here following Dewey (1938) and Ingold (2017), reflecting more complex and pragmatic notions of experience (see Miettinen, 2000). Engaged as experiential learners the students recurrently mentioned four aspects of this experience.

First, the experience enhanced their evaluation of themselves as collaborative learners. The experience of working as part of an engaged learning team (including fellow students and staff) had opened their eyes to the benefits of such collaboration. Students referred to the value of their experience of working with others, perhaps especially with others who approached things in different ways. In such a context they ‘get inspired by other people’. They recognise that ‘everyone’s ideas are helpful because ... everyone thinks different’. The project helped them to ‘learn about each other’, and to ‘develop your own listening’. They discovered that learning together they could ‘make better ideas’. Growth in confidence in collaboration was the first hallmark.

Second, the experience enhanced their evaluation of themselves as creative learners. The experience of being challenged to generate new ideas and to test ideas generated by others had opened their eyes to their own capacity for creativity and to the capacity of their peers for being creative. Students referred to taking pride in what they were creating, with one student saying, ‘I’m more proud of what I’m doing’. At the same time they took pride in the creativity of others, with one student remarking on how another student was starting to ‘learn what s/he is good at’, that s/he is ‘smarter than she thinks’, ‘more clever’, and ‘more intelligent’. Another student explained how her creativity grew through group talk, ‘I felt they were improving my idea’, noting that one key visual element ‘never really had a meaning until the group started talking about it’. Growth in confidence in creativity was the second hallmark.

Third, the experience enhanced their evaluation of themselves as competent learners. The experience of being encouraged to test out their ideas and to put their ideas into practice had opened their eyes to their own capacity to implement their creativity to good effect. Students referred to their growing confidence in their competence, with one student saying, ‘I think that now I know more about myself ... I can challenge myself to do more things. Another student articulated this improved self-belief in her own capabilities when she said, ‘Before I’d be “well I won’t be able to do that”, but Imagineering has opened my eyes and I think “I can do this!”’. Growth in confidence in personal competence was the third hallmark.

Fourth, the experience enhanced their evaluation of themselves as general learners. The experience of being engaged in the experiential learning environment, shaped by the project, had opened their eyes to their personal capacity as learners, extending well beyond the frame of the project itself. Students spoke about this improved appetite for learning in other areas. They spoke about maths, football and social situations, with one student saying, ‘sometimes I wouldn’t really like, believe in myself, but like now I believe in myself more’. Students thought they were ‘just more confident in everything’ and ‘more ready to have a go’. This was also evident in observations made by the project team and acknowledged by teachers in interviews. Growth in confidence in general learning was the fourth hallmark.

Developing a compact measure of student confidence as learners

Building on the insights from the qualitative studies reported by Trowsdale (2014, 2016), Trowsdale et al. (2019) developed a quantitative study designed to capture three of the four hallmarks of student confidence as experiential learners as identified in the previous section. In this study, Trowsdale et al. (2019) focused on the three components of confidence in learning, confidence in creativity, and confidence in competence. The resultant Trowsdale Indices of Confidence in Competence, Creativity and Learning (TICCCL) proposed three separate measures (employing a total of 34 items), each of which reported acceptable levels of internal consistency reliability: the 11-item Scale of Confidence in Learning (α = .89), the 14-item Scale of Confidence in Creativity (α = .86), and the 9-item Scale of Confidence in Competence (α = .75).

Drawing on data provided by 135 9- to 10-year-old students (within five schools) who participated over the 10 weeks of The Imagineerium project during the school year 2017–2018, and who provided full responses to all three scales both at the beginning and at the end of the 10-week period, Trowsdale et al. (2019) reported a significant increase in the scores recorded on all three measures at time two. This finding suggests both that The Imagineerium was achieving the intended educational outcome and that the TICCCL was sensitive to recording that outcome.

Building on this first quantitative study, the research team identified two ways in which an improved outcome measure could be conceptualised. First, it was recognised that, although the TICCCL had captured three of the four hallmarks of student confidence in experiential learning (confidence in competence, confidence in creativity, and confidence in general learning), a more adequate outcome measure should also embrace the fourth hallmark (confidence in collaboration). Second, it was recognised that adding further items to a set of 34 items was not desirable. In light of the high correlations between the three scales proposed by TICCCL, the notion of student confidence in experiential learning was conceptualised as a single construct comprising four components.

To test this conceptualisation the original battery of items employed by Trowsdale et al. (2019) was redrafted, drawing insight from the earlier qualitative studies (Trowsdale, 2014, 2016) with multiple items operationalising the four components of confidence in collaboration, creativity, competence and general learning.

Reconsidering control variables

Since constraints on the resources available for the study reported by Trowsdale et al. (2019) did not allow for a control group, a set of control variables was employed in the design. In this usage of the term ‘control variables’ refer to measures that were completed at the same time as the measures hypothesised as reflecting the intended outcomes of the project, but which were hypothesised as unrelated to the outcomes intended by it. The control variables incorporated in the study were the extraversion, neuroticism, and psychoticism scales proposed by the abbreviated form of the Junior Eysenck Personality Questionnaire Revised developed by Francis (1996). Data provided by the 135 9- to 10-year-old students, before and after the project, demonstrated no significant difference between the scores recorded on these three scales on the two occasions. These findings suggest that the project had no effect on these three variables, confirming stability among the participants on these variables in contrast with the shifts taking place in the three variables hypothesised to be influenced by the project, namely confidence in learning, confidence in creativity, and confidence in competence.

Building on this first quantitative study, the research team identified two reasons for reducing the number of control variables in the second study. First, in the original study there were three control variables to match the three experiential variables. The aim of the second study was to capture the experiential variable with fewer items. It would make sense, therefore, to deal with the control variable in the same way. Second, although the Junior Eysenck Personality Questionnaire Revised, both in the full form (Corulla, 1990) and the abbreviated form (Francis, 1996) is well established and accepted for research among 9- to 10-year-old students, in the earlier project a teacher objected to some of the items comprising the neuroticism scale and the psychoticism scale. This second study tests the use of just the extraversion scale.

Research objectives

Against this background the present study has formulated three clear research objectives to be tested among the students participating over the 10 weeks of The Imagineerium project during the school year 2018–2019.

The first research objective, building on the critique (advanced above) of the Trowsdale Indices of Confidence in Competence, Creativity and Learning (TICCCL), was to test a modified set of items from those used by Trowsdale et al. (2019) to create a compact measure of confidence in learning appropriate for 9- to 10-year-old students (of no more than 20 items). The original set of items was modified in light of a robust conceptualisation of the confident learner during the upper age range of the primary school, embracing confidence in competence, confidence in creativity, confidence in collaboration, and confidence in learning. This compact composite measure will constitute the Trowsdale Index of Confidence in Experiential Learning (TICEL).

The second research objective, designed to replicate and extend the study reported by Trowsdale et al. (2019), was to examine the impact of the educational intervention programme, The Imagineerium, by comparing the scores recorded by a cohort of students on the Trowsdale Index of Confidence in Experiential Learning completed before and after the 10-week educational intervention.

The third research objective, following criticism offered by some teachers during the research reported by Trowsdale et al. (2019) of some of the items presented in the neuroticism scale and the psychoticism scale of the Junior Eysenck Personality Questionnaire Revised (Francis, 1996), was to test the adequacy of using only the extraversion scale as the control variable.

Method

Procedure

The questionnaire was administered twice, by teachers in the five participating schools, to the students taking part in the project, once at the beginning of the programme and again at the end of the programme. Teachers were both trained in the administration of the questionnaire and followed an agreed script to ensure that the same wording was used in all schools. Participants were assured of confidentiality with questionnaires completed anonymously. The programme and the assessment were conducted in accordance with the ethical procedures of the University of Warwick (HSSREC ref:75/16-17) and with parental consent.

Participants

A total of 140 students provided full data at both time one and time two. At time one 56% of students were 9 years old and 44% were 10 years old. By time two 19% of students were 9 years old and 81% were 10 years old. Of the 140 students 49% were male and 51% were female.

Instrument

The Me and My Learning questionnaire comprised three sections. The first section collected basic demographic data on sex and age. The second section comprised 63 questions covering a whole range of attitudinal questions concerning the experience of learning, including specific questions related to confidence in learning. Each item was rated on a five-point scale: agree strongly (5), agree (4), not certain (3), disagree (2), and strongly disagree (1). The third section comprised the six-item extraversion scale from the abbreviated form of the Junior Eysenck Personality Questionnaire Revised (Francis, 1996). Each item was rated on a two-point scale: yes (1), and no (0).

Analysis

The data were analysed by the SPSS package utilising the frequencies, correlations, factor, reliability, and paired t test routines.

Results and discussion

The first step in data analysis explored the factor structure of the 63 items in the second section of the questionnaire to identify the best set of 20 items, comprising five items from each of the four conceptually defined domains as concerned with competence, creativity, learning, and collaboration, that would coalesce to form a unidimensional index of confidence in learning. An iterative process was used removing items that loaded less strongly on the principal factor until five items remained from each of the four conceptually defined groups. Employing principal component analysis, Table 1 presents the best solution to emerge from these 63 items, together with three indicators concerning the psychometric properties of the proposed new Trowsdale Index of Confidence in Experiential Learning (TICEL). The first column presents the factor loadings on the first factor proposed by principal component analysis (accounting for 26% of the variance). The item with the highest loading on this factor (.72) ‘I am good at learning’ clearly roots this factor within the conceptual domain of confidence in learning. The second column presents the correlations between each of the individual items and the sum of the other 19 items as proposed by internal consistency reliability analysis (leading to an alpha coefficient of .84). Here is a good level of internal consistence reliability (Cronbach, 1951). The range of correlations, between .25 and .58 indicates a good bandwidth for the construct being assessed. The third column presents the item endorsements in terms of the proportion of participants rating the item as agree strongly or as agree. Overall, the items displayed variability in endorsement from 25% to 74% indicating a wide range in item discrimination.

Table 1.

The trowsdale index of confidence in experiential learning (TICEL).

	f	r	%
Confidence in learning
I am good at learning	.72	.56	59
I enjoy learning	.63	.48	41
I feel good about myself and how well I learn	.65	.48	68
I feel confident that I can do well at school	.66	.49	74
I feel happy with how well I am learning	.67	.47	66
Confidence in collaboration
I feel I can learn better by working with others	.34	.32	69
I feel it helps me to work with others	.35	.30	65
I feel I can help others to learn by working with them	.67	.58	58
I look forward to working with other people	.45	.42	62
I feel working with a group helps me to develop my ideas	.48	.45	70
Confidence in competence
I can explain how mechanisms work (e.g., lever, cam, pulley, cog and gears)	.35	.34	29
I can explain how ‘resistance’ affects how something moves	.33	.35	25
I can explain the physical properties of different materials (e.g., hard, soft)	.40	.38	59
I can make prototypes for bits of my design ideas	.35	.36	28
I can calculate how to scale something down (or up)	.22	.25	37
Confidence in creativity
I am an imaginative person	.43	.35	64
I am good at coming up with lots of good and new ideas	.55	.45	46
I am good at putting things together to make something new	.56	.51	56
I can often see how to improve an idea	.42	.34	44
I give good reasons for my ideas and answers	.59	.46	40

Note: % = sum of agree and agree strongly responses.

r = correlation between individual item and sum of other 19 items.

f = loading on first factor proposed by principal components analysis.

In terms of the four conceptually defined domains, the items concerned with confidence in learning displayed variability in endorsement from 41% who agree that they enjoy learning to 74% who agreed that they feel confident that they can do well at school. The items concerned with confidence in collaboration displayed variability in endorsement from 58% who feel they can help others to learn by working with them to 70% who feel working with a group helps them to develop their ideas. The items concerned with confidence in competence displayed variability in endorsement from 25% who agreed that they can explain how ‘resistance’ affects how something can move to 59% who can explain the physical properties of different materials (e.g., hard, soft). The items concerned with confidence in creativity displayed variability in endorsement from 40% who agreed that they give good reasons for their ideas and answers to 64% who agreed that they were an imaginative person.

The second step in data analysis explored the potential difference in scores at time one and time two recorded on the 20-item Trowsdale Index of Confidence in Experiential Learning, together with the control variable (extraversion). The data presented in Table 2 demonstrate a significant increase between time one and time two in scores recorded on the Trowsdale Index of Confidence in Experiential Learning, but no significant difference between time one and time two in scores recorded on the extraversion scale. These findings support the connection between the Trowsdale Index of Confidence in Experiential Learning and the outcomes from The Imagineerium that was designed to enhance student confidence in learning as captured by the four themes of competence, creativity, learning and collaboration. Support for this connection can be interpreted as demonstrating construct validity for the new measure and as evidence for the effectiveness of The Imagineerium.

Table 2.

Change over time in the experiential variable and the control variable.

	Time one		Time two		t	p <
	Mean	SD	Mean	SD	t	p <
Experimental variable
Confidence in learning	69.31	10.97	72.34	10.73	3.49	.001
Control variable
Extraversion	4.44	1.41	4.22	1.58	1.64	NS

Conclusion

Building on the earlier study reported by Trowsdale et al. (2019), the present study was designed to address three specific research objectives. The first research objective concerned designing and examining a composite measure of confidence in learning of no more than 20 items that combined confidence in competence, confidence in collaboration, confidence in creativity, and confidence in learning in a way accessible to 9- to 10-year-old students. A sample of 140 students within this age range responded to 63 items, each rated on a five-point scale. From these 63 items a series of factor analyses identified four sets of five items each that mapped onto the domains of confidence in competence, confidence in collaboration, confidence in creativity, and confidence in learning that coalesced to comprise a unidimensional scale. The psychometric properties of the resulting Trowsdale Index of Confidence in Experiential Learning commend the instrument for further use.

The second research objective concerned employing the Trowsdale Index of Confidence in Experiential Learning to assess the impact of an appropriate educational intervention designed to enhance student confidence in learning. The intervention employed The Imagineerium, a 10-week long programme. Data provided by these 140 9- to 10-year-old students both before and after the educational intervention demonstrated a significant increase in scores recorded on the Trowsdale Index of Confidence in Experiential Learning. This finding is of value for two purposes. On the one hand, the finding offers support for the construct validity of the Trowsdale Index of Confidence in Experiential Learning in that the instrument was sufficiently sensitive to detect the impact of the educational intervention. On the other hand, the finding offers support for the notion that The Imagineerium delivers what it sets out to deliver in the sense of enhancing student confidence in learning.

The third research objective concerned identifying an appropriate control variable that could be integrated into the project to compensate for the impracticability of a control group to be set alongside the experimental group within the resource constraints of the present project. In the absence of a control group the theory is that the effectiveness of the intervention would impact the variable specific to the intervention (namely confidence in learning) but at the same time would not impact the control variable. The control variable incorporated in the present study was the six-item extraversion scale proposed by the abbreviated form of the Junior Eysenck Personality Questionnaire developed by Francis (1996). Data provided by the 140 9- to 10-year-old students, before and after the educational intervention, demonstrate no significant differences between the scores recorded on this scale on the two occasions. This finding suggests that the educational intervention had no effect on this control variable, confirming the stability of this variable among the participants in contrast with the shift taking place in the variable hypothesised to be influenced by the intervention.

Together the present study and the earlier study reported by Trowsdale et al. (2019) provide cumulative evidence that the The Imagineerium is effective in enhancing student confidence in learning, and with different outcome measures employed in the two studies. There are, however, clear limitations with both studies since they were conducted among quite small groups of students: 135 students in the first study and 140 students in the second study. Future research would be advised to involve a larger number of students within the experimental environment and to employ a carefully constructed control group of students, not exposed to the educational intervention. What these two small studies have achieved is the demonstration that The Imagineerium is worth further investment and a larger evaluation study.

By learning from the earlier study reported by Trowsdale et al. (2019), the present study has developed the Trowsdale Index of Confidence in Experiential Learning as a conceptually more robust and empirically more economical instrument well designed for assessing the impact of educational interventions intended to embrace student confidence in learning. Here is an instrument that conceptualises confidence in learning among 9- to 10-year-old students as embracing the four components of confidence in competence, confidence in creativity, confidence in collaboration, and confidence in learning. Here is the image of the ideal maturing learner approaching the upper classes in primary education.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Paul Hamlyn Foundation.

ORCID iDs

Jo Trowsdale

Leslie J Francis

References

Colucci-Gray

Burnard

Cooke

, et al. (2017) Reviewing the Potential and Challenges of Developing STEAM Education through Creative Pedagogies for 21st Century Learning: How Can School Curricula Be Broadened towards a More Responsive, Dynamic and Inclusive Form of Education? London: British Educational Research Association.

Corulla

(1990) A revised version of the psychoticism scale for children. Personality and Individual Differences 11(1): 65–76. DOI: 10.1016/0191-8869(90)90169-R.

Cronbach

(1951) Coefficient alpha and the internal structure of tests. Psychometrika 16: 297–334. DOI: 10.1007/BF02310555.

Davies

Trowsdale

(2017) The value of instability: lessons from reviewing how and why creativity and the arts might interact with STEM education. European Journal of Curriculum Studies 4: 584–600.

Davies

Trowsdale

(2021) The culture of disciplines: reconceptualising multi-subject curricula. British Educational Research Journal 47(5): 1434–1446. DOI: 10.1002/berg.3735.

Dewey

(1938) Experience and Education. New York, NY: Macmillan.

D’Agostino

(2005) Measuring learning outcomes: reliability and validity issues. In: Phye

Robinson

Levin

(eds) Empirical Methods for Evaluating Educational Interventions. Burlington, MA: Elsevier Academic Press, pp. 113–146.

Francis

(1996) The development of an abbreviated form of the revised junior Eysenck personality questionnaire (JEPQR-A) among 13- to 15-year-olds. Personality and Individual Differences 21: 835–844. DOI: 10.1016/S0191-8869(96)00159-6.

Heathcote

Bolton

(1995) Drama for Learning: Dorothy Heathcote’s Mantle of the Expert Approach to Education (Dimensions of Drama). Portsmouth, NH: Heinemann.

10.

Imagineer (2016) Behaving like an Imagineer. Unpublished project document.

11.

Ingold

(2017) Anthropology in/as Education. Abingdon: Routledge.

12.

Lave

Wenger

(1991) Situated Learning: Legitimate Peripheral Participation. Cambridge: Cambridge University Press.

13.

Lucas

Hanson

Claxton

(2014) Thinking like an Engineer: Implications for the Education System. London: Royal Academy of Engineering.

14.

Miettinen

(2000) The concept of experiential learning and John Dewey's theory of reflective thought and action. International Journal of Lifelong Education 19 (1): 54–72.

15.

Trowsdale

(2014) The Imagineerium: Pilot Project (With Schools) Report. Coventry: University of Warwick.

16.

Trowsdale

(2016) Imagineering: Re-creating spaces through art-making. Creativity: Theories, Research, Applications 3: 274–291. DOI: 10.1515/ctra-2016-0018.

17.

Trowsdale

(2020) Art-making as a site for education. PhD Thesis, University of Warwick, Coventry.

18.

Trowsdale

McKenna

Francis

(2019) Evaluating ‘the Imagineerium’: the Trowsdale indices of confidence in competence, creativity and learning (TICCCL). Thinking Skills and Creativity 32: 75–81. DOI: 10.1016/j.tsc.2019.04.001.