Teacher evaluation of the impact of The Imagineerium education project on the creativity of individual students: The Trowsdale Index of teacher observation of student creativity

Abstract

In order to evaluate the impact of The Imagineerium a 10-week educational project, teachers were asked to observe and rate the behaviour of a pilot sample of 135 participating students both at the beginning and at the end of the 10-week period. Scores recorded on the seven-item Trowsdale Index of Teacher Observation of Student Creativity (TITOSC) showed a significant increase between time one and time two. In order to test the reproducibility of these findings the same index was employed a year later in a replication study among 139 students. On this occasion also, scores recorded on the seven-item Trowsdale Index of Teacher Observation of Student Creativity showed a significant increase between time one and time two. These data support the effectiveness of educational experience in enhancing teacher perception of creativity displayed by individual students.

Keywords

teacher observation student creativity learning primary school

Introduction

This paper reports on the development of a new instrument designed to access and assess teacher observation of student creativity. This measure was developed within the context of The Imagineerium, a 10-week distinctive arts and engineering education project designed to stimulate and to enhance student creativity. In order to assess the effectiveness of this project the teachers were asked to record their perceptions of the creativity of the participating students both at the beginning and at the end of the 10-week period. The context for this study is set by a discussion of the policy context within which the project is located, by an introduction to the theoretical framework within which The Imagineerium was set, and by reference to a recent study that demonstrated the impact of the project on the students’ self-assessment of their own creativity. This parallel study is designed to assess whether the students’ self-assessment is reflected in the assessment made by their teachers.

The context

Creativity, although declared a ‘core British value’ (Blair, 2000), has been out of political favour in English schooling for several decades. It has been brought back into focus by the recent proposal by the Organisation for Economic Cooperation and Development (OECD n.d.) to test for creativity alongside english, maths and science in its 2021 PISA tests, as well as by growing critiques of a narrowing of the curriculum (National Association of Head Teachers, 2017; Priestley and Biesta, 2013) and the associated social inequalities of schooling (Kulz, 2017). A global, if not English, interest has developed in claims that creative and cultural learning offer a means to address national economic and technological challenges, as well as social and personal ones (NASUWT, The Teachers Union, 2011; National Advisory Committee on Creative and Cultural Education, 1999). This narrative offers a normative account, characterising young people as emergent creative innovators, developing entrepreneurial mindsets and habits, and thereby contributing to the politically important mission of ‘ensuring national economic competitiveness in the “global race” ’ (Ball, 2017: 16). The Imagineerium is to be seen as both a critical response to, and product of, such a context: seeking as it does to promote a broader conception of creativity within a broad and balanced curriculum, as well as addressing economic concerns.

The prospect of a global creativity test has revived old debates about whether creativity can be evidenced through testing in any meaningful way (Cropley, 2000; Said-Metwaly et al., 2017; Zhao and Meyer, 2013). Critics suggest that the very concept of testing is counterproductive: that the criticality of judging effectively freezes the conditions which enable creative behaviours, that tests focus upon singular dimensions or disallow the significance of context and conditions (Beghetto, 2005; Zhao, 2012). Nonetheless the endeavour continues and several models have gained attention by the OECD, one developed by English researchers (Lucas et al., 2013). The study on which their research draws was one which also informed the development of The Imagineerium. This model relies upon the engagement of the learner in self reporting their progress and thus, whilst valuable in developing reflective habits in relation to learning, it does not employ the received scientific ‘rigour’ of quantitative methods. In this study we sought to design a tool which recognised these tensions, was related to, but sat outside, the educative practice of the project and qualitative data gathering and could thus be distinct.

The Imagineerium

During The Imagineerium, 9- to 10-year-old students work with professional ‘imagineers’ (artists, performers, designers, and engineers) and with their teachers, to imagine, to design, and to create partially working models of a mechanical, moving machine. In the project children’s designs were inspired by a story from local history. Through an imaginative ‘mantle of the expert’-like frame (see Heathcote and Bolton, 1995), adult imagineers induct the children into a community of practice of art-making, where they are encouraged to think and behave ‘like an imagineer’ through art-making activities (Trowsdale, 2020). The curriculum suggested by The Imagineerium, hybridises subjects, with knowledge drawn on and emphasised as necessary to the task. As the task is central, the processes involved in making in the arts (which frequently and eclectically draw from beyond the arts) frame children’s experience. Children act out ideas, imagine themselves into another character and situation. Physical theatre is also used to enable children to experience and understand principles of physical sciences in relation to forces and mechanisms (through construction and movement) using their bodies. They work practically, in groups, manipulating and testing how materials might combine to provide robust structures, effect movement and be aesthetically pleasing. Drawing on their imaginations, shaped by the possibilities of materials, they explore challenges in making, and come to understand what is involved in realising ideas and motion. The emphasis on ‘behaving like an imagineer’ (X, 2016) encapsulated a number of desired dimensions of being creative such as ‘dreaming: letting your imagination fly’, ‘sharing and being open to new ideas’, ‘seeing the good things about “failing” ’, ‘persevering’, ‘taking responsibility’ ‘negotiating’, and ‘supporting others’, many of which are noted in analyses of creativity in engineering (Lucas et al., 2014: 24–29). Idea generation, thinking and understanding are supported by the use of personal sketchbook/journals, by questioning and by talking.

The Imagineerium is designed to stimulate children’s aspirations, to channel their inventiveness, and to feed their appetite for the engineering sciences and arts as related to creative practices and professions. It has been recognised as a particular kind of STEAM education (Colucci-Gray et al., 2017) and reported on elsewhere as a particular, arts-rich, educational experience (Trowsdale, 2016) and ‘practice-based’ curriculum (Davies and Trowsdale, 2017).

Previous research

The Imagineerium was conceived, developed and evaluated within an arts-, practice- and research-based educational environment. In the initial phase of research, qualitative methods (generating data through interviews, questionnaires, and journal entries, completed by students and also by teachers) suggested a number of positive effects on learning (see Trowsdale, 2014, 2016). In particular these data indicated an enhanced sense of the students’ confidence in their capability and appetite for learning. Students referred to the value of their experience of working with others who approached things in different ways. They appeared to benefit from this experience because they ‘get inspired by other people’, ‘everyone’s ideas are helpful because … everyone thinks different’; you ‘learn about each other’, ‘develop your listening’ and together you ‘make better ideas’. In the course of the project the students recognised a growing sense of pride in their own personal learning, with one student saying ‘I’m more proud of what I am doing’. At the same time they also recognised the way in which their fellow students were also growing and developing, with one student saying that the project can help another student to ‘learn what s/he is good at’, that s/he is ‘smarter than s/he thinks’, ‘more clever’, ‘more intelligent’, and is making ‘more progress in learning’. The effects of such improved confidence in their capability and appetite for learning appeared to extend beyond the project itself, with students speaking about maths, football and social situations in which ‘sometimes I wouldn’t really, like, believe in myself, but like now I believe in myself more’. Such improved confidence in their capability and appetite for learning also appeared to generate a sense of resilience, with one student saying ‘I think that now I know more about myself … I can challenge myself to do more things’.

Developing earlier and ongoing qualitative studies, (Trowsdale, McKenna and Francis 2019) reported on a quantitative study designed to assess the impact over the 10 weeks of The Imagineerium on the students’ self-assessment of three outcomes defined as confidence in learning, confidence in creativity, and confidence in competence, as assessed by The Trowsdale Indices of Confidence in Competence, Creativity, and Learning (TICCCL). First, this paper describes the way in which these three measures were constructed and reports on their psychometric properties. Drawing on data provided by the 135 9- to 10-year-old students who participated in the 10-week project and who provided full responses to these measures both at the beginning and at the end of the 10-week period, (Trowsdale, McKenna and Francis 2019) provided the following data regarding internal consistency reliability for these three scales: the 11-item Scale of Confidence in Learning (α = .89), the 14-item Scale of Confidence in Creativity (α = .86), and the 9-item Scale of Confidence in Competence (α = .75).

Second, data provided by these 135 9- to 10-year-old students, both before and after The Imagineerium, demonstrated a significant increase in all three measures of confidence in competence, creativity, and learning over the course of the project. Since constraints on the resources available for this study did not allow for a control group, a set of control variables were employed in the design. In this usage of the term, control variables refer to measures that were completed at the same time as the measures hypothesised as reflecting the intended outcomes of the project, but which were hypothesised as unrelated to the outcomes intended by it. The control variables incorporated in the study were the extraversion, neuroticism, and psychoticism scales proposed by the abbreviated form of the Junior Eysenck Personality Questionnaire Revised developed by Francis (1996). Data provided by the 135 9- to 10-year-old students, before and after the project demonstrates no significant difference between the scores recorded on these three scales on the two occasions. These finding suggest that the project had no effect on these three variables, confirming stability among the participants on these variables in contrast with the shifts taking place in the three variables hypothesised to be influenced by the project, namely confidence in learning, confidence in creativity, and confidence in competence.

Research question

Among the three outcome measures completed by the 9- to 10-year-old students in the study reported by (Trowsdale, McKenna and Francis 2019) the key outcome from the perspective of The Imagineerium was that of creativity. Against this background, the aim of the present study is to explore whether the teachers involved in the project observed similar increase in student creativity to that observed by the students themselves. The aim addresses the weakness inherent in all self-assessment measures, namely that of subjectivity.

This broad aim, however, involved three sequential steps. The first step involved the development of a reliable measure of teacher observation of student creativity. The second step involved comparing scores on this measure both before and after the project. The third step involved repeating the project for a second group of students a year later.

Research context

The Imagineerium educational intervention project was located in a Midlands city ranked as the eighty-first most deprived area in England, with one in five children living in relative poverty. Within this city 12 of the 86 primary schools self-selected to participate in The Imagineerium project. Within the 12 schools involved in the project, 9 were sited in wards rated as the 10% or 20% most deprived in the country. The city also reflects a diverse ethnicity, and students attending these schools reflect a high number of ethnic minority students, for many of whom English is an additional language.

Self-selection to participate in The Imagineerium committed schools to involvement in research and evaluation to the benefit of the project. Within the participating schools teachers were selected by the headteachers of the schools. Over the two years of data collection, ten qualified and experienced year five class teachers were involved, 3 male and 7 female, each with between two and 25 years teaching experience. These year five class teachers were trained and well versed in judging student activity and behaviour, and doing so in relation to applying externally defined markers, such as the behaviours listed in the Trowsdale Inventory of Student Behaviour. Teacher ratings were monitored and overseen by the project coordinator (the corresponding author). The students involved in the project comprised all the students in the ten participating teachers’ year-five classes.

The Trowsdale Inventory of Student Behaviour comprised 18 short statements concerning different aspects of student behaviour. While a core of items was designed to tap student creativity (eg, Contributed a lot of good ideas), other items covered areas such as enjoyment (eg, Enjoyed most learning activities), self-esteem (eg, Appeared proud of self as a learner), hard work (eg, Worked hard to develop new skills), and collaboration (eg, Was a reliable team player). In the design stages of this inventory, teachers were consulted regarding clear wording of behaviours, and consequently teachers had either been part of the development of a clear definition or were given clear definitions of terms during a training session for teachers as part of the project, led by the project coordinator (the corresponding author). Guidance was also summarised in a written form.

Method

Procedure

Teachers were invited to observe individual students and to rate their participation in the project twice, once at the beginning and again at the end. The project and the assessment were conducted on two occasions in two successive years, each involving five schools. The project and the assessment were conducted in accordance with the ethical procedures of the University and with parental consent.

Participants

In the first year, class teachers within the five participating schools provided ratings on the 135 students participating in The Imagineerium. In the second year, 139 students participated in the project. The number of participants was determined by the resources available for the project.

Instrument

The Trowsdale Inventory of Student Behaviour invited teachers to rate 18 behavioural items on a five-point scale of frequency: never (1), seldom (2), sometimes (3), often (4), and always (5).

Analysis

The data were analysed by the SPSS package utilising the frequencies, correlations, factor, reliability, and paired t-test routines.

Results and discussion

The first step in the data analysis employed factor analysis to explore the dimensionality of the 18 behavioural items. On the basis of this factor analysis a group of seven items was identified to comprise the Trowsdale Index of Teacher Observation of Student Creativity (TITOSC). The scale properties of these items are presented in Table 1 in terms of the alpha coefficient (Cronbach, 1951), the correlation between each individual item and the sum of the other six items, and the item endorsement in terms of the sum of the ‘often’ and ‘always’ responses. These data demonstrate a good internal consistency reliability (α = .93). The correlations between the individual items and the sum of the other six items (ranging from .68 to .88) demonstrate that each item is playing a coherent part in the homogeneous scale. The item endorsements (ranging from 15% to 37%) demonstrate a fair spread of item discrimination. Among the seven items for this index, the item that recorded the highest correlation with the sum of the other six items was ‘Came up with different ideas from others’. This item suggests a high level of face validity for the scale. The other items that loaded on this factor all demonstrated aspects of creativity. The items that did not load on this factor related to the other aspects of student behaviour included on the Trowsdale Inventory of Student Behaviour, including such areas as enjoyment, self-esteem, hard work, and collaboration.

Table 1.

Trowsdale Index of Teacher Observation of Student Creativity (TITOSC): scale properties.

	r	%
Was confident in what he/she did	.68	37
Contributed lots of ideas	.82	32
Came up with different ideas from others	.88	25
Developed/elaborated on ideas	.83	21
Asked lots of questions; wanted to know more	.80	31
Tried out/experimented with ideas	.80	15
Had a go/took risks	.72	21
alpha	.93

Note: % = proportion of students rated ‘often’ or ‘always’.

r = correlation between individual item and sum of other six items.

The second step in the data analysis compared the mean scale scores recorded at time one and at time two in the first study among 135 students (see Table 2). The t-test demonstrates a significant increase in the mean scale score between time one and time two (p < .001). This finding supports teacher perception of the effectiveness of The Imagineerium in terms of increase in teacher observation of student creativity.

Table 2.

Change over time.

		Time one		Time two		t	p <
	N	Mean	SD	Mean	SD
Sample 1	135	20.57	5.24	23.50	6.29	6.87	.001
Sample 2	139	22.51	5.22	23.96	4.95	4.00	.001

The third step in the data analysis turned attention to the study conducted in the second year among 139 students. On this occasion the seven-item Trowsdale Index of Teacher Observation of Student Creativity recorded an alpha coefficient of .93 (Cronbach, 1951). Again the t-test demonstrated a significant increase in the mean scale score between time one and time two (p < .001), replicating the outcome of the initial study.

Conclusion

Against the background of The Imagineerium this paper set out to address two related and sequential research questions. The first research question concerned exploring and examining the design and development of a measure relevant for assessing teacher observation of student creativity. Drawing on data provided by teachers on assessing creativity of 135 9- to 10-year-old students, this study identified seven items from an initial pool of 18 items that cohered to produce a scale recording an alpha coefficient of .93, demonstrating a high level of internal consistency reliability. Among these seven items, the item that recorded the highest correlation with the sum of the other six items was ‘Came up with different ideas from others’. This item suggests a high level of face validity for the scale.

The second research question concerned comparing scores on this measure recorded before and after the project. The data provided by teachers in the initial study on assessing creativity of 135 9- to 10-year-old students both at the beginning and 10-weeks later at the end of The Imagineerium demonstrated a significant increase in the scores recorded on the seven-item scale. The data provided by teachers in the replication study among 139 9- to 10-year-old students confirmed the findings from the initial study. These findings support the value of the project, in the sense that, according to teacher assessment the students were observed to be more creative at the end of the project than at the beginning. These findings also suggest the construct validity of the seven-item measure of teacher observation of student creativity, in the sense that the educational intervention was hypothesised to enhance student creativity.

Two conclusions can be drawn from these findings. The first conclusion is that the new seven-item measure styled the Trowsdale Index of Teacher Observation of Student Creativity (TITOSC) can be commended for further application and further scrutiny, having in this study recorded satisfactory levels of internal consistency reliability, face validity and construct validity. The second conclusion is that the scores recorded on this measure of teacher observation of student creativity, together with the scores of student self-assessment of confidence in competency, creativity and learning reported by (Trowsdale, McKenna and Francis 2019) supports the effectiveness of The Imagineerium in achieving its objective in enhancing student creativity.

There are clear limitations with a pilot study of this nature that can be addressed in future research, involving a larger number of students within the experimental environment. What this initial study has achieved is that both the educational intervention and the newly designed Trowsdale Index of Teacher Observation of Student Creativity are worth further investment.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Paul Hamlyn Foundation.

ORCID iD

Jo Trowsdale

References

Ball

(2017) The Education Debate. Bristol: Policy Press.

Blair

(2000) Tony Blair’s Britain speech. The Guardian, 28 March. Available at: https://www.theguardian.com/uk/2000/mar/28/britishidentity.tonyblair

Beghetto

(2005) Does assessment kill student creativity? The Educational Forum 69(3): 254–263.

Colucci-Gray

Burnard

Cooke

, et al. (2017) BERA Research Commission Report. Reviewing the Potential and Challenges of Developing STEAM Education through Creative Pedagogies for 21st Century Learning: How Can School Curricula Be Broadened towards a More Responsive, Dynamic and Inclusive Form of Education? London: British Educational Research Association.

Cronbach

(1951) Coefficient alpha and the internal structure of tests. Psychometrika 16(3): 297–334.

Cropley

(2000) Creativity in Education and Learning. London: Routledge.

Davies

Trowsdale

(2017) The value of instability: Lessons from reviewing how and why creativity and the arts might interact with STEM education. European Journal of Curriculum Studies 4: 584–600.

Francis LJ (1996) The development of an abbreviated form of the Revised Junior Eysenck Personality Questionnaire (JEPQR-A) among 13- to 15-year-olds. Personality and Individual Differences 21: 835–844. DOI:10.1016/S0191-8869(96)00159-6.

Heathcote

Bolton

(1995) Drama for Learning: Dorothy Heathcote’s Mantle of the Expert Approach to Education (Dimensions of Drama). Portsmouth, NH: Heinemann.

10.

Imagineer (2016) Behaving like an imagineer. Unpublished project document.

11.

Kulz

(2017) Factories for Learning: Making Race, Class and Inequality in the Neoliberal Academy. Manchester: Manchester University Press.

12.

Lucas

Claxton

Spencer

(2013) Progression in student creativity in school: First steps towards new forms of formative assessments. OECD education working papers, No. 86, OECD Publishing.

13.

Lucas

Hanson

Claxton

(2014) Thinking like an Engineer: Implications for the Education System. London: Royal Academy of Engineering.

14.

NASUWT, The Teachers Union (2011) Creativity and the arts in the curriculum: A report of policies and practices in England, Northern Ireland, Scotland and Wales. Available at: www.nasuwt.org.uk/uploads/assets/uploaded/3535be2c-801c-46cb-b4410810472b52a3.pdf (accessed 4 July 2020).

15.

National Advisory Committee on Creative and Cultural Education (1999) All Our Futures: Creativity, Culture and Education Report. London: Department for Education and Employment.

16.

National Association of Head Teachers (2017) Schools are ‘narrowing’ the curriculum, say Ofsted. Available at: www.naht.org.uk/news-and-opinion/news/curriculum-and-assessment-news/schools-are-narrowing-the-curriculum-says-ofsted/ (accessed 4 July 2020).

17.

Organisation for Economic Cooperation and Development (OECD n.d.) About PISA. Available at: www.oecd.org/pisa/aboutpisa/ (accessed 4 July 2020).

18.

Priestley

Biesta

(eds) (2013) Reinventing the Curriculum: New Trends in Curriculum Policy and Practice. London: Bloomsbury.

19.

Said-Metwaly

Kyndt

den Noortgate

(2017) Approaches to measuring creativity: A systematic literature review. Creativity! Theories – Research – Applications 4(2): 238–275.

20.

Trowsdale J (2014) The Imagineerium – pilot project (with schools) report. Coventry: University of Warwick.

21.

Trowsdale J (2016) Imagineering: Re-creating spaces through art-making. Creativity! Theories – Research – Applications 3: 274–291. DOI: 10.1515/ctra-2016-0018.

22.

Trowsdale J (2020) Art-making as a site for education. PhD thesis. University of Warwick, UK.

23.

Trowsdale J McKenna U and Francis LJ (2019) Evaluating ‘The Imagineerium’: The Trowsdale indices of confidence in competence, creativity and learning (TICCCL). Thinking Skills and Creativity 32: 75–81. DOI: 10.1016/j.tsc.2019.04.001.

24.

Zhao

(2012) Doublethink: Creativity, entrepreneurship and standardized tests. Education Week, 17 July. Available at: www.edweek.org/ew/articles/2012/07/18/36zhao_ep.h31.html (accessed 9 July 2020).

25.

Zhao

Meyer

(2013) High on PISA low on entrepreneurship? What PISA does not measure. In: Meyer

Benavot

(eds) PISA, Power, and Policy: The Emergence of Global Educational Governance. Didcot: Symposium Books Ltd., pp.267–278.