Making judgements about students making work: Lecturers’ assessment practices in art and design

Abstract

This research study explores the assessment practices in two higher education art and design departments. The key aim of this research was to explore art and design studio assessment practices as lived and experienced by art and design lecturers. This work draws on two bodies of pre-existing research. Firstly this study adopted methodological approaches that have been employed to good effect to explore assessment in text based subjects (think aloud) and moderation mark agreement (observation). Secondly the study builds on existing research into the assessment of creative practice. By applying thinking aloud methodologies in a creative practice assessment context the authors seek to illuminate the ‘in practice’, rather than espoused, assessment approaches adopted. The analysis suggests that lecturers in the study employed three macro conceptions of quality to support the judgement process, namely the demonstration of significant learning over time, the demonstration of effective studentship and the presentation of meaningful art/design work.

Keywords

art and design assessment Crit higher education judgement think aloud protocol

Introduction: Studio-based pedagogy

The studio is at the heart of art and design education in higher education.

Typically undergraduate art and design students are set open-ended project briefs that direct the development of their individual art/design practice. This project-centered learning approach is a defining element of studio based pedagogy. Students interpret the project brief and carry out research and development activity that culminates in the submission of a body of work. This pedagogy can be characterized as ‘self directed … and very free form curriculum’ (Svensson and Edstrom, 2011: 1). At different points over the course of study there will be studio Crits. The term ‘Crit’ describes studio-based formative or summative feedback in small or large group situations with students and lecturers (Blythman et al., 2008). These Crits offer an opportunity for staff and students to discuss and evaluate the work.

There is an increased interest in pedagogic research that is premised on the need to understand ‘disciplinary specific ways of thinking and practicing in higher education’ (Harman and McDowell, 2011: 42). One result of this is an increased interest in studio-based pedagogy. This research aims to develop a deeper understanding of the elements of studio-based teaching and learning. Thus Blair (2006) explores the strengths and weaknesses of the Crit as a key site for studio feedback and Blair et al. (2008) offer an overview of creative studio pedagogy which they build on to propose a new model of creative pedagogy premised on interdisciplinary ‘innovative connectivity’ (2008: 68). This approach, they argue, is better suited to the twenty-first-century creative arena. Shreeve et al. (2008) explore the particularities of the discipline-specific aspects of teaching and learning in the visual arts. For them these elements include authentic learning activities, aligned curricula, practice-based work, team-based events, opportunities for skills development, the use of sketchbooks and formative feedback.

In a literature review of educational journal articles that focused on the studio, de la Harpe et al. (2009) located 118 articles that fitted the parameters of the study, namely that the article had as its focus art, design or architecture studio practice. They used the resultant sample of articles to analyse the particularities of studio-based assessment practices in art, design and architecture, and identified 11 key indicators that underpin holistic assessment in the studio. These were: product; process; person; content knowledge; hard skills; soft skills; technology; learning approaches; reflective practice; professional and innovative practice; and interdisciplinary collaboration. The key conclusion of this study was that a focus on process (as well as the product) and the person (i.e. the student) is central to studio assessment. The research reported here builds on this literature review to explore the particularities of art and design assessment in two university contexts. All university assessment practices are situated within the broader policy contexts of higher education.

In the UK, the national organization that assures standards and quality in higher education, the Quality Assurance Agency (QAA), sets out expectations for higher education curriculum and assessment policies and practices. The QAA states that ‘Good programme design creates programmes that facilitate the delivery of the intended learning outcomes’ (Quality Assurance Agency, 2006b: 9). In addition, in its guidance on assessment it states that institutions need to provide for the ‘effective and appropriate measurement of students’ achievement of intended learning outcomes’ (Quality Assurance Agency, 2006a: 31). When new programmes of study are developed and validated, the documentation will set out lists of learning outcomes for each level of study. In the field of creative practice the role of predetermined written learning outcomes and assessment criteria are problematized. Orr (2007), Cannatella (2001) and Gordon (2004) argue that learning outcomes are an inappropriately prescriptive approach to a subject that needs to allow for creativity and innovation in ways that are not easily captured. Orr (2010b) interviewed fine art lecturers about their assessment practices and she notes that lecturers want students’ creative work to have an unnamable quality of creativity which one fine art lecturer calls ‘zing’. Orr concludes that it is hard to reduce this elusive creativity to a written learning outcome. In the words of Fryer (2010), written in the context of performing arts, ‘prescriptive assessment criteria may seem to fit particularly awkwardly with creative work which is inevitably about outcomes that cannot be predicted in advance’ (2010: 549). He goes on to ask:

How might I deal with my own ‘cultural arbitrary’ whilst assessing practical work when students create moments that have a sense of something I find difficult to understand or measure. These are the moments I treasure as a teacher, the moments when I know the students may have found something really exciting, and yet they are the moments when my assessment criteria seem redundant. (Fryer, 2010: 548)

Svensson and Edstrom (2011) study the role of studio conversation and its relation to artwork in the context of students studying fine art. Their study underlines the centrality of language in art and design assessment. Within a Crit the judgements made about student artwork are communicated through language. Elkins (2001) talks about the importance of conversation in art and design education. In a chapter on the Crit, in a book with the teasing title of Why Art Cannot Be Taught, Elkins talks about the ‘ultimate terms of art critiques: they are the final goals, the ideals, the ultimate terms of praise’ (2001: 113). Elkins offers the examples of the following terms that might be used to evaluate student work in a Crit: ‘Interesting, powerful, moving, strong, compelling’ (2001: 113). He calls these words ‘rhetorical criteria’ because the words used do not tell you what the work looks like, but the words do tell those who are used to reading them how the work can make you feel. Thus we cannot evoke the specifics of the artwork by looking at the ‘rhetorical criteria’ used.

The role of language in art and design education is foregrounded in Shreeve et al.’s (2010) study of art and design signature pedagogies, which characterizes art and design teaching and learning as ‘a kind of exchange’. This is interesting given student work is premised on the non-verbal artefact, but, according to the authors, a central ‘truth’ of the discipline is in the exchange about the artefact. In a micro study of the language of a Crit in design, Mitchell (1998: 30) discusses the ways that students are prepared to enter the ‘Discourse of design’. For Mitchell, ‘the reality of the design … exists in and through the exchange’ (1998: 32). What this means is that the design making itself may be non-verbal but the meaning making that surrounds this design making resides in language. Mitchell uses this case study to explore the relationship between verbalizing and non-verbal making and she comments on the role of language in the assessment of art and design: ‘Language is playing a constructive as much as a representational role. It is being used to actively create and clarify the design’ (1998: 31). This is a view of language as constitutive rather than representational; ‘it is as though the word as it is transacted between speakers both allows for shared meanings and yet remains to be interpreted’ (1998: 32). The constitutive role of language means that ‘talk can bring objects into being and allow them to be continually made and remade’ (1998: 34). Mitchell concludes her paper with the claim that ‘the work only begins to be art when talk breaks out’ (1998: 34). These studies suggest that an analysis of assessment talk will offer useful insight into art and design approaches to assessment. Using assessment talk as research data is a particularly effective way to explicate assessment judgement.

Aim of study

Grainger et al. (2008: 133) comment that ‘decisions by markers about quality in student work remain confusing to most students and markers’. As de la Harpe et al.’s (2009) literature review evidences, there is a small body of research that looks at studio-based assessment. Nevertheless de la Harpe et al. cite Ehmann (2005: 107), who points out that assessment is a ‘somewhat neglected area of design education’. In creative disciplines it is common for lecturers to talk about using holistic assessment approaches (de la Harpe et al., 2009). In the words of one lecturer in Orr (2010a) ‘Learning outcomes are useful, but the assessment process is looking holistically’. This study investigates the holistic assessment approaches adopted in two UK-based university art and design communities. It draws on and links to two bodies of research by the authors. Firstly it is part of a wider study that is exploring assessment practices in a range of disciplines (Bloxham et al., 2011). Secondly it builds on previous work carried out by one of the authors specifically in the area of art and design assessment (Orr, 2010a; 2011a, b). This study aims to examine the indicators of quality used to make judgments about student work; its focus is on the process of judgement making in the studio. Typically, the process of judgement is ephemeral. When a mark is put in a box and feedback is written the process of assessing the quality of student work disappears. It does not leave a trace (Brooks, 2009). An interest in assessment process is evident in Crisp’s (2008) study of examiner thinking during the process of examination marking and in van der Schaff et al.’s study into ‘the role of assessment criteria during teacher’s collaborative judgement processes of students’ portfolios’ (2011: 1). The aim of the study reported here is to bring out, record and analyse the process of judgement making to establish the key indicators of quality that are employed by a group of art and design lecturers in their marking practice.

The challenges of researching the process of studio assessment judgement making

Art and design lecturers’ studio-based assessment practices are a key site for carrying out this grounded, bottom-up analysis of lecturer judgement. Two post-1992 English university departments were selected and the art and design course teams in each university were invited to be involved in the project. The resulting opportunity sample consisted of ten lecturers who taught Fine Art, Graphic Design or Product Design. Ethical approval was sought and agreed for the study.

Brooks points out that assessment judgement is ‘elusive’ and notoriously difficult to study. ‘The choice of research methods is a particularly important consideration because judgment is a tacit process which leaves no trace of its working under normal circumstances’ (2009: 2). To understand the methodological approach adopted it is important to locate the study’s epistemology. Methodology and epistemology are intertwined (Orr, 2007) and the study reported here situates itself within an interpretivist paradigm.

The key objective was to research assessment as it is played out in naturalistic and situated studio-based contexts. As a result we rejected the use of (quasi-) experimental approaches where assessments are set up for the sole purpose of research. Grainger et al. (2008) adopted a quasi-experimental approach in a study where they asked multiple markers to assess the same pieces of student work. In their study the data consisted of the transcripts of the ‘professional conversations’ about the work. Van der Schaff et al. (2011), Amabile (1996) and Suto and Greatorex (2008) have also carried out quasi-experimental studies. These studies have utility and their findings enhance the literature base, but for the purposes of this study more naturalistic methods were judged appropriate. The study reported on here does not seek to judge the judgers, rather it seeks to investigate the judgements that lecturers made in situ without preconceptions about how it should be done. For this reason, the study is not concerned with pre-determined indicators of quality and therefore the written learning outcomes and assessment criteria (referred to by Bloxham et al. (2011) as assessment artefacts) were not viewed as part of the research. The researchers did not have pre-existing expectations about what the indicators of quality might be or should be. (See Bloxham et al. (2011) for a fuller discussion of the role of written assessment artefacts in the assessment.)

Seeking to study assessment in a naturalistic setting meant that we needed firstly to understand how assessment was managed locally and then to design a methodical approach that matched the local practices. Previous studies (Orr, 2007) underlined that each university has its own ‘shared repertoires’ (Wenger, 2004) in relation to assessment, i.e. locally understood approaches and practices. For example, in art and design, lecturers may assess student work alone and/or in groups and they may assess work in the presence or absence of the students. At the onset of the research project the participating lecturers were asked about the assessment approaches adopted in their departmental context. It was noted that the approaches to assessment in the two institutions were very different, and this led to the adoption of slightly different data collection approaches, each aligned to the different departmental marking contexts.

As a consequence of this variety the aim of this study was to study the assessment practices as they occurred, rather than artificially to establish consistent approaches for the purposes of research.

In one university (Department A) we noted that the student work was marked by colleagues working alone who then met to discuss and agree the final mark. In these cases the lecturers were asked to think aloud and audio-record their marking while they were alone in the studio. The subsequent moderation meeting was also audio-recorded. In this department students were not invited back into the marking context so no student feedback meetings were recorded as part of the study. The data collection method used was think aloud protocol (see below) and audio-taping of conversation.

In Department B the first and second marking took place as part of a conversation between two or three lecturers. This meant that student work was not assessed individually by lecturers. As soon as the work had been assessed by the lecturers the students were invited back into the space to receive feedback. In this department the data consisted of recordings of these conversations. The authors considered videoing the studio assessment so that visual analysis could be carried out but were not able to identify a way of doing this that secured student anonymity.

We selected these approaches to data collection (as opposed to carrying out interviews) because of a concern emerging from work by Orrell (2004) where she identified disjuncture between the ways lecturers talked about their assessment practices and the ways they actually carried out assessment. This finding reflected the mismatch between ‘espoused’ and ‘in use’ pedagogic practice. We wanted to identify ‘in use’ practice, and therefore first-hand recording of assessment in action in the naturalistic context was judged preferable to lecturers reporting secondhand what they do.

An earlier study carried out by the authors explored the judgement making associated with the assessment of essays in the context of the humanities and education (Bloxham et al., 2011). In this study all marking was a solitary activity where lecturers sat alone marking their essays. In the current study, think aloud protocols were adopted as the data collection approach. The method has been used in other essay marking contexts; see for example Orrell (2004) and Suto and Greatorex (2008). In think aloud methodology, also known as Verbal Protocol Analysis (Suto and Greatorex, 2008), participants are asked to verbalize the thinking process while they are carrying out a task, in this case the task of assessing students’ work. As alluded to above, this methodology can be used in an experimental situation (Suto and Greatorex, 2008) or in a live assessment context (Bloxham et al., 2011). This approach to data collection limits the participants’ reliance on memory (Suto and Greatorex, 2008) and the findings are less likely to be manipulated reports of marking practice. For a fuller exploration of the merits of using ‘think aloud’ protocols see Bloxham et al. (2011). Think aloud methodology commonly situates itself within a cognitivist paradigm but it is being reframed for use within this social practices and interpretive study.

Using recorded conversations

Creating data that can be analysed by recording conversations is not common in assessment research, but studies that do exist suggest that this is a useful way to access and analyse assessment practices. For example, van der Schaff et al. (2011), Orr (2007) and Grainger et al. (2008) have all adopted this approach and a key benefit is its implicit naturalness. In the study reported here the conversations were happening in any case, which contrasts with the Grainger study where the conversations are elicited by the researcher through the creation of an assessment scenario.

In both universities, the recording of marking was supplemented by semi-structured interviews with the lecturers which focused on staff views about marking and their espoused behavior in that regard. Field notes were also recorded by the research assistant in the case of the paired or group marking and included information regarding non-verbal behaviour and use of documentation. All recordings were transcribed and uploaded into the qualitative analysis application Atlas ti for analysis. The recording of marking in action, which is the prime source of data for this study, amounted to approximately 7 hours of material.

Analysis

The recordings were analysed using a qualitative thematic method (Ritchie and Lewis, 2003) and a grounded approach to seeking meaning from the data (Charmaz, 2006). A qualitative, grounded methodology, avoiding preconceived categories, is demanded by the nature of the data and the purpose of the research in terms of discovering new ideas, creating a rich understanding of the topic and gaining in-depth information (Norton, 2009). The aim was to identify the application of common judgemental elements that enabled us to identify the macro-qualities that lecturers look for in student work. As befits a study of this type we view data analysis as craft-based artistry (Wengraf, 2001), acknowledging the subjective role of the researcher in analysing and interpreting the data.

The analysis was carried out in three stages: coding the data, developing emergent categories from groups of codes and, finally, using these categories to determine key themes regarding lecturers’ perceptions of quality. This study was the second stage of a larger project where similar data collection methods had been used for recording individuals’ marking in other subject disciplines. The codes used for that analysis were derived by three researchers independently immersing themselves in the transcripts in order to generate potential codes. Discussion and testing of these codes resulted in a coding framework which was also used to code the art and design lecturers’ recordings. Two researchers immersed themselves independently in the latter recordings in order to test the appropriateness of this coding framework and this led to several additional codes which emerged from the specific nature of art and design marking practices.

Coding was carried out independently by two researchers using this coding framework and disagreements were discussed to ensure consistent interpretation of the codes. The process of identifying the emerging categories was necessarily more interpretive and involved discussion in the research team to identify the meaningful connections between codes which led to the key themes set out in the findings below. These themes represent categories which were frequently repeated in the data but are not presented as exclusive or as definitive ‘proof’ of marker standards. Instead, they are rendered visible through this methodology as a prompt for further research and debate in the field of art and design assessment. In order to test these themes as a holistic representation of the data, the researchers returned frequently to the complete individual transcripts to avoid losing sight of these as unique representations of an individual marker or marking episode.

Findings

The analysis led to the identification of three interrelated themes relating to conceptions of quality as used by lecturers in judging student work.

These are:

The demonstration of significant learning over time;

The demonstration of effective studentship;

The presentation of meaningful art/design work

The demonstration of significant learning over time

The lecturers’ understanding of, and interest in, the students’ individual learning journeys were evident in this study. Lecturers commended and gave value to student work that explicated the student’s learning process and progress. Thus they responded positively to student work that demonstrated a progression and learning over time. Our analysis suggests that the lecturers are looking at the final summative project as a means to tell a story about the arc of learning. One might describe this as the narrative of the learning. This narrative may relate to the student’s overall progress and learning, as in the extract below in which a lecturer comments to his colleague:

As a student, you could say [she] is almost kind of a model student where she starts weaker and you can really see now where she’s come on.

In some cases the learning progression relates more specifically to a particular area, as in the following extract that relates to technical skill development:

You can see a progression in the drawing style from the beginning.

Or a combination:

He’s erm, he’s got a good balance of some mathematics, technical layouts, he’s obviously learnt quite a few things in order to come up with erm, a proposal. Then here, there’s a really fantastic sheet of er, six or seven images, which shows the basic concept being formatively trialled. And then, er, some testing. So … that’s really good to see. And some really superb little models.

In the extract below two lecturers discuss the ways that a student’s work evidences transformatory learning:

L1: I think you know he’s obviously experimenting with different types of media and trying to come into his own, umm …

L2: Yeah, he’s making some breakthrough there for himself isn’t he?

In art and design education it is common to discuss the balance between process (art making) and product (the art) (Cowdray and de Graff, 2005). The extracts above suggest that this may be a false binary and that the lecturers are looking for and rewarding traces of the process in the product. A high product mark is actually a high process grade because the product is the sum total of the journey of learning. Thus perceptions of the product appear to be influenced by the judgements of the process. The student submission is a body of work that seamlessly encompasses the development and realization of a particular project.

In the extracts above, the lecturers ‘see’ the learning journey represented in the work itself. The learning process is manifest in the submitted work. However, some evaluative comments about students’ learning relate more to the prior knowledge the lecturers have about the students, linked to studio based learning that precedes the assignment submission. The lecturers in this study reveal a keen interest in, and assign value to, the students’ learning prior to assessment submission. While assessing the work, the lecturers talk about the students’ personalities and their assumed approaches to study and practice. This knowledge is drawn from their encounters with these students in the studio. This leads on to the second indicator of quality which explores the students’ approaches to study.

The demonstration of effective studentship

Analysing the assessment transcripts we are able to build a rich picture of the approaches to studentship that are viewed as indicators of quality. The lecturers’ assessment talk repeatedly slides between the student work and the students as creators. In the extract below the lecturer looks at the student work and he speculates about the student’s approaches to study. The work appears to offer the lecturer clues about its creation, or indeed clues about the creator:

I would say, on the basis of the amount of work that she’s done, she’s tirelessly experimenting with the medium that she’s using.

In this extract the lecturer comments on the amount of work and the experimentation of the work, but essentially our key point is that this is a comment about the student rather than the work. This leads us to an exploration of the ways that lecturers talk about the students as part of the assessment conversation. In the transcripts there were a number of references to the students’ approaches to studying and making. In some cases these are what might be described as ‘throw away comments’ about the student. These comments do not appear to bear on assessment but are offered as contextual comment in passing. In the extract below the lecturer makes a comment about a particular student, but this is not an indicator of quality.

It’s an unresolved submission, which is a shame because she is a nice student.

In the next extract the lecturer comments more specifically on the student’s struggle with the work. To some extent the work appears to be ‘telling’ the lecturer that the student has struggled:

So, yeah, no spatial depth, not good palette at all, umm drawing sadly lacking, even colour mixing sadly lacking, so we have a rather jumbled submission that ummm, I think it is clear she was struggling.

Again in the extract below the lecturer attempts to deduce the approach adopted in the work’s creation:

To my eyes they are paintings that have been made in quick succession.

The lecturers’ prior knowledge of the students emerges in the following extract where the lecturer’s comments draw on her wider knowledge of working with a particular student:

She has a fascinating sensibility and sense of colour running through all her work … She is comfortable with drawing.

The artwork offers a view of its maker. Thus the student work continually points back to its maker. The presence of the student is underlined by the comment made by one lecturer when he viewed a portfolio:

What I am looking at now is someone who can not only, you know creatively, erm interact with the assignment and I am now looking at the final presentational drawings which show some real control and some emerging understanding about how to present your ideas graphically with a convincing treatment.

In Department A one lecturer informed a colleague that a particular student ‘has done some good studenting’. This illuminating phrase offers an insight into the ways that lecturers comment on, evaluate and assess the study approaches adopted by the students. The lecturers are using assessment as a site to comment on the ways that students make work. This suggests that assessment concerns itself directly with tracing the development of the student artist or the student designer. From this perspective the student work produced represents a proxy that allows the lecturers to make judgements about the development of the artist/designer. The extract below underlines that the student is responding to feedback given in tutorials, that her drawing skills are improving as a result and that she has a good study ethic. This is evidence of good ‘studenting’, which is an indicator of quality:

The [drawings] do look quite fluid and her presentational drawings are much, much better. She has grown from the tutorials that she was given within class and her final drawings are much more what would be acceptable to see at this level. Overall a very conscientious student, quite well presented, quite well laid out. I would like to have seen a lot more adventure, a few more risk taking, but as I said, one comes back to the fact that she’s a first year.

As the extract above indicates, lecturers look favourably on students who listen to and act on feedback given:

I think he has definitely taken on board, umm, the constructive advice whilst on the module.

The lecturers speculate about students’ approaches to study and they discuss whether or not the students have listened to and acted upon the feedback given. At one level this is to be expected. Students come to university to learn and one key demonstration of learning is the ability to elicit and act upon feedback given. However, in the context of art and design this is more complicated. There is a paradoxical discourse in relation to the ways that students respond to their lecturers’ tutorial advice. In this study students are again and again complimented or admonished for listening to/or ignoring the lecturers’ tutorial advice or Crit feedback. The extract below suggests that, to some extent, the students who take on feedback affirm the lecturers’ approach to teaching:

L1: Do you remember those early weeks where we were talking about [how] you use the page and stuff?

L2: Yeah.

L1: She’s really taken that on.

To conclude this section, in the context of art education Elkins usefully reminds us that ‘an artwork implies a creator’ (Elkins, 2001: 159). There are certain approaches to making that, if adopted by the students, are viewed as indicators of quality.

The presentation of meaningful art/design work

These drawings … they’re speaking to you.

If grading student work is a form of problem solving for lecturers, it is interesting to explore the ways that some submissions make the lecturers’ job easier and some harder. This appears to be a determinant of marks awarded. The lecturers reward work that ‘speaks’ to them. Communicative work is an indicator of quality. The lecturers want the student work to present meaning to them. When this happens the work is rewarded. The team do not want to have to deduce the meaning or speculate about the meaning. For the lecturers in this sample, the student work needs to be articulate about the student intention:

I think to be honest you can see the way he is thinking.

I can see the way the student is thinking through all the aspects.

Immediately [the work] speaks to me.

You can see exactly … the realistic point she is [making].

In all of the four extracts above the lecturer is confident that they are deducing the intended meaning of the work.

In some cases the student work communicates the student’s development of ideas:

Now moving through the portfolio and we’re starting to see some sketches. So these are initial thoughts, rough sketches but look well, communicate well, nice initial ideas.

From this perspective the work is attempting to present an argument to the marking team that the markers understand, as in the extract above, or not, as in this oral feedback to a student:

From reading your rationale we know that there is a very good idea, if not several very good ideas within your project, but we’re not convinced that it comes through in the final piece to somebody who does not have your rationale to read, so therefore is that communication very clear to someone who just picks that up?

In cases where the lecturers express the view that they have to deduce the meaning or work things out themselves, this is seen as a negative indicator of quality. In the extracts below the lecturers are left puzzled as to the works’ meaning or the students’ intention:

Not too sure what this represents … so I’m a little confused …

So [the work is] quite puzzling … paintings kind of slightly random … which is odd.

If these […] were lit … they’d speak to me more.

It could use more narrative.

The extract below suggests that lecturers want meaning delivered clearly:

He’s not telling you much, he’s depending on people’s interpretation … you’re second guessing.

This need for the work to communicate effectively links directly to lecturers’ assumptions about the role of the professional designer/artist:

We’re having to do the job of the designer. He’s [the student] not doing that job for us.

Although the lecturers want meaning delivered to them, it is clear that they also make meaning. Echoing Mitchell’s (1998) view of meaning making, Strathern (2000) and Hussey and Smith (2002) take the view that language is meaning making and co-constructive. The artwork does not deliver meaning through a conduit from student to lecturer; meaning is constructed around the artefact. This discoursal creation of the value of the student work is illustrated by one lecturer, who recorded her thoughts about a student’s submission in the studio alone. She spoke for some time and then commented, ‘See I am starting to talk the work up!’ This suggests that the artwork’s value is constructed not measured.

Discussion

We collected convenience sample data from two universities, so our findings are suggestive. However it is our view that art and design practitioners and researchers will be able to recognize assessment keynotes within this analysis. Shreeve et al. (2010) discuss signature pedagogies in art and design and the quality indicators reported on in this study point to a rendering of signature assessment in art and design. Some studies of lecturers’ assessment practices point out their apparent deficiencies. For example, in van der Schaff et al.’s (2011) study the researchers imply that the assessors’ assessment approaches fell short of researcher expectation. In contrast, our study did not set out expected assessment approaches against which assessment practices are measured.

This study explores art and design studio-based assessment of creative practice; however, we recognize that practice is assessed in other disciplines (e.g. medical education and performing arts) and creativity is assessed in other disciplines (e.g. creative writing courses and music composition). The focus here is on creative practice in art and design, so claims for generalizability to assessment in other subjects that assess creativity and/or practice are very limited.

Methodological challenges

The two data collection approaches employed (think aloud and recording conversation) responded to the particularities of the given assessment context. However, these data collection approaches are imperfect. Some participants are better at thinking aloud than others, which can lead some to grade student work more slowly (Crisp, 2008). Some participants in Department A said very little while they were assessing the work, which suggests that they struggled to articulate their thinking. Brooks (2009) found that participants say less when the task they are being asked to complete is difficult. Whilst the combination of (cognitive) individually based think aloud with the discursive (social) conversation recording was challenging, this approach was a necessary consequence of the practices found, and this dual approach is also used in Suto and Greatorex (2008) to good effect.

Overview

We have identified three meta indicators of quality that characterize assessment. These have their parallel in Grainger et al.’s (2008) conceptions of quality. We employed the label indicators of quality to foreground the emergent, dynamic, co-constructive and situated nature of the indicators. The indicators identified are not freestanding; they are interrelated and co-dependent.

The first indicator of quality, ‘the demonstration of significant learning over time’, foregrounds the view of learning as a journey in art and design. The project-based learning approach commonly adopted in art and design education leads to an emphasis on the mapping and tracking of students’ individual learning journeys over time. This focus has been noted in other studies (Logan, 2006). Harman and McDowell’s study into design assessment discourse unpacks the ‘discourse of personal development’ associated with design assessment practices; within this discourse ‘learning is represented as the flow of a river’ (2011: 44). This indicator of quality points to the use of ipsative assessment where work is judged in relation to distance travelled by each student.

The second indicator of quality brings out the value lecturers assign to effective studentship. Students were expected to listen carefully and respond to the feedback they received. Engaging with and learning from lecturers’ feedback is viewed as a ‘valued disposition’ (Hay and MacDonald, 2008). Successful designers, particularly in a commercial setting, will elicit and use feedback effectively; however, in this study this seems to be more concerned with the students listening to the lecturers and following advice given. Arguably, this response does not sit easily within a pedagogic community that prides itself on being non-directive and anti-canonical (Danvers, 2003). Put at its simplest, there is an espoused view that students are encouraged to ‘do their own thing’ but an in-practice view that lecturers value what Barrow (2006), echoing Foucault (1979), refers to as docile bodies and obedient souls.

The study suggests that prior knowledge about the students’ approaches to study and artistic practices was drawn into the assessment process and that these approaches and practices were used as a measure of quality. Looking for evidence of studentship and the learning journey in the work submitted highlights the relation between the work and the student who makes the work. Work and the student elide. The lecturers look at the work and the work evokes the student. In Brooks’ overview of assessment judgment studies across disciplines she noted that ‘markers have been observed … striving to read the student in their work’ (2009: 14). Hay and Macdonald (2008) identified that prior knowledge about the student imbued the teachers’ assessment judgements in a study of PE teachers, so this focus on the student is not particular to art and design.

These findings fracture the commonly held view that the student and the work should be viewed as separate when work is assessed. This study suggests that when a design portfolio is assessed the student designer and her portfolio are being assessed. We may need to reframe the ways we articulate the relationship between the student and the work in art and design. It is all about the student. Over the last decade there has been a trend that has sought to remove any reference to the student from assessment. For example, Sadler (2009) argues that our focus should concern itself solely with the student work and not the student. For Sadler, it is not all about the student, instead it is all about the work. The current requirement for anonymous marking in some universities is one consequence of the idea that the student and their work can be fully separated. This points to an assessment dilemma (Havnes and McDowell, 2008) that reflects a possible gap between the ways that assessment is set out in course literature and the way it is practised in situ.

Whilst the lecturers’ interest in the students’ development as artists/designers is to be welcomed, a note of caution is necessary given Burke and McManus’ (2009) research into art and design admissions interviews. They observed 70 selection interviews in five art colleges. These selection interviews provide an opportunity for prospective higher education art and design students to bring their portfolio of work for discussion with the admissions tutors. Their study offers a blistering account of admission interviews. They conclude that ‘art and design has a deeply embedded, institutionalized class and ethnically biased notion of a highly idealized student against whom they measure candidates’ (Burke and McManus, 2009: 7). No bias of this type surfaced in the study reported here, but Burke and McManus’ report reminds us that lecturers’ close relationship with students and their perceptions of their studentship bring with it a responsibility to recognize and address the potential for discrimination.

The students are expected to present work that ‘communicates’ effectively. This focus on the artwork being ‘articulate’ may have parallels in other disciplines. For example, students are frequently exhorted to write ‘more clearly’ in their essays (Lillis, 2001). Focusing on student meaning-making has a degree of ambiguity and elusiveness that sits comfortably in disciplinary territory that celebrates the ‘pedagogy of ambiguity’ and resists attempts to capture and fix creativity (Austerlitz et al., 2008). The quality indicators discussed in this study are not precise. They are not transparent. The indicators leak into each other and are ‘non-discrete’ (Sadler, 2009: 170). Yorke (2008) discusses the need to recognize the fuzziness of assessment standards. The quality indicators reported in this study are, by necessity, fuzzy. This fuzziness is relatively uncontentious in the field of art and design but is more problematic for those who may be seeking total transparency (see Orr (2005) for a fuller discussion).

It is likely that these quality indicators are used tacitly and as such they may be undocumented and not shared with students. Sadler explores the challenges associated with the use of ‘emergent’ criteria. Sadler argues that the use of unstated criteria cannot be admitted because ‘to do so would breach the implicit contract with students, which is that only the preset criteria will be used in deciding the grade’ (2009: 167). Sharing these quality indicators with students would help the students to become part of the assessment community.

Assessment for learning

A traditional model of assessment poses assessment as post hoc. In other words learning occurs and then it is tested. In contrast a more contemporary view of assessment known as Assessment for Learning (AfL) characterizes assessment as a learning tool. In this model assessment is employed as a means to offer feedback to support ongoing learning. In this case assessment seeks to improve learning (McDowell et al., 2011). Assessment for learning studies have led to increased interest in formative assessment. Usually formative and summative assessment are viewed as discrete entities. The separation of formative and summative assessment is challenged in this study, where we have identified that students’ development work and final work are assessed in their totality. This echoes the assessment practices in the design studio observed by Harman and McDowell, which ‘cannot be summed up in the binary divide of formative and summative assessment’ (2011: 42). In art and design a binary that divides assessment for learning and assessment of learning may not apply.

Conclusion

The quality indicators identified underline that the student and her practices are being assessed as well as the artwork. These findings align well with the first three elements of art and design assessment identified by de la Harpe et al. (2009). The interrelatedness of the three indicators reported on suggests that, for the lecturers in this study, there is no categorical difference or distinction to be made between the work made, how the work was made and the work’s maker.

Footnotes

Author biographies

Susan Orr is Assistant Dean in the Faculty of Arts, Computing, Engineering and Science at Sheffield Hallam University. She holds a chair in Creative Practice Pedagogy and was awarded a National Teaching Fellowship in 2010. Susan researches pedagogy in art and design with a particular focus on the assessment of creative practice. Her recent publications have explored issues of identity, subjectivity and objectivity in relation to the assessment of art and design and the studio Crit. Currently Susan is collaborating with Professor Mantz Yorke and Professor Bernadette Blair on a funded study that is exploring art and design students’ experiences of the National Student Survey.

Sue Bloxham is Professor of Academic Practice and Director of Educational Research at the University of Cumbria. She has taught in higher education for many years, developing a particular interest in research related to assessment, active learning and widening participation. Sue published Developing Effective Assessment in Higher Education: A practical Guide (Open University Press) with her Cumbria colleague Pete Boyd in 2007, the same year that she was awarded a National Teaching Fellowship. She has researched and published on matters such as academic standards, marking, student skills development, group assessment, feedback and course design.

References

Amabile

(1996) Creativity in Context, Boulder, CO: Westview Press.

Austerlitz

Blythman

Grove-White

Jones

Morgan

Orr

Shreeve

Vaughan

(2008) Mind the gap: Expectations, ambiguity and pedagogy within art and design higher education. In: Drew

(ed.) The Student Experience in Art and Design Higher Education: Drivers for Change, Cambridge: Jill Rogers, pp. 125–148.

Barrow

(2006) Assessment and student transformation: Linking character and intellect. Studies in Higher Education 31(3): 357–372.

Blair

(2006) “At the end of a huge crit in the summer, it was ‘crap’ – I’d worked really hard but all she said was ‘fine’ and I was gutted”. Art, Design and Communication in Higher Education 5(2): 83–95.

Blair

Cummings

Dunbar

Hayward

Woodman

(2008) Bau-Wow! A model for creative practice, thinking, learning, research and innovation in the 21st century. In: Drew

(ed.) The Student Experience in Art and Design Higher Education: Drivers for Change, Cambridge: Jill Rogers, pp. 65–95.

Bloxham

Boyd

Orr

(2011) Mark my words: The role of assessment criteria in UK higher education grading practices. Studies in Higher Education 36(6): 655–670.

Blythman M, Blair B and Orr S (2008) Critiquing the Crit: Student guide. Available at: www.adm.heacademy.ac.uk/library/files/adm…/crit-student-guide.pdf (accessed 3 August 2011).

Brooks V (2009) Marking as judgment. Research Papers in Education iFirst Article pp. 1–18.

Burke

McManus

(2009) Art for a Few: Exclusion and Misrecognition in Art and Design Higher Education Admissions, London: National Arts Learning Network.

10.

Cannatella

(2001) Art assessment. Assessment and Evaluation in Higher Education 26(4): 319–326.

11.

Charmaz

(2006) Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis, London: SAGE.

12.

Cowdroy

deGraff

(2005) Assessing highly-creative ability. Assessment and Evaluation in Higher Education 30(5): 507–518.

13.

Crisp

(2008) Exploring the nature of examiner thinking during the process of examination marking. Cambridge Journal of Education 38(2): 247–264.

14.

Danvers

(2003) Towards a radical pedagogy: Provisional notes on learning and teaching in art & design. International Journal of Art and Design Education 22(1): 47–57.

15.

de la Harpe

Peterson

Frankham

Zehner

Forsyth

Musgrave

Neale

McDermott

(2009) Assessment focus in studio: What is most prominent in architecture, art and design? International Journal of Art and Design Education 28(1): 37–51.

16.

Ehmann D (2005) Using assessment to engage graphic design students in their learning experience. Paper presented at the 2005 Evaluations and Assessment Conference, 30 November –1 December, Sydney.

17.

Elkins

(2001) Why Art Cannot Be Taught: A Handbook for Art Students, Chicago, IL: University of Illinois Press.

18.

Foucault

(1979) The History of Sexuality Vol.1: An Introduction, London: Allen Lane.

19.

Fryer

(2010) From reproduction to creativity and the aesthetic: Towards an ontological approach to the assessment of devised performance. Research in Drama Education, the Journal of Applied Theatre and Performance 15(4): 547–562.

20.

Gordon

(2004) The ‘wow’ factors – The assessment of practical media and creative arts subjects. Art, Design and Communication in Higher Education Journal 3(1): 61–72.

21.

Grainger

Purnell

Zipf

(2008) Judging quality through substantive conversations between markers. Assessment and Evaluation in Higher Education 33(2): 133–142.

22.

Harman

McDowell

(2011) Assessment talk in Design: The multiple purposes of assessment in HE. Teaching in Higher Education 16(1): 41–52.

23.

Havnes

McDowell

(2008) Balancing Dilemmas in Assessment in Contemporary Education, Oxford: Routledge.

24.

Hay

MacDonald

(2008) (Mis)appropriation of criteria and standards- referenced assessments in a performance based subject. Assessment in Education: Principles Policy and Practice 15(2): 153–168.

25.

Hussey

Smith

(2002) The trouble with learning outcomes. Active Learning in Higher Education 3(3): 220–233.

26.

Lillis

(2001) Student Writing: Access, Regulation, Desire, London: Routledge.

27.

Logan

(2006) ‘Circles of Practice’: Educational and professional graphic design. Journal of Workplace Learning 18(6): 331–343.

28.

McDowell

Wakelin

Montgomery

King

(2011) Does assessment for learning make a difference? The development of a questionnaire to explore the student response. Assessment and Evaluation in Higher Education 36(7): 749–765.

29.

Mitchell

(1998) Making through talking. National Association for Fine Art Education 2(3): 30–34.

30.

Norton

(2009) Action Research in Teaching and Learning, Oxford: Routledge.

31.

Orr

(2005) Transparent opacity: Assessment in the inclusive academy. In: Rust

(ed.) Proceedings of the 200 412th International Symposium: Improving Student Learning: Diversity and Inclusivity, Oxford: Oxford Brookes University Press, pp. 175–187.

32.

Orr

(2007) Assessment moderation: Constructing the marks and constructing the students. Assessment and Evaluation in Higher Education 32(6): 645–656.

33.

Orr S (2010a) Making marks: Assessment practices in art and design. Networks 10: 9–13. Available at: http://www.adm.heacademy.ac.uk/resources/publications/networks-magazine (accessed 10 August 2011).

34.

Orr

(2010b) The Role of Professional Judgement in Assessment: A Journey from Accuracy to Zing, Inaugural Professorial Lecture. York: York St John University.

35.

Orr

(2011a) ‘We kind of try to merge our own experience with the objectivity of the criteria’: The role of connoisseurship and tacit practice in undergraduate fine art assessment. Journal of Art, Design and Communication in Higher Education 9(1): 5–19.

36.

Orr

(2011b) ‘Being an artist you kind of, I mean, you get used to excellence’: Identity, values and fine art assessment practices. International Journal of Art and Design Education 30(1): 37–44.

37.

Orrell

(2004) An exploration of congruence and disjunction between academics’ thinking when assessing and their beliefs about assessment practice. In: Rust

(ed.) Improving Student Learning: Theory, Research and Scholarship, Oxford: Oxford Brooks University Press, pp. 186–199.

38.

Quality Assurance Agency (2006a) Code of Practice for the assurance and quality of academic standards in Higher Education. Section 6: The assessment of students. Available at: http://www.qaa.ac.uk/Publications/InformationAndGuidance/Documents/COP_AOS.pdf (accessed 10 August 2011).

39.

Quality Assurance Agency (2006b) Code of Practice for the assurance and quality of academic standards in Higher Education. Section 7: Programme design, approval, monitoring and review. Available at: http://www.qaa.ac.uk/Publications/InformationAndGuidance/Documents/programmedesign.pdf (accessed 10 August 2011).

40.

Ritchie

Lewis

(2003) Qualitative Research Practice, London: SAGE.

41.

Sadler

(2009) Indeterminacy in the use of preset criteria for assessment and grading. Assessment and Evaluation in Higher Education 34(2): 159–179.

42.

Shreeve

Sims

Trowler

(2010) ‘A kind of exchange’: Learning from art and design teaching. Higher Education Research and Development 29(2): 125–138.

43.

Shreeve

Wareing

Drew

(2008) key aspects of teaching and learning in the visual arts. In: Fry

Ketteridge

Marshall

(eds) A Handbook for Learning and Teaching in Higher Education, 3rd edn. London: KoganPage.

44.

Strathern

(2000) The tyranny of transparency. British Educational Research Journal 26(3): 309–321.

45.

Suto

Greatorex

(2008) What goes through an examiner’s mind? Using verbal protocols to gain insights into the GCSE marking process. British Educational Research Journal 34(2): 213–233.

46.

Svensson

Edstrom

(2011) The function of art students’ use of studio conversations in relation to their artwork. International Journal of Education and the Arts 12(5): 1–29.

47.

van der Schaff M, Baartman L and Prins F (2011) Exploring the role of assessment criteria during teachers’ collaborative judgement processes of students’ portfolios. Assessment and Evaluation in Higher Education iFirstArticle doi: 10.1080/02602938.2011.576312.

48.

Wenger

(2004) Communities of Practice: Learning, Meaning and Identity, Cambridge: Cambridge University Press.

49.

Wengraf

(2001) Qualitative Research Interviewing: Biographic Narrative and Semi-Structured Methods, London: SAGE.

50.

Yorke

(2008) Grading Student Achievement in Higher Education, London: Routledge.