‘Involve Me and I Learn’: Development of an Assessment Program for Research and Critical Analysis

Abstract

Evidence-based medical practice is best achieved by developing research understanding in medical practitioners. To this end, medical councils worldwide increasingly recognise the importance of medical schools graduating students with well-developed research skills and research capacity. To meet this need, the principles of programmatic assessment were implemented in designing a research and critical analysis curriculum and assessment program that aimed to enhance the research and critical analysis skills of medical students. The program was developed by mapping assessment tasks to a research capabilities framework that was in turn scaffolded to different levels of Miler’s pyramid. The curriculum and assessments were integrated with the science, clinical, and professional aspects of the medical course. The progressive longitudinal development of research skills, with feedback and academic mentoring, culminated in the students’ capacity to undertake an independent research project. Designing an assessment program for learning encouraged students to develop their research capacity by involving them in their learning.

Keywords

evidence-based medicine research capacity programmatic assessment curriculum integration

Background

Research capacity is recognised as an important educational standard for graduates in many health professions, and research skills are fundamental to evidence-based medical practice.^1,2 Accrediting bodies worldwide are highlighting the need for medical schools to develop students’ research competencies by increasing their expectations for research-capable graduates. In Australia, this is regulated by the Australian Medical Council³ which requires medical graduates to be able to ‘apply knowledge of scientific methods to formulate relevant research questions and select applicable study designs’ (AMC standard 3.4). Similarly, the Royal College of Physicians and Surgeons of Canada⁴ recently updated the CanMEDS framework (2015) to include a requirement for medical students to be able to ‘contribute to the creation and dissemination of knowledge and practices applicable to health’. In the United Kingdom, the General Medical Council⁵ has specified that their medical graduates should have the ability to ‘apply scientific method and approaches to medical research’.

These aspirations of medical councils worldwide to improve research competencies of graduating medical doctors highlight the need for medical schools to develop students’ research capacity during training. Research skill development contributes to student learning and future professional practice⁶ and may also encourage future engagement in academic medicine.^7-9 It has been recognised, however, that engaging students in research skill development and training associated with clinical medicine is a challenge.¹⁰ Based on this premise, and Benjamin Franklin’s statement ‘Tell me and I forget, teach me and I may remember, involve me and I learn’, the research and critical analysis (RCA) curriculum and assessment program at the University of Wollongong (UOW) has been successfully integrated within the 4-year medical course which commenced in 2007. The 4 phases of the RCA curriculum and assessment program, developed and implemented between 2007 and 2010, have been iteratively refined, based on student, faculty, and peer feedback, to provide students with an authentic and meaningful research experience.

The principles of programmatic assessment were deliberately included in the development of the RCA curriculum. These principles are based on the foundation that assessment is for learning rather than assessment of learning.¹¹ In programmatic assessment, the assessment tasks are designed to ensure a thorough mapping of the tasks to a competency framework. Integrating the assessment tasks with the curriculum delivery allows for the progressive longitudinal development of students’ skills to be demonstrated.¹² This progressive skill development is further supported by regular and informative feedback regarding the assessment tasks.¹³

By adhering to the principles of programmatic assessment, the UOW RCA curriculum and assessment program represents a departure from more conventional attempts to develop medical student research capacity, such as assigning students defined tasks within research teams or existing projects or limiting students to write draft research proposals. The RCA curriculum is characterised by its alignment with other aspects of the medical course, including its longitudinal nature and the students’ progressive development of research capabilities. It culminates in the students’ capacity to effectively undertake their own research projects, from the development of a research question through to dissemination of findings. The curriculum and assessment program is based on the understanding that facilitation of independent, investigator-led research promotes a culture of research awareness in new graduates^14-17 and can influence involvement in future research.^7,18

Competency Framework for Research Capacity Development

A key feature in the design of the UOW RCA curriculum and assessment program was determining the competency framework on which it would be based. This competency framework was premised on the research capabilities (Table 1) required to conduct an independent research project, as this was one of the desired end points. These research capabilities are developed via approximately 300 hours of lectures and learning modules delivered face-to-face or online and supported by the longitudinal programmatic assessment.

Table 1.

Research capacity framework for RCA skill development.

RCA knowledge/skills/attributes	Phase 1	Phase 2	Phase 3	Phase 4
1. Ethics
1.1 Ethics application	1. Knows	1. Knows	2. Knows how – 4. Does
2. Information literacy
2.1 Conduct a comprehensive search of the relevant literature	3. Shows how	4. Does	4. Does
2.2 Evaluate and search for relevance, comprehensiveness, and scientific merit	3. Shows how	4. Does	4. Does
3. Research methods
3.1 Define a research question/idea	1. Knows – 2. Knows how	2. Knows how	3. Shows how – 4. Does	4. Does
3.2 Write a research proposal	1. Knows	1. Knows	2. Knows how – 4. Does
3.3 Use and understand quantitative research methods	1. Knows – 2. Knows how	2. Knows how	3. Shows how – 4. Does	4. Does
3.4 Use and understand qualitative research methods	1. Knows	1. Knows	2. Knows how – 4. Does
3.5 Use and understand basic statistical concepts	1. Knows – 2. Knows how	2. Knows how	3. Shows how – 4. Does	4. Does
3.6 Analyse and interpret results	1. Knows – 2. Knows how	2. Knows how	3. Shows how – 4. Does	4. Does
4. Critical analysis
4.1 Literature review	2. Knows how – 3. Shows how	3. Shows how	3. Shows how – 4. Does	4. Does
4.2 Research design (methods)	1. Knows – 2. Knows how	3. Shows how	3. Shows how – 4. Does	4. Does
4.3 Data analysis	1. Knows – 2. Knows how	3. Shows how	3. Shows how – 4. Does	4. Does
4.4 Interpretation of study results	1. Knows – 2. Knows how	3. Shows how	3. Shows how – 4. Does	4. Does
5. Presentation and publications
5.1 Write a research report	1. Knows	1. Knows	2. Knows how – 4. Does
5.2 Oral and poster presentation of research results	1. Knows	1. Knows	2. Knows how – 4. Does	4. Does

Abbreviation: RCA, research and critical analysis.

The graduate-entry UOW medical course, based on a spiral curriculum, progresses through 4 phases of increasing clinical complexity founded on 4 key curriculum themes: clinical skills, personal and professional development, medical sciences, and RCA (Figure 1). Rather than provide one-off or disconnected lectures about teaching research methods, the RCA curriculum is anchored to, and integrated with, the other themes across all 4 phases of the medical course. Thus, the students progress through the course via an RCA curriculum and assessment program that has been scaffolded based on the Miller’s¹⁹ pyramid.

Figure 1.

Structure of the University of Wollongong medical degree.

Pyramid is a competency-based framework usually applied to the development of clinical competency and workplace-based assessment.²⁰ However, for the purposes of the RCA program, this framework (Figure 2) was applied to assist with identifying the development of the students’ research capabilities (Table 1). The framework helps to describe how the students progressively develop their research capabilities from ‘knows’, to ‘knows how’, ‘shows how’, and, finally, ‘does’ (Table 1). Furthermore, the course components of the RCA curriculum are supported by a key principle of programmatic assessment, and that assessment is for learning¹¹ and, as such, should be interweaved with the medical curriculum delivery.¹² As part of the RCA curriculum and program, the assessment tasks were specifically chosen to allow students to develop and demonstrate the knowledge, skills, and attributes expected of an independent researcher. This is achieved through the progressive completion of a portfolio of assessment tasks that have been specifically selected to demonstrate the development of research capacity, as has been previously described for the development of clinical competency.²¹ The individual assessment tasks and their alignment with the RCA capacity framework are described in Table 2. Representation of the scaffolding and programmatic assessment in terms of skill development across each phase of the UOW medical degree is described below.

Figure 2.

Miller’s pyramid (adapted from Miller¹⁹).

Table 2.

Alignment of RCA assessment tasks with the research capacity framework.

Assessment task	Aim of assessment task	RCA knowledge/skills/attributes from Table 1
Phase 1
Phase 1: Social determinants of health essay	Develop skills and capacity to identify and effectively utilise various sources of referenced information to construct a coherent and critically reflective argument around a given topic. This task requires an effective literature search and review of relevant literature found. The task provides students with a broader, ‘real-life’ social context in which to place their theoretical and practical experiences of clinical medicine within the MD programme.	2.1 Conduct a comprehensive search of relevant literature
Phase 1: POEM (Patient-Oriented Evidence That Matters)	Develop skills in applying critical analysis principles to a source article in an attempt to resolve a clinically relevant question. This task requires the student to perform a valid literature search, recognise high-quality research, and analyse and present the findings of the research in a clinically meaningful way. This task requires students to demonstrate ‘critical thinking’ and evaluation and to use their own words for interpretation of the research.	2.1 Conduct a comprehensive search of relevant literature2.2 Evaluate and search for relevance, comprehensiveness and scientific merit
Phase 1: Critical analysis of a research paper	Develop skills in applying critical analysis principles to a research paper. The focus is on analysing the research, not on criticising the research, although that may be necessary. This task requires students to demonstrate ‘critical thinking’ and evaluation. Analysis of a research paper can help decide if sufficient evidence is provided by the research to potentially influence medical practice.	2.2 Evaluate and search for relevance, comprehensiveness and scientific merit3.5 Use and understand basic statistical concepts
Phase 2
Phase 2: POEM	Develop skills in applying critical analysis principles to a source article in an attempt to resolve a clinically relevant question applicable to a hospital setting. This task requires the student to perform a valid literature search, recognise high-quality research, analyse and present findings of the research in a clinically meaningful way. Students are expected to demonstrate ‘critical thinking’ and evaluation and use their own words in interpreting the research.	2.1 Conduct a comprehensive search of relevant literature2.2 Evaluate and search for relevance, comprehensiveness and scientific merit
Phase 2: Critical appraisal of a drug advertisement	Develop skills in recognising the implicit and explicit claims made in drug advertisements, searching and appraising the relevant literature, improving knowledge and awareness of the advertised drug or drug class, its actions, and side effects and where it fits in the treatment options for the condition for which it is approved. This task helps to increase the student’s awareness of ethics in drug advertising, paying particular attention to the explicit World Health Organization advertising guidelines and making valid conclusions as to the therapeutic role of the advertised product.	2.2 Evaluate and search for relevance, comprehensiveness and scientific merit4.1 Interpretation of study results4.2 Research design (methods)4.3 Data analysis4.4 Interpretation of study results
Phase 3
Phase 3: Project proposal and ethics application	Develop skills in writing a research proposal (inclusive of background information, research question, methods, participants, stakeholders, ethical considerations, and expected outcomes) and submitting a human research ethics application.	1.1 Ethics application3.1 Define a research question/idea3.2 Write a research proposal4.2 Research design (methods)
Phase 3: Literature review	Develop skills in locating literature relevant to the research topic, reviewing it critically and identifying its relationship with the research topic.	2.1 Conduct a comprehensive search of relevant literature4.1 Literature review
Phase 3: Final report as journal-style article	Develop skills in writing a final report, in the format of a manuscript in preparation for a peer-reviewed journal submission.	3.3 Use and understand quantitative research methods3.4 Use and understand qualitative research methods3.5 Use and understand basic statistical concepts3.6 Analyse and interpret results4.3 Data analysis4.4 Interpretation of study results5.1 Write a research report
Phase 3: Conference-style poster presentation and abstract	Develop skills in, first, designing a presentation that succinctly incorporates the key findings of the research project and, second, present it to a cohort of peers.	5.2 Oral and poster presentation of research results
Phase 4
Phase 4: Report and conference-style poster presentation	Demonstrate the capability of undertaking independent research through a self-initiated and managed research project resulting in a report and conference-style presentation to a cohort of peers.	3.1 Define a research question/idea5.2 Oral and poster presentation of research results

Abbreviation: RCA, research and critical analysis.

RCA Curriculum and Assessment Program Integration for Development of Research Capacity

Phase 1 (18 months)

Phase 1 represents the preclinical component of the course and occurs during the first 18 months of the medical degree. During this phase, the RCA curriculum focusses on delivering content typical of the research paradigm through lectures, a student-led journal club, discussions, and assessment tasks. Students are taught and practise skills in literature searching, critical analysis, interpreting statistics, study design, research methods, and evidence-based medicine. The RCA concepts on which these skills are based are integrated with the scientific and clinical content of the medical degree. For example, students learn how to interpret systematic reviews and meta-analyses, while studying the cardiovascular system, by discussing the scientific and clinical evidence around the use of statins for hypercholesterolaemia and cardiovascular mortality prevention. Simultaneously, they are learning about the basic and clinical pharmacology of statin actions. Other examples include learning principles of screening specificity and sensitivity in the context of bowel cancer and burden of disease in the context of diabetes. Thus, RCA principles, such as evaluating evidence and understanding research design, are embedded into the medical curriculum. In 2010, the success of this approach was acknowledged by external national accreditors who commented that ‘presentation of research methodologies, biostatistics, and epidemiology within the context of a variety of medical sciences disciplines [and this] integration has been successful in preventing marginalisation of the material and presents an opportunity for development of a research capacity’.

The first RCA assessment task in phase 1 is an essay on the social determinants of health (Table 2). As all UOW medical students have a prior degree, this task was specifically chosen as the initial assessment because it allows students to undertake tasks with which they are somewhat familiar (eg, literature searching and applying the evidence to the social context of health). The second component of the phase 1 RCA assessment program is the completion of a POEM (Patient Oriented Evidence that Matters) (Table 2), which requires students to address a clinical problem/question based on evidence from a systematic review and meta-analysis. The POEM topics chosen for phase 1 typically relate to clinical scenarios that students may encounter during their phase 1 clinical placements in primary care settings. The third phase 1 RCA assessment task is a critical analysis of a research paper, allowing them to demonstrate their skills in applying critical analysis principles practised through the student-led journal club.

The student-led journal club was included for the third cohort of phase 1 students in 2009. This was in response to the first 2 cohorts who found the critical analysis of a research paper assessment task challenging. Educationally, the strength of a journal club is its ability to provide practice in developing and refining critical analysis skills.²² Initially, the journal club articles are presented by research-active academic staff to model what is expected. Thereafter, the students are divided into small groups (8-10 students) to present their critical analysis of the research papers to their peers. Journal club articles are specifically chosen to integrate with the topics the students are studying in other aspects of the course and to demonstrate the application of different research methods (eg, randomised controlled trials, qualitative methods, economic evaluations, and survival analysis). Journal club helps to extend and refine student learning of RCA principles and to broaden their knowledge of clinical and medical sciences. Self-assessment evaluations of the journal club, by students of 2 cohorts (N = 90) experiencing journal club for the first time, revealed improved research capacity: 74% reported that their confidence in dealing with medical literature had improved, 69% reported using the skills learnt through journal club in other aspects of their course learning, and 68% indicated improved critical analysis skills. The following quote also suggests that journal club has assisted with the development of student research capabilities:

I feel like my knowledge on statistics and the research designs most important to medicine improved and I am much better equipped to navigate the literature and conduct research myself in future. (Phase 1 student, 2013)

Phase 2 (12 months)

In phase 2, students engage in off-campus, multisite hospital rotations, across 5 specialty areas (medicine, surgery, obstetrics and gynaecology, paediatrics, and psychiatry). During this phase, the students are given the opportunity to apply the knowledge gained in phase 1 to critically appraise clinical issues relevant in a hospital setting. For example, the POEM topics for phase 2 (Table 2) are selected from current literature which reflects the context of the speciality hospital rotations. This strategy of topic choice aligns the assessment task with other aspects of the medical course and increases the authenticity of the task by clearly demonstrating its relevance to the students’ current clinical experiences, once again supporting the premise that the RCA program is based on the principle of assessment for learning.²³

The second component of the phase 2 RCA assessment program is the critical appraisal of a drug advertisement (Table 2). This assessment task aims to evaluate the implicit and explicit claims made in different drug advertisements by applying students’ skills in critical analysis as described in the RCA capacity framework (Table 1). This task also increases the students’ awareness of the advertised drugs or drug classes, their actions and side effects, and where they fit in the treatment options for different medical conditions. This task is important as evidence demonstrates doctors’ prescribing habits are influenced by pharmaceutical company advertising, often unwittingly.²⁴ Furthermore, it has been demonstrated at a number of medical schools that this influence may already be established among medical students from their early exposure to pharmaceutical sales representatives, prior to graduation.^25,26

Phase 3 (12 months)

The capstone of the RCA program occurs during phase 3 of the medical course, which includes a 12-month placement within a regional (ie, noncapital city), rural, or remote community. During this phase, students initiate and complete an individual research project. The research project provides students with authentic learning experiences and builds on their research capabilities and knowledge gained from the earlier phases, as well as their understanding of key research priorities within clinical and community settings. The project culminates in a final journal article–style report and a conference-style presentation to the cohort. Students are also encouraged to seek publication of their research findings.

Phase 3 students are provided with individual supervision by a research-experienced faculty academic mentor with a PhD, who gives ongoing advice and feedback while the students complete each of their research project–related assessment tasks (Table 2). The first task that the students undertake is to submit a project proposal and ethics application (Table 2). This task helps students to define the topic of their research project and ensures that they can justify their choice of research topic and design, as well as to consider the ethical issues associated with their project. Following the submission of their project proposal and ethics application, students submit a review of the literature (Table 2), which allows them to define their research question, establishing the importance of the topic and providing background information that justifies the need for the project. The final report is submitted as a journal-style article, with an accompanying conference-style abstract and poster presentation to their peers (Table 2). To further facilitate their learning experiences, these final journal-style submissions and conference-style presentations are marked by research-qualified academics, who provide the students with journal-level, peer-review–style feedback. Overall, these authentic learning experiences are preparing students to take part in future research and to understand how they will be able to disseminate their findings in peer-reviewed journals and conference abstracts.

It is their phase 3 placement that provides the students with the opportunity, and often inspiration, to experience the RCA curriculum and assessment program in a way that is practical, distinctive, and learner-centred. Although most student projects have been modest in scope, this has been a deliberate evidence-based approach^27,28 to help build research capacity slowly and gain experience and confidence by starting with small-scale studies investigating useful, practice-based problems.²⁹ Furthermore, providing students with the opportunity to be involved in all aspects of their research project is an essential part of the process because they develop a sound appreciation for research.⁷

Phase 4 (6 months)

In the fourth and final phase of the UOW medical degree students spend 6 months in a choice of clinical rotations with the overall aim of preparing the students for medical internship. During this phase, students placed in Australia or overseas work independently (ie, without an academic supervisor/mentor) to complete a critical and reflective report of their clinical experiences. This culminates in the preparation of a conference-style abstract and oral presentation given to their peers, academic and clinical staff at the end of the phase (Table 2). The research capacity development in the earlier phases of the medical degree ensures that these students are not disadvantaged by location or access to communication media as they should have the skills, capacity, and independence required to complete these tasks. Furthermore, they also have the capacity to support other health workers to develop skills in RCA.

Strategies Used to Support the Alignment of Programmatic Assessment Within the RCA Curriculum

In addition to the scaffolding of the assessment tasks, several strategies associated with programmatic assessment have been used to implement the RCA curriculum and assessment program. These include regular and informative feedback, a comprehensive set of marking rubrics, and the aforementioned academic mentoring and supervision.

Regular and informative feedback¹³ is a crucial aspect of programmatic assessment because it allows students to incorporate and apply what they learn from completing assessment tasks to their future learning. Within the RCA assessment program, regular and informative feedback is an integral component of the course. For example, the research skills developed and feedback received after undertaking the critical analysis of a research paper (phase 1) can be applied to the critical analysis of a drug advertisement (phase 2) and then into providing evidence from the literature to support the research proposal, literature review, and final report (phase 3) (Table 2). The scaffolding of the assessment tasks, therefore, provides a process for iterative feedback to be received by students continually throughout the RCA program, with the intention of supporting and enhancing student learning.^30–32

A comprehensive set of marking rubrics are used to provide detailed feedback to the students and to help clarify what is required to achieve competency in each research skill area. The marking rubrics support assessment for learning as they allow assessors to give detailed feedback relevant to the individual student’s performance³³ and help to identify specific areas where students require further development of their research capabilities. Moreover, the marking rubrics ensure that all the students are provided with consistent and clear feedback, which they can then incorporate into their future assessment tasks,³⁴ building upon their research capabilities as they progress through the course. The feedback provided in the marking rubrics, together with the careful scaffolding of the assessment tasks through the phases, is an important principle of programmatic assessment because, as suggested in the literature, it encourages students to take responsibility for directing their own learning.¹¹

During the earlier phases of the RCA program, the students’ research capabilities are supported by the delivered RCA curriculum content and detailed feedback they receive following the completion of their assessment tasks (Table 2). However, in phase 3, additional support is provided to the students by way of a series of online resources relating to aspects of research that students may have difficulties with, as well as individual supervision and mentoring from a research-experienced academic with a PhD. In addition to fostering a trusting relationship between the academic and the student,²⁰ academic mentoring allows for monitoring of the students’ research skill development and, when required, remediation for individual students. On the odd occasion, where such mentorships have faced challenges, such as when a mentor and mentee do not get along or when mentors are time poor, members of the RCA team have taken on the additional responsibility of mentoring these students to ensure that they are not disadvantaged in any way. Overall, the academic supervisors/mentors in the RCA program play an essential role in fostering and nurturing the development of the students’ research capabilities and their engagement with the research process.³⁵

Iterative Improvements of the RCA Curriculum and Assessment Program

Progressive waves of student, faculty, and peer feedback have iteratively shaped the improvement of the RCA curriculum and assessment program. Initially, the delivery of biostatistics and epidemiology principles was contextualised to the concurrently taught cases and body system blocks to intellectually engage the students. Subsequently, this approach was expanded, and all RCA principles were taught within the context of medical and clinical sciences, including population health. In addition, a team-teaching format was adopted for phase 1 RCA sessions to enhance the capacity for interactive workshops and expand the emphasis on integration of research concepts within particular body system blocks. At the same time, the phase 1 and phase 2 assessment tasks were revised to better align with the students’ RCA skill set and concurrent clinical placements (primary care or hospital). The marking rubrics for assessment tasks across the first 2 phases were also refined to increase clarity about what was expected from the students, along with the provision of online annotated examples of past assessment tasks graded ‘Excellent’.

On completion of phase 3 by the first student cohort in 2010, it was clear that additional online resources were required to assist students’ research skill needs, dependent on the different individual research projects they had chosen to undertake. In response to these needs, the RCA team actively developed additional online research resources for the students and other interested parties (eg, academic supervisors and placement preceptors). A comprehensive list of these online resources, which also included how to develop a survey, how to enter data on a spreadsheet, using descriptive statistics to analyse data, analysing qualitative data, intellectual property, and authorship guidelines, have been previously published.³⁵

Outcomes of the RCA Program

The desired outcome of the RCA curriculum and assessment program is to graduate medical students who are capable of undertaking independent research, critically appraising the evidence and becoming evidence-based practitioners (Figure 3). Through undertaking this course, students come to appreciate the value of research. They are able to understand the importance, implications, and limitations of research and, importantly, will have developed the skills to apply research methods and findings in the broad range of contexts that they will encounter as future medical practitioners.

Figure 3.

The place of research and critical analysis in the spiral University of Wollongong medical degree.

The statistically significant improvements in research capabilities, based on students’ self-assessment, that occurred within the first 3 student cohorts (ie, from cohort 1 to cohort 3, n = 221)³⁶ highlight the success of the UOW RCA curriculum and assessment program. Similarly, an upward trend in the correct answers for RCA-related end-of-phase examination questions since 2007 reflects the continual improvements and revisions to the RCA curriculum and program. Further evidence to support the success of the course can be seen in the significant improvements (P < .05) in self-assessed scores of mean RCA knowledge/skills/attributes for the first 5 student cohorts (n = 349) who completed their phase 3 research projects (Table 3). The only area of research with no increase recorded was ‘applying for research funding’, which is not part of the RCA program and therefore served as a test of internal validity.

Table 3.

Assessment of research capacity development in 5 cohorts of medical students (n = 349) during phase 3 of the RCA program.

RCA knowledge/skills/attributes	Pre-phase 3 mean response	Post-phase 3 mean response
Defining a research question/idea	2.3	3.6*
Writing a research protocol	1.7	3.3*
Finding relevant literature	3.3	4.1*
Critically reviewing literature	3.1	3.8*
Using quantitative research methods	2.3	3.3*
Using qualitative research methods	2.0	2.9*
Analysing and interpreting results	2.7	3.4*
Writing and presenting a research report	2.4	3.7*
Publishing results	1.6	2.3*
Applying for research funding	1.3	1.6

Abbreviation: RCA, research and critical analysis.

Responses related to students’ self-assessed scores using a 5-point Likert scale (1: none; 5: very experienced). Responses from first 3 University of Wollongong medical student cohorts have been published previously.³⁶

Paired t test, P < .05.

The dissemination of the findings from the students’ research projects through presentation at national and international conferences and publication in peer-reviewed journals are further indicators of the success of the RCA curriculum and assessment program. To date, findings from more than 50 student research projects have been disseminated to the scientific and/or clinical communities by way of publications and conference proceedings. These experiences have also inspired students to consider doing future research:

Thank you for your help with my phase 3 project. It has been published . . . really made me interested in doing research in the future. (Phase 3 medical student, 2015)

Conclusions

The RCA curriculum and assessment program progressively develops the medical students’ research capabilities using a longitudinal assessment program intimately linked to the medical course and scaffolded against a research capabilities framework. The program, which aligns with the principles of programmatic assessment for learning and is mapped against the different levels of Miller’s¹⁹ pyramid, was purpose built to achieve its aim of contributing significantly to the development of research skills and research capacity of medical graduates. The RCA curriculum and assessment program builds students’ knowledge and analytical skills while encouraging a research attitude and culture by involving students in their own learning. It successfully integrates research concepts throughout the medical degree and allows for these concepts to be revisited throughout the spiral nature of the curriculum, culminating in the students’ capacity to undertake research and to become future evidence-based practitioners.

Footnotes

Acknowledgements

The integrated development of the RCA curriculum and assessment program would not be possible without a dedicated team of academic staff. The authors would like to acknowledge the contributions of Ms Shelley Crowther, as well as the past and present academic staff at the UOW Medical School for their support for the RCA curriculum and assessment programme.

peer review:

Three peer reviewers contributed to the peer review report. Reviewers’ reports totalled 490 words, excluding any confidential comments to the academic editor.

Funding:

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests:

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Author Contributions

All the authors made substantial contributions to drafting, revising and approving the final version of the manuscript.

References

Irby

Cooke

O’Brien

. Calls for reform of medical education by the Carnegie Foundation for the advancement of teaching: 1910 and 2010. Acad Med. 2010;85:220–227.

Laidlaw

Guild

Struthers

. Graduate attributes in the disciplines of Medicine, Dentistry and Veterinary Medicine: a survey of expert opinions. BMC Med Educ. 2009;9:28.

Australian Medical Council. Assessment and Accreditation of Medical Schools: Standards and Procedures. Kingston, ACT, Australia: Australian Medical Council Ltd; 2009.

Royal College of Physicians and Surgeons of Canada. CanMeds Framework. http://canmeds.royalcollege.ca/en/framework. Published 2015.

General Medical Council (UK). Outcomes for graduates, tomorrow’s doctors. http://www.gmc-uk.org/Outcomes_for_graduates_Jul_15.pdf_61408029.pdf. Published 2015.

Laidlaw

Aiton

Struthers

Guild

. Developing research skills in medical students: AMEE Guide No. 69. Med Teach. 2012;34:e754–e771.

McLean

Saunders

Velu

Iredale

Hor

Russell

. Twelve tips for teachers to encourage student engagement in academic medicine. Med Teach. 2013;35:549–554.

Pangaro

. A new vocabulary and other innovations for improving descriptive in-training evaluations. Acad Med. 1999;74:1203–1207.

Ried

Farmer

Weston

. Setting directions for capacity building in primary health care: a survey of a research network. BMC Fam Pract. 2006;7:1–11.

10.

Funston

Piper

Connell

Foden

Young

O’Neill

. Medical student perceptions of research and research-orientated careers: an international questionnaire study. Med Teach. 2016;38:1041–1048.

11.

Schuwirth

LWT

Van der Vleuten

CPM

. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 2011;33:478–485.

12.

Van der Vleuten

CPM

Schuwirth

LWT

. Assessing professional competence: from methods to programmes. Med Educ. 2005;39:309–317.

13.

Van der Vleuten

CPM

Schuwirth

LWT

Driessen

. A model for programmatic assessment fit for purpose. Med Teach. 2012;34:205–214.

14.

Millar

Baker

Howden-Chapman

Wilson

Dickson

. Involving students in real-world research: a pilot study for teaching public health and research skills. BMC Med Educ. 2009;9:45–51.

15.

Webster-Wright

. Reframing professional development through understanding authentic professional learning. Rev Educ Res. 2009;79:702–739.

16.

Wiggins

. Educative Assessment: Designing Assessments to Inform and Improve Student Performance. San Francisco, CA: Jossey-Bass Publications; 1998.

17.

O’Tuathaigh

Duggan

Khashan

Boylan

O’Flynn

. Selection of student-selected component [SSCs] modules across the medical undergraduate curriculum: relationship with motivational factors. Med Teach. 2012;34:813–820.

18.

Riley

Morton

Ray

Swann

Davidson

. An integrated model for developing research skills in an undergraduate medical curriculum: appraisal of an approach using student selected components. Perspect Med Educ. 2013;2:230–247.

19.

Miller

. The assessment of clinical skills/competence/performance. Acad Med. 1990;65:63–67.

20.

Van der Vleuten

CPM

Schuwirth

LWT

Driessen

Govaerts

MJB

Heeneman

. Twelve tips for programmatic assessment. Med Teach. 2015;37:641–646.

21.

Dannefer

Henson

. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Acad Med. 2007;82:493–502.

22.

Deenadayalan

Grimmer-Somers

Prior

Kumar

. How to run an effective journal club: a systematic review. J Eval Clin Pract. 2008;14:898–911.

23.

Henneman

Oudkerk Pool

Schuwirth

LWT

. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 2015;49:487–498.

24.

Spurling

Mansfield

Montgomery

. Information from pharmaceutical companies and the quality, quantity, and cost of physicians’ prescribing: a systematic review. PLoS Med. 2010;7:22.

25.

Monaghan

Galt

Turner

. Student understanding of the relationship between the health professions and the pharmaceutical industry. Teach Learn Med. 2003;15:14–20.

26.

Sierles

Brodkey

Cleary

. Medical students’ exposure to and attitudes about drug company interactions: a national survey. JAMA. 2005;294:1034–1042.

27.

Cooke

. A framework to evaluate research capacity building in health care. BMC Fam Pract. 2005;6:1–11.

28.

Farmer

Weston

. A conceptual model for capacity building in Australian primary health care research. Aust Fam Physician. 2002;31:1139–1142.

29.

Weston

Mullan

McLennan

. Building research capacity through community-based projects. Med Educ. 2010;44:496–497.

30.

Kluger

DeNisi

. The effects of feedback interventions on performance: a historical review, a meta-analysis and a preliminary feedback intervention theory. Psychol Bull. 1996;119:254–284.

31.

Hattie

Timperley

. The power of feedback. Rev Educ Res. 2007;77:81–122.

32.

Shute

. Focus on formative feedback. Rev Educ Res. 2008;78:153–189.

33.

Sargeant

Armson

Chesluk

. The processes and dimensions of informed self-assessment: a conceptual model. Acad Med. 2010;85:1212–1220.

34.

Vardi

. Effectively feeding forward from one written assessment task to the next. Assess Eval High Edu. 2013;38:599–610.

35.

Weston

Mullan

Rich

. Graduating work-ready professionals: research competency as a critical curriculum component. Curric Teach. In press.

36.

Mullan

Weston

Rich

McLennan

. Investigating the impact of a research-based integrated curriculum on self-perceived research experiences of medical students. BMC Med Educ. 2014;14:161.