The Development of 21st-Century e-Learning Module Assessment Tool

Abstract

The emergence of 21st-century skills convinced educational institutions in modifying teaching styles, which lead to the introduction of blended, flipped, and personalized learning. Studies demonstrated present-day best practices, but none provided a practical assessment procedure. Thus, this study aimed to develop a tool in evaluating learning modules—the 21st-century e-Learning Module Assessment Tool. Identification of components for the instrument revealed the interrelating nature of blended, flipped, and personalized learning; hence, it considered nine components of 21st-century teaching instead of assessing the teaching styles separately. Content validity is established through expert evaluation while reliability is established through pilot testing. 21st-Century e-Learning Module Assessment Tool uses a 5-point scale to rate learning modules based on its compliance to the nine components of 21st-century teaching. The developed assessment tool is recommended to be used as standard in evaluating existing learning modules and a guide for future 21st-century learning modules.

Keywords

assessment tool blended learning flipped learning learning module personalized learning 21st-century skills

Introduction

The 21st century offers new opportunities and new challenges due to the emergence of new ideas and technology in which students can attain knowledge from information that is readily accessible through online technology. Hence, educational institutions were convinced that knowledge is no longer inadequate, and a new set of skills is needed to be successful in the 21st century (Schrader & Lawless, 2011). Students must have the skills on how to use their knowledge. These skills include information and communication skills, thinking and problem-solving skills, and interpersonal and self-directional skills (Partnership for 21st Century Skills, 2011a).With this development in education, it is imperative to acquire new approaches in methodology that subsequently resulted to the 21st-century teaching styles known as blended learning, flipped learning, and personalized learning (Bishop & Verleger, 2013; Courcier, 2007; Graham, 2004; Hamdan et al., 2013; Hodges & Weber, 2015; Monteiro & Morrison, 2014; O’Donnell et al., 2015; Singh, 2003; Tucker, 2012).Assurance in implementing these teaching styles is a necessity, thus assessing learning plans is equally important.

Blended learning is described as a method of shifting teacher instruction by incorporating technology in useful and meaningful ways (Kleber, 2015). It is important to note that in blending, shifting of instruction will not totally change the approach. Traditional classroom time is reduced but not eliminated and is only replaced with some e-learning strategies (Ike & Okechukwu, 2015). Nevertheless, blending learning is not just mixing e-learning to instruction; it still requires an effective pedagogy (Ike & Okechukwu, 2015). Studies suggest that blended learning gives more opportunity for meaningful interactions in the classroom (Cottle & Glover, 2011; Kleber, 2015).In addition, students have positive attitude toward blended environments, thus improving the quality of students’ output (Auster, 2016; Cho & Shin, 2014). Also, immersion of technology to instruction gives students more access to both content and teacher support (Auster, 2016; Hughes, 2007).

Blending traditional and online content could be done through flipped classroom—a technique to switch the common instructional approach (Tucker, 2012). An educational strategy that consists of two parts: interactive in-class activities and direct computer-based out-of-class activities (Bishop & Verleger, 2013). However, the method is not a merely interchanging usual out-of-class homework activity with in-class lecture. Flipping means providing students with an online content that introduce a topic, so that when they get to class, they can get into a real workshop of learning (Boyer, 2013). Several studies suggest that the technique have ability to humanize the classroom by providing better student–teacher interaction (Correa, 2015; Forsey et al., 2013; Fulton, 2012; Peterson, 2016; Steed, 2012; Tucker, 2012).Furthermore, literatures also suggest that flipped classroom promotes differentiated teaching and personalization of instruction (Correa, 2015; Forsey et al., 2013; Steed, 2012; Tucker, 2012).

The one-size-fits-all education is no longer appropriate to the varied learners of the 21st century. Hence, educators developed a learning approach the fits the individual needs, skills, and interests of learners—a personalized approach (Childress & Benson, 2014; Courcier, 2007; Jenkins, 2009). Personalizing learning promotes the development of independent, fulfilled, and life-long learners (Courcier, 2007). According to O’Donnell et al. (2015), this teaching style would realize the full potential impact of technology to learning experience. Various e-learning systems are designed to promote personalized learning (Chen et al., 2005; Roccetti et al., 2001; Sivakumar et al., 2015; Steiner et al., 2009; Thyagharajan & Nayak, 2007).

While the modern workplace demands new set of skills, the dominant teaching style used in schools remains to be traditional. Existing learning plans are still dependent on teacher-centered and direct instruction. Compounding the challenge of noncompliance to the recent teaching styles is the absence of an assessment procedure for these new teaching styles. Hence, this study aimed to develop an assessment tool readily available for intended users—the 21st-Century e-Learning Module Assessment Tool (21CELMAT).

Literature Review

Rich bodies of published works have suggested methodologies on assessing their implementation of 21st-century teaching. These teaching styles are a relatively new approach in teaching; thus, it is still a debatable topic whether this approach is effective or not. The following discussion presents how some selected studies measured the effectiveness of their own implementation of blended learning. In the study of Cho and Shin (2014), an educational program for Good Research Practice (GRP) is developed for both nurses and nursing students. The result of the pretest was used as the baseline data to compare with that of the posttest to evaluate the program. The significant increase in both knowledge and perception of research ethics educational program established the program’s success. The study of Hughes (2007) aimed to determine the effect of using blended learning in increasing learner support and improving retention. The method was assessed by comparing his method with another that did not focus on learner support. Difference on pass rates, submission rates, and tutor time showed that blended module with proactive tutor support improved module retention without increasing overall teaching time. Different from the latter studies, Auster (2016) did not use comparison as a method of assessing effectiveness of her blended model. The impact of screencast blended learning was determined through students’ responses. The study showed that screencast was generally helpful, improved performance on exams, and upheld positive feelings about the course. Similarly, Bodell and Hook (2014) assessed blended learning package for developing online professional networks qualitatively. Value was assessed through students’ perceptions on the helpfulness of the program in dealing with coursework.

A number of educators who tried flipping their classroom showed evidences of promise on the said method through research. Nevertheless, the method of proving the potential of flipped learning is still varied and ununiformed. Love at al. (2014) assessed their sophomores-level applied linear algebra flipped learning course. Effectiveness was measured through a survey on students’ perceptions and scores from exams. Comparison showed that flipped classroom holds significant promise for successful implementation of Science, Technology, Engineering and Mathematics (STEM) courses and improves students’ outcomes in linear algebra. Similarly, Peterson (2016) evaluated flipped course in an introductory statistics course at Knox College. The method of evaluation also included the comparison of traditional lecture and flipped class, with results revealing that students in flipped classroom outperformed their lecture peers by more than a letter grade. Another method of appraising flipped classroom was presented on the study of McEvoy et al. (2014) where they assessed the feasibility of a flipped classroom program for communicating nutrition education to fourth-grade school students. A multiple-choice test was administered as both pre- and posttest before and after the implementation of the module. Success was verified through comparison of pretests and poststests scores, where significant improvement in nutrition knowledge occurred. Similarly, Sayeski et al. (2015) assessed the efficacy of IRIS STAR Legacy Modules which follows a flipped model by using pretest–posttest-maintenance design. The module was appraised based on strong effect sizes from pretest to posttest across all modules and across all conditions.

Various studies to personalize their learning reported promise regarding its impact to student development. Assessment of various personalized methods of teaching became a trend in educational research due to the on-going debate on the teaching styles’ effectiveness. Kucirkova et al. (2014) aimed to determine whether books that contain personalized content are better facilitators of word acquisition than books, which are not personalized. The study made use of repeated-measures experimental in which a group of children was read a book containing both personalized and nonpersonalized sections. Findings revealed that children showed significantly better knowledge about the words that were in personalized sections of book than the words in the nonpersonalized sections. Different from the latter study, Courcier (2007) explored the effectiveness of personalized learning through teachers’ perception instead of students’. The study was based on semistructured interviews with 13 teachers that went through 5-year strategy for children and learners promoting personalized learning. The teachers interviewed believed that it was very difficult to implement personalized learning in their classes in practice although the idea was very good. Another study that evaluated personalized learning was done by Song et al. (2012) to determine the effectiveness of fostering personalized learning in science inquiry supported by mobile technologies. This study assessed personalized learning through the quality of various student outputs. Proofs of effectiveness were gathered through students’ artifacts (portfolios and other outputs), Know-Want to know-Learned, and worksheets. Different sources of data were triangulated to determine overall effectiveness of the module, with results indicating favorable results on the use of such teaching approach.

Review of existing literature showed that assessment of blended, flipped, and personalized learning is not an uncommon subject of research. However, current studies still showed gaps on methods of assessing 21st-century teaching styles. First is the absence of a wide-ranging assessment method. Existing assessment methods are tailored to appraise their best practices but inapplicable for other implementations. Second is the nature of existing assessment methods that requires initial implementation. Current studies always assess 21st-century teaching after implementation, where there is also a need for an assessment feasible before the actual teaching-learning process. The following research gaps emphasize the necessity of this study, the development of an assessment tool for 21st-century teaching in learning modules—the 21CELMAT.

Framework of the Study

The 21CELMAT is an instrument that evaluates integration of 21st-century skills in learning modules. Such skills are grouped into three: learning and innovation skills; information gathering, media and technology skills; and life and career skills (Partnership for 21st Century Skills, 2011b). To evaluate integration of 21st-century skills, 21CELMAT is designed to assess the integration of blended, flipped, and personalized learning in learning modules. The aforementioned teaching styles are considered in the framework due to the assumption that following promote 21st-century skills.

Blended learning builds up students’ learning and innovation skills. This approach uses e-learning for acquisition of tool skills and technical information while use face-to-face class time to focus on application, case studies, and develop decision-making skills (Graham, 2004). Employing blended learning gives teachers more time to promote critical thinking, problem-solving, social interaction, and collaboration in the classroom (Graham, 2004). Furthermore, blended learning develops information gathering, media, and technology skills. A blended learning experience combines off-line and online forms of learning where online learning ensues over the internet or intranet while off-line learning happens in a more traditional classroom setting (Singh, 2003). The use of online learning gives students opportunity to practice, gather, and verify information on their own using technology. Moreover, blended learning promotes the development of life and career skills among learners. The approach enables learners to take greater responsibility for managing their own time, self-regulation, and to take greater ownership of their learning, with self-direction and independence (Monteiro & Morrison, 2014).

Flipped classroom gives learners the opportunity to develop learning and innovation skills. In this approach, activities that aims to develop lower levels of the learning where moved out-of-class time. Here, learners work on mastering concepts where they can pause, rewind, and review lessons (Hamdan et al., 2013). On the other hand, in-class time is spent working through problems, advanced concepts, and collaborative activities (Tucker, 2012). In addition, flipped learning practices information gathering, media, and technology skills. Direct computer-based individual instruction outside the classroom is part of this approach (Bishop & Verleger, 2013). Learners are given references from which they could gather information. Direct instruction occurs online, which in turn, employs learners’ information, communication, and technology skills. Flipped learning also cultivates learners’ life and career skills, specifically initiative and self-direction. The approach requires learners to independently study materials that traditionally delivered by teachers (Hodges & Weber, 2015).

Personalized learning is focused in developing student’s life and career skills. The main purpose of this approach is to realize an individual learner’s potential to be a life-long learner (Courcier, 2007). A feature of this approach is to transfer responsibility of learning to learners. It promotes self-learning, which is an essential life and career skill. Moreover, personalized approach allows learners to improve their learning and innovation skills. It is very useful in adapting learning to suit individual learner’s preferences (O’Donnell et al., 2015). Learners tend to discover innovative ways on using their individual aptitudes and interests in completing a task, thus making them think creatively. Furthermore, personalized learning could realize the full potential of technology to learning experience (O’Donnell et al., 2015). The teaching style is best complemented with information, communication, and technology tools, thus promoting media and technology skills. Personalizing allows learners to evaluate the meaning of information and to decide on the media they prefer to use.

The integration of blended, flipped, and personalized learning highlights the integration of 21st-century skills in instruction. Hence, to assess quality of the aforementioned teaching styles means to assess suitability of learning modules in promoting these skills.The evidence for attaining quality blended learning is based on the alignment of learning modules to quality indicators provided by various studies (Carman, 2002; Donnelly, 2006; Kuo et al., 2014; Lee et al., 2007; Monteiro & Morrison, 2014; Sims et al., 2008; Welker & Berardino, 2005).The same with flipped learning (Correa, 2015; Flipped Learning Network (FLN), 2014; Hall & DuFrene, 2015; Hodegs & Weber, 2015; Moran & Young, 2015) and personalized learning (Garn & Jolly, 2013; O’Donnell et al., 2015; Peters, 2009).The indicators of these teaching styles form the components of 21CELMAT.

Figure 1 shows the diagram that illustrates the conceptual framework. The three 21st-century skills at the center of the diagram signify that the core of the study is to make an assessment in integrating 21st-century skills in instruction. The learning styles—blended, flipped, and personalized learning—overlap in the center where the 21st-century skills are placed. This represents the compatibility of the three learning styles in promoting such skills. Assessment encompassing the diagram denotes that the study revolves to the concept that assessment of blended, flipped, and personalized learning tantamount to the assessment of 21st-century skills in instruction.

Figure 1.

Conceptual Framework of the Study.

Methods

This instrument development study followed a multiphased approached, as outlined by Velayutham et al. (2011), that involved (a) identifying components and items from literature and experts and (b) establishing validity and reliability through expert’s consultation and pilot testing. The study implemented the two-phased procedure to answer the following research questions:

What are the components and quality indicators of 21CELMAT?

Could 21CELMAT establish content validity and instrument reliability?

The first phase of the study included the development of 21CELMAT, an instrument developed to assess the quality of 21st-century learning modules. The instrument initially considered three teaching styles, namely, blended learning, flipped learning, and personalized learning. Properties and quality indicators of each teaching style were initially defined by current literature. The assessment tool development panel, a group of content experts, was consulted to finalize the components of the instrument. Members of the panel, all older than 18 years old, willingly signed their consent forms prior to their participation to the study. Then, once components were identified, quality indicators were provided by the latter panel and existing literatures. The collection of indicators from the two sources served as the basis of items of the developed assessment tool.

The second phase involved the establishment of content validity and instrument reliability. Content validity of the assessment tool is established by evaluating each item’s appropriateness in assessing the nine components of 21st-century teaching. For this purpose, members of the assessment tool development panel responded to a questionnaire to establish the appropriateness of each indicator where it followed a scale of 1 to 4: very appropriate (4 points), appropriate (3 points), inappropriate (2 points), and very inappropriate (1 point). The panel was given 2 weeks to assess the content validity. Consequently, to establish instrument reliability, a pilot test of the developed instrument was conducted where seven existing learning modules are assessed. The existing module assessment panel, a separate group of content experts, assessed the following modules using the 21CELMAT. All panel members and authors of existing learning modules, all older than 18 years old, willingly signed their consent forms to participate in the study and use their learning modules respectively. Seven weeks were given to evaluate all learning modules. Modifications based on both content validity and instrument reliability were made in drafting the final form of 21CELMAT.

Discussions

Development of 21CELMAT

Review of literature and expert consultations were conducted to determine the components of the instrument. Existing literature initially considered five properties of blended learning, namely, live events, self-paced learning, collaboration, assessment, and performance support material (Carman, 2002). None of the assessment tool development panel suggested their own definitions of “live events.” Thus, this study made use of the preexisting definition which refers to synchronous, instructor-led learning events in which all learners participate at the same time (Carman, 2002). Two definitions of “self-paced learning” were given by the assessment tool development panel. One suggested definition is very similar to the definition offered by the literature where it refers to learning experiences in which the learner is working on and completing the tasks individually (Carman, 2002). However, the other proposed definition was not considered as allowing each student to follow his or her own learning path and pace is more on personalization rather than self-paced learning (Childress & Benson, 2014).Therefore, the definition considered in the study was the definition given by the literature. Four definitions were recommended for “collaboration.” The definition given by panel is similar to what the literature suggested. However, it is also slightly different because the instrument emphasized activities and learning environments. Referring to collaboration as an act of communication among students is perceived to be rigid. Communicative platforms may not automatically guarantee that collaborative blended learning will be used or is used effectively (Monteiro et al., 2015). Collaboration is more than communication giving a reason to alter the existing definition. Thus, collaboration in the instrument refers to learning activities in which students work cooperatively which each other. Only a single definition for “assessment” is suggested by the panel, but the property itself is no longer included in the instrument.Instead, the definition of Keefe and Jenkins (2002) is adopted as it divided assessment into two specified properties—diagnostic and summative. Three definitions of “performance support materials” are presented, but it was still not considered. The learning module is about the lesson flow and assessments to be taken within a period; thus, the instrument will not include the availability and feasibility of the materials to be used.

Four properties of flipped learning were primarily considered by existing literature, namely, flexible environment, learning culture, intentional content, and professional educator (FLN, 2014). The panel did not provide any definition for “flexible environment.” Thus, the definition in the literature was used in the assessment where the term refers to the variation of learning modes and arrangement of learning spaces (FLN, 2014). Although the property is initially for flipped learning, this is not the whole idea of the concept because altering instruction to fit the needs of each student is more of a property of personalized than flipped learning (Courcier, 2007). Thus, flexible environment is placed under personalized learning. Two definitions for “learning culture” were recommended by the assessment tool development panel, but the instrument still used the definition from existing literature where the term is defined as the use of in-class time in creating rich learning opportunities (FLN, 2014). Given the considered definition, the panel still responded that the term learning culture is vague and not appropriate. Thus, the definition was retained, but the term learning culture is changed to “in-class activities.” Three definitions of “intentional content” were considered, and it is collectively defined as activities done outside the classroom in preparation for in-class activities. The term intentional content, aligning it to its definition, is changed to a more comprehensible term—“out-of-class activities.” Two definitions of “professional educator” were suggested, but the instrument aimed to assess the quality of activities in the learning module. The other tasks of a teacher other than instruction were not stated in a leaning module; thus, the property is deleted.

Review of literature originally presented six properties of personalized learning, namely, dual teacher role, diagnosis of characteristics, culture of collegiality, interactive learning environment, flexible scheduling/pacing, and authentic assessment (Keefe & Jenkins, 2002). No definition of “dual teacher role” was given by the panel as for them, this property is vague. Also, the instrument aimed to assess learning modules and not to assess teacher performance in and out of the classroom. Hence, dual teacher role was not included as a component of the instrument. The panel had a single definition for “diagnosis of characteristics” which they referred to as the process of identifying and measuring each learner’s characteristics. This definition is used in the instrument, but the term for the property is changed to “diagnostic assessments.” Two definitions of “culture of collegiality” were presented by the panel, but the property was no longer included in the instrument because it is redundant as it is similar to that of blended learning which is collaboration. Given three definitions of “interactive environment,” the panel agreed that term refers to the interactions in the physical or online classroom. However, interactive learning environment is broad and specified in different properties in the instrument, namely, in-class, out-of-class, and collaborative. Thus, the property “interactive learning environment” had been excluded in the final draft of the instrument. Two definitions of “flexible scheduling/pacing” were given, where they focus on the variety of learning experiences in addition to the accessibility of learning materials. The term flexible scheduling/pacing is changed to “flexibility,” with the definition referring to the variation of learning experiences to suit the need of the learner. Three definitions of “authentic assessment” were suggested, but the term itself is renamed to “summative assessment” to differentiate the property to diagnostic assessment. The instrument refers to the term as the method of measuring learners’ competencies.

Definitions of each property of 21st-century teaching styles were operationalized in accordance with the input of the experts and literature. These are then considered as components of the instrument. The final copy of the instrument no longer divided it into three teaching styles (blended learning, flipped learning, and personalized learning). Instead, the instrument revised, combined, or removed some properties and formed nine components: self-paced learning, out-of-class activities, live events, in-class activities, collaborative activities, diagnostic assessments, summative assessments, flexibility, and personalization. Table 1 shows the definition of each component with the 21st-century teaching style it represents.

Table 1.

Definition of the 21st-Century Teaching Style Components.

Component	Definition	21st-century teaching style
Self-paced learning	Refers to learning experiences that the learner completes individually	Blended
Out-of-class activities	Refers to activities done outside the classroom in preparation for in-class activities	Flipped
Live events	Refers to synchronous, instructor-led learning events in which all learners participate at the same time	Blended
In-class activities	Refers to the use of in-class time in creating rich learning opportunities	Flipped
Collaborative activities	Refers to learning activities in which students collaborate which each other	Blended/personalized
Diagnostic assessments	Refers to the process of identifying and measuring each learner’s characteristics	Personalized/blended
Summative assessments	Refers to the method of measuring learners’ competencies	Personalized/blended
Flexibility	Refers to the variation of learning experiences to suit the need of the learner	Personalized/flipped
Personalization	Refers to the learners’ ownership of their learning experiences	Personalized

Once the definitions of each property of 21st-century teaching styles were finalized, indicators were gathered from two sources—experts and related literature—to serve as items. Each quality indicators of blended, flipped, and personalized learning are categorized into nine components to serve as items for the instrument. Quality indicators for “live events,” “self-paced learning,” “collaboration,” and “assessment” were all gathered from existing literature and were used as items for their respective components in the instrument. However, the “assessment” property of blended learning was divided into two components—diagnostic and summative. Three quality indicators from personalized learning are transferred to “diagnostic assessments” component, while one indicator is transferred to “summative assessments” component.

All quality indicators for flipped learning were drawn from the literature. Two quality indicators under the property of “flexible learning” were used as an item for “flexibility” component, while one indicator is used as an item under “personalization” component. All indicators under the property of “learning culture” served as items for “in-class” component, while all indicators under property of “intentional content” served items for “out-of-class” component. None of the indicators under the property “professional educator” was used in the instrument as the property is no longer considered in the instrument.

Quality indicators of an effective personalized learning were gathered as items for the instrument. Indicators under the property of “diagnosing of characteristics” were gathered from both experts and literature, where two indicators are used as items for “diagnostic assessments” component. Indicators for the property of “interactive learning environment” were also from both experts and literature where an indicator transferred to “flexibility” component, and four indicators are transferred to “personalization” component. The indicators under the property of “flexible scheduling/pacing” were taken completely from experts’ recommendations. A single indicator is used as an item for the “flexibility” component. The indicators for “authentic assessment,” similar to property of flexible scheduling/pacing, were also gathered from experts. All indicators under this property were used as items for the “summative assessments” component of the instrument.

Identification of components and quality indicators of 21st-century teaching styles resulted to an assessment tool for learning modules, the 21CELMAT. The developed instrument adopted the nine components deliberated by the assessment tool development panel. Collected quality indicators provided six items for out-of-class activities component; five items each for self-paced learning, live events, in-class activities, and personalization components; and four items each for collaborative activities, diagnostic assessments, and summative assessments components.

Content Validity and Instrument Reliability

The content validity of the 21CELMAT was established through evaluation of its item’s appropriateness in assessing the nine components of 21st-century teaching. For this purpose, members of the assessment tool development panel were sent a questionnaire to agree among the experts the extent of appropriateness of each indicator. Table 2 shows the summary of ratings panel members gave for each item.

Table 2.

Results of Content Validation.

Indicators	M1	M2	M3	M4	M5	M6	Ave	Interpretation
Live Event 1	4	4	4	2	3	3	3.33	Appropriate
Live Event 2	4	4	3	4	3	4	3.67	Very appropriate
Live Event 3	3	4	4	3	4	4	3.67	Very appropriate
Live Event 4	3	4	3	3	4	4	3.50	Very appropriate
Live Event 5	4	4	3	2	4	4	3.50	Very appropriate
Self-Paced Learning 1	4	4	4	4	3	4	3.83	Very appropriate
Self-Paced Learning 2	4	4	3	4	3	4	3.67	Very appropriate
Self-Paced Learning 3	4	4	4	4	4	4	4.00	Very appropriate
Self-Paced Learning 4	4	4	4	4	4	4	4.00	Very appropriate
Self-Paced Learning 5	4	4	4	4	3	4	3.83	Very appropriate
Collaborative Activities 1	4	4	4	4	3	4	3.83	Very appropriate
Collaborative Activities 2	4	4	3	4	4	4	3.83	Very appropriate
Collaborative Activities 3	4	4	3	4	4	4	3.83	Very appropriate
Collaborative Activities 4	4	4	3	4	4	4	3.83	Very appropriate
In-Class Activities 1	3	4	4	2	4	4	3.50	Very appropriate
In-Class Activities 2	4	4	3	4	4	4	3.83	Very appropriate
In-Class Activities 3	4	4	4	2	4	4	3.67	Very appropriate
In-Class Activities 4	4	4	4	4	4	4	4.00	Very appropriate
In-Class Activities 5	4	4	4	4	4	4	4.00	Very appropriate
Out-of-Class Activities 1	4	4	4	2	3	4	3.50	Very appropriate
Out-of-Class Activities 2	4	4	3	2	4	4	3.50	Very appropriate
Out-of-Class Activities 3	4	2	3	2	3	4	3.00	Appropriate
Out-of-Class Activities 4	4	4	3	4	4	4	3.83	Very appropriate
Out-of-Class Activities 5	4	4	4	4	4	4	4.00	Very appropriate
Out-of-Class Activities 6	3	4	4	4	4	4	3.83	Very appropriate
Flexibility 1	4	4	4	4	3	4	3.83	Very appropriate
Flexibility 2	3	4	4	4	3	4	3.67	Very appropriate
Flexibility 3	3	4	3	4	4	4	3.67	Very appropriate
Flexibility 4	3	4	3	4	4	4	3.67	Very appropriate
Personalization 1	4	4	4	4	4	4	4.00	Very appropriate
Personalization 2	4	2	3	4	4	4	3.50	Very appropriate
Personalization 3	4	4	4	4	3	4	3.83	Very appropriate
Personalization 4	4	2	4	4	3	4	3.50	Very appropriate
Personalization 5	4	4	4	4	4	4	4.00	Very appropriate
Diagnostic Assessment 1	4	4	4	4	3	4	3.83	Very appropriate
Diagnostic Assessment 2	4	4	4	4	4	4	4.00	Very appropriate
Diagnostic Assessment 3	4	4	3	4	4	4	3.83	Very appropriate
Diagnostic Assessment 4	3	4	4	4	4	4	3.83	Very appropriate
Summative Assessment 1	4	4	4	4	3	4	3.83	Very appropriate
Summative Assessment 2	4	4	3	4	4	4	3.83	Very appropriate
Summative Assessment 3	4	4	3	4	4	4	3.83	Very appropriate
Summative Assessment 4	4	4	4	4	3	4	3.83	Very appropriate

Results of content validity showed that 40 items, having average ratings of more than 3.50, were interpreted as very appropriate, while 2 items, having a mean of 3.33 and 3.00, respectively, were interpreted as appropriate. None of the items was interpreted as inappropriate and very inappropriate. These results established the validity of the items to determine quality of the components they represent. The instrument was revised in accordance to the experts’ comments during the content validity phase. Also, items were rewritten into statements that would make them grammatically parallel.

The 21CELMAT was pilot tested to seven existing learning modules to establish reliability. The members of the existing modules assessment panel were tasked to rate each learning modules using the assessment tool. The ratings of per-item-response of the panel members were correlated to show agreement between their ratings. Through this process, interrater reliability was established. Intraclass correlation coefficient and p value were used to verify significant correlation between the ratings of the panel member. Table 3 shows the results of intraclass correlation analysis for interrater reliability.

Table 3.

Results of Interrater Reliability.

Module no.	Intra-class correlation coefficient	Value	p
1	.523	6.029	.001
2	.778	11.769	.000
3	.230	2.951	.031
4	.324	2.302	.074
5	.832	14.941	.000
6	.789	16.226	.000
7	.459	3.429	.017

Results of intraclass correlation analysis show that Module 1 (r = .778, p < .05), Module 5 (r = .832, p < .05), and Module 6 (r = .789, p < .05) had very high and significant correlation of ratings between panel members (Hemphill, 2003). Module 1 (r = .523, p < .05) had a high and significant correlation between member ratings. Module 7 (r = .459, p < .05) had moderate correlation while Module 3 (r = .230, p < .05) had low correlation, but both are still considered significant. Only Module 4 (r = .324, p > .05) with a moderate correlation had a not significant result. The general strength and significance of correlation among panel members ratings of seven different learning modules established the interrater reliability of the instrument.

Furthermore, intrarater reliability of the assessment tool was tested through the assessment of the same module twice with 3 weeks interval in between. Intraclass correlation analysis was used to determine statistical correlation between the ratings of the evaluations. The strength and significance of the correlation established the interrater reliability of the tool. Table 4 shows the results of intraclass correlation analysis for intrarater reliability.

Table 4.

Results of Intrarater Reliability.

Member no.	Intra-class correlation coefficient	Value	p
1	.935	26.778	.000
2	.806	9.000	.003
3	.774	11.105	.001

Results of intraclass correlation analysis shows that member 1 (r = .935, p < .05) had almost perfect and significant correlation between her ratings before and after the 3-week interval (Hemphill, 2003). On the other hand, Member 2 (r = .806, p < .05) and Member 3 (r = .774, p < .05) had a very high and significant correlation. The general strength and significance of correlation between ratings amid an interval establishes the intrarater reliability of the assessment tool.

Result

Data gathering and analysis of this study culminated to the development of 21CELMAT, an instrument developed to assess the quality of 21st-century teaching styles (blended learning, flipped learning, and personalized learning) incorporated in a learning module. Specifically, the instrument measures the quality of nine components of 21st-century learning modules: self-paced learning, out-of-class activities, live events, in-class activities, collaborative activities, diagnostic assessments, summative assessments, flexibility, and personalization. It is important to emphasize that the instrument for evaluating teaching plans for instruction could not be used to assess actual teaching.

The 21CELMAT uses a 5-point scale to rate each component of a 21st-century learning module in which ratings are based on the compliance of the learning module to the standards stipulated in the instrument. Each component has four to five components. Self-paced learning, live events, in-class activities, and personalization have five indicators. Collaborative activities, diagnostic assessment, summative assessment, and flexibility have four indicators. Out-of-class activities has six indicators. Table 5 shows the interpretations and points for the 5-point scale.

Table 5.

Instrument 5-Point Scale.

Excellent	5 points	100% of the indicators are present
Very satisfactory	4 points	99%–75% of the indicators are present
Satisfactory	3 points	74%–50% of the indicators are present
Moderately satisfactory	2 points	49%–25% of the indicators are present
Needs improvement	1 point	24%–0% of the indicators are present

In assessing the learning module, the assessor must identify the nine components in the module and rate each component by detecting if the corresponding indicators are present. Once each component is rated, the assessor will get the average rating of the components. To get the average rating, the assessor will add all the ratings for all components which will total to 45 points. Then, divide results to nine which is the number of components. A learning module could be rated as a total or per component depending on the objective of the assessment procedure. Table 6 shows the format and selected items of 21CELMAT.

Table 6.

Instrument Format and Selected Items.

	Excellent	Very satisfactory	Satisfactory	Moderately satisfactory	Needs improvement
Self-Paced Learning: Refers to learning experiences that the learner completes individually.
Indicators:• Considers the diverse needs of learners	5 indicators are present	4 indicators are present	3 indicators are present	2 indicators are present	1 to 0 indicator is present
Out-of-Class Activities: Refers to activities done outside the classroom in preparation for in-class activities.
Indicators: • Provides memorization component with examples used in context and shows learners how to annotate a piece of text.	6 indicators are present	5 indicators are present	4 to 3 indicators are present	2 indicators are present	1 to 0 indicator is present
Live Events: Refers to synchronous, instructor-led learning events in which all learners participate at the same time.
Indicators: • Creates opportunities for deeper learning and discussion of the lesson. • Offers learners numerous meaningful interactions with the teacher.	5 indicators are present	4 indicators are present	3 indicators are present	2 indicators are present	1 to 0 indicator is present
In-Class Activities: Refers to the use of in-class time in creating rich learning opportunities.
Indicators: • Gives students opportunities to engage in meaningful activities without the teacher the center of the class’ attention.	5 indicators are present	4 indicators are present	3 indicators are present	2 indicators are present	1 to 0 indicator is present
Collaborative Activities: Refers to learning activities in which students collaborate which each other.
Indicators: • Promotes group interactions that allow learners to dissect information together.	4 indicators are present	3 indicators are present	2 indicators are present	1 indicator is present	0 indicator is present
Diagnostic Assessments: Refers to the process of identifying and measuring each learner’s characteristics.
Indicators: • Uses data from learners’ profile and standardized tests results as a reference in adjusting learning experiences.	4 indicators are present	3 indicators are present	2 indicators are present	1 indicator is present	0 indicator is present
Summative Assessments: Refers to the method of measuring learners’ competencies.
Indicators: • Measures attainment of lesson objectives comprehensively.	4 indicators are present	3 indicators are present	2 indicators are present	1 indicator is present	0 indicator is present
Flexibility: Refers to the variation of learning experiences to suit the need of the learner.
Indicators: • Uses diversified learning activities that provides learners with choices on their learning experiences.	4 indicators are present	3 indicators are present	2 indicators are present	1 indicator is present	0 indicator is present
Personalization: Refers to the learners’ ownership of their learning experiences.
Indicators:• Gives opportunities to learner to take part in setting objectives of module.	5 indicators are present	4 indicators are present	3 indicators are present	2 indicators are present	1 to 0 indicator is present

Conclusion

The purpose of this study is to develop an efficient procedure in assessing existing learning modules vis-à-vis 21st-century teaching styles, leading to the development of a prototype e-learning module. Literatures presented various indicators and assessment procedures in implementing 21st-century skills in instruction. However, there is still a gap in providing assessment procedure that could be done before implementation regularly. This study resulted to the development of 21CELMAT, an instrument that could provide pre-implementation procedure suited to be used regularly. The 21CELMAT determines the quality of a learning module in terms of self-paced learning, out-of-class activities, live events, in-class activities, collaborative activities, diagnostic assessments, summative assessments, flexibility, and personalization. The study revealed that blended learning, flipped learning, and personalized learning are complementary entities requiring holistic assessment. During the development of the 21CELMAT, the researcher found out that indicators of these three teaching styles overlap. The overlying objectives of the 21st-century teaching styles prompted the research to assess learning modules in terms of nine components instead of per teaching style. Pilot testing 21CELMAT through evaluation of existing learning modules established validity and reliability of the instrument.

Recommendations

The study is limited in assessing mathematics learning modules of one school. Hence, it is recommended to assess modules from other disciplines and institutions to bring more extensive and conclusive results on the condition of 21st-century teaching. Furthermore, the study involved only a limited number of experts. The reliability of 21CELMAT should be tested with a larger pool of content experts to further standardize the developed instrument. For future studies, it is recommended to use 21CELMAT in validating learning modules and instructional materials that could be applied as interventions. Also, continue studies on improving the out-of-class activities is suggested due to their low quality, although exploring on easing the creation of out-of-class activities must be addressed. It is suggested that 21CELMAT be used as standards for 21st-century teaching styles.

Footnotes

Declaration of Conflicting Interests

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Jose Mari M. Calamlam

Author Biography

Jose Mari M. Calamlam is a mathematics and research senior high school teacher in De la Salle Santiago Zobel school. He completed his bachelor's and master's degree in Philippine Normal University, where he graduated with flying colors. His current research endeavors were focused in blended learning. He has presented his studies in various research fora nationally and internationally. He is also active in facilitating seminar-workshops in blended learning and research.

References

Auster

C. J.

(2016). Blended learning as a potentially winning combination of face-to-face and online learning: An exploratory study. Teaching Sociology, 44(1), 39–48.

Bishop

J. L.

Verleger

(2013). The flipped classroom: A survey of the research. American Society for Engineering Education.

Bodell

Hook

(2014). Developing online professional networks for undergraduate occupational therapy students: An evaluation of an extracurricular facilitated blended learning package. British Journal of Occupational Therapy, 77(6), 320–323.

Boyer

(2013). The flipped classroom. TLN Journal, 20(1), 28–29.

Carman

J. M.

(2002). Blended learning design: Five key ingredients. https://www.it.iitb.ac.in/∼s1000brains/rswork/dokuwiki/media/5_ingredientsofblended_learning_design.pdf

Chen

C. M.

Lee

H. M.

Chen

Y. H.

(2005). Personalised e-learning system using item response theory. Computers & Education, 44(3), 237–255.

Childress

Benson

(2014). Personalised learning for every student every day. Phi Delta Kappan, 95(8), 33–38.

Cho

K. C.

Shin

(2014). Operational effectiveness of blended e-learning program for nursing research ethics. Nursing Ethics, 21(4), 484–495.

Correa

(2015). Flipping the foreign language classroom and critical pedagogies: A (new) old trend. Higher Education for the Future, 2(2), 114–125.

10.

Cottle

N. R.

Glover

R. J.

(2011). Teaching human development: A case for blended learning. Teaching of Psychology, 38(3), 205–208.

11.

Courcier

(2007). Teachers’ perceptions of personalised learning. Evaluation & Research in Education, 20(2), 59–80. https://doi.org/10.2167/eri405.0

12.

Donnelly

(2006). Blended problem‐based learning for teacher education: Lessons learnt. Learning, Media and Technology, 31(2), 93–116.

13.

Forsey

Low

Glance

(2013). Flipping the sociology classroom: Towards a practice of online pedagogy. Journal of Sociology, 19, 471–485.

14.

Fulton

K. P.

(2012). 10 reasons to flip. Phi Delta Kappan, 94(2), 20–24.

15.

Garn

A. C.

Jolly

J. L.

(2013). High ability students’ voice on learning motivation. Journal of Advanced Academics, 25(1), 7–24.

16.

Graham

(2004). Blended learning systems: Definition, current trends, and future directions. In Handbook of blended learning: Global Perspectives, local designs. Local Designs, 2, 3–18‌.

17.

Hall

A. A.

DuFrene

D. D.

(2015). Best practices for launching a flipped classroom. Business and Professional Communication Quarterly, 79(2), 234–242.

18.

Hamdan

McKnight

Arfstrom

(2013). A white paper based on the literature review titled a review of flipped learning. Flipped Learning Network/Pearson/George Mason University.

19.

Hemphill

(2003). Interpreting the magnitude of correlation coefficients. The American Psychologist, 58, 78–79. https://doi.org/10.1037/0003-066X.58.1.78

20.

Hodges

T. S.

Weber

N. D.

(2015). Making heads or tails of classroom flipping. Kappa Delta Pi Record, 51(2), 57–63.

21.

Hughes

(2007). Using blended learning to increase learner support and improve retention. Teaching in Higher Education, 12(3), 349–363. https://doi.org/10.1080/13562510701278690

22.

Ike

Okechukwu

I. B.

(2015). Effective learning systems through blended teaching modules in adult secondary education systems in developing nations: Need for partnership. Journal of Adult and Continuing Education, 21(1), 18–30.

23.

Jenkins, H. (2009). Confronting the challenges of participatory culture: Media education for the 21st century (p. 145). The MIT Press.

24.

Keefe

J. W.

Jenkins

J. M.

(2002). Personalised Instruction. Phi Delta Kappan, 83(6), 440–448.

25.

Kleber

(2015). Differentiation through BLENDED LEARNING. Leadership, 44(3), 20–24. http://search.proquest.com/docview/1645566280?accountid=173015

26.

Kucirkova

Messer

Sheehy

(2014). Reading personalised books with preschool children enhances their word acquisition. First Language, 34(3), 227–243.

27.

Kuo

Y. C.

Belland

B. R.

Schroder

K. E.

Walker

A. E.

(2014). K-12 teachers’ perceptions of and their satisfaction with interaction type in blended learning environments. Distance Education, 35(3), 360–381.

28.

Lee

C. H.

Yeh

Kung

R. J.

Hsu

C. S.

(2007). The influences of learning portfolios and attitudes on learning effects in blended e-learning for mathematics. Journal of Educational Computing Research, 37(4), 331–350.

29.

Love

Hodge

Grandgenett

Swift

A. W.

(2014). Student learning and perceptions in a flipped linear algebra course. International Journal of Mathematical Education in Science and Technology, 45(3), 317–324.

30.

McEvoy

C. S.

Cantore

K. M.

Denlinger

L. N.

Schleich

M. A.

Stevens

N. M.

Swavely

S. C.

Novick

M. B.

(2014). Use of medical students in a flipped classroom programme in nutrition education for fourth-grade school students. Health Education Journal, 75(1), 38–46.

31.

Monteiro

Morrison

(2014). Challenges for collaborative blended learning in undergraduate students. Educational Research and Evaluation, 20(7–8), 564–591. https://doi.org/10.1080/13803611.2014.997126

32.

Moran

C. M.

Young

C. A.

(2015). Questions to consider before flipping. Phi Delta Kappan, 97(2), 42–46.

33.

Flipped Learning Network (FLN) (2014). The four pillars of FLIP™. www.flippedlearning.org/definition

34.

O’Donnell

Sharp

Wade

(2015). A review of personalised e-learning: Towards supporting learner diversity. International Journal of Distance Education Technologies, 13(1), 22–47.

35.

Partnership for 21st Century Skills. (2011a). Framework for 21st century learning.

36.

Partnership for 21st Century Skills. (2011b). Learning for the 21st Century.

37.

Peters, M. A. (2009). Personalization, Personalized Learning and the Reform of Social Policy: the prospect of molecular governance in the digitized society. Policy Futures in Education, 7(6), 615–627.

38.

Peterson

D. J.

(2016). The flipped classroom improves student achievement and course satisfaction in a statistics course: A quasi-experimental study. Teaching of Psychology, 43(1), 10–15.

39.

Roccetti

Salomoni

Bonfigli

M. E.

(2001). A design for a simulation-based multimedia learning environment. Simulation, 76(4), 214–221.

40.

Sayeski

K. L.

Hamilton-Jones

(2015). The efficacy of IRIS STAR legacy modules under different instructional conditions. Teacher Education and Special Education: The Journal of the Teacher Education Division of the Council for Exceptional Children, 38(4), 291–305.

41.

Schrader

P. G.

Lawless

K. A.

(2011). Research on immersive environments and 21st century skills: An introduction to the special issue. Journal of Educational Computing Research, 44(4), 385–390.

42.

Sims

D. E.

Burke

C. S.

Metcalf

D. S.

Salas

(2008). Research-based guidelines for designing blended learning. Ergonomics in Design: The Quarterly of Human Factors Applications, 16(1), 23–29.

43.

Singh, H. (2003). Building effective blended learning programs. Educational Technology-Saddle Brook Then Englewood Cliffs NJ-, 43(6), 51–54.

44.

Sivakumar

Praveena

Saranya

(2015). Improving content personalisation through ant optimisation in E-learning. Advances in Natural and Applied Sciences, 9, 581–587.

45.

Song

Wong

L. H.

Looi

C. K.

(2012). Fostering personalised learning in science inquiry supported by mobile technologies. Educational Technology Research and Development, 60(4), 679–701.

46.

Steed

(2012). The flipped classroom. Hassocks: Economics, Business and Enterprise Association.

47.

Steiner

C. M.

Nussbaumer

Albert

(2009). Supporting self-regulated personalised learning through competence-based knowledge space theory. Policy Futures in Education, 7(6), 645–661.

48.

Thyagharajan

K. K.

Nayak

(2007). Adaptive content creation for personalised e-learning using web services. Journal of Applied Sciences Research, 3(9), 828–836.

49.

Tucker

(2012). The flipped classroom. Education Next, 12, 82–83.

50.

Velayutham

Aldridge

Fraser

(2011). Development and validation of an instrument to measure students’ motivation and self‐regulation in science learning. International Journal of Science Education, 33(15), 2159–2179. https://doi.org/10.1080/09500693.2010.541529

51.

Welker

Berardino

(2005). Blended learning: Understanding the middle ground between traditional classroom and fully online instruction. Journal of Educational Technology Systems, 34(1), 33–55.