Development and Validation of a Health Information Technology Curriculum: Toward More Meaningful Use of Electronic Health Records

Abstract

The Health Information Technology for Economic and Clinical Health (HITECH) initiative of the 2009 American Recovery and Reinvestment Act in the United States was intended to promote meaningful use of electronic health records (EHRs). This article reports on a comprehensive, three-stage model employed to develop, validate, and facilitate regional implementation of a health care information technology curriculum for workforce development as part of that coordinated national effort. Building on needs assessed at the national level, the the stages involved: (a) curriculum design, (b) assuring quality of curricular products through validation and revision, and (c) design of a systems-based, curriculum implementation and evaluation protocol. The objective of the project was to prepare health care professionals with competencies necessary to implement EHRs meaningfully, thereby improving patient care. We produced content-validated and usable versions of curriculum goal frameworks, student learning outcomes, instructional materials, multiple-choice tests, and performance assessments for 40 HITECH units across four curriculum components: (a) Public Health IT, (b) Vendor Specific Systems, (c) Usability and Human Factors, and (d) Training and Instructional Design. The three-stage approach is an innovative, replicable, and stakeholder-oriented method that fills a gap in the health care literature for curriculum design.

Keywords

assessment curriculum development health information technology program evaluation

The Health Information Technology for Economic and Clinical Health (HITECH) initiative of the American Recovery and Reinvestment Act (ARRA) in the United States, enacted in 2009, was intended to promote “meaningful use” of electronic health records (EHRs) by clinicians and hospital workers, thereby making improvements to health care delivery systems and the overall care of patients (Menachemi et al., 2011). Implicit in the federally legislated requirement for using EHRs meaningfully is an assumption that the quality of care would improve in substantive ways because of EHRs. The ARRA also assumes that current practices and behaviors of health care professionals would change in significant ways in the workplace, incorporating use of EHRs with ease. Such behavioral changes, however, further assume that health care professionals possess the attitudes, knowledge, and capacity to integrate EHRs within their day-to-day practices and existing patient care infrastructures. At a minimum, an important determinant of successful EHR implementation is a well-trained professional workforce demonstrating best practices with respect to EHR adoption (Hersh, 2010). Shortly, an estimated 40,000 health information technology (HIT) professionals will be required to exhibit “meaningful use” of EHRs in practice settings (Hersh & Wright, 2008).

Theoretical and Policy Rationale

Available research evidence suggests that quality of health care can indeed be improved by the effective use of EHRs, and more generally, through adding HIT (Buntin, Burke, Hoaglin, & Blumenthal, 2011; Chaudhry et al., 2006; Hillestad et al., 2005; Menachemi & Brooks, 2006; Wang et al., 2003). Positive outcomes include the enhancement of preventive care, reduction of medication errors, and a decrease in health care costs. Researchers have recommended that the goal should be to use EHR technology in ways that will improve the quality, safety, and efficiency of health care (Classen & Bates, 2011). For example, electronic data can be aggregated and responsibility of patient outcomes shared at different levels, which in turn can improve the quality of health care systems at each point of service (Buntin, Jain, & Blumenthal, 2010). Little is known, however, regarding the precise kinds of knowledge and skills the current HIT workforce need. Moreover, there is little information regarding the best ways to prepare and train the existing workforce for optimal use of HIT (Buntin et al., 2011). Concerns have also been raised that classroom-based instruction may need to be supplemented with more on-the-job training (DeVore & Figlioli, 2010).

Regardless of the mode of training, workforce development programs must begin with the design of curricula and a comprehensive plan for implementing the curricula via existing education and health care system infrastructures. In anticipation of the greater demands on the health care workforce to use EHRs in the United States, federal grants totaling $118 million were allocated under President Barack Obama’s Administration to expand the cadre of professionals needed to develop, implement, and evaluate the EHR systems being adopted under HITECH (Blumenthal, 2010).

National Needs for HIT Curriculum Development

Specifically, based on federal-level assessments of needs, the Office of the National Coordinator (ONC) provided funds for four initiatives to prepare the HIT workforce using community colleges as a major vehicle. In the United States, the term community college refers to 2-year public institutions that grant certificates and 2-year associate’s degrees in technical fields, and also prepare students for transfers to a college or university for 4-year degrees in the liberal arts, sciences, and other fields. Historically, community colleges have played a key role in workforce development programs as part of economic expansion.

The funded initiatives were (a) establishment of Curriculum Development Centers (CDCs) to develop workforce training materials that would be implemented via regional community colleges; (b) Community College Consortia to educate prospective HIT professionals in 6 months or less; (c) a Competency Examination Program to implement HIT competency testing of individuals who completed short-term, non–degree training programs or other types of training if they already had the relevant job experience; and (d) university-based training to increase the availability of individuals qualified to serve in specific health IT professional roles, rapidly and sustainably (HITECH Programs & Advisory Committees, 2012).

At the time these programs were funded, the ONC determined that 12 “workforce roles,” grouped in three broad categories, would be required: mobile personnel who moved from site to site to assist in implementing EHR systems; permanent staff to maintain and support the implemented EHR systems; and clinical and public health informatics experts who would manage, evaluate, educate, and perform further research and development of EHR systems. Half of the HIT workforce roles were deemed suitable for 6-month training programs offered via community colleges, leading to HIT certificates, while the other half were to be addressed through 1- to 2-year-long university-based programs.

Purpose

This article presents a comprehensive, three-stage model to develop, validate, and facilitate regional implementation and evaluations of HIT curriculum components in 6-month workforce development programs. It reports on the implementation of key portions of the proposed model as a part of the coordinated national effort funded by the ONC and presents illustrative curriculum products and empirically collected evidence of stakeholder perceptions of their quality.

The study focuses specifically on the development of four curriculum components or modules, all of which were incorporated in a Community College HIT certificate program. The objective of the overall project was to facilitate the operation of programs that help prepare health care and information technology professionals with competencies necessary to implement EHRs meaningfully. As mentioned, promoting “meaningful use” of EHRs was mandated by ARRA in 2009 and expected to improve patient care in hospitals and clinics.

A Comprehensive Model for Curriculum Development, Validation, and Implementation

At the Columbia University CDC, curriculum development was informed by principles for designing com-petency-based educational programs drawn from the literatures on standards-based education reforms, assessment design, and participatory evaluation (Chatterji, 2003; Cousins & Whitmore, 1998; McLaughlin & Shepard, 1995; Shepard, 2000). Premised on information from needs assessments already completed at the national level in the HIT area, the three main stages and embedded steps in our method were as follows. Terms in quotes below and in referenced figures are defined immediately following the presentation of the model next.

Stage 1: Curriculum Design

Development of domains of expected learning outcomes and instructional goal/objective frameworks

Designing of instruction and assessment materials matched with “domains of expected learning outcomes”

Alignment of expected learning outcomes, instruction, and assessment materials to ensure “content-based validity” (Figure 1)

Figure 1.

Integrated model of curriculum development to ensure alignment of objectives, instruction, and assessments. HITECH = Health Information Technology for Economic and Clinical Health.

Stage 2: Assuring Quality of Curricular Products

Selecting and training of “expert” teams of internal and external validators, including “stakeholders” in field settings

Conducting of guided and structured reviews by experts to “validate” the content and ensure quality of curricular “products”

Revision and improvement of products iteratively

Stage 3: Systems-Based Protocol for Facilitating Curriculum Implementation and Evaluation

Development of a systems-based “logic model” to coordinate and guide different aspects of new curriculum development, implementation, and evaluation in organizational or community contexts (Figure 2)

Development of a long-term curriculum evaluation plan with formative and summative components, guided by the logic model

Refinement of the evaluation protocol and logic model through discussions with stakeholders to facilitate adoption, implementation, use, and continuous improvement of the new curriculum in organizational or community contexts

Dissemination of the new curriculum in field settings through appropriate channels, supported with necessary orientation, training, and resources

Figure 2.

HITECH project viewed in a system-wide setting: A logic model to guide curriculum adoption, implementation, and evaluation by stakeholders. HITECH = Health Information Technology for Economic and Clinical Health; HIT = health information technology; EHR = electronic health record.

Definitions

Stage 1

Salient terms in the aforementioned model are now defined. In our approach to curriculum design, “domains of expected learning outcomes” denote formally organized statements of what students would be expected to know and be able to do following exposure to the new curriculum or an instructional module. Eventually, these statements served as the objectives that guided the delivery of instructional units.

A first step in curriculum design is to operationally define students’ learning outcomes as statements of observable behaviors, actions, tasks, or responses that competent students would likely demonstrate following training. These statements are then organized as concentric or developmental hierarchies (e.g., as tree-diagrams), suggesting how specific types of knowledge/skill mastery would lead to more broad and culminating performance outcomes at the end of a program of study. Outcome statements, as mentioned, specify explicitly what students should know and be able to do in subject area domains following exposure to an educational program. For instructors, these domains of learning outcomes eventually serve as instructional goals and objectives to which all other instructional materials can be tied (Chatterji, 2003).

Curricular “products” include three main sets of items generated in Stage 1 of the model: frameworks of expected learning outcomes, instructional materials, and assessments in particular units of instruction or curricular segments. In the HIT project, the term also included technology-based systems of product delivery that are not the focus of this article.

Stage 2

Content-based validity is typically pursued when designing educational tests and assessments (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education [AERA, APA, & NCME], 2014; Chatterji, 2003; Shepard, 2000). It is a process that helps ensure a match exists between the targeted learning outcomes and material to be taught, and the content to be covered on assessments. Figure 1 illustrates the concept.

The term validate refers to blind expert- or peer-reviews of the content, format, presentation, usability, and all other aspects of curricular products, such as the HIT content selected for an instructional module or unit. Content-based validity results from applying the “alignment” principle in Stage 1 of the Curriculum Design process. In practice, this occurs by systematically matching the outcomes, instruction, and assessments during the design phase and confirming the match through an expert review process called “validation.”

In the context of validation, “experts” are knowledgeable academic or subject-matter specialists, curriculum developers, instructional and assessment specialists, and practitioners who are not involved in the curriculum design processes, but are deemed able to provide objective and constructive feedback to improve the quality of products. Such experts are carefully selected by the project leaders and curriculum design team as appropriately qualified. In our project, these individuals had PhD or master’s-level training in fields of health informatics, health care systems, health education, curriculum design, and assessment-evaluation, or were experienced practitioners in HIT areas as classroom instructors, leaders, or senior managers.

Stage 3

References to “stakeholders” in the model include all experts mentioned above, and also incorporate all individuals and entities with a vested interest in the outcomes of the overall curriculum development effort. Figure 2 contains a graphic presenting the system-wide architecture within which the present HIT curriculum was expected to operate, showing all the potential stakeholders who were expected to be involved in different stages of our project (Cousins & Whitmore, 1998). Stakeholders, in our project, are actors involved in various aspects of curriculum design, delivery, dissemination, adoption, evaluation, and implementation, as well as students and health care professionals.

A “logic model” in Stage 3 (Figure 2) is a tool to facilitate adoption, implementation, evaluation, and use of a new curriculum. It is a causal path diagram that shows how different components and variables associated with a new curriculum or educational program would likely interrelate, leading to expected outcomes in the targeted population (Donaldson, 2007). As shown in Figure 2, assumptions about the new curriculum, expected processes, and how trainee outcomes will be manifested can be mapped with a logic model. These pathways are then tested as a part of evaluating a new program.

There could be more than one way to conceptualize logic models for this purpose. We adopted a systems-informed approach (Donaldson, 2007; Stufflebeam, 2004). Such logic models help in guiding various stakeholders toward more effective implementation of new programs. When placed in the larger organizational or community context as shown in Figure 2, such logic models raise greater awareness of system-wide factors that can either facilitate or impede the implementation of a new curriculum

Theoretical Support for the Proposed Model

Our proposed approach is based on a synthesis of education, assessment design, and evaluation literature bases. Classical guidelines for curriculum design (Tyler, 1949) recommend an objectives-based approach to curriculum design as the ideal, guided by needs assessments of larger social, community, and student factors. Tyler (1949), known as the “father” of curriculum development and evaluation, recommended that we draw on studies of philosophical, sociological, and psychological factors that could affect learning and learner development to formulate the objectives, learning experiences, and expected learning outcomes of a curriculum. In his approach, objectives were also statements of proficiencies that students should be capable of demonstrating as a result of exposure to organized learning experiences.

In designing medical education curricula more recently, Kern, Thomas, Howard, and Bass (1998) also proposed a similar six-step process. Their steps focus on (a) problem (or need) identification and a general needs assessment, (b) needs assessment of targeted learners, (c) goals and objectives, (d) educational strategies, (e) implementation, and (f) evaluation and feedback.

Our proposed three-stage model builds on both these traditions, but extends the same with Stage 3. In our case, the need or problem identification occurred at the federal level, leading to the 2009 ARRA legislation and HIT policy context for facilitating meaningful EHR use. Those national needs instigated the establishment of CDCs and the present project at the Columbia CDC.

However, as seen, missing in both historical and more recent models of curriculum development is adequate attention to factors that could affect new curriculum adoption, dissemination, and implementation by stakeholders at all levels of the organizational infrastructure as it is taken to scale (in our case, the health care system). Many new reforms, including legislatively driven standards-based reforms in education, have been known to fail due to a lack of attention to this critical area.

For example, standards-based curriculum reforms are continuing today under the auspices of the 2001 No Child Left Behind and Race to the Top (see http://www2.ed.gov/nclb/landing.jhtml) initiatives at the elementary and secondary education levels in the United States (McLaughlin & Shepard, 1995; U.S. Department of Education, 2010; Wiggins & McTighe, 2005). However, there has been a fierce national backlash to these reforms, now called the Common Core State Standards reforms (see Thousands Refuse Common Core Testing at http://www.huffingtonpost.com/alan-singer/thousands-refuse-common-c_b_9631956.html). Currently, several state education systems have withdrawn because of a lack of readiness among key stakeholders of schools and school districts to adopt the proposed changes on a large scale. Our proposed framework attempts to address this well-documented issue.

Of note, and consistent with our approach, the standards-based reforms approach called for challenging performance standards and clearly defined statements of expected learning outcomes to serve as goal frameworks for guiding educational program development (McLaughlin & Shepard, 1995). Also consistent with our approach, standards-based reforms emphasized the need for a close match of curriculum goal frameworks and expected learning outcomes with instructional and assessment activities in the classroom (Figure 1). This alignment principle, recommended by assessment and curriculum experts from the AERA, APA, and NCME (2014; Chatterji, 2003; Nitko, 1989; Shepard, 2000; Wang et al., 2003; Wiggins, 1998), ensures that classroom assessment exercises have the essential property of “content-based validity.” That is, what is tested is also what is taught, and vice versa.

The backlash to Common Core testing suggests, instead, that key stakeholders and classroom instructors on a system-wide scale did not have access to the resources, training, or the orientation necessary to disseminate, implement, and use the new curriculum effectively. In our conceptualization of the model, Stage 3 was explicitly added to avoid such contingencies.

Figure 2 depicts how the HIT training curriculum was expected to be implemented within a larger system of health care delivery. To account for this complex infrastructure, the project team believed a systems-based logic model and evaluation protocol was necessary to complete our comprehensive design methodology.

The logic model presented in Figure 2 and a long-term evaluation protocol were added products intended for use during pilot-testing of the curriculum on a small scale, as well as beyond, by stakeholders and project leaders (Stage 3). The curriculum components were expected to be implemented at community colleges. Because of the historically strong emphasis these institutions place on workforce development, the potential for trainees to find internship and job-placement opportunities through them was higher. These two products were intended to inform both formative and summative evaluations of the curriculum as the program was taken to scale in field settings (Chen, 1990; Donaldson, 2007).

Formative evaluations, conducted in the course of developing or improving new products, were begun with the content validation of the HIT curriculum materials that is reported here (Scriven, 1991). As the new curriculum is introduced in college settings, site-based formative and process evaluations to inform continuous improvements to the educational program within the organizations were also expected to follow. Summative evaluations, conducted to make final pronouncements of the merits of a new program or curriculum, typically focus on gathering evidence of learner outcomes (Scriven, 1991). Such studies, although outside the scope of the reported project, were also expected to follow at suitable junctures.

We employed a participatory approach to curriculum design at the Columbia University CDC. This meant working closely with all stakeholders of the project. Variously labeled as responsive evaluation (Stake, 1967), utilization-focused evaluation (Patton, 1997), stakeholder-based evaluation (Cousins & Earl, 1995), or participatory evaluation (Cousins & Earl, 1995), participant-oriented approaches to curriculum development, implementation, and evaluation involve partnerships of assessment and evaluation experts with various stakeholders. In the HIT project, our partners included curriculum designers, community college program leaders, health system policy makers and leaders, teachers, and subject-area specialists in HIT.

Finally, at the Columbia University CDC, we also produced all the curriculum, instruction, assessment, and evaluation products with an iterative design and validation process. This method is borrowed from traditions in formal test development (AERA, APA, & NCME, 2014; Chatterji, 2003). The utility of this iterative methodology in designing curriculum outcome frameworks and curriculum-based assessments was recently demonstrated in a graduate medical education project (Chatterji, Graham, & Wyer, 2009).

Method

This section describes our implementation of the comprehensive, three-stage model within the Columbia CDC as a demonstration case. The institutional review boards of Columbia University Medical Center and Teachers College, Columbia University, approved this study. The project proceeded in three stages: (a) designing of curriculum materials, (b) content validation and revision of products through expert and external user reviews, and (c) development of a program implementation and evaluation protocol, guided by a systems-based logic model.

“Blueprints,” based on ONC terminology, represented complete course syllabi for future instructors, identifying the culminating performance outcomes, objectives, recommended readings, instructional resources, and student assessment exercises. The Columbia CDC was charged with designing four HIT modules, called curriculum components hereafter. Figure 3 and Table 1 may be useful as references as the procedures are described.

Figure 3.

Product samples. EHR = electronic health record; CPOE = computerized physician order entry; IT = information technology; HIPPA = Health Insurance Portability and Accountability Act.

Table 1.

Mean Ratings on Importance of Workforce Roles.

Relevance of workforce role to unit	M	SD	Max	Min
Units in Component 13—Public Health IT
1. Practice Workflow and Information Management Redesign Specialist	4.10	0.97	5	2
2. Clinician/Practitioner Consultant	4.30	0.98	5	2
3. Implementation Support Specialist	4.50	0.58	5	3.5
4. Implementation Manager	4.35	0.67	5	3
5. Technical/Software Support Staff Trainer	4.35	0.58	5	3.5
6. Trainer	4.05	1.42	5	1
Units in Component 14—Vendor-Specific Systems
1. Practice Workflow and Information Management Redesign Specialist	4.81	0.37	5	4
2. Clinician/Practitioner Consultant	4.50	0.60	5	3.5
3. Implementation Support Specialist	4.75	0.71	5	3
4. Implementation Manager	4.44	0.73	5	3
5. Technical/Software Support Staff Trainer	4.69	0.46	5	4
6. Trainer	4.81	0.37	5	4
Units in Component 15—Usability and Human Factors
1. Practice Workflow and Information Management Redesign Specialist	4.42	0.87	5	2
2. Clinician/Practitioner Consultant	3.71	1.18	5	2
3. Implementation Support Specialist	3.88	1.00	5	2
4. Implementation Manager	3.88	0.80	5	2
5. Technical/Software Support Staff Trainer	3.92	0.90	5	2
6. Trainer	4.21	0.72	5	3
Units in Component 16—Training and Instructional Design
1. Practice Workflow and Information Management Redesign Specialist	3.63	1.41	5	1
2. Clinician/Practitioner Consultant	3.88	1.36	5	1
3. Implementation Support Specialist	3.25	1.39	5	1
4. Implementation Manager	3.88	1.25	5	1
5. Technical/Software Support Staff Trainer	3.75	1.28	5	1
6. Trainer	4.63	1.06	5	2

Note. Rating scale: 1 (low) to 5 (high). M = mean ratings for all units; SD = standard deviation; IT = information technology.

Stage 1: Curriculum Design

Domain Specifications

Design of the four HIT curriculum components began with specification of domains of learning outcomes in consultation to subject area specialists and academics involved in the project teams, along with leaders. These were (a) Public Health IT (10 units), (b) Vendor Specific Systems (8 units), (c) Usability and Human Factors (13 units), and (d) Training and Instructional Design (9 units; Figure 3).

The iterative design and validation process led to Versions 1.0 and 2.0 of learning outcomes organized by unit of instruction. Tree-diagrams were represented in terms of a culminating performance outcome(s) for each unit of instruction, under which there were lists of embedded competencies labeled as “objectives” in the blueprints.

“Cognitive demand” is defined as the level of mental processing expected of the trainee, as given in statements of desired learning outcomes. Recall and understanding of concept knowledge are at lower levels, while higher order thinking and complex procedural skills are at the highest levels. New curricula are often criticized for being too simple or easy. By design, the cognitive demands of the HIT learning outcomes ranged from high to low.

Instructional Design

Concurrently with the blueprint design, teams of content area specialists and instructional designers designed PowerPoint slides and scripts for each unit of instruction under the four components. These scripted lessons were prepared with instructor audiences in mind.

Assessment Design

Following guidelines in educational test design, we then developed multiple-choice tests for each unit, tapping less demanding cognitive competencies at the recall and understanding levels. To tap more complex performance outcomes, the team designed performance assessments or constructed response tasks labeled as “application activities.” Each application activity was accompanied with scoring rubrics incorporating 4- and 5-point structured rating scales (AERA, APA, & NCME, 2014; Chatterji, 2003).

Stage 2: Assuring Quality of Curricular Products: Iterative Validation and Revisions

To produce final versions of the materials (Version 3.0), two levels of content-validation studies were conducted, both internally and externally. As many validators were unexposed to the principles of formal curriculum design and measurement, a semistructured survey was designed to guide the collection of useful data. This validation instrument consisted of four sections, dealing respectively with (a) Perceived Relevance of HITECH Workforce Roles, (b) Perceived Alignment of Curriculum Materials with Stated Objectives and Component Blueprints, (c) Perceived Quality of Instructional Materials, and (d) Perceived Quality of Classroom Assessments. Specific items focused on reviewer opinions of alignment of assessment materials with HITECH goals, objectives, match of instructional materials, and assessments with objectives, and overall usability. Outside the rating scales tied to items, the validation instrument also invited open-ended comments to each segment of the questionnaire.

Internal Content Validation

The first iteration involved an internal validation by experts. The internal product validation teams (N = 10) consisted of in-house content area experts, instructors, and/or measurement specialists on the teams who had not designed particular curriculum products. We initiated the internal validation process immediately after the first version of materials was ready.

External Content Validation

For each HITECH unit, next, two independent validators were asked to review an integrated package of materials comprising the blueprint, learning outcomes, instructional content, and assessment. All 40 units were reviewed and validated.

For the second phase of validation, we invited a panel of outside institutional leaders (Deans), HIT content experts, and potential community college instructors to conduct structured reviews (N = 38). The group was selected to represent all user groups and regional stakeholders broadly, and included project leaders, members of the New York City Department of Health and Mental Hygiene, and HIT instructors from community colleges. None of these members had participated on the design teams.

Analysis

The quantitative data generated by the items were summed to yield scale scores in each domain of the validation instrument. Results were summarized with descriptive statistics (means, standard deviations, maximum and minimum scores on each scale). Some variability among raters was expected due to subjective factors and differences in background and professional expertise among raters. For example, we expected different perspectives of university experts, deans, and classroom instructors on the same materials; however, each was valuable to have a multifaceted picture of the product quality.

Internal consistency reliability (using Cronbach’s alpha estimates) and interrater agreement studies provided estimates on the range of variability that could be expected with the obtained results. The reliability estimates were found to lie above the .70 criterion for acceptability (Chatterji, 2003; Crocker & Algina, 2006).

Using sentences as the unit of analysis, we then coded all open-ended comments by section and organized the major themes and subthemes qualitatively (Miles & Huberman, 1994). The guiding question for this analysis was: What curriculum attributes do stakeholders and prospective users value the most—content, utility, other factors? In addition, to provide curriculum designers with specific directions for making revisions to materials, frequencies of each major theme and subtheme were tallied and percentages were reported with unit and component identifiers. Ongoing product development and refinement activities continued thereafter to generate Version 3.0 of all the products.

Stage 3: Design of a Systems-Based Protocol for Facilitating Curriculum Implementation

We convened an evaluation task force in Year 2 of the project to engage local stakeholders in the development and review of an implementation plan and related materials that could facilitate future program implementation and evaluation in the northeastern region and, eventually, in similar projects elsewhere. The products developed included a logic model, an evaluation plan, an instructor survey, and focus group interview schedules for students and community college participants. At project advisory board meetings and conference calls during the project period, the consortium leader and ONC staff communicated that ONC-sponsored evaluations of the project were forthcoming at the national level, eliminating a need for a formal evaluation to be initiated at the regional level during the project period.

Results

Products

Product samples are illustrated in Figure 3. They illustrate how we achieved alignment between curriculum–instruction–assessment materials to optimize “content-based validity” through our design process. Products were intended to tap different cognitive processes and levels. A total of 84 curricular and assessment products were designed and content-validated for 40 instructional units in four components. These included four “blueprints” listing culminating performance outcomes and embedded exiting competencies (objectives) organized as hierarchical “tree-diagrams”; multiple-choice tests for units aligned with particular objectives and instructional content; and performance assessments with scoring rubrics, also aligned with specific objectives and instructional content. We released results from the external content validation study of Version 2.0 products to domain leads and consortium leaders through meetings and workshops, so as to inform the production of final products.

Content Validation Results

Quantitative Results

Following the first round of internal validation and revision, the Version 2.0 products were rated positively by outside users. Using summated rating scales with maximum possible total scores of 5 or 15 depending on the domain, the perceived overall quality of Version 2.0 of the HITECH materials was deemed to be quite high by the external validation team, yielding mean scores from 3 to 5 or 13.0 to 15.0 (Tables 1 and 2).

Table 2.

Mean Ratings for Component and Unit Objectives, Quality on Instructional Materials, and Content-Based Validity of Assessments, by Component.

Component	N (units)	M	SD	Max	Min
Panel A: Alignment of component and unit objectives
13. Public Health IT	10	14.18	1.04	15	12
14. Vendor-Specific Systems	8	14.13	1.25	15	12
15. Usability and Human Factors	12	13.49	1.76	15	10
20. Training and Instructional Design	8	13.50	2.39	15	8
Total	38	13.81	1.63	15	8
Panel B: Quality ratings of instructional materials
13. Public Health IT	10	13.76	1.55	15	10
14. Vendor-Specific Systems	8	13.32	1.65	15	10.5
15. Usability and Human Factors	12	13.33	1.36	15	11
20. Training and Instructional Design	8	13.90	1.06	15	12.5
Total	38	13.56	1.39	15	10
Panel C: Content validity of assessments
13. Public Health IT	10	13.90	2.12	15	9
14. Vendor-Specific Systems	8	13.81	1.60	15	11
15. Usability and Human Factors	12	13.88	1.30	15	11
20. Training and Instructional Design	8	13.59	1.74	15	10
Total	38	13.81	1.63	15	9

Note. Rating scale: 1 (low) to 5 (high). M = mean ratings for all units; SD = standard deviation; IT = information technology.

Table 1 presents a quantitative summary of the perceived importance of workforce roles by component. The Cronbach’s alpha reliability estimated for the items on the Workforce Role scale was high at .88, well above the threshold for acceptability of .70 (Crocker & Algina, 2006). The mean ratings speaking to the relevance of various Workforce Roles for Components ranged from 2.88 to 4.81 on a 5-point scale. Consistent with designer intentions, the degree of match of particular Workforce Roles varied on different curriculum components. For example, on Component 13, the Implementation Support Specialist was rated as the most important Workforce Role with the highest mean rating of 4.5. Table 2 shows the mean ratings for component and unit objectives (Panel A), quality of instructional materials (Panel B), and perceived validity of assessments for each of the components (Panel C), all of which were also high.

Qualitative Findings

The topmost general themes extracted to answer the qualitative question may potentially be useful for future curriculum developers. They were Content Relevance, Usability, Clarity, and Connectivity among curriculum parts, and Link of Theory to Practice and Workforce Needs. Additionally, narrative commentaries revealed several specific areas that needed more detail, correction, or clarification, making a third iteration necessary. For example, 50 text units or sentences with reviewers’ open-ended comments were coded that spoke to the Relevance of Workforce Roles.

14 (28%) were categorized as Positive Comments, such as “information adequately supports roles”; for particular modules like Component 14, the match was perceived to be “Strongest for workforce roles 1, 3, and 6”

24 (48%) were categorized under the theme, Concerns, Recommendations, and Questions, and included suggestions, such as “need less theory” or “(units) not particularly useful for most training roles,” or “need more discussion of meaningful use”; and

7 (14%) made specific suggestions for better alignment with Workforce Roles.

Similarly, the instrument yielded detailed feedback from reviewers/users on Relevance of Objectives, Alignment of Instructional Materials with Objectives and Roles, and Perceived Quality of Assessments. In sum, while the project generated largely positive overall quantitative ratings of products, validators provided adequately detailed and component-specific qualitative information to help team leaders, and content experts revise the materials by unit and generate a much-improved Version 3.0 of the curriculum products. (The validation instrument and complete data set are available on request.)

System-Wide Logic Model and Protocol: Future Program Implementation and Evaluation

A few more details of Figure 2, which was developed by stakeholders and the project design and evaluation team, now follow. The HITECH curriculum is viewed as one part of a larger whole consisting of many dynamic elements. The variables/factors in columns and boxes were deemed minimally necessary for successful curriculum delivery, program implementation, and evaluation on a system-wide scale. The arrows suggest directions in which particular factors will likely influence others, based on the assumptions underlying the HITECH initiative.

Programs are expected to be in place coordinated via the regional CDCs, and operated in community colleges with community outreach organizations. Our logic model demonstrates that appropriate conceptualization, training of stakeholders, planning, and coordination among relevant agencies are all critical for the HITECH program to succeed on a system-wide level.

The ultimate goals of the overall HITECH policy are reflected in the outcomes column of the logic model. A main outcome is “meaningful use” of EHRs in hospitals and health care systems, as practiced by a HITECH-trained workforce. According to the model, the knowledge and attitudinal outcomes of this primary target population—students/participants who enroll in the HITECH courses—will be mediated by knowledge, behavioral, or attitudinal outcomes in community college instructors. Measurement of all these outcomes in relevant populations would become relevant in eventual field studies assessing the program’s impact. Continuing field-testing of the curriculum should examine participant outcomes, as well as program processes within organizational contexts.

Discussion

In 2010, five university-based CDCs were funded and charged with the responsibility of developing curricular components that would be used by the Community College Consortia to educate the workforce to undertake six roles, slated for 6-month training (HITECH Programs & Advisory Committees, 2012). All five universities selected (Columbia University, Duke University, John Hopkins University, Oregon Health & Science University [OHSU], and the University of Alabama at Birmingham) had established biomedical and health informatics programs as well as faculty expertise in teaching the curricular components. In addition, OHSU was designated as the National Training and Dissemination Center responsible for developing the website for dissemination of the materials and carrying out training activities for participating community colleges.

There were 20 curriculum components stipulated for development by the ONC, with a few including a laboratory segment; each CDC was responsible for designing five components. In addition, a matrix was established that stipulated which components should be required or optional in each of the workforce roles. Because each CDC had well-developed methods for curriculum design and unique expertise areas in biomedical informatics, ONC made no effort to standardize curriculum development approaches across centers. Rather, each CDC developed components using methods aligned with their specialties, with some mandatory requirements. For example, all CDCs followed a standardized blueprint for each component; lectures had to be available in both PowerPoint and Flash-format voice-over-slide narrated formats; and exercises, classroom-assessment activities, and an instructor’s manual had to accompany each component with a specified format. CDCs were required to produce Version 1.0 of curriculum materials, which then would undergo two rounds of review and revision over the 2-year period of funding. After the first revision, Version 2.0 was distributed for public use under a Creative Commons Attribution-NonCommercial-ShareAlike License, for use by regional educational programs without restrictions. The final version (Version 3.0) of the products was released in the Spring of 2012, following external validation by users.

Using an iterative, systems-based and participant-oriented approach, the Columbia University CDC was able to develop instructor-friendly versions of a curriculum. We should note that the logic model provided is one representation of the assumptions underlying the HITECH initiative. Nevertheless, by embedding curriculum design and workforce training within the overall health care systems infrastructure, the comprehensive logic model is expected to lead to the best practices in EHR use.

Systems-based logic models, as opposed to a simple linear link of a program to outcomes, help guide curriculum design, program implementation, and evaluation more holistically, accounting for the fact that programs are housed and operate in large and complex organizations (Chen, 1990; Donaldson, 2007). In the case of HIT, a “systems-based” logic model helped all actors look at the curriculum and workforce development program within the larger health care system in the community. Participants could see relationships among various contextual factors, resources and inputs or expected program processes, as these were expected to influence outcomes in trainee populations. Such logic models also serve as useful tools that enhance the capacity of program management and delivery personnel (leaders and instructors in community colleges in the HIT case) in gathering relevant data for improvement of operations and decision making (Stufflebeam, 2004).

Evaluations should not be conducted prematurely, or prior to appropriate preparation of organizations and personnel that will implement the materials developed by the CDCs. For meaningful implementation and for evaluation results to be useful, “evaluability assessments” or assessments of the readiness of all partners and stakeholder agencies should first be gauged (Worthen, Sanders, & Fitzpatrick, 1997).

In conclusion, our proposed three-stage model of curriculum development builds on past traditions, but adds new dimensions to the program implementation, evaluation, and dissemination steps of the process. We view the three-stage, curriculum development, implementation, and evaluation model following needs assessments as a potentially replicable method. In the context of the project described here, it was aimed at improving workforce preparation in EHRs and quality of patient care in field settings. Finally, the approach is also sufficiently general to be a guide for the now ongoing HITECH policy implementation nationally (not just restricted to the Columbia CDC), and may also serve as a useful framework for similar health-related workforce training efforts outside HIT. As few such models are available in health care, medical education, and health promotion fields today, we hope there will be interest in continued field-testing of the method demonstrated here.

Footnotes

Acknowledgements

First and foremost, we acknowledge and thank the principal investigators and staff of the Curriculum Development Centers from the Johns Hopkins University (Patricia Abbott, PI, with Tricia Francis, Michael Vaughn); Oregon Health and Science University (William Hersh, PI, with Shelby Acteson, Corkey Devlin, Chris Weldon, Nathan Skidmore, Vishnu Mohan); Duke University (William E. Hammond, PI, with Serae Thomas, Vivian West); and University of Alabama, Birmingham (Eta S. Berner, PI, with Amanda Dorsey, Meg Bruck, Kay Clements, and Beth Elias). We acknowledge and thank Charles Friedman, who in the role of the Office of the National Coordinator (ONC) Chief Science Officer conceptualized and executed the ONC Workforce Development Initiatives, including the Curriculum Development Centers initiative. We also acknowledge Michelle Murray and Chitra Mohla who served as ONC project officers.

Authors’ Note

Conception and design: MC (Co-PI), JPA (Co-PI), and RK (PI). Analysis and interpretation of data: MC, JT, SJ, NK, and SS. Drafting of manuscript: MC and JPA. Critical revision of manuscript for important intellectual content: MC, JPA, and RK. Statistical analysis: MC, JT, SJ, and NK had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The work on this article was partially supported by ONC DHHS No. 1U24OC000003.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing: Approaches and practical guidelines. Washington, DC: Author.

Blumenthal

(2010). Launching HITECH. New England Journal of Medicine, 362, 382-385.

Buntin

M. B.

Burke

M. F.

Hoaglin

M. C.

Blumenthal

(2011). The benefits of health information technology: A review of the recent literature shows predominantly positive results. Health Affairs, 30, 464-471.

Buntin

M. B.

Jain

S. H.

Blumenthal

(2010). Health information technology: Laying the infrastructure for national health reform. Health Affairs, 29, 1214-1219.

Chatterji

(2003). Designing and using tools for educational assessment. Boston, MA: Allyn & Bacon.

Chatterji

Graham

M. J.

Wyer

P.C.

(2009). Mapping cognitive overlaps between practice-based learning and improvement and evidence-based medicine: An operational definition for assessing resident physician competence. Journal of Graduate Medical Education, 1, 287-298.

Chaudhry

Wang

Maglione

Mojica

Roth

Morton

S. C.

Shekelle

P. G.

(2006). Systematic review: Impact of health information technology on quality, efficiency, and costs of medical care. Annals of Internal Medicine, 144, 742-752.

Chen

H. T.

(1990). Theory-driven evaluations. Newbury Park, CA: Sage.

Classen

D. C.

Bates

D. W.

(2011). Finding the meaning in meaningful use. New England Journal of Medicine, 365, 855-858.

10.

Cousins

J. B.

Earl

L. M.

(Eds.). (1995). Participatory evaluation in education. Bristol, PA: Falmer Press.

11.

Cousins

J. B.

Whitmore

(1998). Framing participatory evaluation. New Directions for Evaluation, 80, 5-23.

12.

Crocker

L. M.

Algina

(2006). Introduction to classical and modern test theory. Mason, OH: Cengage Learning.

13.

DeVore

S. D.

Figlioli

(2010). Lessons premier hospitals learned about implementing electronic health records. Health Affairs, 29, 664-667.

14.

Donaldson

S. I.

(2007). Program theory-driven evaluation science. New York, NY: Erlbaum.

15.

Hersh

(2010). The health information technology workforce: Estimations of demands and a framework for requirements. Applied Clinical Informatics, 1, 197-212.

16.

Hersh

Wright

(2008). What workforce is needed to implement the health information technology agenda? Analysis from the HIMSS analytics database AMIA Annual Symposium Proceedings/AMIA Symposium, 303-307.

17.

Hillestad

Bigelow

Bower

Girosi

Meili

Scoville

Taylor

(2005). Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Affairs, 24, 1103-1117.

18.

HITECH Programs & Advisory Committees. (2012). Health IT adoption programs. Retrieved from http://www.healthit.gov/policy-researchers-implementers/health-it-adoption-programs

19.

Kern

Thomas

P. A.

Howard

D. M.

Bass

E. B.

(1998). Curriculum development for medical education: A six-step approach. Baltimore, MD: John Hopkins University Press.

20.

McLaughlin

M. W.

Shepard

L. A.

(1995). Improving education through standards-based reform. A report by the National Academy of Education Panel on Standards-Based Education Reform. Stanford, CA: National Academy of Education.

21.

Menachemi

Brooks

R. G.

(2006). Reviewing the benefits and costs of electronic health records and associated patient safety technologies. Journal of Medical Systems, 30, 159-168.

22.

Menachemi

Yeager

V. A.

Bilello

Harle

C. A.

Sullivan

C. B.

Siler-Marsiglio

K. I.

(2011). Florida doctors seeing Medicaid patients show broad interest in federal incentives for adopting electronic health records. Health Affairs, 30, 1461-1470.

23.

Miles

M. B.

Huberman

A. M.

(1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage.

24.

Nitko

A. J.

(1989). Designing tests that are integrated with instruction. In Linn

(Ed.), Educational measurement (3rd ed., pp. 447-474). New York, NY: Macmillan.

25.

Patton

M. Q.

(1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.

26.

Scriven

(1991). The theory behind practical evaluation. Evaluation, 2, 393-404.

27.

Shepard

L. A.

(2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4-14.

28.

Stake

R. E.

(1967). The countenance of educational evaluation. Teachers College Record, 68, 523-540.

29.

Stufflebeam

D. L.

(2004). The 21st-century CIPP model. In Alkin

M. C.

(Ed.), Evaluation roots: Tracing theorists’ views and influences (pp. 245-266). Thousand Oaks, CA: Sage.

30.

Tyler

R. W.

(1949). Basic principles of curriculum and instruction. Chicago, IL: University of Chicago Press.

31.

U.S. Department of Education. (2010). A blueprint for reform: The reauthorization of the Elementary and Secondary Education Act. Washington, DC: Author.

32.

Wang

S. J.

Middleton

Prosser

L. A.

Bardon

C. G.

Spurr

C. D.

Carchidi

P. J.

. . . Bates

D. W.

(2003). A cost-benefit analysis of electronic medical records in primary care. American Journal of Medicine, 114, 397-403.

33.

Wiggins

G. P.

(1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco, CA: Jossey-Bass.

34.

Wiggins

G. P.

McTighe

(2005). Understanding by design (2nd ed.). Alexandria, VA: Association for Supervision & Curriculum Development.

35.

Worthen

B. R.

Sanders

J. R.

Fitzpatrick

J. L.

(1997). Educational evaluation: Alternative approaches and practical guidelines (2nd ed.). White Plains, NY: Longman.