Evaluation & the Health Professions

The volatility in the U.S. health care system due to unprecedented changes in its organization, financing, and delivery, coupled with a growing physician surplus in certain areas, suggests the need for a research agenda to investigate the impact of these forces on the educational programs of medical schools. This article discusses the potential impact of trends in the health care environment on the following key aspects of undergraduate medical education: admissions, faculty, curriculum, and educational outcomes. A representative set of research questions intended to stimulate inquiry and guide empirical studies in each of the four domains is proposed.

Perceptions of medical school seniors about changes occurring in the health care environment were investigated. A survey was completed by 196 Jefferson Medical College seniors in the class of 1997. Of the respondents, 79% believed that cost reduction rather than quality of care is the primary consideration behind recent changes, 78% felt that managed care organizations hamper physicians’ abilities to render optimal care, 83% maintained that the control of health care by insurance companies would lead to lower quality of care, 69% agreed that patients should have the freedom to seek a specialist’s care without being referred by a primary care physician, 82% recommended that mentally ill patients should be referred to a mental health professional, and 82% believed that learning to work in a managed care environment should be an essential component of medical education. Assessment of student perceptions can assist in the development and implementation of appropriate curricular changes.

This study’s objective was to measure variation in physicians’ practice styles and policies. Family physicians and general internists were surveyed about evidence-based medicine in the areas of asthma, congestive heart failure, and diabetes mellitus. They were asked about clinical recommendations where standards of practice were uncertain, controversial, or changing in response to published guidelines. Also included were items dealing with managed care. Although there was wide variation in responses to 20 of 36 items, some responses were consistent with practice guidelines. Responses to several items indicated a tendency to overuse expensive tests. Overall, the results indicate that a brief, open-ended survey can assess practice variation quickly and economically, as contrasted with more expensive analyses of medical records or claims data. With proper validation such assessments can be used as baselines to guide interventions, as well as measures of the outcomes of these interventions to change practice styles.

With the increasing shift to community-based ambulatory education, it is essential to gain a better understanding of the impact of these changes. To assess the impact of the location and structure of an ambulatory internal medicine clerkship rotation on cognitive knowledge and clinical performance, students were assigned to one of the following: (a) a multi-disciplinary ambulatory clerkship (MAC), (b) the office of a community-based general internist, or (c) a university-based internal medicine ambulatory clinic. The groups were compared on the internal medicine clerkship examination and preceptor ratings controlling for introduction to clinical medicine course performance via analysis of covariance. MAC students were rated lower than the other two groups by their preceptors. There were no other statistically significant differences. The structure and location of the rotation had little impact on cognitive knowledge. The impact of the structure of the rotation on clinical performance is less clear suggesting that further research is needed.

This study examined the psychometric properties of an assessment tool for measuring attitudes toward physician-nurse collaboration. A survey addressing areas of responsibility, expectations, shared learning, decision making, authority, and autonomy was administered to first-year medical and nursing students. Factor analysis of the survey indicated that the survey measured four underlying constructs of shared education and collaborative relationships, caring as opposed to curing, nurse’s autonomy, and physician’s authority. A scale was developed in which 15 items of the survey with large factor loadings were included. The alpha reliability estimates of the scale for medical and nursing students were .84 and .85, respectively. The mean of the scale was significantly higher for nursing than medical students. Results supported the construct validity and reliability of the scale. This scale can be used to evaluate the effectiveness of programs developed to foster physician-nurse collaboration, and to study group differences on attitudes toward interpersonal collaboration.

A method for measuring interrater agreement on checklists is presented. This technique does not assign individual scores to raters, but computes a single agreement score from the concordance of their check mark configurations. An overall coefficient of agreement, called phi, is derived. The agreement coefficient that is expected by chance and the statistical significance of phi are determined by statistical simulation. Despite the dichotomous nature of the checklist agreement (raters either agree or disagree on items), we show that the binomial distribution does not provide a means for testing the statistical significance of phi. A medical education study is used to illustrate the phi methodology.

In 1995-1996, an evaluation was conducted of publicly funded HIV counseling, testing, referral, and partner notification sites in the Commonwealth of Pennsylvania. A collaborative effort of public and private entities, this evaluation used multiple methods that proved to be more powerful in eliciting results than any of the methods used alone, and it did so with budget parameters determined by the public funding agency. This article examines the results of two methods: written mail surveys sent to all of the agencies in the population of publicly funded sites and interviews conducted at a random sample of sites. Results were found to be similar along lines of questions posed similarly by both approaches, and complementary in other areas. Benefits of triangulation in statewide evaluation of HIV/AIDS services are discussed.

Longitudinal monitoring of individual patient data is becoming routine in physician office practice. This study compares three different methods for evaluating clinical outcomes for individual patients: raw change score analysis versus normative and ipsative statistical analyses. Two discrete samples of intermittent claudication patients making vascular surgery office visits—drawn from interventional management versus stable, routinely followed control groups—were tested four times using both generic and disease-specific functional status measures. Results indicated that the ipsative method was most consistent with several different types of a priori hypotheses that are often evaluated in analysis of repeated measures data.