Enhancing Job Analysis Surveys in the Medical Specialties With CMS Data

Abstract

Job analysis surveys are a key component in validating certification examinations in the medical specialties. Few organizations, however, confirm the survey data-gathering process using external data. This article demonstrates how an organization can use data from the Centers for Medicare & Medicaid Services (CMS) to supplement job analysis survey results using a real example from medical imaging. Organizations can also use CMS data longitudinally to predict which procedures are increasing and decreasing in frequency. This prediction can greatly assist with future comprehensive job analyses or smaller, more targeted updates between comprehensive job analyses.

Keywords

job analysis certification medicare medicaid CMS validation surveys

Job analysis, often called practice analysis in medical specialties, is the keystone of validation for job credentialing examination programs (Raymond, 2001). There are many data-gathering methods for job analysis, including surveys, public work records, work journals, and job shadowing. When job analysis involves surveying practitioners, a panel of content experts uses the job task survey results to form examination content specifications (Raymond & Neustel, 2006). Validating or confirming the survey results, namely the assumption that the practitioners accurately filled out the surveys (Colton, Kane, Kingsbury, & Estes, 1991), is important to the job analysis process (Laduca, 1994). Certification programs also may want to track practice trends with job analysis data, but predicting trends with few data points is difficult.

The article has three purposes. First, this article demonstrates how data from the Centers for Medicare & Medicaid Services (CMS; Centers for Medicare & Medicaid Services, 2011) can confirm and enhance data gathered by job analysis surveys. Second, this article shows how CMS data can highlight differences between a target and an overall group if a job analysis targets a special group of practitioners (say entry level instead of all practitioners). Finally, this article demonstrates how methodologists can apply longitudinal analyses to CMS data to highlight trends in medicine. This article uses hierarchical linear modeling (HLM) for longitudinal analysis, but any number of approaches could be used. All examples in this article come from a job analysis from the field of medical imaging, namely radiography. The results of the above analyses are helpful to certification board methodologists, executives, and board members, who are trying to make informed decisions about the appropriate content on certification exams.

Method

Sources of Data

A committee of experts in radiography assisted throughout the Radiography job analysis process. A stratified random sample of 2,000 full-time employed, entry-level radiographers was selected to receive the Radiography survey mailed in March 2009. Respondents indicated the frequency with which they performed each procedure (i.e., daily, weekly, monthly, quarterly or less often, and not responsible). Some computed tomography (CT) procedures were included on the survey in addition to more “traditional” radiologic procedures, because the committee felt that CT had increased in frequency among entry-level radiographers.

The CMS Physician/Supplier Procedure Summary (PSPS) data file was used for all CMS analyses. There was a data set for each year from 2000 to 2008. The PSPS data set contained the number of times providers billed 55 medical imaging procedures that were on the job analysis survey in addition to a variety of other data.

Analyses

The job analysis tasks were ranked by frequency from the survey results in order to compare them to the frequency ranks from the CMS data. The analysis used rankings for two reasons. First, the number of procedures billed to CMS and the survey frequencies were on very different scales, so using ranks placed the two data sources onto a comparable numbering system. Second, the CMS data contained an important outlier, which will be discussed later in the article. Using ranks decreased the influence of the outlying point.

The most important analysis to the Radiography program as related to the CMS data was the comparison between the survey frequency rankings and the 2008 CMS billing frequency rankings. A high correlation between the survey results and the CMS data would indicate that the survey data corresponded to practice in the radiography profession. Large differences in the rankings would require further investigation.

The authors also used HLM (Raudenbush & Bryk, 2002) to find trends in the CMS data over time. HLM takes all of the procedures into account when estimating growth linear regressions. Organizations can use this information to subsequently conduct focused, smaller scale job analysis updates, which survey only selected procedures based on frequency and volume trend.

The HLM used the year as an independent variable, starting with the year 2000 (set as year 0) and ending at the year 2008. The authors also included the number of people that Medicare served annually as a variable, because this eliminated a problem with correlated residuals. The final model was

\ln (Procedure j) = γ_{00} + u_{0 j} + (Year - 2000) (γ_{01} + u_{1 j}) + (n - \bar{n}) γ_{02} + r_{i j},

where j was the procedure index, i was the year index, γ₀₀ was the fixed effect intercept, u _0j was the random effect intercept for each procedure, γ₀₁ was the fixed effect slope for year, γ₀₂ was the fixed effect slope for the mean-centered natural-log number of people served by Medicare n in a given year, u _1j was the random effect slope for year for each individual procedure, and r was the residual. The HLM used the natural logarithm of the number of procedures in order to minimize the influence of an important but outlying procedure, chest radiographs. The log scale also took into account relative instead of absolute change, so it was ideal for this context.

Results

Comparing CMS With the Radiography Job Analysis

1,008 radiographers returned surveys. Figure 1 is a scatterplot of the Radiography job analysis frequency ranks on the 2008 CMS billing ranks. There was high agreement between the survey data and the CMS data for the radiographic procedures but disagreement concerning the CT procedures. The CT procedures all clustered in the upper left corner of Figure 1. This indicated that the CT procedures ranked as among the most frequent procedures in the CMS data but as relatively infrequent in the Radiography survey data. The reason for this difference was that the Radiography survey targeted entry-level radiographers, while the CMS data accounted for radiographers of all experience levels. Numerous entry-level radiographers reported not being responsible for CT procedures on the survey, suggesting that these tasks were performed by more experienced radiographers. The Spearman rank-order correlation between the CMS and survey data was .52 overall. Excluding the CT procedures increased the correlation to .82, which indicated that there was high agreement between the survey and CMS concerning the non-CT procedures.

Figure 1.

2008 Centers for Medicare & Medicaid Services (CMS) billing rankings and radiography job analysis frequency rankings, computed tomography (CT) procedures as triangles are circled.

Longitudinal Modeling of the CMS Data

The model from Equation 1 fit well, with a pseudo R ² of .97 (Singer, 1998). After examining model fit, the next step was to examine the slopes for the procedures. The mean of the slopes was near 0 (no change), with a minimum of −.22 (on pace to drop by 50% about every 3 years) and a maximum of .19 (on pace to double about every 4 years). The slopes for most procedures were between −.029 and .029 (drop by 50% or double about every 24 years, respectively). The slopes for several borderline procedures, which were lower frequency procedures from the survey that made it onto the task list, were negative. The CT procedures, which did not make it onto the task list, all had positive slopes. Trend information can be helpful for making decisions about tasks that barely made it onto the task list or tasks that did not quite make it onto the list. Based on the trend information, an organization could conduct a smaller, more focused job analysis between comprehensive job analyses that only ask about tasks that are changing in frequency.

Discussion

Job analysis surveys fulfill an important role in determining the content of certification examinations. This research investigated ways to use CMS data to supplement medical job analysis surveys. Examining the level of agreement between the CMS and job analysis survey data can help validate survey results. Places where the two data sources disagree can provide valuable information into how the practice of target groups differs from overall practice.

Data from CMS records are more useful than job analysis survey responses for projecting trends. Many organizations do not have the resources to conduct comprehensive job analyses every year. In contrast, CMS data are inexpensive. The PSPS data files cost only $250 per year at the time of this publication. Because CMS releases data annually, organizations will have finer data over time. This makes it possible to conduct insightful longitudinal analyses. CMS data can highlight the direction of changes in an area of practice. Changing procedures are prime candidates for smaller, more targeted job analysis updates.

One must remember that having CMS data does not eliminate the need for job analysis surveys, particularly for organizations targeting a specific population or experience group. CMS data may not cover key patient demographic groups. CMS data may also be inaccurate to the degree that providers may not bill for the exact procedure performed (e.g., a patient needs a noncovered procedure, so the physician orders the needed procedure, billing for a related but covered procedure). In this example analysis, there was disagreement between the survey and CMS concerning CT procedures, indicating that CT procedures are frequent, but that entry-level radiographers are primarily not responsible for them.

There are also other sources of data besides CMS records that can supplement practice analyses. Among these data sources is the National Center for Health Statistics (NCHS), which publishes a variety of health care data. Boulet, Gimpel, Errichetti, and Meoli (2003), for example, used the National Ambulatory Medical Care Survey from NCHS for a job analysis.

Job analysis is extremely important for certification examinations to remain valid, particularly in the medical professions. Using CMS data with survey data, medical boards can make the most informed decisions possible about their certification exams. CMS data analysis will help validate the job analysis process and point toward more focused periodic updates to make exams even more valid.

Footnotes

Authors’ Note

The views and discussions presented in this research are not necessarily the official views of the The American Registry of Radiologic Technologists.

Acknowledgments

The authors would like to thank the Radiography Practice Analysis Committee (Robin R. Berke, BS, RT(R); Helena A. Coello, MEd, RT(R); Kellie S. Cranfill, MSRS, RT(R)(BD); Jose L. Martinez, BS, RT(R)(CT)(MR); and Debra Reese, MPH, RT(R)), The American Registry of Radiologic Technologists (ARRT) Board of Trustees, and ARRT’s staff for all of their hard work on the Radiography job analysis project. The authors would also like to thank all of the radiographers who completed the job analysis survey. Without their participation, the ARRT could not make properly informed decisions about current practices in radiography.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Boulet

J. R.

Gimpel

J. R.

Errichetti

A. M.

Meoli

F. G.

(2003). Using national medical care survey data to validate examination content on a performance-based clinical skills assessment for osteopathic physicians. Journal of the American Osteopathic Association, 103, 225–231.

Centers for Medicare & Medicaid Services. (2011). Retrieved from http://www.cms.gov

Colton

D. A.

Kane

M. T.

Kingsbury

Estes

C. A.

(1991). A strategy for examining the validity of job analysis data. Journal of Educational Measurement, 28, 283–294.

Laduca

(1994). Validation of professional licensure examinations: Professions theory, test design, and construct validity. Evaluation & the Health Professions, 17, 178–197.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.

Raymond

M. R.

(2001). Job analysis and the specification of content for licensure and certification examinations. Applied Measurement in Education, 14, 369–415.

Raymond

M. R.

Neustel

(2006). Determining the content of credentialing examinations. In Downing

S. M.

Haladyna

T. M.

(Eds.), Handbook of test development (pp. 181–224). Mahwah, NJ: Lawrence Erlbaum.

Singer

(1998). Using SAS PROC MIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Education and Behavioral Statistics, 24, 323–355.