Improving the U.S. Workforce System by Transforming its Performance Measurement System into an Intelligent Information System

A well designed and executed system of performance measures and targets is an integral part of an effective workforce system. It is how the workforce system tracks its results, communicates expectations among its key partners, allocates and aligns resources among its federal, state, and local entities, holds administrators accountable for performance, and drives continuous improvement. However, an effective workforce performance measurement system should and can go further than merely reporting the employment outcomes of participants to higher levels of government. Information within a performance measurement system can also flow in the other direction—to the customers. Using the same administrative data as currently used to record participants’ employment outcomes and other characteristics, a performance measurement system can provide important information to customers to help them better navigate the intricate avenues required of people finding jobs and businesses finding workers.

The purpose of this commentary is to show how technology and the use of administrative data already collected by local American job centers can help participants of the federal workforce programs navigate the process. I do this by showing what is done, both as pilots for the public workforce system and as online platforms developed by the private sector, for individuals and businesses. The gist of my comments is that information should go in both directions—to administrators and to customers. This information can also flow to higher levels of government—state and federal levels—to better inform policy and hold the system accountable for meeting various targets. In addition, the information should be current. Even lags of 6 months to 1 year are too long for the purpose of accountability and performance monitoring.

Adding Tools to Administrative Data

When converted to a longitudinal format, defined as tracking the same type of information on the same individuals at multiple points in time, administrative data from the workforce development system combined with unemployment insurance (UI) wage records and educational outcomes can become a powerful tool. It can offer valuable insights into an individual's future employment prospects and what services may work best. Adding other “real-time” tools, such as the amount of training received from their employer or the career paths an individual may wish to pursue, can provide job seekers and employment services staff with information about other occupations for which a person may be qualified, current job postings, and even what qualifications businesses most prefer.

Unfortunately, the current workforce development system does not live up to these ideals. Many local workforce investment offices are gloomy, stark, places that rely on old technology to help people find jobs and employers qualified workers. By old technology, I mean at best, computer screens where participants can revise or write resumes or search on established platforms for current jobs, and at least, dusting off books full of job postings from yesteryear and old posters listing the top jobs from a survey taken several years ago.

Then why doesn't the federal workforce system wake up to more modern technology? The private sector has. Today, many people go through private companies, such as Indeed, LinkedIn, CareerBuilder, Monster.com, National Labor Exchange (NLx), or Glassdoor, to name a few, that provide online job search engines or job boards, and importantly, a means of social networking. Businesses increasingly use search firms for their key hires and set minimum education requirements to help them find the most qualified applicants. Yet, the public workforce development system, in many cases, is far behind these approaches. One reason is the lack of wherewithal, participants’ financial status, and them simply knowing how to access up-to-date approaches. Another reason is that the federal workforce system is the agency of “last resort” for many of these workers.

Being an agency of last resort, it is not surprising that a large share of participants in the federal workforce development programs is low income. They are likely on or have been on federal assistance programs in the past 6 months. Furthermore, many of these individuals have never held a job and neither have their parents, so they find it difficult to navigate the complex process of finding a job. Many businesses are in the same situation. They are small and do not have the resources to hire qualified human resources (HR) personnel, resulting in many local workforce investment areas acting as an HR consultant.

Congress enacted the current federal workforce development program in 2014 under the Workforce Innovation and Opportunity Act (WIOA), replacing the Workforce Investment Act (WIA), which served as the federal workforce development system since 1978. In passing WIOA, Congress recognized the need for a more intelligent system by directing local boards to “develop strategies for using technology to maximize the accessibility and effectiveness of the workforce development system for employers, workers and job seekers” (H.R. 803, sec. 107, subsec. d [7]). More specifically, the bill requires the development of “strategies for aligning technology and data systems across one-stop partner programs to enhance service delivery … and to improve coordination” (H.R. 803, sec. 101, subsec. d[8]). The bill, however, leaves considerable latitude for designing such a system.

Administrative data from the three WIOA programs—adult, dislocated workers, and youth—come from the Participant Individual Record Layout (PIRL). This is very similar to Workforce Investment Act Standardized Record Data (WIASRD) except that PIRL contains participants of the Wagner–Peyser Employment Service whereas WIASRD does not. The PIRL administrative data elements include everything about the participants that is necessary for staff to help that individual find work. It includes participant information, such as place of residence and workforce board code, equal opportunity (or demographic) information, veteran characteristics, employment and education information, public assistance information, One-Stop center program participation information, services information, and more. For the third quarter of 2021, the PIRL public use file had more than 275 data elements across six federal programs. The internal use file has even more data elements.

Frontline Decision Support System (FDSS)

This pilot program, referred to as FDSS, used administrative information and computer algorithms to better inform customers’ job search efforts. It was implemented in Georgia in the late 1990s after WIA was enacted. The project was a joint effort of the Employment and Training Administration (ETA) of the U.S. Department of Labor, the Georgia Department of Labor, and the Upjohn Institute (Eberts et al., 2002). FDSS focused on job seekers and staff of One-Stop career centers by offering a set of tools that provided job seekers and One-Stop career center staff with customized information about employment prospects and the effectiveness of services.

Using dislocated workers as an example, FDSS offers a systematic sequence of steps participants can use to move through the reemployment process, beginning with understanding their likelihood of returning to work in the same industry, proceeding to exploring job prospects in occupations that require similar skills and aptitudes, accessing information about the earnings and growth of jobs in particular occupations within their local labor market, and ending with an understanding of which reemployment and training services work best for them, if none of the previous steps leads to a job. The tools are based on statistical relationships between a customer's employment outcomes, personal characteristics, and other factors that may affect his or her outcomes, all of which are available from administrative files already collected by various agencies. The statistical algorithms provide an evidence-based approach to determining which services are most effective for specific individuals.

Using administrative data that captures the experience of all customers who have participated recently in the state's workforce system, this evidence-based approach offers a more comprehensive “collective” experience of what works and what doesn't than relying on the narrower experience of individual caseworkers. In addition, FDSS incorporates local labor market information and data about job requirements and available openings.

Barnow and Smith (2004), in a critique of the performance management system in the federal workforce system, recommended using FDSS as the centerpiece for a redesign of the performance system. In what they described as an “ideal” performance system, “randomization would be directly incorporated in the normal operations of the WIA program…through a system similar in spirit to the Frontline Decision Support System” (p. 276). Randomization is an ideal method to construct comparison groups, and they contend that such randomization need not exclude persons from any intensive services, but only assign a modest fraction to low-intensity services (i.e., the core services under WIOA). The randomization would then be used, in conjunction with outcome data already collected, to produce experimental impact estimates that would serve as the performance measures. Other well-established approaches, such as propensity score matching, could be used as well, although randomization is considered the best approach to constructing comparison groups.

Workforce Data Quality Initiative (WDQI)

Since the Great Recession of 2007 to 2009, the federal government has provided federal funds to help states integrate much of their education and workforce data. In conjunction with the U.S. Department of Education's Statewide Longitudinal Data System (SLDS), the U.S. Department of Labor's WDQI funded states to address several priorities, including (1) infrastructure, (2) education choice, and (3) equity. Both programs use competitive grants to link education data to workforce data at the individual level. Through analysis, “these data will demonstrate the relationship between education and training programs, as well as the additional contribution of the provision of other employment services,” according to the WDQI website. When FDSS was first implemented in Georgia, much effort was expended on integrating data from various agencies. Now, with the accomplishments of WDQI and SLDS in the interim, effort can be directed toward other aspects of an intelligent workforce system.

During the past 5 years, the state of Michigan, through the efforts of its LMI office and the Upjohn Institute, updated the process to be used by WIOA participants. The process was funded by the U.S. Department of Labor, the Michigan Department of Labor and Economic Opportunity, and Data for the American Dream (D4AD). With this updated version of FDSS, considerable effort was made to incorporate artificial intelligence (AI) modules into the system, including neural networking. The Michigan version can also benefit from using vendors, such as Burning Glass or the Conference Board, that “spider” the Internet in search of job postings and job-seeker resumes. The result is a comprehensive database of “real-time” job opportunities, which can be used by job seekers to determine the demand for jobs with specific skill requirements and by one-stop career center administrators to identify the demand for various occupations and skill sets.

WIOA, like its predecessor WIA, has many of the basic elements of an intelligent information system, but the various parts still have not been well integrated into a system that can be used simultaneously by customers, managers, and decision makers. The current system integrates employment outcomes from UI wage records and educational attainment from individual workforce customers with training attainment and administrative data elements from PIRL. For most states, information flows in only one direction—from the customer and manager to agencies to which they report. Furthermore, because of reporting lags, primarily the time it takes to compile UI wage records, that information is not very useful to frontline staff or to customers, and there is little effort to make that information available more quickly or to find alternative sources of information. There is little doubt that the private sector would not wait for data so long in coming; the old adage “time is money” is very relevant here.

Value-Added Performance Improvement System (VAPIS)

Long reporting lags for UI wage records has made it difficult for one-stop center and state agency managers to use administrative data to monitor programs and to follow continuous improvement methods. Information regarding participant employment outcomes is not available for at least a year after they exit the program. The mere fact one of the “common measures” of employment outcomes does not occur until the fourth quarter (a year) after exiting the program is one indication of the long lags. Even with the regression-adjusted approach to setting targets, the information is not current enough to be an effective management tool.

The state of Michigan attempted to address this issue using regression techniques to forecast likely employment outcomes of customers based on the outcomes of past participants. With technical assistance from the Upjohn Institute, the state developed VAPIS to help aid local workforce area administrators in making better management decisions. The system was similar to regression-adjusted targets, used for setting targets of federal workforce programs beginning in 2009. The difference in this approach and VAPIS was that instead of adjusting targets for factors outside the control of local administrators, such as the personal characteristics and employment histories of customers, common measures were adjusted. In this way, the performance measures themselves reflect to a greater extent the value added of the workforce system. In addition, VAPIS forecasted the possible outcomes of participants currently receiving services, so that local administrators could get some idea of how their current decisions might affect future outcomes. Michigan provided VAPIS to local workforce administrators for several years (Bartik et al., 2009, 2011).

Businesses as Workforce System Customers

In addition to job seekers, businesses are customers of the workforce system. Increasingly, businesses complain that they cannot find qualified workers. This mismatch may stem from a variety of reasons: lack of specific occupational skills, low wage offers, lack of soft skills, and more stringent hiring requirements. Businesses look to the workforce system to help identify, assess, and train workers to meet their specific requirements. Many local workforce investment areas work closely with businesses to identify needs and assess the qualifications of prospective workers. Businesses and local workforce investment areas are partnering with community colleges to develop curricula and train workers to meet their specific needs. However, such planning requires real-time information and future projections of the demand by businesses for specific skill sets of workers. Traditional methods of gathering this information through surveys typically yields outdated information because of the length of time required to devise, complete, and process surveys. State and local workforce agencies are starting to turn to other means of collecting and assessing information on the skills required by businesses.

An alternative approach uses the same “spidering” method as previously mentioned. Web-based information is timely and comprehensive in that all job postings on the Internet can be searched and compiled. The information can also be reported by individual businesses and classified into highly detailed occupational categories. The detailed occupational categories can be associated with specific skill sets, which in turn can inform community colleges and local workforce investment areas about the specific skills that need to be provided by workforce system training programs. Information on the flow of participants into training programs and those graduating from those programs can offer businesses important insights into the availability of workers with the skills they need. Therefore, completing the loop between the demand for and supply of workers with specific skills and incorporating this information into a system easily accessed by both employers and prospective workers can help improve the match between employers and employees.

Measures of Business Satisfaction and Needs

Job seekers and one-stop center staff rely on “common measures” for information about the employment outcomes of participants. These employment outcomes are considered common measures because they are same across the four basic programs included under WIOA-adult, dislocated worker, youth, and Wagner–Peyser Employment Services. They include the number of individuals employed in various quarters after leaving a program and median wages during the second quarter. However, the ETA of the U.S. Department of Labor has yet to come up with measures of business satisfaction and needs. Obviously, participants of the workforce system who find jobs benefit the companies who hire them, but the current common measures do not record whether an employer used the workforce system to find workers, whether the workers they hired came through the workforce system, or whether they retained the worker after so many months of being initially employed. Many of these measures can be added to the performance measurement system by more fully utilizing the data available from UI wage records. Other measures may need to be derived from other sources.

Several years ago, under WIA, the Commonwealth of Virginia and the state of Washington looked at alternative and additional performance indicators to what was found in the common measures. Of particular interest is a measure they constructed to record the use by employers of WIA services. It is a measure of repeat employer customers and is calculated as the percentage of employers served by WIA who return to the same program for service within 1 year (Hollenbeck & Huang, 2008). More specifically, an employer was categorized as satisfied if they hired someone who had exited from a program in the first quarter of the fiscal year, and then hired another individual from the program before the fiscal year was over. The denominator for this indicator was the number of employers who hired someone in the first quarter of the fiscal year. For example, Hollenbeck and Huang calculated the measure for the WIA adult programs in Virginia and found 52% of employers who hired someone from one of the two programs hired at least one more worker from the same program within the year. Of course, this is contingent upon the number of times an employer hires during the year, but it can be normalized by a state or industry average.

Currently, employers are concerned with finding qualified workers to fill their job openings. Business satisfaction with the WIA programs would depend upon the programs’ ability to place individuals with appropriate skills with employers. The measure adopted by Virginia assumes that employers are repeat customers because the programs have provided them with applicants with the appropriate skills. As mentioned in the previous section, the question for workforce administrators is how this measure helps them guide participants into the skills that are in demand. Moreover, how does this measure help training providers determine the appropriate curriculum and the appropriate capacity to meet employers’ demands? A more precise measure would measure skills needs in the openings posted by employers. A measure of this sort would be more easily attainable if employers posted their openings with the workforce programs, more precisely, the employment service.

Short of this, the growing use of the Internet to post openings offers another solution. Organizations such as the Conference Board and other private sector companies are offering services that search the Internet for job postings. These services can be customized for specific locations and can glean from the job postings requirements related to educational attainment, certifications, experience, and other qualifications. Furthermore, this information is offered in real time, and even daily. Using past data, trends can be detected that give training providers a better sense of the future demand for various skills sets. This information can then be used to have a more meaningful conversation with employers as to their future needs and training providers regarding their future capacity to train individuals to meet those needs.

Private Sector Job Boards and Job Search Engines

Job boards are the most-used tool to search for jobs on the Internet. Hundreds of websites are available to search for job postings and upload resumes, far too many to report in this short brief. I will discuss a few. The largest and perhaps most prominent include the NLx CareerBuilder, Indeed, Monster.com, and ZipRecruiter. The first online job search service was Monster.com, registered in 1994. It is also one of the largest, attracting 35 million unique visits per month from 50 countries. Indeed, launched 10 years later, is the largest job site in the world, with other 250 million unique visitors per month in more than 60 counties and 28 languages (Indeed.com, 2022).

One of the most important job platforms for the public workforce system is the NLx. It has more than 4 million job openings posted on its platform at any given time. Those job postings are contributed by more than 25,000 corporate career websites and state job banks. The NLx was started in 2007 as a partnership agreement between the DirectEmployers Association and the National Association of State Workforce Agencies. When a workforce system participant uses the tool “Job Finder” on the U.S. Department of Labor sponsored website OneStopCareer, NLx is the default source for job searches. The other three websites listed are Indeed, CareerBuilder, and ZipRecruiter. All state workforce agencies have signed participation agreements with DirectEmployers to operate the NLx. A unique aspect of the NLx is that all job openings are unduplicated, currently available, and are gathered from vetted employers, which none of the search engines, such as Burning Glass or the Conference Board, can boast. It gathers currently available and unduplicated job opportunities from verified employers, which includes employers in addition to members of DirectEmployers, and pushes that content into state workforce agency sites to reach a maximum number of job seekers.

A different approach to aggregation is to search the Internet with “spider” methodologies to find job postings among the thousands of sites where they are posted. In addition to pulling job postings from these sites, the “aggregation algorithms” attempt to not duplicate the same job positing that may be found on several sites. Much of the difference between sites, in addition to the services offered and the prices charged, is the way they attempt to not duplicate postings.

For the private sector to operate, someone must pay for their services. Therefore, pricing of services is necessary. For these online platforms, the question is who pays—job seekers (workers), job posters (businesses), or both. Basically, those who post jobs pay, but this varies from the number of clicks by job seekers to a package approach for those who post many jobs a month.

Summary and Research Questions

An intelligent workforce system needs to incorporate five elements: (1) a data-driven system; (2) information that flows to the customers; (3) customized to the specific needs and circumstances of each customer; (4) targeted reemployment and training services; and (5) valued-added performance management. WIOA incorporates various aspects of these five elements, but still significant improvements must be made. The most recent workforce system (WIOA) encourages states to target services, integrate data-driven counseling and assessments into service strategies, more fully integrate programs, and provide easy and seamless access to all programs. It even requires states to periodically evaluate their workforce system using comparison group methodologies. After nearly 8 years under WIOA, most states are still trying to comply with the regulations. FDSS comes the closest to incorporating these functions. It integrates administrative workforce data with education and wage data, it develops statistical algorithms that provide personalized information to help customers understand what various trends and circumstances mean to them, and it allows this information to flow both ways—down to customers and frontline staff who are making decisions and up to policy makers and higher-level managers who need the information to help direct the workforce system.

It is safe to say that development of an intelligent workforce system will not happen all at once. State administrators must know what they are purchasing. Only the most visionary administrators seem to be able to grasp the notion of an intelligent workforce system. First, it takes strong leadership throughout the system, from top management down to frontline staff, to implement an intelligent workforce system. Second, research is required to come up with the various modules available to one-stop centers. From the time FDSS was implemented even until now, many people, who were knowledgeable about the job finding process, could not break out of their mindframe at the time to understand how the various modules could be used. Third, research must also be conducted to understand the benefits of using such modules.

A research agenda for the workforce system must include innovative ideas discussed in this short brief and others not yet conceptualized. These innovative ideas must be researched, implemented, and evaluated. The research questions are difficult to pin down since many ideas have yet to be formulated. However, for those that have already been implemented, one can always find room for refinement. For instance, the updated Michigan version of FDSS looked at different AI approaches, such as neural networking. The research question in this instance is how much better, if any, are the AI approaches to the various modules (or paths) than the more traditional regression analysis methodologies? Another research question is to examine ways to evaluate and display the methodologies used to understand the value to each participant of the services and training provided. One could also construct and evaluate the efficacy of various outcome measures for businesses using Internet spidering or other sources. The third line of research is to develop new paths that have not yet been conceptualized.

Another set of research questions is to investigate possible ways to adjust performance measures as outlined in the WIOA legislation. So far, research has primarily looked at simple approaches using both individual participant and one-stop center aggregated data to adjust performance measures. The WIOA legislation does not specify the methodologies that could be used, and previous research has shown that regression-adjusted approaches or something a little more complicated, such as multilevel mixed-effects linear regression with random sloped and intercepts effects, could be competitors.

Research could also evaluate the entire workforce system at various levels, from local one-stop centers to state programs. The WIOA legislation requires that states evaluate their programs using comparison group methodologies. However, no state has completed such an evaluation. An increasing number of universities and think tanks, along with the U.S. Department of Labor, are ramping up efforts to provide resources to states for this purpose. The public use database PIRL, WIOA's version of WIA's WIASRD, includes Wagner–Peyser Employment Service individual participants, which could be used as a viable comparison group, to evaluate the workforce system. ¹ Of course, there is still much research to be conducted on the use of short-term versus long-term employment measures, incentives of local administrators, cream skimming, the use of merit employees in the employment service and WIOA programs, and many others. ²

Considerable research is required to transform the current workforce system into a more intelligent one. The public sector, under both WIA and WIOA, has made considerable strides in making such a transformation, but today the private sector has surpassed the public sector on most counts, even though the public sector, with funds and direction from the U.S. Department of Labor, is making advances. However, only with the leadership and research that can come from users of the workforce system and experts of the system will we be successful in providing a public workforce system to those who need it most. Without adopting the appropriate technology, the workforce system will remain less efficient that it might be and leave the United States using more resources to keep those who want and need jobs employed.