Abstract
This article examines the current state of compensation surveys, including a review of the broad range of compensation surveys available ranging from large published surveys to magazine surveys and repackaged survey data. The issues surrounding participation, quality and the applicability of the data are discussed. The article also suggests actions that compensation stakeholders can take to improve the integrity of external market pay data as a reliable tool.
Compensation surveys are a relatively modern tool for establishing the pay level for a position. Systematic compensation surveys have increased in prominence over the past 30 to 40 years. Prior to this, the emphasis was on internal equity through point factor systems (or other internal ranking system). Companies raised pay if turnover was high and slowed increases for positions with little turnover without really knowing whether pay was the issue. In the “golden age” of compensation management (the 1990s - early 2000s), the increased availability of market data enabled HR departments to balance the internal focus with the external market. The data were imperfect but credible and very helpful.
Over time, the focus has shifted such that pay levels are calibrated primarily with the external market. This shift has created enormous pressure to obtain accurate, relevant and timely market pay data. In response to the demand, thousands of published surveys now provide pay data for various job families, industries, geographic locations and classifications of employees. In addition, many custom surveys focus on specific niches. While there appears to be a wealth of information, the survey data may not be as credible as the users need them to be in order to support a market-based compensation approach. A disturbing issue is the relatively small size of the data sets. Such small sets call into question the reliability of the data and give compensation professionals limited ability to sufficiently refine the data in order to accurately reflect a specific pool of talent.
Fingers can be pointed to many culprits for the diminished data pools. Rampant cost cutting in compensation departments has left fewer people able to accurately complete surveys. Positions in compensation, benefits and job analysis have dropped 30% since 2008, according to statistics from the Bureau of Labor Statistics. In addition, new types of survey tools have emerged that repackage survey data and allow nonparticipants to purchase them. We are concerned about the diminished integrity of market data and the implications this holds for the management of compensation in organizations.
This article examines the types of surveys available and the issues around data reliability and suggests actions that compensation stakeholders can take to improve the integrity of external market pay data as a reliable tool.
Type of Survey Data Available
There are thousands of surveys available and they come in all shapes, sizes and price points. Table 1 summarizes the various types of surveys, providing an overview of the current landscape and some insights for choosing surveys to use for competitive assessment.
Summary of Survey Data Sources.
Positive Trends
There are many improvements in compensation surveys over the years, for example:
Multiplicity of sources: There are many surveys from which to choose—the WorldatWork database lists almost 1,000.
Helpful formats and tools: Results are often delivered in an electronic format, and the big survey houses often provide sleek online tools to download the data or create custom analysis.
Various types of pay data
Target bonuses are covered by many surveys—this is critical for examining policy pay levels. Long-term incentive pay is almost always included in executive compensation surveys. Sales compensation surveys have better coverage of sales incentives—several major survey houses have made noticeable efforts to obtain and report more accurate sales incentive pay levels. Global data are more readily available.
Improved minimum standards: Survey companies are publishing data with no fewer than four to five company matches to the position.
In spite of the progress, underlying issues still create roadblocks to obtaining data that fully reflect the external market
Issue 1: Participation—Participation—Participation
Without participation there are no surveys; yet only a fraction of companies participate in surveys. As shown in Table 2, there are millions of firms in the United States, but a very low percentage is represented in surveys.
U.S. Companies and Survey Participation.
Source. U.S. Census Bureau (https://www.census.gov/econ/smallbus.html#EmpSize).
This excludes surveys that use data from proxies, Form 990s and/or self-report data.
A survey with 2,500 participants represents less than 1% of all companies that have more than 500 employees. And even this participation number is potentially overstated because the participation count for some surveys includes multiple subsidiaries of the same company. Also, many of the same companies participate in multiple surveys, so adding up participation from all surveys would overstate the universe of participants.
Many companies obtain market data without participating in any surveys. Some companies scour job postings for pay rates. Other companies obtain market pay either by purchasing published surveys that have a nonparticipant fee (though some publishers do not allow this) or by subscribing to services such as Economic Research Institute, Salary.com or PayScale. But where do those services obtain data? The answer is mixed: Some data come from published surveys, some come from Bureau of Labor Statistics and some are self-reported. As a result, more companies use survey data than contribute to the surveys.
To be viable, a survey requires data. Basic economics shows that the fee for a survey does not cover the real cost of quality survey data. The real cost of obtaining quality survey data includes the time the participant invests in submitting accurate data from the company.
The Cost of Surveys Equation is as follows:
Paying more for data rarely effectively offsets lack of participation. Thus, companies that repackage data or sell results to nonparticipants are doing a disservice to the entire industry in two ways: (a) it reduces participation by allowing companies to obtain data without sharing data and (2) it places downward pressure on the price of surveys that require participation because it is easier to purchase data than provide it. Companies that purchase data without contributing are not shouldering their fair share of the effort to ensure a robust data set; they need to make the effort to submit their data. Ultimately higher participation is critical to improve the representativeness of the data.
Issue 2: Representativeness of Data
A fundamental question is this: Does the survey data set reflect the entire market or just the segment of the market that includes companies with the resources to participate in surveys? Even the “big” surveys may not truly represent the entire market given the number of participants relative to the vast numbers of companies in the United States. One concern is that the surveys “self-select” for companies that are big enough to have staff that can complete a survey or are better funded and thus may be better-paying companies. Table 3 summarizes how companies generally fall in the survey participation stratosphere (there are always exceptions).
Survey Participation Summary.
By way of example, Table 4 shows the participation rates by company size for a large published survey. This is a good example as over 60% of the participants have more than 5,000 employees and over 40% of the participants have more than 10,000 employees. This is not unusual of a sample for large published survey.
Example Survey Participation Rate by Company Size.
The underrepresentation of small and moderate-sized firms is startling given that broader participation leads to more reliable data, and it opens the question of what approaches could be used to encourage participation in surveys.
The other question is whether the survey participant companies represent the survey user’s universe of talent. This is typically the first line of inquiry from HR’s clients: What companies make up the data set? The managers, who are trying to attract and retain talent, want to know that the “market” that HR presents reflects the pool from which they are trying to hire. High levels of participation are the only way to develop data sets that offer customized slices of data that focus on a specific market. Even with the “big” surveys, there is typically limited ability to slice the data finely enough to obtain the ideal peer group.
Issue 3: Data Quality
When assessing data quality, often one considers poor job matches or incorrectly tabulated data; however, based on experience, data quality in most surveys is fairly good. Many reputable companies invest time, energy, intellect and technology to produce top-notch surveys. There are definitely survey houses that produce better surveys than others, but overall the data are usually cleaned and tabulated with reasonable accuracy. However, there are other quality issues of concern as discussed below.
A Lack of Company-Size Trended Pay Data for Senior-Level Positions
Many surveys do an excellent job providing pay levels by company size using both tabular and trend line approaches. However, some high-profile surveys do not provide this. That means a company with 1,000 employees could be referencing the same “competitive pay” for their CEO as a company with 10,000 employees.
Mystery Data
There can be a disconnect between the number of participants listed and the number of data points for any given job. For instance, it is perplexing if the number of companies matching to a fairly universal job, such as the top HR or an accountant, is significantly less than the number of participants listed. For example, one survey house listed over 1,800 participants but the most matches to any one job were just over 400. This may be a result of the following:
Subsidiaries listed as primary participants: While subsidiary data are very valuable, they will only show up in subsidiary or division jobs and, as a consequence, actual corporate data may be less robust than expected. For example, a survey listed one participant 16 times: once for the corporate submission and the other 15 for each location/subsidiary. These multiple listings occurred many times in the participant list.
Some survey houses augment their data set (and bolster their “participant” list) with proxy data: There are several issues with this: (a) proxies report just five jobs, so including these companies does not improve the usefulness of the survey for the vast majority of the positions, and (b) proxy jobs can only be matched based on title. It may be fairly easy to obtain relevant data for the CEO, CFO and a few other clearly titled positions, but other jobs might not be easily mapped to a standard survey position.
Career Ladders
Back in the day, career ladders were limited to professions such as engineers, actuaries and computer programmers. The hallmark of these professions is explicit education, training and certification requirements to move up the career ladder. The career levels are defined by the attainment of specific professional achievements (e.g., earning a certificate or passing an exam). Over time, the concept of career ladders has spread into almost all job families.
The dilemma is that most professions do not have a universally defined and accepted career ladder with explicit knowledge and skill requirements. The implications are that: (1) there can be any number of levels: Junior and Senior, Levels I to III, Lead, SMEs and the great Oz; and (2) each company can define the levels to meet their internal needs. In too many cases, levels get created because a manger wants to “promote” an individual. This may be the only way to get more than the 2.89% raise that is available in the merit budget. It is also a way to provide recognition and can be viewed as a retention device; however, it may not be based on a defined and consistent set of criteria
This creates quite the conundrum for the survey administrator. The question ends up being—how many levels in the survey is the right number?
Too few levels and the risk is that the results will show competitive pay ranges that are so broad that they are meaningless.
Too many levels risks inconsistent matching by participants as they try to interpret fine differences between similar jobs. The other risk is that there will be insufficient data to report all the levels.
The leveling issue is exacerbated by companies that repackage survey data as they attempt to map levels across various surveys. This is discussed in more detail below. Reputable survey houses go to great lengths to clearly define professional levels in terms of experience, education, supervision requirements and so on. Industry-specific surveys may have job-matching meetings to help ensure that the jobs are mapped correctly.
Self-Report Data
Self-report data are provided by the jobholder. So an employee of a company can log onto a website and identify the job that he or she thinks is most like his or hers. That person then inputs his or her salary and bonus with myriad other information—making the accuracy of the job matching and the accuracy of the pay data suspect:
Employees are not trained in the process of matching jobs. This process assumes that they matched their job correctly based on content, not just based on title.
Employees may not report their actual pay rate. They may input the after-tax amount from their pay stub or include overtime pay. They also may enter the amount that they hope to earn if they get the anticipated raise. Others may enter fake data because they have a perverse incentive to input higher numbers to influence the market rate.
Overall, there is no way to be confident of the integrity of self-reported data.
Repackaged Survey Data
Several services repackage survey data. Repackaging allows a subscriber to market price jobs knowing that the data are derived from multiple survey sources. At first blush, repackaging may seem like a great way to determine the market rate for a job, but consider the following pitfalls:
Job leveling/calibration: Suppose one survey has three levels of accountants, another survey has four and yet another has five? How were the levels calibrated across the various surveys? Was this calibration done accurately? This is a question for which the user might not get an answer. In addition, the calibration might not best reflect the user’s internal job levels.
Weighting of data: What if one survey has 300 matches to the job and another survey has 10 matches? Users have no way to know and no ability to choose the data that are more robust.
Survey sources: What surveys are included in the data set? In some cases that information might not be available.
Data accuracy: What if some of the market data seem illogical? There may be no way for the user to find out why or check data accuracy.
Participants: Are the user’s peer companies included in the data? Using repackaged data typically means that a participant list is not available. This may be yet another question without a direct answer.
Geographic Data Cuts
Many surveys offer geographic cuts. There is a proven difference in pay levels based on location, so geographic data are important. However, often the number of data points is so small that the findings may not be accurate. In this case, using a survey from the local Chamber of Commerce or Economic Research Institute’s geographic assessor may be a better solution.
Solutions
Participation is the key to reporting and obtaining quality data and all stakeholders have responsibility for improving participation:
Survey houses: Those that allow nonparticipants to purchase the survey results should discontinue the practice or increase the nonparticipant price significantly enough (as some have) to make it financially burdensome for a company to not participate.
Survey data repackagers: This practice should be modified to encourage participation in surveys. Consider developing a process by which subscribers upload their job matches to the surveys.
Industry associations: Associations should discontinue selling published survey data to nonparticipants. If an association wants to offer survey data, it should set a limit on the number of positions that can be purchased (e.g., 10-15 positions per year), rather than sell the entire survey.
Other vendors: Market pricing software vendors, such as MarketPay, facilitate job matching and upload of data to questionnaires—this is a great solution and should be expanded.
Survey users: Companies must commit to participate in compensation surveys. Once a company has purchased a survey and completed the process of mapping their jobs to those in the survey, participation will be easier in subsequent years.
The focus on the external market is not likely to change; therefore, obtaining accurate and reliable market data will continue to be critical for attracting and retaining the right talent as well as for managing compensation costs. The mechanisms for reporting quality analysis of the data are in place in most cases. If various industry stakeholders take steps to improve participation, the data should become more robust and thus more reliable.
Footnotes
Acknowledgements
The author acknowledges the helpful comments and suggestions by Gerry Ledford.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.