Academic journal article Public Personnel Management

Nonrandom Response and Rater Variance in Job Analysis Surveys: A Cause for Concern?

Academic journal article Public Personnel Management

Nonrandom Response and Rater Variance in Job Analysis Surveys: A Cause for Concern?

Article excerpt

Job analyses should provide the foundation for all HR management activities. (1) Job descriptions, consisting of samples of incumbent behaviors, provide the data to describe work conditions, work processes, and worker requirements. Employers use these descriptions when making decisions regarding selection, promotion, and compensation. Because employers, as well as the courts during employment-related litigation, base so many critical decisions on job analysis data, it is essential that the data provide accurate descriptions of the work an incumbent must perform.

Survey instruments are in widespread use and are possibly the single most frequently used means of collecting job analysis data. Surveys are rarely administered to all incumbents, supervisors, and other job experts. Furthermore, a 100% completion rate for any survey is highly unlikely. This means survey results are almost always based on a subset--a sample--of the population. The degree to which these samples are accurate representations of the intended population is crucial to the quality of job analytic data. Given the widespread use of surveys in job analysis research and the importance of the quality of research samples, it would seem that issues surrounding survey methodology would be major concerns for job analysts. We, however, were unable to locate any published studies in the job analysis research literature that specifically address the issue of survey response rates and sample representativeness.

The research available on surveys shows that organizational survey response rates are often low and have declined in recent years. (2) Unfortunately, this appears to be true for job analysis surveys, as well. To illustrate, we performed a small-scale review of 27 published job analysis studies (see Table 1). We found that response rates ranged from 21% to 96%, with a weighted mean response rate of 43%. Poor face validity and lack of acceptance many job analyses could be the result of low survey response rates. However, in technical terms, it is not only the response rate that is important. The sample contacted to complete the survey and/or who returned completed surveys must be representative of the population in all ways that could affect ratings. In the past, U.S. courts have been critical of the racial and gender composition of subject matter expert (SME) panels. (3) The logic being that the results from a panel made up of male job experts might be biased against women.

Rather than focusing on applied practical questions such as the degree to which samples are representative of the intended population, job analysis researchers have typically focused their attention on the technical issues of survey reliability, survey validity, and sources of rater variance. (4) There is a growing body of research examining sources of rater variance in job analysis surveys that shows that, despite some arguments to the contrary, job analysis ratings can vary systematically as a function of rater characteristics. (5) Recent studies

show that job analysis ratings can vary as a function of rater position, performance level, task engagement, job satisfaction, tenure, and race. (6) Even if these effects are small and inconsistent, they can still be important, as well as informative. (7) Moreover, small effects could be exacerbated if the rater characteristics that are related to survey ratings are also related to survey return. For example, suppose age is related to both survey return, with older workers less likely to respond to job analysis surveys, and to job analysis importance ratings, with older workers more likely to deflate the importance of technical competencies. The sample would then consist primarily of younger workers who, in comparison to older workers, tend to rate technical competencies as more important. Thus, these two effects combined would produce much more measurement error than either one individually.

In summary, there is insufficient research to understand the effects of response rate on job analysis results. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.