Academic journal article Public Personnel Management

Frame of Reference Training for Assessment Centers: Effects on Interrater Reliability When Rating Behaviors and Ability Traits

Academic journal article Public Personnel Management

Frame of Reference Training for Assessment Centers: Effects on Interrater Reliability When Rating Behaviors and Ability Traits

Article excerpt

Assessment centers have been widely criticized on the basis of measurement problems. The present study sought to present a methodological piece on the extent to which Frame of Reference (FOR) training would increase the interrater reliability associated with assessment center ratings provided by non-psychologist assessors, Five managerial assessors (with no psychological training) rated the behavior and the ability traits of a contrived participant on the basis of behaviors described in two alternative vignettes (detailing critical incidents of job performance). The ratings were obtained both before and following FOR training. It was found that agreement among assessors on their assessment of both behaviors and traits increased subsequent to the FOR training procedure. The implications of increasing the precision associated with assessment center ratings are discussed. Assessment centers are a popular process, widely applied in the human resource arena. (1) A robust finding in the assessment center literature is that assessment centers engaging managers as assessors tend to yield low construct validity in terms of the extent to which the assessment center reflects the skill or ability traits it was designed to measure. (2,3,4,5,6,7,8,9,10,11,12,13,14,15) This research has found that in general, assessment centers employing managerial assessors exhibit poor convergent and discriminant validity, whereby the same ability traits across exercises do not correlate well across exercises and different traits within exercises show high correlations. Assessor classifications in assessment centers often reflect context driven or method variance rather than hypothesized ability traits (labeled the 'exercise effect').

Lowry's (16) comprehensive survey of assessment centers identified several major uses of ACs, including recruitment and selection, and employee development. The exercise effect presents problems for the recruitment and selection process in terms of a possible lack of precision in determining the ability traits, dimensions or competencies that may be relevant to the job in question. A failure to identify the stable characteristics that may be advantageous in determining who is best suited for a particular position limits the usefulness of the assessment center process. In the developmental context, low construct validity has dire implications because it leads to erroneous feedback being given to participants on the basis of stable ability traits that are not actually being measured. (17) Thus, selection, promotion or developmental decisions made on the basis of assessment center measurement may be fallacious.

Lievens (18) reviewed the factors that have been found to facilitate the construct validity of the ratings in 21 studies of the assessment center process from 1976 to 1997. One of the suggestions Lievens made was that psychologists should be included as a key component of the assessment process. This is because psychologists tend to yield more construct valid ratings than managers, in terms of the extent to which their ratings reflect predetermined constructs. (19) A small number of studies have examined assessor type in relation to construct validity.

Sagie and Magnezy (20) compared the convergent and discriminant validity of psychologist and managerial assessors, and found that psychologist ratings reflected the five predetermined constructs, whereas managerial ratings reflected only two broad dimensions related to performance and interpersonal skills. In their meta analysis of assessment centers, Gaugler, Rosenthal, Thornton, and Bentson (21) reported studies which found that using psychologists as assessors lead to greater predictive validity in assessment ratings. Lievens & Klimoski (22) reported that managerial assessors distinguished dimensions somewhat less well than did industrial and organizational psychology students. However, manager's ratings better reflected the values and norms that were advocated by the organization. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.