... a Viable Option for Selecting People and Not as Bad as It Looks: Two Global Executive Assessment Consultants with Perspectives from a Practitioner Standpoint
Irving, Robert, O'Reilly, Stuart, European Business Forum
The study of effective selection and assessment methods in any rigorous way is, by and large, an academic discipline with often only the most superficial distillations of the results penetrating through to the mainstream business world, despite the huge sums spent by organisations on recruitment activities. However, as a manager; one of the most important things we do is to select our people and at executive level a poor selection decision can often have a major impact on a business. A study cited by Bennis and O'Toole in Harvard Business Review (Autumn, 2000) found that since 1995 one third of the Fortune 100 companies had changed their CEO and that only some of the changes were due to natural reasons. They go on to argue that a major cause of failure in appointments is that insufficient attention is paid to leadership as a selection criterion. Dawes' article, though academic in tone and content, deals with one of the most critical factors facing business.
His article makes two key points the first being that actuarial methods (i.e. the use of mathematical equations) of combining data make better predictions than 'experts' combining data (i.e. individuals using their intuitive understanding and experience). His second key point is that the actuarial method is rejected as a practical tool precisely because it yields specific data, which shows that it makes relatively ineffective predications. His third point implicit in the title is that psychometrics are poor predictors of individuals' success in jobs.
We deliberately do not intend to debate the details of actuarial versus expert decision-making at an academic level but would make the following arguments from a practitioner standpoint. In management positions particularly, jobs are not tightly structured. The nature of the same job can be quite different at various points in time depending on the challenges posed, the individual's manager and the talents and interests of the job incumbents. Organisations initially ask 'can this person do this job?' but they make the final decision 'on the way' in which the person will do the job. Given this fluidity in job structure what organisations want are methods that will give them insight into this type of issue. Clear criteria, structured interviews and psychometrics can all help get at this answer. Where all three are combined, the results can be a powerful description of the individual's abilities and predisposition to adopt certain types of behaviour and thinking styles. This is the case for measurement and rigour and its effectiveness is testified to in the academic statistics. However, the very fluidity of this situation mitigates against an actuarial decision-making process. Our own experience also tells us that when this type of information is available a proportion of managers do make their decisions by weighting this wider set of information rather than simply eliminating factors.
The question therefore is how do we know rigorous assessment of individuals via tests and questionnaires is accurate and how do we know in practice it is predictive. First, there is the sanity test. The organisations we have worked with normally endorse the judgements we make around their people. Secondly, in those cases when people fail in their jobs, not-withstanding clear evidence at the point of selection that they would, our clients ruefully look back at the data, shake their heads and say "you told us he/she would but we decided to run with our gut feelings." This conversation is surprisingly common. Additionally when we discuss their development needs with individuals they generally agree with the different strengths and weaknesses identified in the assessments.
Our response to Dawes's view that psychometrics are poor predictors of performance, is twofold. Firstly, as practitioners it is apparent that individual job competence is in many ways a function of personality and intellectual ability (both measured well by questionnaires and tests) as well as technical knowledge. …