Magazine article Talent Development

Evaluation Biases Could Affect the Credibility of Your Results: Sources of Bias Cut across All Four Levels of Evaluation

Magazine article Talent Development

Evaluation Biases Could Affect the Credibility of Your Results: Sources of Bias Cut across All Four Levels of Evaluation

Article excerpt

The evaluation challenge is one of balance. You must do enough analysis to meet your own needs while also meeting the needs of your client. Too much evaluation is a waste of effort; not enough inhibits good decision making. This is why evaluation planning is so important. As part of the design process, you determine the initial business metric (the data you will track, such as number of sales, number of defects, or turnover rates), what evaluation level of information to gather, when to gather that information, and how it will be used.

But when it comes to evaluation, the original development of your research methods and instruments is subject to several types of bias. Although it is difficult to address bias, it is not impossible. First, recognize that bias exists and then take action to minimize it.

Sampling bias

The first type of bias is sampling bias. It is easy to send surveys or interview certain participants whom you know and like and who are favorably disposed to the program. There is also a tendency to send the information to recent participants, who are usually still enthusiastic about the course and the opportunity to implement their action plans. The realities of the environment have not dampened their spirits.

These practices can result in tainted data by introducing sampling bias. You should always conduct surveys or interviews with participants selected on a random basis.

Insufficient sample size

The second bias comes from not having a large enough sample. Sampling generally applies to Level 3 and 4 evaluations. You could draw your sample from the entire population; in this case, all the participants taking a course.

Alternatively, if a course has several deliveries, you could randomly select particular deliveries to evaluate. Depending on the audience size, you may want to randomly select participants from each delivery to be evaluated. For example, if you designed and delivered a sales training course that is delivered twice a month with an average class size of 24 participants, you could randomly select two or three of the 24 deliveries. Then, you could randomly select participants from each delivery. Make sure you statistically determine the sample size required to provide reliable data, because a sample that is too small is not representative of the entire population or group.

Observation bias

The observation technique is not without bias problems. The more visible the observation process is, the less reliable the data.

Participants will perform differently if they know they are being observed, and an observer who is not trained or given proper instruments adds to the unreliability of the data. Therefore, conduct your observations in the least obtrusive manner possible while still getting the information you need. Provide training for the observers and use some sort of instrument, such as a checklist, to aid in the observation.

Bias in interviews and focus groups

Interviews and focus groups can provide high-quality information. To be most useful and to avoid bias, the interview design must ensure that:

* the sample is representative of the population

* the participants understand the questions

* the participants are willing participants (their participation is not mandatory)

* the interviewer is trained in interviewing techniques and knows how to record the information accurately

* there is a protocol for consistency in questioning

* there is a method to objectively evaluate the results of the interview.

Restriction of range or range error

Some respondents to a survey or questionnaire may engage in the error of restriction or range. This occurs when the respondent, or rater, restricts her ratings to a small section of the rating scale. This could be positive (leniency) or negative (severity).

In some cases, this phenomenon is an unconscious bias on the part of the rater. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.