Academic journal article Psychological Test and Assessment Modeling

Which Person Variables Predict How People Benefit from True-False over Constructed Response Items?

Academic journal article Psychological Test and Assessment Modeling

Which Person Variables Predict How People Benefit from True-False over Constructed Response Items?

Article excerpt

Multiple Choice (MC) and True-False (TF) tests have previously been investigated in the early 20th century and their diagnostic quality has been a controversial topic among researchers. Nonetheless, their popularity among examiners continues to grow. An important advantage is their high economy both in administration and analysis. Furthermore, in comparison to Constructed Response (CR) questions, where examinees are asked to give a short answer, the analysis of MC or TF questions is more objective in evaluation. On the other hand, it is a well-known fact that MC tests encourage guessing what is known to be a huge diagnostic problem. Since different strategies might lead to the correct answer to MC items, they face the possible loss of unidimensionality (Kubinger, 2014). The main question that is addressed in this paper is whether people benefit from answering MC (or TF) items over CR items. More precisely we are interested in a person specific variable, which we call Benefit from TF that indicates to what extent someone benefits from answering TF over CR items.

In the current paper TF items are used which means "each item is a statement to be judged true or false", whereas MC means "items involving a single choice from 3 or more answer options" (Burton, 2002, p.805). If the test type is not specified in the literature, we call it 'MC test'. In the following section we present an overview of former research, which leads to our hypotheses (1) to (3).

When it comes to research on MC tests, studies have focused mainly on the effect of guessing, which is one aspect of what happens when using MC instead of CR exams. Those studies mostly involve some kind of punishment for false answers (e.g. subtracting incorrect answers from the total score) in order to examine guessing behavior related to risk-taking. Nevertheless the present paper does not aim to examine exclusively guessing. We rather intend to find out, which characteristics a person must exhibit in order to benefit from TF over CR items irrespective of his guessing behavior.

Benefit from TF as a systematic variable. The main question of interest in guessing research was, whether examinees benefit from answering items on which they guess. The main methodology for measuring guessing was to ask people to answer items they would have omitted under penalty for guessing (e.g. Bliss, 1980; Cross & Frary, 1977; Lord, 1975; Sherriffs & Boomer, 1954). Although many of these studies revealed that people benefit from guessing, also some of them showed the contrary. These results suggest that not all people benefit from guessing in the same way. Therefore guessing cannot only be seen as a constant contribution to error variance but should rather be seen as a systematic variable on which people differ. For this reasons, we believe that also Benefit from TF is a systematic variable. In the following we are going to define the precise measurement of this variable.

Measurement of Benefit from TF. In some former studies researchers measured guessing by comparing sum scores in CR with those of MC tests concerning the same subject matters (e.g. Kennedy & Walstad, 1997; Kuechler & Simkin, 2010). Thereby it is possible to quantify an overall Benefit from TF in a certain subject. However, in the present study the aim was to be able to tell for every item whether the answer was based on the Benefit from TF, as the aim was to separate achievement with MC items completely from achievement in general. Given the fact that this is not possible with any of the previously used methods, it was necessary in our case to formulate a completely new method.

We decided to operationalize Benefit from TF by designing a CR item to each of the TF items, which contains the same knowledge. An example of a TF item is: "Is the following statement true or false? The binomial random variable is a continuous random variable." And its corresponding CR item is: "Give an example of a) a discrete and b) a continuous random variable". …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.