Academic journal article Educational Technology & Society

Evaluating Knowledge Structure-Based Adaptive Testing Algorithms and System Development

Academic journal article Educational Technology & Society

Evaluating Knowledge Structure-Based Adaptive Testing Algorithms and System Development

Article excerpt

Introduction

During the last two decades, from the functional aspect, many computerized test systems have been developed for estimating abilities of examinees (Chang, Lin, & Lin, 2007; Guzman & Conejo, 2005; Lewis & Sheehan 1990; Sands, Water, & McBride, 1997; Sheehan & Lewis, 1992; Wainer, 2000; van der Linden, 2000; Tao, Wu, & Chang, 2008; Yen, Ho, Chen, Chou, & Chen, 2010) or diagnosing students' learning profiles (Appleby, Samuels, & Treasure-Jones, 1997; Chang, Liu, & Chen, 1998; Hwang, Hsiao, & Tseng, 2003; Liu, 2005; Tsai & Chou, 2002; Tselios, Stoica, Maragoudakis, Avouris, & Komis, 2006; Vomlel, 2004;Yu & Yu, 2006). From the theoretical aspect, some of them are based on item-response theory (IRT) (Chang et al., 2007; Guzman & Conejo, 2005; Lewis & Sheehan, 1990; Sands et al., 1997; Sheehan & Lewis, 1992; Wainer, 2000; van der Linden, 2000; Yen, et al., 2010), some of them are based on artificial intelligence techniques such as Bayesian networks (Liu, 2005; Tselios et al., 2006; Vomlel, 2004), and others are based on knowledge structures. From the operational aspect, some of the computerized tests are adaptive and others are non-adaptive. The focus of this study is to construct computerized adaptive tests based on knowledge structures for diagnosing students' learning profiles.

The computerized adaptive test (CAT) can not only offer examinees customized items in accordance with their aptitudes or cognitive status, but can also shorten the test. The CAT based on IRT models can obtain efficient estimates of subjects' abilities, but it cannot provide the capability to diagnose subjects' cognitive concepts at a detailed level (Tatsuoka, Corter, & Tatsuoka, 2004; Yan, Almond, & Mislevy, 2004). Instead, knowledge structure- or artificial-intelligence-based adaptive tests can provide information about how well subjects performed on specific concepts, so they can achieve the diagnostic function (Appleby et al., 1997; Tatsuoka et al., 2004; Vomlel, 2004).

Diagnosys, developed by Appleby et al. (1997), is a knowledge-based-computer diagnostic test of basic mathematical concepts. In Diagnosys, a method was proposed to estimate the knowledge structure of examinees and then apply this structure to build the adaptive testing process. Chang et al. (1998) have proposed adaptive test algorithms to construct a computerized adaptive diagnostic test based on knowledge structures constructed by the domain experts. The results of these two papers exhibit that the proposed algorithms have the capability of decreasing the use of test items and are able to precisely diagnose the cognitive status of examinees. However, the impact of correct guessing on the diagnoses of concepts is not considered in these studies. Correct guessing means that an item is answered correctly by guessing in multiple-choice tests. In knowledge-based adaptive tests, if an item is answered correctly by guessing, then all prerequisite items of it are assumed to have been answered correctly. But, in actual fact, these prerequisite items may not have been answered correctly. In that situation, the precision of diagnosing results would be decreased. Moreover, the impact of correct guessing in adaptive testing would be greater than that in non-adaptive testing such as the traditional paper-and-pencil test.

Tselios et al. (2006) used the Bayesian network to diagnose students' problem-solving strategies with two distinct problems. The results show that the Bayesian network can estimate students' problem-solving strategies very well, but it is not an adaptive test. Vomlel (2004) and Liu (2005) have proposed adaptive testing algorithms based on the Bayesian network. In their simulation study, the numbers of test items were 10 and 21, respectively. The experimental results show that the Bayesian network is a powerful tool to diagnose students' learning status; however, it is difficult and time consuming to find the optimal adaptive testing strategy when the test is long. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.