Academic journal article The Spanish Journal of Psychology

Computerized Adaptive Testing: The Capitalization on Chance Problem

Academic journal article The Spanish Journal of Psychology

Computerized Adaptive Testing: The Capitalization on Chance Problem

Article excerpt

This paper describes several simulation studies that examine the effects of capitalization on chance in the selection of items and the ability estimation in CAT, employing the 3-parameter logistic model. In order to generate different estimation errors for the item parameters, the calibration sample size was manipulated (N = 500, 1000 and 2000 subjects) as was the ratio of item bank size to test length (banks of 197 and 788 items, test lengths of 20 and 40 items), both in a CAT and in a random test. Results show that capitalization on chance is particularly serious in CAT, as revealed by the large positive bias found in the small sample calibration conditions. For broad ranges of θ, the overestimation of the precision (asymptotic Se) reaches levels of 40%, something that does not occur with the RMSE (θ). The problem is greater as the item bank size to test length ratio increases. Potential solutions were tested in a second study, where two exposure control methods were incorporated into the item selection algorithm. Some alternative solutions are discussed.

Keywords: computerized adaptive testing, capitalization on chance, item parameter estimation.

Se describen varios estudios de simulación para examinar los efectos de la capitalización del azar en la selección de ítems y la estimación de rasgo en Tests Adaptativos Informatizados (TAI), empleando el modelo logístico de 3 parámetros. Para generar diferentes errores de estimación de los parámetros de los ítems, se manipuló el tamaño de la muestra de calibración (N = 500, 1000 y 2000 sujetos), así como la ratio entre tamaño del banco y longitud del test (bancos de 197 y 788 ítems, longitudes del test de 20 y 40 ítems), ambos tanto en un TAI como en un test aleatorio. Los resultados muestran que la capitalización del azar es especialmente importante en el TAI, donde se obtuvo un sesgo positivo en las condiciones de escaso tamaño de la muestra. Para rangos amplios de θ, la sobrestimación de la precisión (Se asintótico) alcanza niveles del 40%, algo que no ocurre con los valores de RMSE (θ). El problema es mayor a medida que se incrementa la ratio entre el tamaño del banco de ítems y la longitud del test. Varias soluciones fueron puestas a prueba en un segundo estudio, donde se incorporaron dos métodos para el control de la exposición en los algoritmos de selección de los ítems. Se discuten también algunas soluciones alternativas.

Palabras clave: tests adaptativos informatizados, capitalización del azar, estimación de los parámetros de los ítems.

(ProQuest: ... denotes formulae omitted.)

The problem of capitalization on chance is common in different psychometric contexts when items or tests have to be selected based on their estimated parameters. An example of this problem occurs when items are selected to optimize test reliability using the classical indices of discrimination. In a subsequent administration, these same items, which were chosen because they had the highest correlation with the rest of the test, may not result in a test as reliable as in the first administration when item parameters were estimated. This happens because items with overestimated discrimination indices in the first administration have a higher probability of being selected. Capitalization also takes place in the estimation of the number of factors to retain in exploratory factor analysis models, in the evaluation of the fit of structural equation models, or in the estimation of the partial regression coefficients in regression models.

The effects of capitalization on chance are directly related to the estimation errors of the parameters included in the statistical model, and depend primarily (though not exclusively) on the available sample size and the estimation method employed (Li & Lissitz, 2004). In item response theory (IRT), Bayesian estimation methods have proven to be effective when sample size is small, by incorporating into the estimation procedure information on the prior distribution of the parameters (Baker, 1992; Gao & Chen, 2005). …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.