Academic journal article Kuram ve Uygulamada Egitim Bilimleri

Comparing the Effectiveness of SPSS and EduG Using Different Designs for Generalizability Theory

Academic journal article Kuram ve Uygulamada Egitim Bilimleri

Comparing the Effectiveness of SPSS and EduG Using Different Designs for Generalizability Theory

Article excerpt

G theory has formed a comprehensive structure by employing variance analysis which provides a broad conceptual framework for social sciences such as psychology and education (Brennan, 2000, 2001a; Cronbach, Gleser, Nanda, & Rajaratnam, 1972; Shavelson & Webb, 1991). It is also a powerful statistical tool for situations where there are numerous measurements. The theory, as an extension of classical test theory and variance analysis, stands as a model where multiple sources of error can be handled (Brennan, 2001a; Shavelson & Webb, 1991).

Generalizability (G) Theory

The reliability of measurement results in education and psychology was previously examined using classical test theory (CTT) in general. It is assumed in CTT that the observed score is composed of the actual score with no separable score for error. The restriction of this assumption, especially in performance measurements where the probability of the existence of more than one source of error is high, reveals the importance of G theory in which more than one source of error is handled and can be predicted simultaneously (Brennan, 2000). Another advantage of G theory in using performance assessment is that while there is a restrictive parallel assumption in CTT, randomly parallel assumption is adopted in G theory (Brennan, 2011; Kretchmar, 2006). The main aim of G theory is to generalize the scores of a specific measurement tool from a specific group to the universe of generalization which consist of 1) the universe of admissible observations and generalizability studies (G studies), 2) the universe of G studies and decision studies (D studies). While G studies provide an estimate of the generalizability coefficient of variances from all facets and this coefficient includes the examinee's universe score, D studies enable one to examine the interactions among all applicable facets (tasks, raters, observations, etc.) and the subject of measurement for calculating the dependability coefficient (Brennan, 2000; Crocker & Algina, 1986; Hsu, 2012).

G theory has four main advantages compared to CTT. 1) It provides simultaneous evaluation of test-retest reliability, internal consistency, inter-rater reliability, and convergent validity. 2) It enables estimates of both individual measurement facets and interaction effects. 3) When assessing an examinee's performance, it gives information about the quality of their absolute structural level of knowledge as well as ranking this information in order. 4) It allows researchers to optimize the reliability of an assessment within the cost constraints of time and money. For example, assessment developers can provide information about how many items, how many raters, and how many occasions are needed to reach a reliable result (Yin & Shavelson, 2008).

When looking at the historical evolution of G theory, its basic principles were first discussed in articles published by Cronbach, Rajaratnam, and Gleser in 1963 and 1965. Indeed, the use of variance analysis in reliability studies started before the work of Cronbach and his colleagues. Burt in 1936, Hoyt in 1941, and Jackson and Ferguson also in 1941 discussed the use of variance analysis in the prediction of reliability. Then the contributions made by Alexander (1947), Ebel (1951), Finlayson (1951), Loveland (1952), and Burt (1955) followed, as cited in Brennan (1992). These were then followed by the book entitled "The Dependability of Behavioral Measurement" by Cronbach, Gleser, Nanda, and Rajaratnam in 1972, which was an extended form of generalizability theory.

In 1983, Brennan's book "Elements of Generalizability Theory" was published. Crick and Brennan designed a computer program called "A Generalized Analysis of Variance System (GENOVA)" in the same year. However, because the theory and the program prepared for it seemed too complex for users, studies concerning the theory remained limited until 1991. Later in 1991, Shavelson and Webb published their book "Generalizability Theory: A Primer," which made the basics of G theory more understandable and the theory more applicable for relevant research studies. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.