Academic journal article Brazilian Political Science Review

Expert-Driven and Citational Approaches to Assessing Journal Publications of Brazilian Political Scientists *

Academic journal article Brazilian Political Science Review

Expert-Driven and Citational Approaches to Assessing Journal Publications of Brazilian Political Scientists *

Article excerpt

Because of the intense theoretical and methodological debates that are underway in Brazil, the discipline of political science1 is experiencing one of its most dynamic and vital periods. The academic research of Brazilian scholars is increasingly relevant to discussions on a variety of pressing questions in the field. At the same time, however, there is an important literature that has shown that the quality of the research produced by Brazilian political scientists has significant and severe methodological shortcomings (SOARES, 2005). Albeit having made significant advances in the last decade (NEIVA, 2015)2, an aspect that has not been analyzed in depth is the determinants of the expert-driven index that is employed by the Brazilian federal government's Coordination for the Improvement of Higher Education Personnel (CAPES) to evaluate the quality of the academic research produced by the faculty and students in the nation's political science graduate programs3.

There are an extensive variety of methods used to assess the quality of academic research. Reputational criteria, which are based on expert opinion, have historically served an essential role as instruments for assessing the quality of political science journals (CREWE and NORRIS, 1991; GILES and WRIGHT, 1975). These methods contrast with citation-based criteria that seek to measure a journal's impact and which have come to predominate as instruments to evaluate individual scholars (KLINGEMANN et al., 1989; MASUOKA et al., 2007), departments (HIX, 2004) and journal quality (GILES and GARAND, 2007; PLUMPER, 2007) in the discipline. Based on the sample of Brazilian political science academic research published in journals from 2010-2014, we use bivariate correlations and ordered logistic regressions to examine if high-quality journals according to the Qualis expert-driven index are also journals with the highest impact according to citation metrics. Furthermore, we explore whether additional structural factors (the language of publication, country of publication, discipline area) might explain divergences between both metrics.

We show that there is a positive, but weak correlation between citational criteria and the Qualis evaluation of the same journals. Indeed, the correlation between the Qualis journal rank and the commonly used impact indexes is 0.35. In ordered logistic regressions, we show that a journal's past Qualis scores are the most important factor for explaining its grades in the next evaluation. We show that once a journal's past Qualis score is considered, a journal's citational ranking does not influence its Qualis score with the exception of the SJR in the 2013-4 evaluation. Moreover, a journal's Qualis score is not influenced by the country of publication, language, or social science focus, all else equal.

The article is structured in four sections in addition to this introduction and the conclusion. The next section presents a brief description of our research design. We then review the literature and discuss the metrics that have been developed to assess academic research quality. The fourth section presents the results of our empirical analysis of the relationship between the CAPES journal quality ranking and the selected international impact indicators. Finally, the conclusion discusses directions for further research.

How should one measure a journal's relevance?

Ideally, an impact assessment system would require the complete reading of each published article in order to measure its quality (GARFIELD, 2005). Given the infeasibility of this approach from a practical standpoint, evaluators adopted the practice of assessing the importance and quality of published articles based on judgments of the journals in which they appear (GILES and GARAND, 2007). In the field of political science, the first-generation of studies to evaluate journal quality were based on reputational criteria. Giles and Wright (1975) were the first scholars to develop a ranking of journal relevance based on expert opinions of 63 journals. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.