Academic journal article Agenda: A Journal of Policy Analysis and Reform

Citations as a Measure of the Research Outputs of New Zealand's Economics Departments: The Problem of 'Long and Variable Lags'

Academic journal article Agenda: A Journal of Policy Analysis and Reform

Citations as a Measure of the Research Outputs of New Zealand's Economics Departments: The Problem of 'Long and Variable Lags'

Article excerpt

Abstract

The paper explores the merits of utilising citation counts to measure research output in economics in the context of a nationwide research evaluation scheme: the New Zealand Performance-Based Research Fund (PBRF). Citations were collected for all refereed papers produced by New Zealand's academic economists over the period 2000 to 2008, and used to estimate the time-lags in between publication and the flow of citations; to demonstrate the impact of alternative definitions of 'economics-relevant' journals on citation counts; and to assess the impact of citation measures on departmental and individual performance. We conclude that under certain scenarios around 60 per cent of papers received no citations over the period. Our findings suggest that the time-lags between publication and citation make it difficult to rely on citation counts to produce a meaningful measure of output in a PBRF-like research-evaluation framework, especially one based on the assessment of individual academics.

Introduction

The external evaluation of the research of university faculty has become an important part of academic life in many countries. The processes used to assess research differ significantly in the way in which evaluations are undertaken.2 Peer-review panels or expert panels are used by a number of countries, but they are costly, both for the assessors and those being assessed. Advances in information technology and the quality of research databases have made citation analysis a more attractive alternative or complement to other assessment processes. The primary purpose of this paper is to explore the merits of utilising citations to measure research output in economics in the context of a nationwide research-evaluation scheme.

To provide a context for the evaluation of citation analysis we discuss its potential use in relation to the New Zealand government's Performance-Based Research Fund (PBRF).3 To assess the quality of research under this program, individual academic staffmust submit research portfolios providing a review of their research output, of their contribution to the research environment and of peer esteem over the previous six years. These portfolios are assessed and scored by peer-review panels formed for each subject area. Individual scores are aggregated to provide assessments of subject areas in an institution and the tertiary institution overall. Individual scores are reported to the academics concerned, and subject and institution results are made public. PBRF evaluations of the quality of research were undertaken in 2003 and 2006, and will be undertaken in 2012.4 The evaluations of research quality determine 60 per cent of the PBRF funding allocations to tertiary institutions, with the remaining 40 per cent based on research degree completions and external research income. In 2010 the results were used to distribute NZ$268 million of research funding; this is equal to 18 per cent of government funding to universities and roughly 9 per cent of total system-wide revenue. Furthermore, and equally important, the PBRF results are aggressively used by the winners in their formal and informal promotional material.

Unlike the Australian ERA process, PBRF peer-review assessments are not supplemented by citation analysis. Over time it is likely that counting citations will play a part. It is our view that after the upcoming 2012 PBRF round, pressure may mount for the government to consider a shift, at least in part, to a metric-based system. If this were to occur, citation counting is likely to be at the heart of any such scheme, given its widespread acceptance as a reliable measure of performance in the physical and biological sciences. This view is based, in part, on the tendency of New Zealand tertiary education policy to follow that of the UK and, to a slightly lesser extent, Australia. In both of these countries, citation counting is now partially incorporated into their nationwide research-evaluation schemes. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.