Academic journal article English Language Teaching

Evaluation by Native and Non-Native English Teacher-Raters of Japanese Students' Summaries

Academic journal article English Language Teaching

Evaluation by Native and Non-Native English Teacher-Raters of Japanese Students' Summaries

Article excerpt


Although the importance of summary writing is well documented in prior studies, few have investigated the evaluation of written summaries. Due to the complex nature of L2 summary writing, which requires one to read the original material and summarize its content in the L2, raters often emphasize different features when judging the quality of L2 summaries. Therefore, this study examines the ratings of English-language summaries written by Japanese university students in order to identify differences in EFL instructors' evaluations. Fifty-one Japanese EFL university students read a passage and then wrote an English summary without receiving any instructions concerning summary composition. The raters included three native English speakers (NESs) and three non-native English speakers (NNESs), who individually evaluated each summary using the Educational Testing Service's holistic rubric. Analysis of inter-rater reliability revealed a lower Cronbach's alpha coefficient for NNES raters (α = .39) when compared to NES raters (α = .77). Comments were collected from raters regarding the difficulty of evaluating summaries, and the causes of such difficulties were examined. Comments from NNES raters more concerned vocabulary use and paraphrasing, whereas the NES raters concentrated on content and language. This study also explores ways to potentially improve the holistic rubric by examining feedback from raters regarding their rating experiences.

Keywords: evaluation, holistic rubric, native English speaker, non-native English speaker, paraphrasing, reliability, summary writing

1. Introduction

Summary writing is widely recognized as an important teaching method, particularly for university students in foreign language classes, and is also an effective tool for measuring L2 proficiency. Moreover, university students are often required to perform writing tasks that involve summarization when taking academic courses conducted in English. Nevertheless, English as a foreign language (EFL) students often struggle to use source texts properly, despite this being an essential academic skill (Hirvela & Du, 2013; Shi, 2012).

To foster summary writing skills among EFL students, and also enhance the quality of L2 summary writing instruction, a research project was launched to focus on the evaluation of written summaries. This study is one from a series of studies comprising a project for the development of efficient and useful rubrics (or rating scales) for L2 summary writing in EFL academic contexts. The flow of this larger project is illustrated in Figure 1.

Study I (Hijikata, Yamanishi, & Ono, 2011) examined the reliability and validity of a holistic rubric developed by the Educational Testing Service (ETS). In the study, three Japanese raters used the ETS rubric to evaluate summaries written by 51 Japanese EFL university students. The Cronbach's alpha reliability coefficient was .51, which did not indicate sufficient reliability among the three raters. The results of Study I revealed that the rubric was difficult to use, and highlighted the need for an analytic rubric specifically targeting non-native English speakers (NNESs) who evaluate and teach L2 summary writing in higher education.

To determine whether an analytic or holistic rubric is better suited to evaluating and teaching L2 summary writing in an academic context, it is first necessary to identify components that NNES raters find particularly difficult to grade, and also examine whether their evaluations differ from those of native English speakers (NESs). Therefore, the purpose of the present study (Study II) is to investigate the feasibility of developing an analytic rubric that is capable of overcoming difficulties associated with the use of a holistic rubric. After identifying the causes of difficulties faced by NNESs when grading summaries, this research project intends to propose a new rubric and refine it in Study III and Study IV. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.