Academic journal article Issues in Informing Science & Information Technology

Formative Assessment Visual Feedback in Computer Graded Essays

Academic journal article Issues in Informing Science & Information Technology

Formative Assessment Visual Feedback in Computer Graded Essays

Article excerpt

Background

The motivation for developing computer supported techniques to assess or grade free text assignments or essays is rather obvious--increased speed, efficiency and consistency, and thus reduced costs and an amelioration of the onerous nature of (humans) marking large volumes of essays in a short time. Of course, this assumes effectiveness, reliability and user (student and teacher) acceptance of 'computer as assessor'. These three aspects have been reported on in the work of Williams & Dreher (2004) for example.

Automated Essay Grading (AEG) is an emerging phenomenon widely documented in the literature (Shermis & Burstein, 2003; Valenti, Neri & Cucchiarelli, 2003; Williams, 2001; Williams & Dreher, 2004). Many of the current AEG systems claim to produce various kinds of feedback regarding the knowledge deficit or other problems in the essays enabling the essay authors to learn, improve, and correct the errors for future submissions. However, much of the feedback is generic in form, for example "this section is inadequate" or "this section needs improvement". This sort of feedback is not very helpful to the learner, and if the truth be known, it is often provided as a justification for the mark, so that when a student queries the grade given, the assessor can offer some further 'soothing' words at least not inconsistent with the original feedback. Of course, the type of evaluation we are concerned with here is formative, and we appreciate that the case of summative evaluation needs to be treated separately--our interest is in the former.

Purpose of Assessment

In our work on grading and assessment we take the view that incremental improvement is an important goal for the learner and the teacher. This implies that when students are given assignments it is the teacher's role to evaluate the work against the stated assignment assessment criteria and provide the student with a grade and some reasons which explain why the particular grade was awarded. An example of such a scheme can be seen in Figure 1 for a course dealing with JavaScript programming and website development.

Naturally, the criteria given in Figure 1 must be distributed with the assignment specification; else the students' would have no goal. The assessment task for such assignments involves considering the assignment from the viewpoint of each of the six criteria and making some judgment and generating relevant comments.

Assignment tasks which can conveniently be subdivided into chunks, an extreme example being Multiple-Choice or True-False Tests, lend themselves to computer scoring. However the more essay-like the assignment task the greater the challenge for automated or semi-automated assessment. Nevertheless, there is a growing body of literature in the field of AEG--see below.

In an interesting case of formative evaluation in a course with well in excess of one hundred students, and the flexibility for the students to choose from a variety of topics or themes (Dreher, Scerbakov & Helic, 2004), the authors claim good support provided by the Learning Management System (WBT-Master), which permits individual and relevant formative evaluation comments to be efficiently generated. Figure 2 is a screenshot of an essay assignment being assessed and commented upon.

[FIGURE 2 OMITTED]

It should be clear that evaluating assignments and providing feedback to students for the purposes of improvement is on the one hand good education practice, and on the other is very 'expensive'. As we have been developing our AEG system (MarkIT) we have had a unique opportunity to ponder on the provision of meaningful, relevant, consistent feedback which students can use to reflect on their own performance in essay writing.

We now present a short section on the state of the art of AEG, making particular note of the nature and extent of feedback which is provided by these systems, and then take the opportunity to explain how our AEG system has been engineered in terms of feedback provision. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.