Quality Measurements of Error Annotation-Ensuring Validity through Reliability

By Dobric, Nikola | European English Messenger, Summer 2015 | Go to article overview

Quality Measurements of Error Annotation-Ensuring Validity through Reliability


Dobric, Nikola, European English Messenger


Major obstacles to achieving high levels of reliability (and by extension validity) of error annotation of learner corpora range from defining errors in general, the lack of an error taxonomy sufficiently applicable in corpus annotation, insufficiencies of any set linguistic norm as background for tagging, to the lack of well-defined measurements of quality of annotation. The paper first looks at the theoretical issues behind the definition of an error. It expands the discussion by focusing on a more practically applicable account of errors aimed at error annotation. It goes on to offer a more robust error taxonomy which could help address issues of interpretability inherent in linguistic categorization and could ensure more consistency. In the end, the paper suggests an alternative definition of an error applicable for corpus annotation, based on inter-annotator agreement and aimed at being the primary indicator of validity.

1. Introduction and background

Error annotation of learner corpora (1) is problematic when it comes to defining errors in practical terms, in terms of the error classification chosen as the background for a tagset, in terms of the annotators' training to use a set linguistic norm, and in terms of assessment of the quality (or 'correctness') of annotation. Assessing the quality of annotation is essential for gauging the validity of the linguistic information we wish to extract from the corpus. Since validity cannot be assessed directly, due to the lack of 'ground truth' in human (linguistic) judgment, the only thing we can assess is the reliability of annotation as indicating validity of the tags assigned (Plaban, Pabitra and Anupam 2000). The reliability is mirrored in consistency of annotation (Brants 2000), done both by each individual annotator (intra-annotator agreement) and by more annotators when compared to each other (inter-annotator agreement). High levels of consistency signify high quality of data processing. It is hence important to ensure high levels of consistency of tagging, which would lead to high reliability of annotation and hence signify valid information in the corpus.

Starting with the issue of accounting for errors, there is a general consensus that errors constitute failures in language competence (Corder 1971; Lennon 1991; Lengo 1995). From the point of view of a practical application in any form of error analysis, as in corpus annotation, such a take on errors is not really useful. Firstly, following such a theoretical premise in practice would lead us to a conclusion (reached already by other authors, such as for example James (1998:79)) that errors cannot be produced by 'native' speakers. Only mistakes could be possible. (2) Secondly, language competence is in broad terms accessible only indirectly through language performance and needs to be observed accordingly. In terms of error annotation (and applied linguistics in general), it is clear we need a more practice-oriented definition of an error. In other words, we need something more tangible that annotators can actually hold on to while tagging.

A common approach to solving this problem is to define errors using a set linguistic norm as a background. The opaque (and often unjustly negatively connoted) term 'error' is then (much more accurately) referred to as a 'non-norm adequate form'. In practice, the linguistic norm can be tied to a standard variety of a language, its grammatical description, dictionaries, and in case of learner corpus annotation, to the training of annotators. This is a more applicable take on errors--there is a common linguistic background set for all annotators. It can clearly indicate to annotators what the acceptable, norm-adequate performance is, at least up to a certain hierarchical level of language description. It makes their determining what is not adequate simpler and ideally produces more agreement between their categorizations.

2. Error annotation in practice

The next stumbling block on the road towards higher consistency of learner corpus annotation is the choice of a taxonomy of non-norm adequate features to be used as a tagset. …

The rest of this article is only available to active members of Questia

Already a member? Log in now.

Notes for this article

Add a new note
If you are trying to select text to create highlights or citations, remember that you must now click or tap on the first word, and then click or tap on the last word.
One moment ...
Default project is now your active project.
Project items
Notes
Cite this article

Cited article

Style
Citations are available only to our active members.
Buy instant access to cite pages or passages in MLA 8, MLA 7, APA and Chicago citation styles.

(Einhorn, 1992, p. 25)

(Einhorn 25)

(Einhorn 25)

1. Lois J. Einhorn, Abraham Lincoln, the Orator: Penetrating the Lincoln Legend (Westport, CT: Greenwood Press, 1992), 25, http://www.questia.com/read/27419298.

Note: primary sources have slightly different requirements for citation. Please see these guidelines for more information.

Cited article

Quality Measurements of Error Annotation-Ensuring Validity through Reliability
Settings

Settings

Typeface
Text size Smaller Larger Reset View mode
Search within

Search within this article

Look up

Look up a word

  • Dictionary
  • Thesaurus
Please submit a word or phrase above.
Print this page

Print this page

Why can't I print more than one page at a time?

Help
Full screen
Items saved from this article
  • Highlights & Notes
  • Citations
Some of your highlights are legacy items.

Highlights saved before July 30, 2012 will not be displayed on their respective source pages.

You can easily re-create the highlights by opening the book page or article, selecting the text, and clicking “Highlight.”

matching results for page

    Questia reader help

    How to highlight and cite specific passages

    1. Click or tap the first word you want to select.
    2. Click or tap the last word you want to select, and you’ll see everything in between get selected.
    3. You’ll then get a menu of options like creating a highlight or a citation from that passage of text.

    OK, got it!

    Cited passage

    Style
    Citations are available only to our active members.
    Buy instant access to cite pages or passages in MLA 8, MLA 7, APA and Chicago citation styles.

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn, 1992, p. 25).

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn 25)

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn 25)

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences."1

    1. Lois J. Einhorn, Abraham Lincoln, the Orator: Penetrating the Lincoln Legend (Westport, CT: Greenwood Press, 1992), 25, http://www.questia.com/read/27419298.

    Cited passage

    Thanks for trying Questia!

    Please continue trying out our research tools, but please note, full functionality is available only to our active members.

    Your work will be lost once you leave this Web page.

    Buy instant access to save your work.

    Already a member? Log in now.

    Search by... Author
    Show... All Results Primary Sources Peer-reviewed

    Oops!

    An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.