Assessing High-Order Skills with Partial Knowledge Evaluation: Lessons Learned from Using a Computer-Based Proficiency Test of English for Academic Purposes

By Aluisio, Sandra Maria; Pizzirani, Rafael et al. | Journal of Information Technology Education, Annual 2003 | Go to article overview

Assessing High-Order Skills with Partial Knowledge Evaluation: Lessons Learned from Using a Computer-Based Proficiency Test of English for Academic Purposes


Aluisio, Sandra Maria, Pizzirani, Rafael, de Aquino, Valeria Tomas, de Oliveira, Osvaldo Novaes, Jr., Journal of Information Technology Education


Introduction

In non-English speaking countries, students enrolled in Master and PhD degree programs are usually required to undertake an English proficiency test (EPT), aimed at assessing their ability to understand and produce technical literature in English as it is the language used in most scientific journals. In practice, such EPT consists in asking students to translate a text passage from a scientific paper or a technical book from English (L2) into their mother tongue (L1), and in some cases to produce a version in L2 of another piece of technical text originally in L1. It is usual, as was the case of the Institute of Mathematics and Computer Science at the University of Sao Paulo (ICMC), Brazil, until April 1998, that the test is prepared each year by a different member of the academic staff. This is disadvantageous since distinct exams may be highly non-uniform due to the high degree of subjectiveness in the evaluation process. The alternative approach used by some Universities and funding agencies is to require the student to pass a general-purpose exam such as TOEFL (http://www.toefl.com) or IELTS (http://www.ielts.org). None of these exams, however, evaluate students' competence in terms of the demands of highly standardized research articles written in English. Furthermore, the need of "genre-consciousness" is not aroused, which is essential for a novice researcher to better and faster perform the reading and writing tasks for his/her own research. In order to obviate these limitations, the first author of this paper proposed a new type of proficiency exam for graduate students, in which students' competences would be evaluated according to four modules: M1: dealing with the analysis of the structure of a given section of a paper, where the students had to identify the section components; M2: dealing with the analysis of relationships among clauses which are signalled by discourse markers; M3: involving knowledge of conventions in the English language for scientific texts; M4: involving knowledge of writing strategies for each component of the sections' paper.

The four modules include questions that assess two cognitive levels from the well-known Bloom's Taxonomy (Bloom, 1956), which divide cognitive objectives into six subdivisions ranging from the simplest behavior to the most complex: knowledge, comprehension, application, analysis, synthesis and evaluation. In addition to the knowledge category, normally used in multiple-choice tests, we included questions of two modules (M1 and M2) to assess competences in the fourth level of difficulty (analysis). In order to evaluate such abilities, particularly those for a higher cognitive level, we employed an information theoretical model of knowledge assessment to measure the student knowledge base. According to this model, a student may have total information, almost total information, partial information, partial misinformation, misinformation or total lack of information about a given topic being assessed. This approach makes use of a scoring procedure referred to as Admissible Probability Measurement (APM) developed by Shuford & Brown (1974) and employed in computerized exams by Bruno (1986, 1987) and Bruno, Holland & Ward (1988) and in manually marked exams by Klinger (1997).

As we shall comment upon later, this new proficiency exam was successful but relied on the work of a lecturer who is knowledgeable of a number of issues in scientific writing. This severe limitation could only be overcome if the exam was fully automated. This prompted us to develop a computer-based assessment system in which the students' ability to read and write scientific literature in English was evaluated with objective questions and the APM scoring procedure. This computer-based system possesses a test management subsystem and a test delivery subsystem that are integrated into a single application. It is web-based and allows access to 5 different types of users: test administrator, instructor, students, master program secretary, and general public. …

The rest of this article is only available to active members of Questia

Already a member? Log in now.

Notes for this article

Add a new note
If you are trying to select text to create highlights or citations, remember that you must now click or tap on the first word, and then click or tap on the last word.
One moment ...
Default project is now your active project.
Project items

Items saved from this article

This article has been saved
Highlights (0)
Some of your highlights are legacy items.

Highlights saved before July 30, 2012 will not be displayed on their respective source pages.

You can easily re-create the highlights by opening the book page or article, selecting the text, and clicking “Highlight.”

Citations (0)
Some of your citations are legacy items.

Any citation created before July 30, 2012 will labeled as a “Cited page.” New citations will be saved as cited passages, pages or articles.

We also added the ability to view new citations from your projects or the book or article where you created them.

Notes (0)
Bookmarks (0)

You have no saved items from this article

Project items include:
  • Saved book/article
  • Highlights
  • Quotes/citations
  • Notes
  • Bookmarks
Notes
Cite this article

Cited article

Style
Citations are available only to our active members.
Buy instant access to cite pages or passages in MLA, APA and Chicago citation styles.

(Einhorn, 1992, p. 25)

(Einhorn 25)

1. Lois J. Einhorn, Abraham Lincoln, the Orator: Penetrating the Lincoln Legend (Westport, CT: Greenwood Press, 1992), 25, http://www.questia.com/read/27419298.

Cited article

Assessing High-Order Skills with Partial Knowledge Evaluation: Lessons Learned from Using a Computer-Based Proficiency Test of English for Academic Purposes
Settings

Settings

Typeface
Text size Smaller Larger Reset View mode
Search within

Search within this article

Look up

Look up a word

  • Dictionary
  • Thesaurus
Please submit a word or phrase above.
Print this page

Print this page

Why can't I print more than one page at a time?

Help
Full screen

matching results for page

    Questia reader help

    How to highlight and cite specific passages

    1. Click or tap the first word you want to select.
    2. Click or tap the last word you want to select, and you’ll see everything in between get selected.
    3. You’ll then get a menu of options like creating a highlight or a citation from that passage of text.

    OK, got it!

    Cited passage

    Style
    Citations are available only to our active members.
    Buy instant access to cite pages or passages in MLA, APA and Chicago citation styles.

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn, 1992, p. 25).

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences." (Einhorn 25)

    "Portraying himself as an honest, ordinary person helped Lincoln identify with his audiences."1

    1. Lois J. Einhorn, Abraham Lincoln, the Orator: Penetrating the Lincoln Legend (Westport, CT: Greenwood Press, 1992), 25, http://www.questia.com/read/27419298.

    Cited passage

    Thanks for trying Questia!

    Please continue trying out our research tools, but please note, full functionality is available only to our active members.

    Your work will be lost once you leave this Web page.

    Buy instant access to save your work.

    Already a member? Log in now.

    Oops!

    An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.