Academic journal article Australian Journal of Language and Literacy

Analysis and Comprehension of Multimodal Texts

Academic journal article Australian Journal of Language and Literacy

Analysis and Comprehension of Multimodal Texts

Article excerpt


As educators are increasingly acknowledging that reading comprehension necessarily entails the integrative construction of meaning from images and language in the majority of contemporary texts (Unsworth, Thomas & Bush, 2004), measures of students' reading comprehension achievement will also necessarily entail their negotiation of inter-semiotic meanings. The theoretical framework for this research sits within the realm of social semiotics. The research concerns an investigation of student comprehension of image-language relations in multi-semiotic texts, using a framework developed from systemic functional linguistics. The site for the investigation is the NSW Basic Skills Test (BST) and specifically the Aspects of Reading part of that test administered to students in 2005 and 2007 (NSW Department of Education and Training, 2005, 2007). The data considered in this investigation includes textual data and state-wide comprehension data. This paper outlines how the textual analysis was conducted.

Context of the study

The specific context for the research was analysis of the 2005 Year 3 (age 8) and Year 5 (age 10) NSW Basic Skills Tests (BST) reading materials and questions. The reading materials were analysed using established linguistic and visual textual frameworks, namely Functional Grammar (Halliday, 1994) and Visual Grammar (Kress & van Leeuwen, 1996), and an emerging framework for image-language relations in texts (Unsworth, 2006a; Unsworth, 2008). Test item difficulties were obtained from Rasch analysis which enabled the difficulty of items (test questions) and ability of students to be placed on the same linear scale across all year groups taking the tests using 'probabilistic equations' (Bond & Fox, 2001, p. 7). Early analysis of the 2005 reading materials suggested the draft model of image-language relations was appropriate. However, the number of items was limited, so reading materials and items from the 2007 BST and the Year 7 English Language and Literacy Assessment (ELLA) were later added to increase the data set for analysis. The Year 7 ELLA items are on the same common scale of difficulty as the Year 3 and Year 5 BST items.

Model of image-language relations and text analysis

The model of image-language relations applied during the research was developed around the notions of 'concurrence' and 'complementarity' in representational or ideational meaning (Unsworth, 2006b, 2008; Chan, in press). The following definitions were used to identify the different types of image-language relations which are further elaborated and exemplified by Unsworth and Chan (2009) and Chan (in press).

Concurrence is a relationship where one mode elaborates on the meaning of the other by further specifying or describing it while no new element is introduced by the written text or image. The elaboration can take four forms:

* exemplification, where the image may be an example or instance of what is in the text, or the text may include an example of what is depicted more generally in the image e.g. when words mention 'destructive behaviour of pets' and photo shows an instance of a puppy chewing a shoe;

* exposition, which refers to the re-expression or reformulation of the meanings of the image or the text in the alternative semiotic resource with both the written text and image representing the same level of generality e.g. when the word 'weighs' is reinterpreted visually as a balance scale (see Figure 1);

* equivalence, where there is ideational redundancy since the ideational content corresponds (completely or partially) across semiotic resources e.g. when a label or caption heading appears next to its image.

* homospatiality, as discussed by Lim (2004), which refers to texts where two different semiotic resources co-occur in one spatially bonded homogenous entity e.g. when the letters of the word 'seaweed' are created using fluid images of strands of seaweed. …

Search by... Author
Show... All Results Primary Sources Peer-reviewed


An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.