Multimodal literacy (Jewitt & Kress, 2003) is about understanding the different ways of knowledge representations and meaning-making, such as on advertisements, posters, web-pages and films.
Multimodal literacy focuses on the design of discourse by investigating the contributions of specific semiotic resources, (e.g. language, gesture, images) co-deployed across various modalities (e.g. visual, aural, somatic), as well as their interaction and integration in constructing a coherent multimodal text (such as advertisements, posters, news report, websites, films).
The multimodal approach takes into account how linguistic and visual (and other) choices fulfill the purposes of the text, the audience and context, and how those choices work together in the organisation and development of information and ideas.
The Systemic Approach pedagogy for the teaching of multimodal texts helps students develop critical thinking and discourse analysis skills.
This pedagogical approach stems from Michael Halliday’s Systemic Functional Theory. The Systemic Approach is an extension to the teaching and learning of media messages, and is developed jointly by researchers from the Multimodal Analysis Lab, National University of Singapore and specialists from Educational Technology Division, Ministry of Education. It recognises that meanings in texts are a result of specific choices made within specific systems. For instance, the system of PROMINENCE in an image is realised by choices made in size, sharpness of focus, colour contrast, lighting and foreground or background techniques.
The Systemic Approach advocates the provision of a set of vocabulary (or meta-language) for teachers and students to describe the choices made in the multimodal text. This supports students’ reading and interpretations of the text by relating them to textual evidence, as a result of the system choices. This serves to develop students’ critical reasoning faculty and discourse analysis skills as they are able to explain the choices made in the text and present an argument for their interpretation. The students’ perspective, based on his or her analysis of the text, is empowered where traditionally the teacher’s perspective was privileged.
Information, particularly in the digital age, is represented not just with language. Instead, language is often nestled amongst a range of semiotic resources in a multimodal text.
O’Halloran, Tan, Smith & Podlasov (2010: 4) explain that “we believe that, in practice, texts of all kinds are always multimodal, making use of, and combining, the resources of diverse semiotic systems to facilitate both generic (i.e., standardized) and specific (i.e., individualized, and even innovative) ways of making meaning”.
Technology accentuates the multimodal nature of text, by facilitating the production and consumption of multimodal text. It also provides the analytical tools to support the annotation and visualisation of the meanings made in multimodal texts.
Educators who understand the multimodal ways in which knowledge is presented, may teach students to access, appraise and appropriate the multimodal texts which they will inevitably encounter.
Current research in multimodal analysis establishes the need and provides the meta-language to develop multimodal literacy in education (see for example readings in the Resources Section).
O’Halloran and Lim (2011) envision that “a ‘multimodal literate’ student must be sensitised to the meaning potential and choices afforded in the production of the text, rendering an enhanced ability to make deliberate and effective choices in the construction and presentation of knowledge.”
Multimodal literacy aims to develop students into discerning readers and savvy producers of multimodal texts by drawing attention to the various meaning-making resources in the texts, and the ways in which specific choices work together to achieve the desired communicative goals.
Since all disciplinary knowledge is constructed using multiple resources which extend beyond language, the multimodal approach may be applied in all subjects, such as the study of English, social sciences, mathematics and physical sciences.
The Systemic Approach pedagogy to the teaching of multimodal texts involves students learning the different system choices available in language and images. This is done through the introduction of the system choices available to the producer of the text via system networks.
Students are taught to identify these choices in the selected multimodal text and to explain how these choices lead to certain meanings being made. This can be facilitated to varying degrees by annotating the text with the technology tools listed. More advanced tools, such as the Multimodal Analysis Image/Video, also provide options for visualisation of the analysis.
Students are encouraged to support their interpretations of the multimodal texts by citing textual evidences from their analysis. This is achieved through worksheets designed with questions to lead students from textual description to textual analysis.
Finally, students are given opportunities to articulate their views and discuss the ideologies in the multimodal text by identifying the strategies used to appeal to the reader. Students may be invited to present their analysis and defend their interpretation of the text to the class.
Analysis of Advertisements
Evidence of learning is shown when students are able to….
- Appropriate the meta-language to describe the different components in an advertisements
- Identify the meanings made in the advertisement through the system choices made
- Critically discuss the messages and ideologies in the advertisement and support with textual evidences
- Teacher introduces the System Choices in Advertisements through a PowerPoint presentation.
- Students work individually to annotate an advertisement. This can be done via the technologies tools. Students are to identify the system choices in the advertisement and relate them to the meanings made.
- Students work in pair to discuss their interpretations of the advertisement. They also complete a worksheet, with structured questions to guide their analysis (See next page).
- Selected students may present their analysis to the class and explain their interpretations of the messages in the advertisement.
- Teacher facilitates a discussion on the meanings made in the advertisements and encourages students to examine the messages and ideologies espoused in the advertisement critically.
Pedagogical and Assessment Considerations
- This activity aims to help students to identify and analyse the meanings in an advertisement.
- Student should appropriate the meta-language to describe the components in an advertisement and understand how these choices realise specific meanings.
- Students should be able to explain their interpretation of the advertisement by citing textual evidences as support.
- Students should identify the messages and ideologies in an advertisement and examine them critically.
- Students are equipped with the theoretical tool-kit – the meta-language, to ask questions about the representations made in the advertisement.
For higher ability students, teacher may provide less scaffolding and facilitate their analysis of more complex texts. They could also be introduced to a wider range of system choices in an advertisement.
 Halliday, M.A.K., and C.M.I.M. Matthiessen (2004). An introduction to functional grammar, 3d ed. London, Arnold
 O’Halloran, K. L., Tan, S., Smith, B. A., & Podlasov, A. (2010). Challenges in designing digital interfaces for the study of multimodal phenomena. Information Design Journal, 18(1), 2-12.
 O’Halloran, K. L. & Lim, F. V. (2011). Dimensioner af Multimodal Literacy. Viden om Læsning. Number 10, September 2011, pp. 14-21. Nationalt Videncenter for Laesning: Denmark.