This article presents a generalized system of image–text relations which
applies to different genres of multimodal discourse in which images and texts
co-occur. It combines two kinds of relations – the relative status of
images and text, and how they relate to one another in terms of
logico–semantics. Every instance of an image–text combination in
the data sample is described by a selection of features from the system. The units
of images and text between which the relations obtain are identified and the
realizations of the logico–semantic and status relations are specified,
both for the human analyst and a machine. Two application scenarios are discussed.
The system should be useful for distinguishing between image–text
relations for (genuinely) new and old media.
Adams, S.
(1995) Always Postpone Meetings with Time-Wasting
Morons.
London: Boxtree
.
2.
Arnheim, R.
(1997) ‘The Two Sources of
Cognition’, in
T. Sebeok
and
J. Umiker-Sebeok
(eds) The Semiotic Web,pp.
253–259.
Berlin: de Gruyter
.
3.
Barthes, R.
(1997a[1961]) ‘The Photographic
Message’, in
R. Barthes
(ed.) Image–Music–Text,pp.
15–31.
London: Fontana
.
4.
Barthes, R.
(1997b[1964]) ‘Rhetoric of the
Image’, in
R. Barthes
(ed.) Image–Music–Text,pp.
32–51.
London: Fontana
.
5.
Bolter, J.D.
and
Grusin, R.
(2000) Remediation: Understanding New Media.
Cambridge, MA: MIT Press
.
6.
Cagan, J.
and
Vogel, C.M.
(2002) Creating Breakthrough Products.
Upper Saddle River, NJ: Prentice Hall
.
7.
Drucker, J.
(1994) The Visible Word: Experimental Typography and Modern
Art, 1909–1923.
Chicago: University of Chicago Press
.
8.
Halliday, M.A.K.
(1985) An Introduction to Functional Grammar.
London: Arnold
.
9.
Halliday, M.A.K.
(1994) An Introduction to Functional Grammar, 2nd edn.London: Arnold
.
10.
Halliday, M.A.K.
and
Hasan, R.
(1976) Cohesion in English.
London: Longman
.
11.
Hillenbrand, R.
(1999) Islamic Art and Architecture.
London: Thames & Hudson
.
12.
Horn, R.E.
(1998) Visual Language.
Bainbridge Island, WA: MacroVU
.
13.
Kress, G.
and
Van Leeuwen, T.
(1996) Reading Images.
London:Routledge
.
14.
Kress, G.
and
Van Leeuwen, T.
(2001) Multimodal Discourse.
London: Arnold
.
15.
Lemke, J.
(1990) Talking Science.
Norwood, NJ: Ablex
.
16.
Lemke, J.
(1998) ‘Multiplying Meaning: Visual and Verbal
Semiotics in Scientific Text’, in
J.R. Martin
and
R. Veel
(eds) Reading Science, pp. 87–113.
London: Routledge
.
17.
Lyons, J.
(1977) Semantics,Vol. 1.
Cambridge: Cambridge University Press
.
18.
Marsh, E.E.
and
White, M.D.
(2003)
‘A Taxonomy of Relationships between Images and Texts’
, Journal of Documentation59(6):
647–672
.
19.
Martin, J.R.
(1983) ‘Conjunction: The Logic of English
Text’, in
J.S. Petofi
and
E. Sozer
(eds) Micro and Macro Connexity of Texts, pp.
1–72.
Hamburg: Helmut Buske Verlag
.
20.
Martin, J.R.
(1992) English Text.
Amsterdam: Benjamins
.
21.
Martin, J.R.
(1994) ‘Macro-genres: The Ecology of the
Page’, Network 21, pp.
29–52, available online at <http://minerva.ling.mq.edu.au/Resources/Network/Network.html>
22.
Martinec, R.
(1998a)
‘Cohesion in Action’
, Semiotica120(1/2):
161–180
.
23.
Martinec, R.
(1998b)
‘Cohesion in Language and Action’
, paper presented at the Sociolinguistics Symposium 12,
Institute of Education, University of London, UK.
24.
Martinec, R.
(2003)
‘The Social Semiotics of Text and Image in Japanese and English
Software Manuals and Other Procedures’
, in T. van Leeuwen and C. Caldas-Coulthard (eds) Critical Social
Semiotics (Special Issue) 13(1):
43–69
.
25.
Martinec, R.
(ed.) (2005) ‘Topics in
Multimodality’, in
R. Hasan
,
J. Webster
and
C. Matthiessen
(eds) Continuing Discourse on Language, Vol. 1.
London: Equinox
.
26.
McCloud, S.
(1993) Understanding Comics.
New York: HarperCollins
.
27.
Miller, A.R.
and
Brown, J.M.
(1998) What Logos Do and How They Do It.
Gloucester, MA: Rockport Publishers
.
28.
Mitchell, W.J.T.
(1994) Picture Theory.
Chicago: Chicago University Press
.
29.
O’Donohoe, E. (2003) ‘Between Image and Text’,
MPhil thesis. School of Art and Design, Swansea Institute of Higher Education, UK.
O’Toole, M.
(1994) The Language of Displayed Art.
Leicester: Leicester University Press
.
32.
Royce, T.
(1998)
‘Synergy on the Page: Exploring Intersemiotic Complementarity in
Page-Based Multimodal Text’
, JASFL Occasional Papers1(1):
25–50
.
33.
Rush, M.
(1999) New Media in Late 20th-Century Art.
London: Thames & Hudson
.
34.
Salway, A.
and
Ahmad, K.
(1998)
‘Talking Pictures: Indexing and Representing Video with
Collateral Texts’
, in D. Hiemstra, F. de Jong and K. Netter (eds) Proceedings of the
14th Twente Workshop on Language Technology – Language Technology for
Multimedia Information Retrieval,pp.
85–94
. ISSN: 0929–0672.
35.
Salway, A.
,
Graham, M.
,
Tomadaki, E.
and
Xu, Y.
(2003) ‘Linking Video and Text via
Representation of Narrative’, in AAAI Spring Symposium
on Intelligent Multimedia Knowledge Management, Palo Alto, 24–6
March, pp. 104–112. ISBN 1–57735–190–8.
36.
Trigg, R. (1983) ‘A Network-Based Approach to Text Handling for the
Online Scientific Community’, PhD dissertation, University of Maryland,
USA. <http://www.Workpractice.com/trigg>
37.
Van Leeuwen, T.
(1991)
‘Conjunctive Structure in Documentary Film and Television’
, Continuum5(1):
76–114
.
38.
Van Leeuwen, T.
(2005) ‘Rank in the Analysis of
Images’, in
R. Martinec
(ed.) ‘Topics in Multimodality’, in
R. Hasan, J. Webster
and
C. Matthiessen
(eds) Continuing Discourse on Language,Vol. 1.
London: Equinox
.
