5diretta da Leonardo Terzo

Collana di sintomatologiadelle apocalissi culturali

10

Nella stessa collana

1. L. Terzo, Pornografia ed episteme. Per una sintomatologia delle apocalis-si culturali.

2. B. Berri, Nathaniel Hawthorne. Dal subliminale al trascendentale.3. S. Monti, Le vicissitudini della corporeità. Anima e anatomia nella nar-

rativa inglese e americana dell’Ottocento.4. L. Terzo, Sublimità contemporanee.5. B. Berri (a cura di), Saggi italiani su Elizabeth Bowen. (Saggi di E.

Cotta Ramusino, S. Granata, S. Monti, L. Terzo, C. Marelli, B. Berri,L. Guerra, J. Meddemmen).

6. Cristina Marelli, The Survival of Literature. Remake Practices fromShakespeare to the Graphic Novel. William Shakespeare, MichaelHoffman, Ingmar Bergman, Woody Allen, Art Spiegelman, MarjaneSatrapi.

7. L. Terzo (a cura di), Assurdo, paradosso, follia. Samuel Beckett, OscarWilde, William Shakespeare. (Saggi di B. Berri, S. Monti, L. Terzo, E.Zuccato).

8. L. Terzo (ed.), Lunatic Giants, (Essays by B. Berri, F. Ceravolo, C.Marelli, S. Monti, L. Terzo, C. Viola).

9. L. Terzo (ed.), La civetta di Venere, (Essays by P. Nerozzi, Hic SuntGroup, L. P. Ellis, L. Goldheim, V. Tavazzani, L. Terzo).

AUGMENTED LINGUISTICS

Mara Logaldo

Language and Communication

in the Age of Augmented Reality

Edizione a cura diArcipelago EdizioniVia Pergolesi, 12

20090 Trezzano sul Naviglio (Milano)info@arcipelagoedizioni.com

www.facebook.com/ArcipelagoEdizioni

Prima edizione novembre 2012

ISBN 978-88-7695-480-1

Tutti i diritti riservati

Ristampe:7 6 5 4 3 2 1 02018 2017 2016 2015 2014 2013 2012

è vietata la riproduzione, anche parziale, con qualsiasi mezzo effettuata, com-presa la fotocopia, anche ad uso interno o didattico, non autorizzata.

TABLE OF CONTENTS

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1. SEMANTIC COMPLEXITY IN AR1.1. Partial overlapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271.2. Mixed reality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311.3. Semantic continuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351.4. Reaching beyond: metaphoric connections . . . . . . . . . . . . 391.5. The Semantic Web Layer Cake on and off line . . . . . . . . . 44

2. SUPERIMPOSITION2.1. You are a Layar®! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 512.2. A brief history of superimposition . . . . . . . . . . . . . . . . . . . 552.3 An aesthetics of concealment . . . . . . . . . . . . . . . . . . . . . . 662.4. Interference patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 682.5. Word and image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 722.6. Verbal and visual superimposition . . . . . . . . . . . . . . . . . . . 752.7. Superimposition in cognition and language . . . . . . . . . . . . 79

2.7.1. Cognitive blending and metaphor . . . . . . . . . . . . . . 812.7.2. Foregrounding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

2.8. Superimposition in AR: encrustation vs. integration . . . . . 882.9. Logical categories in mixed environments . . . . . . . . . . . . . 93

3. THE WORLD IN A TAG3.1. The map is not the territory . . . . . . . . . . . . . . . . . . . . . . . . 973.2. The end of photography? . . . . . . . . . . . . . . . . . . . . . . . . . . 1033.3. Framing and browsing the real . . . . . . . . . . . . . . . . . . . . . 1053.4. Tagging the world . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1143.5. Verbal tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1173.6. Tag clouds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1213.7. QR tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1293.8. AR tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

3.9. Secret language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1383.10. RFID tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1403.11. Mobile tagging and ostensive communication . . . . . . . . . . 142

4. BETWEEN TAG AND SCREEN4.1. Augmented art . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1474.2. Augmented literature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1524.3. Augmented advertising . . . . . . . . . . . . . . . . . . . . . . . . . . . 1584.4. Augmented cinema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1684.5. Augmented bodies, augmented fashion . . . . . . . . . . . . . . . 169

5. AR IN EDUTAINMENT, INFORMATION, AND AUGMENTED ALTERNATIVE COMMUNICATION (AAC)5.1. Relevance theory and AR:

towards a redefinition of context . . . . . . . . . . . . . . . . . . . . 1875.2. Affordance theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1895.3. Locative narratives and locative games . . . . . . . . . . . . . . . 1925.4. Augmented learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1975.5. Augmented journalism . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2025.6. Making reality speak:

AR for the language and visually impaired . . . . . . . . . . . . 208

6. TOWARDS AN ‘AUGMENTED’ LINGUISTICS6.1. Testing perceptive limits: how far can we go? . . . . . . . . . . 2136.2. In praise of noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2206.3. Conversational noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2256.4. An ‘augmented’ idea of felicity conditions . . . . . . . . . . . . . 227

6.4.1. Quantity and quality . . . . . . . . . . . . . . . . . . . . . . . . 2296.4.2. Relation and manner . . . . . . . . . . . . . . . . . . . . . . . . 232

6.5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234

AFTERWORD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239

– 7 –

INTRODUCTION

The title of this work, Augmented Linguistics, was originallyconceived to suggest, admittedly in a rather provocative andironic way, the necessity of a general re-thinking of the funda-mental assumptions of linguistics in the era of augmented real-ity (AR). Although I felt rather pleased with the choice of thisexpression to synthesize the scope of my study, I lay no claimsover the invention of the phrase itself: indeed, there were fewdoubts about its being a neologism which, like many similarones, exploited the adjective ‘augmented’ that has become sofashionable since the spreading of AR to indicate the applica-tion of this technology to different fields. Also in the associationof the adjective ‘augmented’ with the noun ‘linguistics’ thereseemed to be nothing particularly creative: it could be as pre-dictable as that between ‘augmented’ and ‘media,’ or between‘augmented’ and ‘art.’ However, when I typed the phrase in theGoogle search box to check – as was most likely – if somebodyelse had already used this combination of words and exploredthe subject, much to my puzzlement I noticed that, while therewere thousands of webpages containing the expressions ‘aug-mented media’ and ‘augmented art,’ only two of them men-tioned the expression ‘augmented linguistics.

“As stargazer’s avatar may say: an exercise in augmented lin-guistics it was.”1

“The call for augmented linguistics is growing.”2

1 <www.fansofrealitytv.com>2 <www.alternet.org>

Augmented Linguistics

– 8 –

Neither website used the phrase with the particular meaning Iintended to give to it. However prophetic, the two excerptsmainly signalled a lack of studies in the field. The former, dated2003, was a comment made by the fan of a reality TV pro-gramme during a discussion forum: the “exercise in augmentedlinguistics” consisted in the transcription of a dialogue betweentwo participants in the show, with particular reference to thewords spoken by one called Amy. From my point of view theonly interesting aspect of the remark was the connectionbetween the qualifier ‘augmented’ and the idea of difficulty, thereference to something so complex that it verged on the cryptic.In fact, this idea will often recur in this work: indeed, one of thepeculiar characteristics of AR is the systematic search forextreme stimuli, whose challenging quality, and the ways peo-ple respond to them, constitute one of its distinctive traits. Thistrait but emphasises the use of language in the era of Web 2.0.Suffice it to think about the encryption of information in the QRcodes that have been burgeoning in the past few years onlabels, billboards, and magazine covers; or the jagged alterna-tion of fonts of different sizes and intensity of colour in tagclouds, where unpopular items may become so faded to bealmost invisible to the untrained eye. The second quotation was clearly more focused on linguistic

problems. But, if we read the linked page, an article entitled“Set Your Speling Free,” we discover that by “augmented lin-guistics” the author meant only the wished for adoption of asimplified spelling (“speling”) system that would – according tohim - better mirror the way English is actually pronounced,and, by shortening several words (for instance, “yacht” into“yot”), would also make the language fitter for the restrictedwriting space of mobile phones and keypads. This aspect,already widely explored by linguists with reference to differentforms of display-based texting (Crystal, 2008), could also be ofsome relevance in a study on AR, space being a fundamentalcategory of augmented reality. The strategic arrangement inspace of virtual objects or texts is actually necessary to the very

INTRODUCTION

– 9 –

existence of AR. The multimodal notion of framing (Kress –van Leeuwen, 1996; van Leeuwen, 2005: 7-19; 277) can cer-tainly be applied also to AR, since it is the result of a processcarried out by framing parts of the world within the display ofhandheld devices and augmenting them with the digital infor-mation provided by the online connection. On the other hand,we know that AR technologies tend to solve the problem ofspace not by looking for new possible ways of cramming writingand pictures into tiny monitors, but rather by studying how toget rid of displays altogether. Although for the time being wehave to rely on smartphones and other analogous devices, theultimate goal of AR technologies is turning the whole world intoa 3D screen that may receive and ‘englobe’ the computer-gen-erated imagery or text ubiquitously (Boulter - Grusin, 1999:216-9; Hainich, 2009 [2006]: 17).3

While augmented linguistics seems to be, at least so far, arather unexplored land, Augmented Communication appears asa more popular subject, also among linguists. For more thanhalf a century - the first experiments can be dated back to the1950s – the expression Augmentative and AlternativeCommunication (AAC) has been commonly used for all the

3 However, by “ubiquitous computing” Bolter and Grusin mainly meanthe placing of several computer screens in strategic places rather than, forinstance, the use of near-eye projectors, the exploitation of atmosphere par-ticles for the projection of 3D imagery or the superimposition of computer-generated messages over the real. The discussion requires a reflection uponthe differences between virtual and augmented reality (see chapter 1). Inorder to achieve this, however relative, autonomy from screens – includingthose of smart phones – new devices placed closer to the retina are beingexperimented. As Hainich points out: “This technology, getting as near tothe personal communicator as anything so far, will surely hit the wall in thenear future, simply because there is no acceptable display, and rollout dis-plays or projectors won’t do. Long ago I realized this problem, and now tech-nology is ready to solve it. We need to eliminate the screen in favor of anear-eye projector […]. Virtual objects, virtual devices will surround us,everywhere.” (Hainich, 2009 [2006]: 17).

Augmented Linguistics

– 10 –

devices that make communication easier for people with speechand language impairments due either to congenital causes oracquired conditions. Also AR technologies can be used in thisfield, particularly to provide sensory substitutions, such asvisual cues to the deaf or aural information to the blind. Giventhe importance of AR in AAC, some of its applications will beinvestigated in a chapter of this book (§. 5.6.). On the other hand, Augmented Communication considered

as the result of the use of augmented reality devices as such,that is without specific remedial purpose, has not been thor-oughly explored yet. It has certainly been recognized as a newtrend in communication which encompasses all media and dif-ferent contexts, but only since the past year the interest hasgone beyond the study of the marketing possibilities of this newtechnology. Yet, the subject has been pointed at as a compellingone for speculative study, particularly by cognitive linguists. Inthe closing paragraph of “Augmented communication: Thecommunicative potential of the internet”, for instance, theauthor argues that “the access to the internet will one day leadto easier communication online in virtually augmented reality(and later in ‘pure’ virtual reality) than face to face in ‘real’ real-ity, through internet-mediated extensions of the mutual cogni-tive environment” (Philip Diderichsen, 2006). Other studiesare following, but they are still at an embryonic stage.4

The necessity to study how augmented reality is changingour way of viewing language and communication has been thestarting point for this book. Although I do not in the least con-sider it an exhaustive work on the linguistic aspects connectedwith AR − there are far too many facets to be taken into accountand what we are witnessing at present is only the beginning − I

4 The Centre for Research in Media and Cultural Studies of theUniversity of Sunderland has recently launched a call for papers for a pub-lication focused on the influence of Augmented and Mixed Reality in art,history, literature and media studies. The publication, edited by Jay Bolterand Maria Engberg, is due for the beginning of 2014.

wish it to encompass a wide range of problems connected withthe forms of communication and textual organization that aregradually being introduced by AR technologies. One point I would like to make clear is that this is not just

one more book on the language of the Web. Research onInternet-based forms of communication – from websites toemails, from blogs to chatrooms of several kinds - have been(and are still being) excellently carried out by linguists whoconstantly monitor the state of the art of the ever-evolving formsof communication of the Net (Aitchinson, 2001; Crystal, 2001,2008, 2011; Herring, 1996, 2007; Posteguillo, 2003). BeingAR an online form of communication, this work will inevitablytouch on some of these aspects (such as the use of tag cloudsboth in webpages and in AR applications, or the employ of Web2.0 in social networks and in experiments of hybrid narrativedesigns) but it intends to concentrate chiefly on those comput-er-mediated texts that are associated with the textual features ofaugmented reality as distinguished from virtual reality. Theseare, for instance, mobile computing and browsing and the infor-mation made available by augmented reality appliances onhandheld devices (i-Phones, Android smartphones, etc.), tagtexts, which are common both to online communication and toAR mobile applications, the use of QR tags in magazines andadvertising, but also, significantly, in art and storytelling, theuse of AR tags to visualise digital information in different kindsof environments. The principal change that is being produced by the new

technology is that thanks to the multiplication of computerinterfaces the world and the web are increasingly overlapping.However, this does not necessarily involve the virtual in timeentirely substituting the real. The idea at the core of AR is thatthe real and the virtual are not mutually excluding dimensions.“Augmented Life” is neither “Second Life” nor “Virtual Life.”On the one hand Internet communication can no longer remainimmersed in the underworld depths of the computer screen,

INTRODUCTION

– 11 –

Augmented Linguistics

– 12 –

where it ‘naturally’ belongs. On the other hand this does notentail the future of digitalization leading to the total substitu-tion of the real world with a virtual one, either in the claustro-phobic rooms of MUDs (multi-user dungeons) or in similarlyconstraining simulations taking place ‘outdoors’ but made pos-sible thanks to the use of special devices that exclude the sub-ject from the surrounding context (Poster, 2001). The digitali-zation of data and imagery will not automatically lead to a futur-istic dimension in which people wear special goggles and fum-ble in a virtual space without any coordinates and connectionswith everyday life. People have often demonstrated that theirnature is amphibious. Generally speaking (there are of coursesome exceptions) we do not like to consider ourselves either aslife-prisoners, although under virtual identities, or wired pup-pets. We do have aspirations to improve (read ‘augment’) ourlives, but we also love our real selves and our real context. It isvery unlikely that we will choose between opposites. We shallmore probably opt for intermediate states, being attracted byopposite poles and different experiences at the same time. It is undeniable that with AR the Internet ‘jumps’ out of the

screen to mix up with the real world: tagging and geotagging aredoing precisely this. However, even if it were possible to tagevery square metre on earth and all the objects it contains, thiswould not inevitably involve a perfect fusion of the real and thevirtual. A necessary condition for the very existence of AR is anonly partial overlapping of the two dimensions. ParaphrasingCleopatra’s words in Shakespeare’s tragedy, like Antonio’sdelights, AR applications are “dolphin-like”, they show theirbacks “above the element they live in.” 5 Which is the WorldWide Web. With the aid of AR appliances the Internet can eas-ily emerge on this side of the computer interface and, to someextent, be superimposed on the real world. The terms usuallyassociated with AR themselves, such as for instance ‘Supranet,’

5 W. Shakespeare, Antony and Cleopatra, Act 5, Scene II, 79.

INTRODUCTION

– 13 –

entail the coming to the surface of Internet applications, overthe screen and into the world. This extends the mode that char-acterises most social networks, significantly named ‘Brightkite’and ‘Twitter,’ suggesting that the forms of communicativeexchange – and we with them – should raise above the Weband, birdlike, fly with the wind. The immense ether, with itsconnotations of freedom and infinity, will hardly lose its myste-rious essence, however numerous the devices and strings oftexts that inhabit it. Curiously, a fascinating application of ARmade possible by Google Sky Map and Android operating sys-tem, is that of learning the name of stars and planets by point-ing the smartphones in their direction (Furth, 2011: 12).AR demonstrates that we are not looking for totally immer-

sive experiences on either side of the computer interface(Milgram and Kishino, 1994). With reference to Boulter andGrusin’s milestone book Remediation, we could say that sinceits publication in 1999 the contemporary subject has increas-ingly left behind the twin logics based either on a “desire forimmediacy” (Boulter – Grusin, 1999: 9) and transparency or onthe awareness of an all-pervasive and stratified hypermedia-tion. The search for experiences that are uncompromisinglyimmersive has been abandoned in favour of incomplete illu-sions. As Erkki Huhtamo foresaw, “Technology is graduallybecoming a second nature, a territory both external and inter-nalized, and an object of desire. There is no need to make ittransparent any longer, simply because it is not felt to be incontradiction to the ‘authenticity’ of experience.” (Huthamo,1995: 171; see also Boulter – Grusin, 1999: 42). This does nottotally obliterate the difference between the real and the virtu-al but definitely adds a sense of continuity between the twodimensions: their relationship is perceived as metonymic ratherthan as merely metaphoric and substitutive.As the expression AR itself suggests, the emphasis is on the

word ‘reality.’ A new kind of realism, hybrid and hypermediat-ed, is steadily spreading. In an AR perspective, the Web willstill retain its enormous importance, but principally as a reso-

Augmented Linguistics

– 14 –

nance box of what is taking place in the augmented ecosystem.Once again, my thought goes to tag clouds, which visualizemixed occurrences taking place on both sides of the screen.The Web will most probably increasingly become the storingplace of augmented experience, the place where individualmixed existences will be shared. On this account, the most suit-able example of an early application of social networks asrepositories is the creation of sites that try to weave narrativedesigns from individual contributions (photos, writings, videos)perceived as belonging to hybrid ecosystems (Pata, 2010).From this perspective the Web can be considered as the placewhich, at once, triggers, stores, and implements data and emo-tive responses. But although ‘augmented realism’ is bound to become the

episteme (Foucault, 1966) of our time, it would be incorrect tothink that the shift to the new framework will be easy andundisruptive. As history teaches us, the passage from one ideaof realism to another always entails a series of problems. In thiscase, first of all cognitive problems related to visual perception.These are mainly caused by the superimposition of real and vir-

tual objects. Let us take forexample a tag superimposed ona building: which is in the fore-ground? The virtual tag or thereal building? Which is in thebackground? Is the virtual tagembedded in the real buildingor is the real building embed-ded in the virtual dimension ofthe tag (fig. 1)?Fig. 1. Source: <http://futuremediachange.com/

2009/12/augmented-reality-for-non-profits/>

According to Hainich, from a merely technical perspective theproblem may be solved in two opposite ways: either by making the

INTRODUCTION

– 15 –

superimposed device transparent so that the underlying realobject may be seen through; or by overlaying a picture of the realobject over the picture of the virtual object (Hainich 2009 [2006]:47).6 However, Hainich’s starting point is the search for a seam-less integration of virtual objects into everyday life (Hainich 2009[2006] :27). This, according to me, is not the point with popularuses of AR. We are not looking for a virtual world that looks real.We just do not care about noticing the “seams and joints,”(Hainich 2009 [2006] :17) probably because, as we have seen,they have already become ontologically permeable categories. Itwould rather be more interesting to highlight the different para-digm engendered by the superimposition of the two environments,the very possibility of acquiring a sort of stereoscopic vision.7 Thisstereoscopic vision does not have to lead to a perfectly consistent3-D perception of a world which incorporates the virtual intoitself. Actually, AR experts hardly talk about ‘stereoscopic vision’but, rather, about ‘augmented vision’ (Behringer et al, 1999). Thefunction of augmented vision, one may deduce, is primarily thatof making simultaneously sense of the several layers of whichaugmented reality is composed. Some of these layers are real,some are computer-mediated: what is important to find out iswhether, being co-present, they enhance one another rather thanobstruct one another, whether communication proceeds from layerto layer in a way which is “smooth, without catches or serration.”(Memmot, 2000) and how this process may meet felicity condi-tions (Austin, 1962; Searle, 1969).Hence the starting point of this book is augmented reality as

distinguished from both the real environment and the virtual

6 Hainich describes these solutions as follows: “1) to cut out the shapeof the real object from the virtual object and the mask display, in order touncover it; 2) to overlay a picture of the real object, as taken from the posi-tion sensor cameras, over the picture of the virtual object.” (Hainich, 2009[2006]: 47).

7 Stereograms seem to provide this possibility of seeing superimposedobjects at once, thus acquiring a 3-D perception of the image.

Augmented Linguistics

– 16 –

environment. Indeed, experiments in AR have blurred a clear-cut distinction between the real and the virtual world, but theyhave in no way substituted the former with the latter. The com-puter screen has become a portable, thin, sometimes almostimperceptible membrane allowing a two-way passage of mean-ing (Hayles, 2005) in a complex, or hybrid, ecosystem, but hasnot, at least so far, become totally transparent or all-embracing.Urban settings, in particular, present a jigsaw puzzle of screensin which information seems to flow freely, according to a com-plex system of signs which incorporates the reflection of theonlooker and interacts with it.8

The above remarks seem to concentrate on the visualaspects connected with AR. But, as always happens with visu-ally-based media, all problems connected with the act of seeingare also bound to turn into linguistic problems. For every see-ing act is, inevitably, also a model of textualization by which wemake sense of the world. The blurring of the boundary between the real and the virtu-

al environment that I have tried to highlight so far also involvesthe dimension of language. As David Crystal foresaw, the clear-cut distinction between “sign language” and “computer-medi-ated language” can no longer be held: “[…] the Net is only apart of computer-mediated language. Many new technologiesare anticipated, which will integrate the Internet with othercommunication situations, and these will provide the matrixwithin which further language varieties will develop” (Crystal,2001: 225). The conception of the world as a hybrid ecosystemin which the virtual and the real co-exist and often mix can nolonger limit the field of linguistic studies to one dimension of

8 This aspect was discussed by Mauro Carbone (Université Jean MoulinLyon3) in a paper entitled “The dream of living in a screen: from moviescreens to screen-facades. Reflections starting from Forever 21 TimesSquare giant interactive billboard” presented during the international con-ference on Media City: new spaces new aesthetics organized by the Triennaledi Milano (7-9 June 2012).

communication focusing, alternatively, on spontaneous speech,on media discourse or on computer-mediated texts. It has tofocus on mixed languages in a mixed reality, even contemplat-ing ideas that were until a little time ago inconceivable.

If you had said to me, a few years ago, that it was possible to havea successful conversation while disregarding the standard con-ventions of turn-taking, logical sequence, time ordering, and thelike, I would have been totally dismissive. But the evidence isclear: millions are doing just that (Crystal, 2001: 170).

Crystal is here referring to synchronous and a-synchronousforms of verbal exchange that we may find in chatrooms. Now,ten years later, these ‘impossibilities’ have multiplied. Mixedrealities, intelligible overlapping messages, enriching noise arejust a few of the oxymoronic expressions and aporias (Derrida,1996) underlying the world of language-based communicationin our brave augmented world.The experience of AR is so complex that not even a multi-

modal approach (Kress – van Leeuwen, 2001) can thoroughlypay justice to its intricacy. For multimodal linguistics analysesthe interaction between different forms of communication −both visual and aural ones, pictures, videos, written and spokentexts, in media such as printed magazines, cinema andtelevision, the Internet − but not the short-circuit deriving fromthe superimposition of the modes which concur to create virtu-al multimodal texts to the modes which concur to create ‘real’multimodal texts (in the semiotic sense, that is including bothmediated texts and objects belonging to the experiential worldalike). On the other hand I think that, in order to analyse theinteraction between virtual and real layers, it can be useful toborrow from the multimodal approach some of the categories itutilized to study the relationship between word and image. Forinstance, the overlap which, in advertising “is likely to occurbetween the fantasy envisaged in the picture and the realitygiven by the text and the picture(s) of the product” (van

INTRODUCTION

– 17 –

Leeuwen, 2005: 12) can be translated in AR in a reflectionupon the overlap occurring between the virtual dimension ofthe tag and the real world. More generally, the notions ofanchorage and relay, and the notions of permeability, porosity,and overlap (Barthes, 1977: 40 ff.; van Leeuwen, 2005: 229),which in multimodal linguistics are often adopted in the studyof the interaction between the text and the image, will oftenrecur in this text with reference to AR.A fundamental point in common with the multimodal

approach is the belief that the distinction between text andvisual image can no longer be held. This is particularly evidentin the use of typographic signs (van Leeuwen, 2005: 27-9). Tagclouds, for instance, are pictures and texts at the same time (see§ 3.6.). The complexity in the relationship between the wordand the image can be seen in any kind of AR application. In theuse of handheld devices, for instance, the interaction betweenthe information framed by the display and the surrounding – orembedded - real world is also problematic from a linguistic per-spective. As Rodowick finely argued in his essay Reading theFigural or, Philosophy after the New Media (2001) - it is thevery status of the image in relation to discourse which has beenquestioned by the advent of the digital era. The separationbetween the temporal dimension of language and the spatialdimension of the image can no longer be unambiguous: “[…] inthe era of electronic and digital communication, the figural isincreasingly defined as a semiotic regime where the world ofthings is penetrated by discourse.” And, recalling Foucault’sreading of the figural as similitude: “Here the figural disturbsthe collateral relation that defines figure and text into two sep-arate streams, one characterized by simultaneity (repetition-resonance), the other by succession (difference-affirmation)”(Rodowick, 2001: xii). As a result, he concludes “text was spa-tialized, thus losing its uniform contours, fixed spacing, andlinear sense, and, at the same time, “space was textualized”(Rodowick, 2001: 3).

Augmented Linguistics

– 18 –

This anticipates a possible definition of AR: a spatializationof text or a textualization of space. This idea, of course, is notnew: since the advent of semiotics, space has always been treat-ed as a text (Sebeok, 1997a). Nonetheless AR shakes theground of this distinction even more violently and makes thehybridization even more cogent, because it does not identify orsubstitute text and space, it has them partially overlap. Whatactually happens is that mediated discourse penetrates theobjects that surround us. And, I would add, it is not importantwhether these objects are virtual or real, because discourseinfiltration works in either case, thanks to a technological han-dling which involves images and text strings alike.All the issues rather clumsily – I am afraid – illustrated in

this introduction will be extensively dealt with, and in a moreorderly fashion, in the ensuing chapters of this book. The first chapter will set the framework to the study of lan-

guage in augmented reality. It will introduce the fundamentalconcept of AR: the existence of a multi-layered discourse andits way of affecting the world of communication. It will focus oncontext as a hybrid ecosystem which blends real and virtualobjects, paying particular attention to the problematic notion ofsemantic continuity between digital contents and the actualworld. Chapter two will concentrate on the historical, aesthetic and

cognitive aspects involved by superimposition. This issue willbe first explored with reference to the visual aspects, then inrelation to the overlap of verbal and visual layers and, finally,within language itself, proposing an overview on the common-est linguistic phenomena that imply the idea of superimposi-tion, such as cognitive blending, metaphor and foregrounding.Chapter three investigates the process by which the world

and the web can be made isomorphous, a condition which isnecessary to the very existence of AR. After a survey on thenotions of representation, framing and browsing, it will focus onthe meanings of ‘tag’ and some of the most relevant applications

INTRODUCTION

– 19 –

of ‘tagging.’ Particular emphasis will be given to the tag con-ceived as a kind of sign that, on the one hand, strongly shakesthe Saussurean idea of the arbitrariness of the signifier in rela-tion to the signified (Saussure, 1916), while, on the other hand,it confirms Peirce’s position which stresses the crucial role ofthe interpretant (Peirce, 1934). The linguistic and communica-tive implications of the tagging process will be mainly observedin the kind of text which can best visualize them: the tag cloud.Chapter four explores both the aesthetic and the practical

changes brought about by AR in the field of the media, art, lit-erature, and fashion. In art, a blatant example is the sessionrecently dedicated to AR art at the MoMA in New York. In lit-erature, it will be interesting to analyse first the hypertextswhich seem to forerun the modes of AR by showing an exten-sive superimposition of texts and images, such as Memmot’sFrom Lexia to Perplexia (2000), and then those that overtlyapply AR technologies to the literary text, as in the experimentBetween Page and Screen recently carried out at the OtisCollege of Art and Design in Los Angeles.Chapter five will re-propose the notions of relevance and

affordance by applying them to mixed environments, givingexamples of the related possible applications of AR in edutain-ment, information and augmentative alternative communication(AAC), particularly to provide sensory substitution to thevisually impaired. Chapter six will revisit some of the main theories of linguis-

tics in augmented reality environments. An important assump-tion that needs to be re-examined is the notion of ‘noise,’ par-ticularly in view of the positive value attached to interference,which in AR is no longer seen as an infringement to successfulcommunication but as an augmentation of meaning, as addedvalue. Finally, starting from Grice’s maxims of quantity, quali-ty, relevance and manner (Grice, 1975), I will concentrate onthe changes provoked by AR vis-à-vis the pragmatic aspects ofcommunication.

Augmented Linguistics

– 20 –

My attempt cannot but be cross-disciplinary. This meansthat, at worst, my way of proceeding will be far from systemat-ic, and that, at best, it will open up some new perspectives forfurther analysis.

INTRODUCTION

– 21 –

AUGMENTED LINGUISTICS

in the Age of Augmented RealityLanguage and Communication

“Yet is not language itself taking on new forms in the eraof virtual reality and cyberspace? Can one rest with a self-styled deconstructive stance when the material infrastruc-ture of the sign is being so drastically reconfigured?”(Poster, 2001: 125-126).

“[…] it is at least possible that new technologies, increas-ingly ubiquitous, multi-purpose and ‘natural’ in terms oftheir interfaces, will help create a fourth dimension ofcommunication in the same way that writing created athird” (Kress – van Leeuwen, 2001: 11).

“[…] programmable media provide arbitrarily numerousmeans to realize, in program and performance, complexrelationships between the symbolic realm of language andthe world it dwells within, represents and constitutes. Toachieve this we require a textuality of complex surfaces,capable of conveying a multi-dimensionality that is com-mensurate with lived human experience, including thestructured culture of human time” (Cayley, 2005).

– 27 –

1.

SEMANTIC COMPLEXITY IN AR

1.1. Partial overlapping

“It is like Internet, but it is outside the Internet.”(Sterling, 2009: 139)

“… a distinction between clear-cut unambigu-ous boundaries and fuzzy boundaries. But theterms I have used differ – ‘overlap’ in the onecase, and ‘permeability’ in the other. Clearlyanother, slightly more general term is needed, aterm which can encompass the many ways […]in which boundaries can blur and categoriesoverlap.” (van Leeuwen, 2005: 19)

According to researchers who have concentrated on the possi-bilities of AR, it is difficult to support the idea that computer-generated data and imagery will forever be perceived as some-thing that “stands out” of the real context or is simply superim-posed on it. For the time being, for instance, holograms are justfascinating intruders, but they will probably not be recognizedas such in the long run, nor will they exhaust all the potential ofthe medium (Hainich, 2006). One possible question is whetherwe shall get accustomed to augmented reality as we have got

Augmented Linguistics

– 28 –

accustomed to our television sets, whether virtual objects willsoon simply become part of the furniture. Maybe they will, evenwith a vengeance. While the distinction between the real worldrepresented, say, by the room we are in and the images flicker-ing on the television screen can still be felt, even after more thanhalf a century of daily fruition (Fiske, 1987), ubiquitous comput-ing might soon turn digital imagery into something which is partand parcel of the surrounding context (Hainich, 2006). For, how-ever used we have become to television, we know that its imagesexist only within the circumscribed space of a screen, while ARowes its very definition to its being – potentially – unframed. AsI mentioned in the introduction, ubiquitous computing may beresolved, rather than with the wearing of helmets and goggles,which inevitably fall short of the sophistication and precision ofeyes and ears, with the use of tablets that can be compared to“portable windows” (see, for instance, the project iTacitus,Sterling, 2009: 138-142). This would bridge the gap between thevirtual and the real, at least from the point of view of visual per-ception. Also other systems of visualization such as Mirage TM

try to minimize the discrepancy between the real environmentand the superimposed recorded image by adopting lightweight,ergonomic and see-through devices.

The merging of the virtual and the real actually seems to bethe primary aim of these systems of technological visualization,often with useful practical applications, as in medicine, sur-gery, e-learning and teleconferencing. However, in spite ofthese functional uses, AR has mainly been surrounded by ahalo of futuristic awe, opening in front of us fantastic scenarios:windows with virtual views on the ocean, the possibility of turn-ing a grey sky into a perfectly blue one “and add little patchesof sunlight to the scenery” (Hainich, 2009 [2006]: 140); eventhe perspective possibility of getting rid of people we do notlike has been envisaged.1 Given this general kind of approach,

1 See www.openthefuture.org.

1. ~ SEMANTIC COMPLEXITY IN AR

– 29 –

it is not surprising that scholars have described AR by resort-ing to analogies with science-fiction: while Boulter and Grusin,for instance, refer to the film Strange Days (Boulter – Grusin,1999), others frequently remind the reader that the futureapplications of AR could transform the world into the setting ofone of Gibson’s novels (Tagliagambe, 2009: 80).

In actual fact, most of the AR applications described so farare just fascinating hypotheses that deliberately go beyond thescope of augmented reality as such. Significantly, the subtitle ofHainich’s book is Augmented Media and Beyond (2009 [2006]),thus acknowledging that he is trying to foresee possible futuresituations rather than simply record what is available on themarket today. At least for the time being, AR is far from entail-ing completely virtual experiences. I am not denying the exis-tence of cyberspace (Gibson, 1984; Levy, 1997): the very per-ception of space as a mixed dimension made of both real andvirtual objects must change our view of the ordinary world. ButI intend to consider AR, as I think it actually is now, as adialectic space, as a hybrid dimension in which the virtual con-stantly tends to actualization. By ‘actualization’, I mean, withPierre Lévy, a “dynamic configuration” involving “the produc-tion of new qualities, a transformation of ideas, a true becomingthat feeds the virtual in turn” (Lévy, 1998: 25). Or, we couldalso define it as a borderline on which the tension between thereal and the virtual can constantly be felt. As SilvanoTagliagambe writes:

[…] rather than ‘going beyond’ the real world, plungingdirectly into a virtual reality, we should operate on the border-line between the outside and the inside of the latter, that is, onthe interface between virtual reality and ordinary reality. […]a liminal reality, a frontier, a place where different states meetand are mediated. A place dominated by the category of the in-between.” (Tagliagambe, 2009: 88; my translation)

The idea that we live in a hybrid ecosystem has graduallybecome dominant. Actually, it is not an utterly new concept.

Augmented Linguistics

– 30 –

Tagliagambe reminds us that, as early as in the 1920s, theRussian philosophers Vernadskji and Florenskij alreadyclaimed that the world should be considered as a totality, awhole, a biosphere in which information is transmitted fromevery organic and non-organic category to other organic or non-organic categories. No real gap should be perceived betweenthe geosphere, the living world, and the noosphere, which is thedimension of thought and culture: they all influence one anoth-er in a complex “mechanism of transformation and translation.”This happened well before the discovery of quantum physics(Tagliagambe, 2009: 15). Today the notion has been widened,including all computer-mediated forms of communication; onthe other hand, the holistic perspective that was implicit intwentieth-century theories has been left behind in favour ofmore relativistic ones. Kai Pata, for instance, speaks of a“deliberate blending of geographical spaces with collaborativeenvironments (such as blogs, microblogs, social repositoriesand social networks).” She believes that there are no longer anyboundaries between the real and the virtual world, actually, thatall boundaries have blurred. Among these, the boundarybetween the subject, the computer, and the augmented world:“Augmented reality makes us distributed beings spatially, andactivity based” (Pata, 2010). This all-pervasive technologicalspace strongly affects cognitive and linguistic processes.Katherine Hayles was, with Pierre Lévy (1994) one of the firstphilosophers to theorize about the intrinsic relationshipbetween mind and technology: language “does not exist apartfrom its penetration by code, so the subject does not exist apartfrom the technology that produces it and that it also produces”(Hayles, 2005 ).

It will by now have become clear that it is not easy to delim-it the space of “ordinary reality;” and that it is impossible notto go somehow beyond it: we actually do so all the time, and inthe most disparate ways. Yet, this does not jeopardize the ideathat the achievement of a perfectly seamless continuity between

the real and the virtual may not be the primary aim of AR. Ifthis complete integration were actually feasible, it would leadto the obliteration of reality itself in the name of an all-embrac-ing virtualization. And this would mean the end of virtualiza-tion, too. Resorting to linguistic categories, we could makeabout AR the same remark that has been articulated aboutmetaphor: it is difficult to sustain that there is a place in lan-guage which is uncompromisingly literal; yet, if we discard theexistence of this dimension altogether, we also deny the exis-tence of figurative language: “If we deny the literal in language,we deny the possibility of metaphor as well” (Kittay, 1987: 20).Similarly, we could say that it is difficult to find a place in theworld which is not virtual, yet, if we deny the existence of thereal altogether, we also deny the possibility of the virtual.

1.2. Mixed reality

The idea of a continuum existing between the real and thevirtual environments, not only due to technology but also as theresult of an ontological change in the perception of realityitself, is a far more stimulating approach. We can find this kindof representation in Paul Milgram’s and Fumio Kishino’s defi-nition of AR (1994), where the continuum is graphically repre-sented as follows (table 1):

1. ~ SEMANTIC COMPLEXITY IN AR

– 31 –

Table 1. AR graph by Paul Milgram and Fumio Kishino (1994)

Augmented Linguistics

– 32 –

According to this diagram, augmented reality belongs to MixedReality. It is therefore a hybrid dimension, a computer-mediat-ed ecosystem spanning between the two poles, though its placeis closer to the Real Environment than to Augmented Virtuality,which is closer to the Virtual Environment.

In slight contrast with this graph, which suggests the idea ofseveral degrees of ‘reality’ or ‘virtuality’, Ronald Azuma (1997)proposes a general description based on the following threeconditions 1) AR combines real and virtual; 2) is interactive inreal time; 3) is three-dimensional.

Let us consider both positions: in the above graphAugmented Reality is included in the span containing thewhole wide category of Mixed Reality. In Azuma’s definition,emphasis is given not only to the hybrid nature of AR, but alsoto its being interactive and three-dimensional. Both, however,start from the idea that in AR levels of reality are ‘mixed’ or‘combined’. This is perfectly consistent, though I would acceptthe idea of a mixing or combination of levels of reality only if itdoes not entail a complete confusion of the levels themselves.On the one hand AR is bound to remain a mixed experience (atleast in the short and medium terms): one composite contextwill derive from its extensive application, in which computer-generated texts and images will be perceived neither as a mereexcretion nor as belonging integrally to the context. On theother hand this continuum would be better represented as aseries of superimposed three (but also two)-dimensional planesrather than by an unbroken line. This diagram (table 2) is moreindicative of my perspective, which is based, rather than on theidea of a continuum between the virtual and the real environ-ment, on the belief in the partial overlapping of these dimen-sions.

Table 2. Mixed reality diagram.

The tension between real and virtual objects should be preser-ved, if only to keep us interested in what surrounds us. As KenPerlin remarked in one of his blogs, entitled “You can’t live inthe future for more than five minutes:”

[…] the human brain is simply not wired to sustain a sense ofnovelty. Unfortunately, all new things on our event horizonbecome reduced to the mere normal with astonishing rapidity,and our voracious and fickle appetite for the new and differentcan all too quickly lead us to consume the very change we wishto enjoy. We eat the future for breakfast, by mid-morning wehave indigestion, and by lunchtime we are hungry again. […]Even if you were to build a time machine, put on your silverlamé suit, set your flux capacitor to full forward thrust, andemerge two hundred years in the future, you would have onlyabout five minutes to enjoy the sensation, more or less. Duringthat time you might marvel at the wonders of antigravity, thegraceful arc of the protective energy dome over your city, theglint of sunlight off the floating skyscrapers in the sky above,or the way your brain tickles from the seamless techno-tele-pathy that appears to have rendered both TV and the internetobsolete. But after about a minute or so, your brain’s novelty

1. ~ SEMANTIC COMPLEXITY IN AR

– 33 –

Augmented Linguistics

– 34 –

normalization filter will begin to kick in. Within three minuteseverything around you will start to seem obvious, even prosaic.After five minutes you’ll once again simply be living in theordinary present. Yes, it will be a present that contains floatingcities, free infinite energy, shimmering holograms you can con-trol with pure thought. But none of that will matter once youget used to it. It will just be normal.2

However open to future possibilities mixed reality may be, weshould keep within the bounds of what is technically availabletoday, and, which is even more important, within the bounds ofwhat people’s communicative and technological demands makeof AR today. The latter is, I think, an essential point. Becausetechnological innovations are usually accepted only to theextent to which they meet our desires. Producers of AR systemsknow this, as the following online advertisement shows:

Our mission is to bring augmented reality technology to theuser level, and offer complete solutions that are simple to useand do not require knowledge of the underlying aspects of thetechnology. We want our clients to focus on their own ARapplications and reach their goals as fast as possible and withminimum effort. Giving excellent customer service is our pri-ority and we are committed to helping our customers integrateour technology to their projects.3

Some technologically available innovations have disappearedinto nothingness or found practical applications much later intime simply because people found them too difficult to use, orbecause they did not need or like them (Crystal, 2001: 226;Fuhrt, 2011). A crucial example is television: technically avail-able since the 1930s, it became popular only in the 1950s,when the economic and cultural conditions were favourable to

2 http://blog.kenperlin.com/?m=201001&paged=3>; posted on January11th, 2010.

3 http://www.arcane-technologies.com/en/

it. In his last book, Culture and Explosion (1992), Juri Lotmanbrilliantly summarized this process which involves the turningof revolutionary scientific discoveries (the ‘explosive’ event)into gradual technological progress: “But at the very momentwhen the explosion releases its internal energy it is changed bythe chain of cause and effect – and enters into the time of tech-nology. Logical development selects from the explosion thoseideas whose time has arrived and which can then be used.”(Lotman, 2009 [1992]: 61).

1.3. Semantic continuity

The awareness of the necessity of a semantic continuitybetween physical and digital data and the importance of aknowledge concerning the user’s demands has already led tosignificant experiments within the Internet. Tim Berners-Leewas the first to conceive the idea of a semantic web where dataand information should be associated according to complexprocesses of interpretation building up a network of relation-ships and cross-references that are far more sophisticated thana mere syntax or key-word-based linking (Berners-Lee et al,2001). This aim could only be achieved by substituting syntax-based linking procedures with semantics-sensitive ones.Intelligent agents should be able to recognize the meaning ofonline texts and therefore direct the user to the needed infor-mation, moving from link to link in a logical and coherent way,mapping data according to the classes and concepts belongingto particular domains. Significantly, these agents are not theoutcome of experiments in Artificial Intelligence but rathercomplex applications, such as URI (Uniform ResourceIdentifiers) and other XLM-based standards such as RDF(Resource Description Framework) and RDF Scheme (Dorati –Costantini, undated).

1. ~ SEMANTIC COMPLEXITY IN AR

– 35 –

Augmented Linguistics

– 36 –

If the problem of defining semantic coordinates is crucial tothe Internet, it becomes even more vital in AR, where the con-nection of different dimensions, of virtual and concrete worlds,makes the need for semantic coherence and cohesion evenmore urgent. Indeed, AR is activated by the physical context.The augmentation, consisting of digital imagery or strings oftexts retrieved from the Internet is the result of the act of point-ing at physical objects in the surrounding environment with ahandheld device provided with AR technology. Obviously, theinformation called about by this pointing must be consistentwith the existing world.

Studies concerning semantic continuity between real anddigital data are routinely made by marketing experts who areaware of the importance of context to give users the informationthey want. The following example, taken from a website, showsthe results of research made in the field of Mobile TouristInformation Systems. The description specifically refers to thenotion of “Context concepts”:

For a mobile information system, several aspects of contextcan be considered […], such as the characteristics of the par-ticular mobile device (storage and screen size) and network(bandwidth and peers), context of the application (require-ments in storage, download and display capability), context ofthe user of the system (e.g., time, location, interests), context ofinformation objects (e.g., location). The handling of the conceptdata depends on the intended usage: information about themobile device could be used to adapt the networking mode (togain efficiency in the system communication) or to adapt theinformation display (to gain effectiveness in the user commu-nication); […]. We believe, that a system’s concept of contextneeds to be open and extensible in order to address variousapplication environments. In general, the concept of context isonly pertinent if the system supports context-adaptation orcontext-awareness. That is, change must be an inherent andexplicit concept in the system. Also, the changes should not be

directly predictable; otherwise a simple parameterisationwould be more appropriate.4

Context-awareness and context-adaptation are here identifiedwith functionality: knowing the characteristics of the mobiledevice, the context of the application, the context of the userand of the information objects is a necessary pre-condition forproducing effective systems. Another point is that such notionof context needs to be open and extensible, that is, ready toinclude whatever new contextual elements might come intoview. But the most interesting part of this advertisement is theconclusion: context-awareness should not lead to a merelymechanical application of parameters. The author of the textsseems to be conscious of the fact that complete predictability inthe whole procedure would take to the antipodes of the informa-tion system scope: non-informativity. Indeed, to be effectiveinformation needs to contain an element of novelty and unpre-dictability, otherwise it becomes totally redundant. This, too,was clearly envisaged by linguists (Shannon & Weaver, 1949).On the other hand, a lack of context-awareness (including aknowledge of both the ‘environment’ and the user’s intentions)would lead to complete unpredictability, which is also identifi-able with uninformativeness. In other words information stemsfrom the balanced interplay between the known and theunknown. This notion has been principally systematized in lin-guistics resorting to the theme-rheme distinction in sentence-construction (Halliday, 1967-8; see also Sperber and Wilson,1996: 215-217).

Technically speaking, in AR spatial and temporal continuityis provided by tags: particularly by Radio Frequency, QR(Quick Response), and AR tags (see chapter 4). All these typesof tags contain finder, alignment and timing patterns which

1. ~ SEMANTIC COMPLEXITY IN AR

– 37 –

4 http://www.cs.waikato.ac.nz/~hinze/isdb/publications/hinze_buchanan_CONTEXTmHCI.pdf

Augmented Linguistics

– 38 –

allow the AR device to retrieve semantically-related informa-tion. At the same time, systems of geolocalization allow the userto be tracked, thus making the supply of information spatiallyand temporally related to the user’s environment.

But beside being spatial and temporal the relationshipbetween AR and physical reality should also be semantic(Milgram – Kishino, 1994). Obviously, there should be asemantic continuity between AR and the environment: if I amplaying chess, for instance, the computer-generated text orimagery should be semantically consistent with the game itself:what will appear in front of me to be moved is most probably avirtual chess piece, not, say, a virtual baseball club or a tennisracket. This aim may be achieved thanks to the pattern mark-ers contained in ARtags. These permit the virtual camera of thecomputer or of the handheld device to be aligned with realobjects in real time by allowing a computer vision algorithm tocalculate the camera pose. Thanks to these tags computer-gen-erated imagery can be coherently positioned over the realworld, so that we do not feel the incongruence between the 3Dgraphics, animations, and videos and the surrounding context.

Virtual objects should also possess, as far as possible, thephysical properties of real objects and behave accordingly. Inthe section entitled “Behavioural Association of the Real andVirtual” in the Handbook of Augmented Reality (Fuhrt, 2011)the process is described in the following terms:

We semantically modellise virtual objects by taking into accounttheir physical properties according to the laws of gravity, contact,elasticity, fluidity, etc. so as to enrich the scene. Prior knowledge isused in the real scene and its objects. This functionality defines geo-metrical and physical interactions between real and virtual objects.For example, we can mention the behaviour and attraction functionsof virtual objects with real objects (Fuhrt, 2011: 53).

On the other hand, in particular situations I could see some-thing which has little to do with the surrounding context. ARcan aim at creating artificial environments rather than

real(istic) ones. Rather than augment the real world, AR canattempt to create possible alternative worlds. For instance, abeautiful tropical island can be made to appear in front of agrey block of flats to provoke in me the desire to set out on ajourney (maybe there is a travel agent’s just around the corner).Here the category of semantic continuity is overtly called intoquestion.

In other words, artificial AR environments can be obtainedthrough a semantic gap produced by the incoherence of theoverall meaning of the mixed environment (Fuhrt, 2011: 58).As we shall see, this semantic gap can be studied in terms ofmetaphoric substitution.

1.4. Reaching beyond: metaphoric connections

The possibility of obtaining additional information about theplaces of interest (houses, restaurants, companies, etc.) thatsurround us in a specific geographical area depends on the glo-bal positioning system (GPS) provided by our handheld device,which can localize us and supply context-related information (§2.2.). From this perspective the notion of context becomes fun-damental. AR is deeply contextualized; more precisely, it isactivated by context. The question therefore is: in what kind ofrelationship do AR texts and imagery stand to context? Andhow does this relationship differ from that existing in otherforms of mediated communication?

The relationship between AR and the surrounding context isspatial and temporal. Its very existence is determined by coor-dinates that make it happen “there and then.” In oral commu-nication this relationship is mainly expressed through deictics:“The fruit is in the bowl over there”, where the “over there”acquires meaning only if the speakers share the same physicaland cultural context. In AR spatial interconnectedness is givenby the alignment process made possible by videocameras and

1. ~ SEMANTIC COMPLEXITY IN AR

– 39 –

Augmented Linguistics

– 40 –

ARtags while the temporal interconnectedness is given by thereal time response of the interface.5 Thus in AR contextualsalience is, in a sense, less ambiguous than in other kinds ofverbal exchange, because the association of real and computer-generated objects is directly (using an old-fashioned term wemight say ‘automatically’) called upon by the technologicaldevice itself. If I point at the Coliseum, for instance, the pop-up window appearing on my smartphone will most certainly tellme something about the historical and artistic relevance of thatmonument, rather than about other sites, at least not about sitesthat have nothing to do with the Coliseum. In other words if Iam likely to get the fruit I want by saying that it is “over there”I will most certainly get what I want with AR, because deicticsare physically actualized by the use of my handheld device: Iphysically6 point at things.

Yet, the process is not as simple as that. AR is based onindexicality: on the one hand it is true that AR is activated bypointing a handheld device at an object, thus establishing aconnection between discourse and the world; on the other hand,AR shares with the forms of indexicality we find in natural lan-guage the ambiguous relationship existing between discourseand the world. As Roderick, interpreting Lyotard’s thought,argues:

“ […] the eye is in the word because there is no articulationwithout the appeal to an outside constituted as a visibilitywhere objects are designated in space, as well as spatializationthat resides at the heart of discourse as an unconscious force-desire. […] Lyotard finds that figure resides in discourse as

5 A discourse apart should be made on the content of AR texts andimagery. Time can make them slightly obsolete if they are not constantlyupdated. For instance, a pop-up could tell me that I am in front of an Indianrestaurant while the site has now become a private home.

6 I am sticking to the present situation. Researchers are studying waysof activating AR through eye-blinking or even with thought.

the intractable opacity of the visible […] plasticity and desire,an extensive horizon. Indexicality means that discourse is shotthrough with the visible. […] the énoncé must point beyond itsborders to objects positioned in space with respect to it. It isplunged into a textual space that surrounds it, and it is riddledfrom within by deictic holes whose function is to indicate posi-tionality in space (here/there) and in time (now/then). […]Nonetheless there is a negativity of a special type, an openingin space between eye and object as a kind of moving frame thatis formal or formalizing. Indexicality gives us a formed space.”(Rodowick, 2001 : 6)

In the effort to make sense of reality, indexicality creates a net-work of cross-references that give unity to otherwise unrelatedobjects, simply by connecting the eye to the world and sig-nalling (i.e. phatically stating) the connection. On this accountthe semiotic interpretation of deictics is far more effective indealing with the complexity of the process (Greimas, 1968), butit is not the only applicable one.

Indeed in AR semantic continuity is difficult to define. Ofcourse the problem is also cogent in other forms of language-based communication and is definitely not new. Metonymies,for instance, do stand in a semantic relationship. Collocation isanother linguistic category based on the idea of continuitybetween the elements of an utterance (Firth, 1957). This conti-nuity is given either by the code itself (“as a matter of fact”) orby linguistic habit (“a gorgeous day”). But what would we makeof metaphors, for instance, especially of far-fetched ones, inwhich the relationship between the tenor and the vehicle is notimmediately detectable (Richards, 1936)? This problem sur-faced also with reference to the Internet. In a recent issue ofWired (Priolo – Maggiato, 2009: 54 ff.), which stressed theimportance of context-aware search engines, the authors under-lined that most of the experiments carried out in this directionare based not only on the study of logical associations, but alsoon the possibility that users might reason indirectly andmetaphorically.

1. ~ SEMANTIC COMPLEXITY IN AR

– 41 –

Augmented Linguistics

– 42 –

To widen the perspective of this research by adding ele-ments that can throw light on this problematic aspect whichinvests also digital contents, I may resort to other ways bywhich spatialization and indexicalization are actualized incomputer-mediated texts. Let us think about hypertexts. Aparallel has been made between hypertexts and suburbs on thebasis of the concept of linkage to distant items. According toAmerican thinker David Kolb, just as chunks of suburban buil-dings acquire a meaning from their linkages to distant sites, sohypertext chunks owe their meaningfulness to their being con-nected to other chunks, even if they are not immediatelyobvious or visible:

I want to make, but also qualify, a parallel between suburbs andhypertexts, as a way of emphasizing that the meaning of a givenchunk of suburban building or real estate usually depends onits linkages to distant items. The basic comparison with hyper-text – a non-linear assemblage of chunks of text with linksamong them – is that the form of the text is not the same as theform visible on any one page or screen. It reaches beyond, justas form of the suburb is not the same as the immediately visiblespatial connections. Immediate architectural form is not thesame as the place form of suburban locations, because theyreach out beyond the local horizon, and form wholes and net-works that are not architecturally obvious. We are not sure howto express this linkage architecturally, and most suburbanarchitectural types celebrate isolation rather than connection.[…] This parallel is useful, because the armature of links in ahypertext creates a “spatiality” that has more complex interre-lations and dimensions than the linear one-thing-after-anotherof physical space, or of pages in a novel.7

“It reaches beyond, just as form of the suburb is not the sameas the immediately visible spatial connections.” But isn’t thisprecisely the definition of metaphor, particularly of the far-

7 www.dkolb.org/sprawlingplaces/suburbsa.html. My italics.

fetched metaphor? If, pillaging Shakespeare (Sonnet 18), I sayto my husband “you are a summer day” what semantic relation-ship connects my husband to the weather condition he is figu-ratively identified with? Rhetoricians speak of the existence ofa tertium comparationis which unites the two semantic fields(Richards, 1936): in the above example the human being andthe weather situation could be semantically connected by theattributes of brightness, lovingness and light-heartedness.However, the criteria by which this term is chosen among thou-sands to represent certain attributes are not self-evident.

An interesting question might be whether AR contents canalso work indirectly, that is following metaphorical forms of rea-soning and distant connections. In other words, whether AR isprone to metaphorical uses. I think it is, and in two possibleways: 1) either by a deliberate choice of AR users, when theywant to generate virtual imagery or texts that are not directlyrelated to the context but are the result of their own desires or2) when AR texts and images are proposed/imposed to the useras in virtual forms of advertising, anticipating or provoking theuser’s needs and desires.

Literally speaking, as I am likely to get the fruit I want bysaying that it is “over there,” I will most certainly get what Iwant with AR, because deictics are actualized by the use of myhandheld device, either through digitalization or through eye-twitching movements. But AR – like the Internet – could alsobe designed so as to react to metaphorical thinking by respond-ing to people’s cognitive complexity or playing with it for com-mercial advantage. In the ‘real’ world I can say that my col-league is “beating about the bush” and the sentence would beaccepted as perfectly plausible even if I uttered it during abusiness meeting on the top floor of a New York skyscraper:because everybody would understand that I am speaking figu-ratively. In AR a bush will hardly be aligned to the furniture ofmy office by simply pointing at the physical context. Unless I(or other people for me) want to play with virtual imagery by

1. ~ SEMANTIC COMPLEXITY IN AR

– 43 –

Augmented Linguistics

– 44 –

placing a virtual bush in my city office, thus provoking “asemantic gap produced by the incoherence of the overall mean-ing of the mixed environment” (Fuhrt, 2011: 58).

But we could also interpret the incoherence (or have it inter-preted) as metaphorical thinking and adapt AR to our associa-tion of ideas and to our dreams, thus attributing to objectsother-than-literal meanings. Or, more radically, we could con-clude that metaphors are all-pervasively and inextricably inter-woven with the perception of semantic spaces. In the end, asLotman stated:

The problem of the intersection of semantic spaces is compli-cated by the fact that the circles we draw on paper represent aparticular visual metaphor rather than a precise model of theobject. […] In the distribution of these intersections across theentire space of a language, so-called linguistic metaphors aregenerated.” (Lotman, 2009 [1992]: 19).

Semantic continuity must come to terms with all this: not onlyspace is invested with cultural and symbolic meaning; in cog-nition, in its translation into language and then during commu-nicative processes it goes through stages of modellisation thatare inseparable from metaphorisation, so that it is hardly adatum which is given once and for all.

1.5. The Semantic Web Layer Cake on and off line

The same complexity in the activation of linking proceduresis at work both inside and outside the Web, both within anentirely digital context and within mixed dimensions involvingthe interaction of the virtual with the real. This would alsoexplain why researchers encounter serious technical problemsin finding software that takes into account the inevitablesemantic gaps existing between real and computer-generated(or computer-processed) objects (Behringer et al, 1999;Hainich, 2006). The mapping of objects from reality in the

model (as in ARtags) can minimize this gap but not obliterateit entirely.

The difficulty of defining context univocally is due to thecomplex processes by which we perceive and make sense of theworld. As Lotman remarked, semantic spaces “are linked toindividual consciousness” (Lotman, 2009 [1992]: 19), althoughthey may partly intersect. As Kai Pata echoes, place is not auniquely physical category: it is something that is meaningfulfor me, it is related to some people, emotions, actions (Pata,2010). Thus considered, also space in hybrid ecosystems (suchas AR) is better defined as an ontospace which contains, con-currently, a semantic and a pragmatic dimension (table 3).

1. ~ SEMANTIC COMPLEXITY IN AR

– 45 –

Table 3. Ontology Dimensions Map. Source: <http://accuracyandaesthetics.com/?p=370>

The idea of a Semantic Web (Tim Berners-Lee, 2001) is insep-arable from the acknowledgement of the existence of an onto-logical level. But can this ontospace be measured or defined?As experience is inextricably entangled with this dimension,and therefore, as the graphs show (tables 3 and 4), the pragmat-

ic aspects are inseparable from the semantic ones, it would bemuch more fruitful to achieve environments that are capable ofreacting to people’s communicative needs. Experiments in thedirection of the creation of “reactive environments” have actu-ally already been made. Among these, an interesting one aimedat defining a semantic topology which takes into account allpossible implications of hybrid environments has been recent-ly carried out by Pierre Lévy in his IEML project (2010). Butdifficulties remain: successful communication can be madeprobable but never guaranteed (Sperber and Wilson, 1986: 17).

AR ultimately replicates the same pragmatic problemsinvolved by every communicative act. As we shall see inChapter 6, communicative intent, cooperative principle, andinferential strategies are at work also here (Grice, 1975). ARsystems should be as good in applying inferential strategies tomessages as people are in normal interaction.

These strategies appear to be inseparable from the aware-ness that communication is a multi-layered dimension, a “layercake”, as the following graph representing the dynamics of theSemantic Web shows (table 4).

Augmented Linguistics

– 46 –

Table 4. Semantic Web Layer Cake. Source:<http://www.semanticfocus.com/blog/entry/title/introduction-to-the-seman-tic-web-vision-and-technologies-part-2-foundations/>

Though graphically conceived in different ways the two graphs(tables 3 and 4) show a similar interest towards those specificcategories that play a vital role in communicative processes.

1. Code or standard: RDF, XML Query Language, XMLSchema, Uri, Iri, Unicode. Grammar rules, vocabulary,syntax, sentence structures and models; encryption; formalcategories (expression, design methods, granularity, etc.),Namespace*;

2. Modality: rigour, validity, strictness, trust, proof, norms,authority, signature, governance;

3. Context: domain, Namespace (*it is also known as ‘context’because the correct operative meaning of the name dependson context)

4. Pragmatic aspects: shared beliefs, ontology, knowledge; con-tent as related to the users’ encyclopedia and emotions(things, stuff, relationships), intended use <-> functionaldomain;

5. Logical and cognitive patterns: reasoning, schema, models.

All these categories work together to produce meaning. Thishappens both in natural and in computer-mediated communi-cation: form and content, rules belonging to code, cognitive pat-terns, modality, conversational implicatures, shared knowled-ge, etc., all contribute to the creation of meaning.

While the first graph visualizes the process as a set system,the second one figures it out as a stratified structure whoselayers are all necessary to the making of sense. Indeed, theawareness of the ‘multi-layeredness’ of discourse is vital notonly to the Web but to cognitive processes and communicationin any context of situation. When we communicate we actaccording to mental frames (Goffman, 1974), “knowledgeschemas” and “interactive frames” which are both prior to theactual verbal exchange and inherent to it (Tannen, 1993). As aresult, different planes of talk are integrated (Shiffrin, 1994:

1. ~ SEMANTIC COMPLEXITY IN AR

– 47 –

Augmented Linguistics

– 48 –

21). Communication is therefore based on a multi-layerednesswhich is both semantic and pragmatic; it is “a multi-layeredframework of framing, in which the various facets of situationcan be seen in qualitatively different types of overlappingnotions of frames” (Johnson, 2008).

On this point, semiologists and linguistis seem to agree. AsLotman stated:

Semiotic space appears before us as the multi-layered inter-section of various texts, which are woven together in a specificlayer characterised by complex internal relationships andvariable degrees of translatability and spaces of untranslatabi-lity. The layer of “reality” is located underneath this textuallayer – the kind of reality that is organized by a multiplicity oflanguages and has a hierarchical relationship with them.

The existence of superimposed levels in cognitive and conse-quently on communicative processes is similarly stressed bycognitivists. As Sperber and Wilson remark: “While grammarsneutralise the differences between dissimilar experiences,cognition and memory superimpose differences even on com-mon experiences” (1986:16).

The idea that communication is a stratified or multi-layereddimension is of course crucial also in a multimodal perspecti-ve. In Kress and van Leeuwen’s words:

[…] communicative practices are seen as multi-layered andinclude, at the very least, discursive practices, productionpractices and interpretive practices, while they may alsoinclude design practices and/or distribution practices. […]each of these layers contributes to meaning. […] we assumethat meaning is made everywhere, in every ‘layer’, in phonolo-gy and in grammar/syntax. In any mode all realisational ele-ments are available for the making of signs” (Kress – vanLeeuwen, 2001: 111).

AR is just one more facet added to the situational frame and assuch, it partly overlaps and interacts with all the other facets

belonging to the context in which communication takes place.If the theory of framing is applicable to any kind of spontaneousor mediated communication, it is even more apt to describe themodes of AR, which is by definition a hypermediated dimen-sion based on a stratification of levels. Not only those that, aswe have seen, pertain to cognition and communication in gene-ral, but also those resulting from the different strata or levels ofreality involved in this particular medium: the virtual, the real,and all the layers in-between.

Finally it is important to highlight that while in the Internet,at least so far, the necessity to make choices between a pletho-ra of potential meanings has been signaled by the presence ofdisambiguation pages (Crystal, 2001: 210) in AR, by contrast,the ambiguity becomes part and parcel of the experience, sothat making choices seems to be no longer required by themedium.

This further complicates the problem. It is therefore on thenotion of superimposition and on other phenomena founded onoverlapping processes that we shall concentrate in the ensuingsection.

1. ~ SEMANTIC COMPLEXITY IN AR

– 49 –