The main human-centered activities in multi-media include content production, annota-

tion, organization, archival, retrieval, sharing,analysis, and multimedia communication. Withinthis group of activities I identify three key factorsin the development of future computing systems:

❚ culture,

❚ integration of sensors and multiple media,and

❚ access outside the desktop by a wide range ofusers.

When someone enters any establishment inJapan, where I live, there’s an immediateirashaimase greeting (“Welcome!”) by storeemployees. But it’s also often automatic: When Ienter an elevator, approach an ATM, a photobooth, or a metro ticket vending machine, a sen-sor activates (see Figures 1 and 2), and I am greet-ed by multimedia cartoon characters that speakto me: “Going down.” “All information will bedisplayed in English.” The characters do notspeak like computers—they speak like Japanesesales clerks (high-pitched voices with specificcharacteristics). They even bow.

It’s interesting to consider these systems whilethinking about multimedia. Although the inter-

faces are primitive and some of them are not real-ly multimedia computing systems, they exhibitseveral important characteristics:

❚ they act according to the cultural context inwhich they’re deployed;

❚ they integrate different types of sensors forinput and communicate through a combina-tion of media; and

❚ they’re deployed outside the desktop andthey’re meant to be accessed by a diversity ofindividuals.

These examples highlight that ubiquitous com-puting (see the “Definitions” sidebar on p. 14 for aclarification of what I mean when I refer to theterm ubiquitous computing and other terms inthis article) is becoming a reality, and that the dis-tinctions between the physical and the digitalworld are blurring, as are the distinctions betweenmultimedia computing and computing. Considerthe process for taking the train: I approach amachine that greets me. By pressing buttons I geteither a paper ticket or an electronic card, which isthen inserted into another computer that process-es it and allows me to enter the system. What partof the process was digital, analog, or multimedia? Itbecomes clear why companies like Google want toorganize the world’s information without makingany explicit reference to digital information.

Multimedia technologies are key in accessingthe world’s resources, particularly if we extendour notion of multimedia to what it really is—acombination of digital, analog, spatial, and sen-sory inputs and outputs. Perhaps the ultimateexample of this is the automatic toilet (see Figure3), a device installed in most restaurants andhomes throughout Japan. Such toilets have sen-sors and produce music and various other

12 1070-986X/06/$20.00 © 2006 IEEE

Visions and Views Nevenka DimitrovaPhilips Research

Alejandro JaimesFXPAL Japan,

Fuji Xerox

Human-Centered Multimedia: Culture, Deployment, and Access

Cultural setting is an intrinsic part of what we’re trying to capture anduse in multimedia systems. However, being in our own culture (both every-day culture and professional culture) we forget that multimedia interfacesand communication are culture-specific. This article gives some greatinsights that stem from diversity in countries (and cultures) as well as insidethe interdisciplinary multimedia community.

—Nevenka Dimitrova

Editor’s Note

sounds to aesthetically improve time spent inthe bathroom.

The toilets are available in several makes andmodels. They include a shower spray, and somenext-generation toilets offer to give users a per-sonalized health analysis, sent directly over theInternet to their doctor for monitoring andbefore their scheduled check-up.

Cultural factorsCulture plays an important role in

human–human communication because the waywe generate signals and interpret symbolsdepends entirely on our cultural background.Multimedia systems should therefore use cultur-al cues during interaction (such as a cartooncharacter bowing when a user initiates a transac-tion at an ATM), as well as during analysis (suchas algorithms to automatically analyze newsbroadcasts from different countries, meetingvideos, or any other content).

The majority of work in multimedia analysisand interaction, however, assumes a one-size-fits-all model, in which the only difference betweensystems deployed in different parts of the world(or using different input data) is language. Thespread of computing under the language-onlydifference model means people are expected toadapt to the technologies, imposed arbitrarilyusing Western thought models.

The problem is that this approach discouragescreativity and leads not only to a technologydivide, but also to a content gap between rich andpoor countries (and/or upper and lower classes).Although our research community is interna-tional, the majority of international conferenceorganizers and researchers that publish interna-tionally work in developed countries, so it’s rareto see highly innovative technical work thataddresses local cultural issues. Most researchers inmultimedia end up following a single model.

With such a narrow view, we lose robustnessand more importantly, limit the access of tech-nology and content to a small percentage of theworld’s population.1,2 Technology is increasing-ly acting as the gateway to all basic resources,giving those that have it a real competitiveadvantage—for instance, what would happen toour food, water, healthcare, security, and com-munications if the computing infrastructure sud-denly collapsed? Thus, the problem with anarrow view is not only limiting access to tech-nology, but limiting access to all the benefitsassociated with it.

Cultural factors play a role in every aspect ofcomputing, but let’s examine two importantareas in multimedia.

13

Figure 1. The ATM

machine activates

when it’s approached.

Unlike ATMs in the US

or Europe, the customer

first chooses the desired

option and then inserts

the card. Incidentally,

people can buy tickets

for concerts or movies

at an ATM-like kiosk in

any convenience store.

Figure 2. Japanese

photo booths are

popular. Users receive

instructions (cartoon

character that speaks)

and can customize and

modify their photos

prior to printing. The

screen turns on and the

cartoon character bows

as the machine is

approached.

Figure 3. The toilet seat goes up automatically as the customer opens the door to

the restroom. It closes when he closes the door. Many of these toilets warm the

toilet seat, have a water jet spray, and play a range of melodies while in use.

Automatic analysis The TREC Video Retrieval Evaluation 2005

(TRECVID; http://www-nlpir.nist.gov/projects/trecvid) set serves as a good example of how cul-ture affects content production and automaticanalysis techniques. In news programs in someMiddle Eastern countries there are mini-soap seg-ments between news stories. The direction of textbanners differs depending on language, and thestructure of the news itself varies from country tocountry, so a technique developed for news fromthe US is not likely to perform well in news fromsome of these countries. Thus, it’s important touse varied data sets and consider cultural factorsin multimedia production during automaticanalysis (that is, we should use culture-specificcomputational models).

Without a doubt, the cultural differences insemantic content span every level of the multi-media content pyramid and the content produc-tion chain,3,4 from low-level features (colors havestrong cultural interpretations) to high-level

semantics (consider the differences in communi-cation styles between Japanese and Americanbusiness people). The contrast is even greater infilm: colors, music, and all kinds of cultural sig-nals convey the elements of a story. Consider thedifferences between Bollywood and Hollywoodmovies (colors, music, story structure, and so on).Content is knowledge, and to make this knowl-edge widely accessible, we must develop culture-specific automatic analysis techniques.

Interaction Significant research has been done in the field

of human–computer interaction (HCI) on cultur-al factors. However, the general emphasis in HCIresearch has been on novel techniques or applica-tions, so cultural issues haven’t been explored indepth and little agreement exists on how orwhether culture-specific techniques should bedeveloped. Unfortunately, the multimedia com-munity hasn’t applied work in HCI and otherfields where culture has been studied. Furthermore,

14

IEEE

Mul

tiM

edia

Visions and Views

The following are definitions of some of the terminology ref-fered to in this article:

❚ User-centered design. This term has been used widely in thehuman–computer interaction (HCI) community for severalyears (a brief history is available elsewhere1). Althoughthere’s no general agreement in the field as to the focusareas of UCD (or the definition), the activities generally focuson understanding the needs of the user as a way to informdesign.1 Unfortunately, the UCD work in the HCI communi-ty has had little impact on the development of multimedia.I conjecture that this is in part because there isn’t that muchoverlap between the two communities, and because in theshort history of multimedia, interest has been mainly onanalysis, retrieval, and production of multimedia content.

❚ Multimedia. This term, generally referred to as “a fusion ofmultiple types of data sources used to acquire, process,transmit, store, and utilize information,”2 has been used ina wide variety of contexts. In fact, the term multimedia oftenincludes many types of media, but not necessarily a combi-nation of media. I view multimedia as a combination of dig-ital, analog, spatial, and sensory inputs and outputs.

❚ Human-centered computing. A human-centered computingsystem is a system that involves any human activity (such asmultimedia indexing for retrieval), or whose design parts

from human models or gives special consideration to humanabilities (such as human memory or subjectivity). This differsfrom user-centered computing, which assumes an explicituser (see also Flanagan et al.3 for related definitions).

❚ Multimodal system. This refers to a system that responds toinputs in more than one modality or communicationchannel (for example, speech, gesture, writing, and others).

❚ Ubiquitous computing and ambient intelligence. There is muchoverlap between these two terms. Ubiquitous computingplaces emphasis on connecting devices to devices, peopleto devices, and people to people, anywhere and anytime.Ambient intelligence is based on similar principles, but seeksto make people's environments intelligent.4

References1. J. Karat and C.M. Karat “The Evolution of User-Centered Focus

in the Human-Computer Interaction Field,” IBM Systems J., vol.

42, no. 4, 2003, pp. 532-541.

2. L. Rowe and R. Jain, “ACM SIGMM Retreat Report,” ACM Trans.

Multimedia Computing, Communications, and Applications, vol. 1,

no. 1, 2005, pp. 3-13.

3. J. Flanagan et al., eds., Human-Centered Systems: Information, Inter-

activity, and Intelligence, tech. report, Nat'l Science Foundation, 1997.

4. E. Arts, “Ambient Intelligence: A Multimedia Perspective,” IEEE

MultiMedia, vol. 11, no. 1, 2004, pp. 12-19.

Definitions

the majority of tools for content production followa standard Western model, catering to a small per-centage of the world’s population and ignoring thecontent gap (see the World Summit Award—http://www.wsis-award.org—which is an initiativeto create an awareness of this gap).

Our technical developments must be alignedwith social and cultural developments, in a newmodel that embraces multiculturalism and seesthe human impact of the technologies we’redeveloping. One of the big research challenges,however, is how to translate what we learn aboutculture into technical approaches that can beapplied computationally. There’s a large gapbetween the work of sociologists, psychologists,and anthropologists, and the work of researchersand developers of multimedia technologies.

Although social scientists and nontechnicalresearchers have used technology for years, thisuse has been limited and has not been reciprocal(we start from the technology). The most promis-ing research direction to address this problem isfurther integration between the development ofmultimedia systems and techniques and socialstudies. An excellent example is the work ofPentland5 on social computing and attempts atbenchmarking using culturally diverse data sets,such as the TRECVID 2005 set.

In academia, the importance of this has beenrecognized, and several programs integrate artsand engineering. In some universities graduateprograms in human-centered computing havebeen established (for example, Georgia Tech).Collaboration with people in other fields is posi-tive, but to really change the way we think abouttechnology, it’s necessary to create programs thatintegrate disciplines from start to finish. It’s nec-essary to integrate approaches used in otherhuman-centered fields and build new method-ologies to incorporate cultural knowledge.Ontologies, multicultural knowledge bases, andtechniques that use machine learning (becauseof their ability to be used with diverse trainingsets) can be possible starting points. Such inte-gration will hopefully lead to new computation-al models and design methodologies.

To summarize, in our own interactions we rec-ognize cultural differences and act accordingly(for example, company culture, social status, andso on), so why not develop highly adaptive orculture-specific systems? It’s clear that usingdiverse content can help us gain a better under-standing of how cultural differences are mani-fested in multimedia content production and that

understanding how users of different culturesinteract will make our systems more effective.

Culture defines a large part of who we are,what we do, and how we interact with our envi-ronment, so it should be considered whendesigning multimedia systems, whether it’s mul-timedia production, annotation, organization,retrieval, sharing, communication, or contentanalysis. Multimedia systems should include cul-ture-specific models at every level of content andinteraction in multimedia communication.

Integrating sensors and multiple media New ATMs in Japan use biometric technology

(palm and index readers) to verify identity, andsome tour buses use GPS technology to automat-ically project tour-guide videos as the bus passestourist attractions. Some restaurants use wirelesstouch screens so customers can order as soon asthey’re ready (see Figure 4).

What’s interesting about these applications ishow, even at primitive levels, information fromnetworks and sensors is integrated with othertypes of inputs and outputs. Despite great effortsin the multimedia research community, inte-grating multiple media (in analysis and interac-tion) is still in its infancy. Our ability tocommunicate and interpret meanings dependsentirely on how multiple media is combined(such as body pose, gestures, tone of voice, andchoice of words), but most research on multime-dia focuses on a single medium model.

Most of the systems I describe employ simplemotion sensors that have been available for years(for example, in washrooms and for automatic

15

January–M

arch 2006

Figure 4. Some

restaurants have

wireless touch screens

so customers can order

when they’re ready. The

screen can be passed

around the table just

like a menu. The

waiters only show up

when the food is ready

or when someone

presses the waiter icon

to call them.

doors). Nonetheless, many novel applicationshave been developed integrating multiple sen-sors. In this context, multimodal interactionbecomes a crucial part of a multimedia system.In the past, interaction concerns have been leftto researchers in HCI—the scope of work oninteraction within the multimedia communityhas focused mainly on image and video brows-ing. Multimedia, however, includes many typesof media and, as evidenced by many projectsdeveloped in the arts, multimedia content is nolonger limited to audiovisual materials. Thus, Isee interaction with multimedia data not just asan HCI problem, but as a multimedia problem.

Our ability to interact with a multimedia col-lection depends on how the collection is indexed,so there is a tight integration between analysisand interaction. In fact, in many multimedia sys-tems we actually interact with multimedia infor-mation and want to do it multimodally.

Two major research challenges are modelingthe integration of multiple media in analysis, andin multimodal interaction techniques. Statisticaltechniques for modeling are a promisingapproach for certain types of problems. Forinstance, Hidden Markov Models have been suc-cessfully applied in a wide range of problems thathave a time component, while sensor fusion andclassifier integration in the artificial intelligencecommunity have also been active areas ofresearch. In terms of content production, wedon’t have a good understanding of the humaninterpretation of the messages that a systemsends when multiple media are fused—there’smuch we can learn from the arts and communi-cation psychologists.

Because of this lack of integration, existingapproaches suit only a small subset of the prob-lems and more research is needed, not only onthe technical side, but also on understandinghow humans actually fuse information for com-munication. This means making stronger linksbetween fields like neuroscience, cognitive sci-ence, and multimedia development. Forinstance, exploring the application of Bayesianframeworks to integration,6 investigating differ-ent modality fusion hypothesis7 (discontinuity,appropriateness, information reliability, directedattention, and so on), or investigating stages ofsensory integration8 can potentially give us newinsights that lead to new technical approaches.

Without theoretical frameworks on integrat-ing multiple sensors and media, we’re likely tocontinue working on each modality separately

and ignoring the integration problem, whichshould be at the core of multimedia research. Weneed new mathematical models that truly inte-grate multiple sources and media, both in analy-sis and interaction, and a better understandingof how humans perceive and interpret multiplemodalities.

Ubiquitous accessOn one hand we can use mobile systems such

as third-generation mobile phones to create andaccess multimedia content. On the other handwe have nonmobile systems (such as ATMs andticket vending machines). An interesting charac-teristic of this second group of devices is thatbecause they’re deployed in public spaces, they’redesigned to be used by anyone (no need to readextensive manuals).

Computing is migrating from the desktop, atthe same time as the span of users is expandingdramatically to include people who wouldn’tnormally access computers. This is importantbecause although in industrialized nationsalmost everyone has a computer, a small per-centage of the world’s population owns a multi-media device (millions still do not have phones).The future of multimedia, therefore, lies outsidethe desktop, and multimedia will become themain access mechanism to information and ser-vices across the globe.

Mobile devicesEveryone seems to own a mobile device (see

Figure 5)—there’s a new wave of portable com-puting, where a cell phone is no longer a cellphone but rather a fully functional computerthat we can use to communicate, record, andaccess a wealth of information (such as location-based, images, video, personal finances, andcontacts). Although important progress hasbeen made, particularly in ambient intelligenceapplications8 and in the use of metadata frommobile devices,9,10 much work needs to be doneand one of the technical challenges is dealingwith large amounts of information effectively inreal time.

Developing effective interaction techniquesfor small devices is one of our biggest challengesbecause strong physical limitations are in place.In the past, we assumed the desktop screen wasthe only output channel, so advances in mobiledevices are completely redefining multimediaapplications. But mobile devices are used for theentire range of human activities: production,

16

IEEE

Mul

tiM

edia

Visions and Views

17

January–M

arch 2006

annotation, organization, retrieval, sharing, com-munication, and content analysis.

Shared resourcesWhile mobile phone sales are breaking all

records, it’s increasingly common for people toshare computational resources across time andspace. As in the examples previously discussed,public multimedia devices are becoming increas-ingly common. In addition, it’s important to recognize that—particularly in developing coun-tries—sharing of resources is often the onlyoption. Many projects exist on using communityresources, particularly in rural areas, for educa-tion and other important activities. One of themain technical research challenges here is con-structing scalable methods of multimodal inter-action that can quickly adapt to different typesof users, irrespective of their particular commu-nication abilities.

The technical challenges in these two casesseem significantly different: mobile devicesshould be personalized, while public systemsshould be general enough to be effective formany different kinds of users. Interestingly, how-ever, they both fall under the umbrella of ubiq-uitous multimedia access: the ability to accessinformation anywhere, anytime, on any device.

Clearly, for these systems to succeed we need toconsider cultural factors (for example, text mes-saging is widespread in Japan, but less popular inthe US), integration of multiple sensors, and mul-timodal interaction techniques.

In either case, it’s clear that new access para-digms will dominate the future of computing andubiquitous multimedia will play a major role.Ubiquitous multimedia systems are the key in let-ting everyone access a wide range of resourcescritical to economic and social development.

Research agenda for human-centeredmultimedia

Human-centered multimedia systems shouldbe multimodal (inputs and outputs in more thanone modality or communication channel). Theymust also be proactive (understand cultural andsocial contexts and respond accordingly), and beeasily accessible outside the desktop to a widerange of users.

A human-centered approach to multimediadeparts from user models that consider howhumans understand and interpret multimediasignals (feature, cognitive, and affective levels),and how humans interact naturally (the culturaland social contexts as well as personal factorssuch as emotion, mood, attitude, and attention).

Figure 5. Maybe

nowhere in the world

more than in Shibuya,

a crowded, young area

of Tokyo, are

pedestrians bombarded

with videos, images,

and sounds. It has

being said that the

Shibuya Crossing

(pictured) is the place

with the highest density

of mobile phone use in

the world.

Inevitably, this means considering some of thework in fields such as neuroscience, psychology,communications research, HCI, and others, andincorporating what’s known in those fields inmathematical models that we can use to con-struct algorithms and computational frameworksthat integrate different media.

Machine learning integrated with domainknowledge, automatic analysis of social net-works, data mining, sensor fusion research, andmultimodal interaction11 will play a special role.More research into quantifying human-relatedknowledge is necessary, which means developingnew theories (and mathematical models) of mul-timedia integration at multiple levels. In partic-ular, I propose the following research agenda:

❚ Create new interdisciplinary academic andindustrial programs, as well as workshops thattackle the issues discussed and involveresearchers across disciplines.

❚ Use culturally diverse data sets for commonbenchmarking and evaluation (includingbehavioral and multisensory data).

❚ Make software tools available (for annotation,feature extraction, and so on).

❚ Work on new human-centered methodologiesfor the development of algorithms in each ofthese aforementioned areas (not just systems).

❚ Focus research efforts on the integration ofmultiple sensors and media, with the humanas the starting point (based on studies, theo-ries in neuroscience, and so on).

❚ Instead of just considering the impact oftechnology, consider the social, economic,and cultural context in which it might bedeployed (see Bohn et al.12 for an interestingdiscussion on implications of ubiquitouscomputing).

Human-centered approaches have been theconcern of several disciplines,13 and some of theinitiatives I’ve mentioned have been undertakenin separate fields. The challenges and opportuni-ties in the field of multimedia, however, aregreat, not only because so many of the activitiesin multimedia are human-centered, but becausemultimedia data itself is used to record and con-vey human activities and experiences. It’s only

natural, therefore, for the field to converge in thisdirection and play a key role in the transforma-tion of technology and human livelihood.

ConclusionsMany technical challenges lie ahead and in

some areas progress has been slow. With the costof hardware continuing to drop and the increasein computational power, however, there havebeen many recent efforts to use multimedia tech-nology in entirely new ways. One particular areaof interest is new media art. Many universitiesaround the world are creating new joint art andcomputer science programs in which technicalresearchers and artists create art that combinesnew technical approaches or makes novel use ofexisting technology with artistic concepts. Inmany new media art projects, technical noveltyis introduced while many issues are considered:cultural and social context, integration of sen-sors, migration outside the desktop, and access.

Technical researchers need not venture intothe arts to develop human-centered multimediasystems. In fact, in recent years many human-centered multimedia applications have beendeveloped within the multimedia domain (suchas smart homes and offices, medical informatics,computer-guided surgery, education, multimediafor visualization in biomedical applications, andso on). However, more efforts are needed and wemust realize that multimedia research, except inspecific applications, is meaningless if the user isnot the starting point. The question is whethermultimedia research will drive computing (withall its social impacts) in synergy with humanneeds, or be driven by technical developmentsalone. MM

AcknowledgmentsI’d like to thank Nicu Sebe for fruitful discus-

sions on this topic, and the attendees of our recenttutorials on this topic (at ACM Multimedia,International Conference on Computer Vision,International Conference on Multimedia andExpo, and the Pacific Rim Conference onMultimedia). I would also like to thank NevenkaDimitrova for insightful comments on this article.

References1. E. Brewer et al., “The Case for Technology for

Developing Regions,” Computer, vol. 38, no. 6,

2005, pp. 25-38.

2. R. Jain, “Folk Computing,” Comm. ACM, vol. 46,

no. 4, 2003, pp. 27-29.

18

IEEE

Mul

tiM

edia

Visions and Views

3. A. Jaimes and S.-F. Chang, “A Conceptual

Framework for Indexing Visual Information at

Multiple Levels,” Proc. Soc. for Imaging Science and

Technology and Int’l Soc. for Optical Eng. (IS&T/SPIE)

Internet Imaging 2000, IS&T/SPIE, vol. 3964, 2000,

pp. 2-15.

4. N. Dimitrova, “Context and Memory in Multimedia

Content Analysis,” IEEE MultiMedia, vol. 11, no. 3,

2004, pp. 7-11.

5. A. Pentland, “Socially Aware Computation and

Communication,” Computer, vol. 38, no. 3, 2005,

pp. 33-40.

6. S. Deneve and A. Pouget, “Bayesian Multisensory

Integration and Cross-Modal Spatial Links,” J.

Physiology Paris, vol. 98, nos. 1–3, 2004, pp. 249-258.

7. T.S. Andersen, K. Tiippana, and M. Sams, “Factors

Influencing Audiovisual Fission and Fusion Illusions,”

Cognitive Brain Research 21, 2004, pp. 301-308.

8. C.E. Schroeder and J. Foxe, “Multisensory

Contributions to Low-level, ‘Unisensory’

Processing,” Current Opinion in Neurobiology, vol.

15, 2005, pp. 454-458.

9. S. Boll, “Image and Video Retrieval from a User-

Centered Mobile Multimedia Perspective,” Proc.

Int’l Conf. Image and Video Retrieval, Springer LNCS

3568, 2005, p. 18.

10. M. Davis and R. Sarvas, “Mobile Media Metadata for

Mobile Imaging,” Proc. IEEE Int’l Conf. Multimedia

and Expo, IEEE CS Press, 2004, pp. 936-937.

11. A. Jaimes and N. Sebe, “Multimodal HCI: A

Survey,” Proc. Int’l Conf. Computer Vision (ICCV

2005), N. Sebe, M.S. Lew, and T.S. Wang, eds.,

Springer LNCS 3766, pp. 1-15.

12. J. Bohn et al., “Social, Economic, and Ethical

Implications of Ambient Intelligence and

Ubiquitous Computing,” Ambient Intelligence,

Springer-Verlag, 2005, pp. 5-29.

13. J. Flanagan et al., eds., Human-Centered Systems:

Information, Interactivity, and Intelligence, tech.

report, Nat’l Science Foundation, 1997.

19

Sandy Brown10662 Los Vaqueros CircleLos Alamitos, California 90720-1314USAPhone: +1 714 821-8380Fax: +1 714 821-4010sbrown@computer.org

For production information, conference, andclassified advertising, contactMarian Anderson10662 Los Vaqueros CircleLos Alamitos, California 90720-1314Phone: +1 714 821-8380 Fax: +1 714 821-4010manderson@computer.org

http://www.computer.org

Advertising Sales Offices

AXMEDIS 2006 Cover 2, Cover 3

Game Editor 91

Iconico 91

Ideal 91

InterVideo 92

Konica Minolta 91

KWorld Computer 92

MagicScore 91

PQDVD.com 91

Advertiser / Products Page Number

FUTURE ISSUE

April–June 2006Evolving Media