Amajor challenge to making creative leaps in mediainnovations is demonstrating the magnitude of

human impact that each new media technology willbring.

If an American foundation had been commis-sioned to research the prospects of Gutenberg’sprocess called printing they would probably havesaid, “Interesting, but of limited value becausemost people cannot read.”

Anonymous, TerraMedia.co.uk quotations

The leading media companies today tend to base theirfuture decisions on repeating financial successes, a prac-tice that discourages radical innovation. LuckilyGutenberg had the Bible to drive sales, because the noveland the newspaper took hundreds of years to evolve.Even when we celebrate new technical capabilities, wefollow the celebration with hype and expectations thatscientific or technical solutions can’t meet alone. Alltechnology is inherently limited. For media to havehuman impact it must affect the imaginationthat is, itmust have an associated magic “behind the eyeballs”that the artist cultivatesto push new technologicalcapabilities to their limits and invent new media con-ventions. This bridges technological limitations with thehuman imagination, leading to true innovation that, asa result, forces us to change the way we do business com-mercially, academically and even personally, a prospectthat frightens most traditional and funded operations.Now that we are in a state of constant innovation, newmedia is a dangerous business.

It is the business of the future to be dangerous.Alfred North Whitehead, philosopher

Inventors bring new capabilities into our hands in theform of inventions; innovators bring new possibilitiesto inventions by adapting them in unintended ways thatexpand their application. History has shown that we areinventors and innovators, but we are rarely the innova-tors of our own inventions. It’s the generation followingthe invention that can examine its possibilities, unfet-tered by current perceived limitations. (For example,Douglas Engelbart invented the mouse, but it was Appleand their customers that innovated his invention some

20 years later.) In lieu of waiting for the next genera-tion, the artist who is teamed with the inventor canmake these creative leaps.

The infusion of artistic expression within scientificexploration of new media can be a process that defiessome traditions and embraces a dangerous naïveté ofthe artists that appears to be an unpredictable alchemyof the mind, machine, and magic. Like the alchemist,we don’t need to see it to believe it; we see it because webelieve it. Even though new science and technologicalcapability spark every media revolution, its possibilitiesare proven by artistic imagination, the consequenteffects inevitably changing life as we know it, and neverin ways we expect.

The emerging media of mixed realitythe blendingof real and synthetic environments and objectshasmelted the boundaries between physical and virtual real-ities, letting us interweave simulated characters andscenes into real-world experiences (see, for example,Figures 1 and 2). So why do we treat mixed reality as wedo more limiting media technology such as virtual reali-ty, television, cinema, radio, or print media? A new visionmight borrow from tradition, but by making creativeleaps, we can find the magical power that lies within.

Making creative leaps

All media, as extension of ourselves, serve to pro-vide new transforming vision and awareness.Marshall McLuhan, Oracle of the electronic age

We must not use new technology in the old ways, butrather use new technology to see in new ways. Ourresearch goal is to discover the magic that transformsmixed reality from technological invention into mediainnovation. Because the “Ministry of Magic” isn’t fund-ing any projects this millennium, we approached agen-cies and companies that are most directly affected orwhose constituencies are most impacted by media (inparticular, entertainment, education, and training orga-nizations). We then cross-fertilized insights from eachapplication to formulate tools, conventions, and infra-structure that stimulate innovation.

Mixed reality differs from traditional linear media.It’s real time, immersive, multisensory, and intended toengage, not capture or pacify, the audience’s imagina-

ChristopherStapleton andCharles E.Hughes

University ofCentral Florida

Believing is Seeing: Cultivating Radical Media Innovations __________________________

Graphically Speaking

Editor: Miguel Encarnação

88 January/February 2006 Published by the IEEE Computer Society 0272-1716/06/$20.00 © 2006 IEEE

tion. Mixed reality also aims to embracereal experience, unlike traditional elec-tronic or virtual media viewing. Theimpact of this future participatory expe-riential media requires less that weempower the creator’s imagination thanwe spark and interact with the user’simagination.

We also quickly learned that mediainnovation transforms each of the appli-cations we explored as much as each appli-cation informs the evolution of the mediainnovation. Our approach aims to melt theboundaries not only between physical andimagined reality, but also between appli-cations, disciplines, and markets, increas-ing potential economic rewards byapplying core technologies across a spec-trum of diverse industries. By using thecommon denominator of human experi-ence, we avoided reinventing technologiesfor each of these varied industries.

We changed the criteria of successfrom achieving technological function tomeasuring human impact through per-formance. We structured the innovationprocess’ priority to what the technologyshould do, not what it could do, and theninvented the technology and artistic con-vention to match. This concept goesbeyond “form follows function” to themore applicable expression, “form fol-lows function that serves a purpose.” Thisis the requisite condition so that technol-ogy becomes transparent to the user.

Innovating innovation

Any sufficiently advanced technolo-gy is indistinguishable from magic.

Arthur C. Clarke, science fiction writer

How do you inspire the crazy ideasthat produce radical innovations thatchange not only the product, but also themarket, the business, and even the users? What will fos-ter the ideas with which people can make creative leapsinto the future and envision the next transformationalinnovation that will change how we work, play, learn,and live? What does it take to make the technology dis-appear so the result becomes indistinguishable frommagic?

Truly radical media innovations stem from an alche-my of the mind (cognitive and imaginative), themachine (science and technology), and the magic(media and money) over a long and involved process oftechnological convergence, interdisciplinary creativity,and cross-industry transfer. In the past, this evolutionhas had as much to do with blind faith, dumb luck, andfailed investments as it did with anybody’s precon-ceived, well-organized innovation road map.

With all of the horror stories of these inventors’ dilem-mas, is there a better way of crossing the chasm to trans-form ideas into inventions and to neutralize adoptionobstacles to true innovations? Do all radical innovationsneed to be disruptive? Without such a process, can wekeep up with the ideas and inventions backlogged inacademic laboratories and commercial proprietaryarchives? What can serve as a logistical guide for trans-fer? What is a measure of success? What can we do tointegrate diverse genius and innovate innovation itself?

When newly invented capabilities overcome limita-tions in capturing, rendering, replicating, and trans-mitting sensory stimuli, the artists can transform theinvention into a blockbuster media outlet that we can’tlive without. Edison invented motion pictures (projec-tor and camera), but never made a successful film. It

IEEE Computer Graphics and Applications 89

1 Family research subjects play-testing Mixed Reality Aquarium for next-generation main-stream audiences.

2 Mixed Reality for Military Operations in Urban Terrain (MR MOUT) for close-quartered,dismounted soldier training.

took the Charlie Chaplins, Mary Pickfords, and D.W.Griffiths to create the movie media experience thattransformed the novelty of moving pictures to the mar-ket phenomenon of cinema. The same is true for radio,television, cable, and now for simulation and videogames. Thus, it isn’t the technology’s function but thecontent’s purpose that transforms inventions into lucra-tive media innovations. The application creates demandand drives the economics and then the adoption of newmedia technology.

With science and technology’s accelerated advance-ment comes a demand for constant media innovation.How do you integrate areas of study into disciplinarycollaboration, versus sequential cross-disciplinary devel-opment, so that each exploration informs the otherssimultaneously? How do you measure the impact ofeach development on human performance to validateresults across disciplines? If we could just see the inno-vation’s impact beforehand, we could gain this insight.

MR modeling human experience Mixed reality is the spectrum of physical reality (par-

ticipating in the real world), VR (procedural media stim-ulation), and imaginary reality (sparking cognitiveperception and creation by the user). By spanning thesensory modalities and richly layering the realities in alldirections and dimensions, we can effectively capture,replicate, render, display, and distribute human experi-ence. What we thought of as science fiction only a fewdecades ago is now possible with mixed reality.

Our ability to melt the boundaries between our realand familiar physical world and the simulated fantasyof the virtual world, engaging the emotions and psy-chology of the imaginative world, gives us a powerfulcapability to impact human performance. This is most

evident in “life or death” professionssuch as law enforcement or the military.For example, the Mixed Reality inMilitary Operations in Urban Terrainproject uses mixed reality and themepark entertainment techniques to sim-ulate future capabilities (see Figure 2).1

Such experimental projects let us eval-uate human performance before wefully develop the technology’s functionand direction. Human testing is nolonger an afterthought to invention;rather, it’s a prerequisite, helping todefine any future innovation’s functionand form.

Authoring MR interplayconventions

The three nodes of the mixed reality,represented in a circular continuum,are the physical, virtual, and imaginaryrealities, as Figure 3 illustrates. Themachine’s participatory function medi-ates the interaction with physical reali-ty. The media’s procedural conventionsparks the VR’s magic. The story’s emo-tional empathy paints with the user’s

imaginary reality, drawing in the mind. In applying mixed reality to mainstream educational

and entertainment venues, we found that authoringthese diverse spectrums of reality required the full rangeof age-old technical and artistic conventions of story,game, and free play, each working interdependently.These techniques of the media artist needed to clarifyenough to be integrated into the computer’s algorhyth-mic process to produce the magic.

For the sea creatures exhibit (see Figure 4), we decon-structed traditional structural elements of conventioninto an innovative convention that addresses both artis-tic purpose and technical function. We transferred thismodel from an earlier marketing/entertainment scenariomodeled in partnership with the Human MachinePerception Laboratory (formerly the MR SystemsLaboratory) at Canon, for MGM marketing, where theexpectation was for the model to feel like a movie, playlike a game, and be immersive like a theme park.

The diagram in Figure 3 doesn’t define a solution asmuch as it defines the challenge of mixed reality con-vention. Story conventions draw on the participant’sempathy and deal with characters, environments, andevents. Unlike a traditional linear story, this process bor-rows from the interactive and nonlinear techniques and theories of role-playing games. Game conventions,on the other hand, deal with procedure (or mechanics),using rules, tools, and goals to engage the media’s com-puter rendering. Free play draws on the intuitive physi-cal interaction’s participatory aspect and generatescause, effect, and consequences. These integrated con-ventions engage the heart, mind, and body, respective-ly. Ideally, these conventions use the entire spectrum ofinteractivity, converging conventions instead of invent-ing new ones.

Graphically Speaking

90 January/February 2006

Magic of mixed realityconventions

Media

Machine

Mind

Storypathos

Free playparticipation

Gameprocedure

Worlds

Imaginaryreality

Events

Tools

Characters

Rules

Goals

Effect

Passive:absorbs

Active:participates

Reactive:changes

Interactive:contributes

Experiential:immerses

Consequences

Physicalreality

VR

Cause

3 Interplay conventions for transdisciplinary poesies of mixed reality.

This approach’s pitfall is that those not intimatelyversed in all three conventions find it difficult to medi-ate. Most storytellers are fearful of interactivity and gamemechanics because of their ability to lead the audience’sinteractivity down a wrong path. Most gamers know lit-tle about interactive story versus linear full-motion video,and thus marginalize the story’s power to engage. MostHollywood game and story masters are unaccustomedto inviting the user to contribute outside of a passive orreactive role as in free-play techniques. This is where thenext generation of transmedia story creators plays a crit-ical role in defining future conventions. Transmedia pio-neers such as Brenda Laurel purposely interweave thetechniques of story, game, and play, or theater, cinema,and simulation to be complementary versus competitivedisciplines.

An important scientific (and artistic) innovationrarely makes its way by gradually winning overand converting its opponents. What does happenis that its opponents gradually die out, and thatthe growing generation is familiarized with theideas from the beginning.

Max Planck 1858-1947, theoretical physicist

This transmedia-based approach interweaves psy-chology (mind/cognition/imagination), computer sci-ence (machine/algorithms/story engine), and media(magic/experiential story conventions) into one processthat can simulate human experience. A system is need-ed to mediate all of these.

Human experience modelerThe technology for creating innovations in training,

education, and entertainment also applies to cognitiverehabilitation. Human experience modeling in all mix-tures of realities, modalities, directions, and dimensionsis a mixed reality capability that lets us capture, codify,and replicate a wide range of experiences in great detail.Experiencing activities as mundane as making break-fast, as demanding as protecting your comrades in urbancombat, and as playful as interacting with prehistoricmarine reptiles lets us model capabilities not yet invent-ed to explore their impact on human performancebefore developing real processes and products.

Adapting mixed reality human experience modelingfor use in cognitive rehabilitation (see Figure 5) clearlydemonstrated the limitations of pure VR. The test sub-ject was comfortable with the virtuality; however, boththe instructor’s and the subject’s need for human-to-human interaction and full body and sensory percep-tion was critical. Balancing virtuality and realityproduced an adaptable system that met the application’sobjective. When purely virtual environments (VEs) areintegrated with reality, you can utilize the realworldwith its eye-to-eye contact, a therapist’s gentleguiding hand, the arrangement of actual mementosfrom home, the interaction with real appliances, thesmell of coffee brewing and a bagel toasting, the sensa-tion of spreading the cream cheese, the refrigerator’scoldness, and the toaster oven’s heatto help both userand instructor with the tasks at hand. With the VE’s

dynamics, users can adapt and transfer real day-to-dayactivities into adjustable experiences that capture, sim-ulate, replicate, and analyze the experience in an excep-tionally detailed way. We modeled, animated, tracked,and integrated real and virtual worlds, allowing for anout-of-body after-action-review of the session.

Rehabilitation centers typically have only one generickitchen, requiring the patient to generalize skills thattransfer to the home. The adjustable modularity of ourkitchen human experience model allowed for a more com-plete and realistic rendering of the specific home experi-ence, with interaction enhancing the learning experience.Contextualizing the patient’s real kitchen allowed thedirect transfer of skills learned. Results from the pilot testshow promise for effective training that could shorten apatient’s stay at the rehabilitation center. Our pilot testsubject reduced the time spent preparing breakfast bynearly in half after only five training sessions.2,3

Capturing the subject’s viewpoint also let friends andfamily gain better insight and empathy for the challengesthe patient faced. Graphical read-outs support the recog-nition of patterns for automating behavioral analysis intodetailed quantitative and qualitative aspects. The conti-nuity of diverse environments in the same facility lets uscompare aggregate data across subjects.

Mapping vision into realityEvery innovation occasions more harm andderangement of order by its novelty, than benefitby its abstract utility.

Legal maxim, Thinkexist.com quotations

IEEE Computer Graphics and Applications 91

4 Sea CreaturesJourney, novelconcept for MRExperientialLearningLandscapes.

©2005 Univ of Central Florida; John Nadaeu, illustrator

5 Traumaticbrain injurypatient workingwith instructorwithin MRduring cogni-tive rehabilita-tion for dailyactivities.

Managing a culture of disruptive change to stimulateand nurture media innovation from idea to utility pushedus to create a repeatable innovation infrastructure, orinterdisciplinary roadmap, involving academia, govern-ment, and business. The goal was to track, nurture, andanticipate change before discarding good ideas becauseof practices that resist radical change. By exploring anddeveloping the technology, media, and human modelingin parallel, we could gather critical feedback to informeach discipline and evolve a solution that can overcometypical obstacles to adoption. Human experience modelscreated in the lab produced adaptable and demonstrableprototypes that incorporate critical field requirementsand expertise to better match purpose and function.

The belief that a killer app might appear and changethe world just by the fact of its invention lost all credi-bility during the Internet’s dot bomb phase. Venture cap-italists seized the opportunity to get rich quick, thinkingthat they could throw an idea into the marketplace andit would innovate by itself. They assumed that if theymade it, consumers would buy it, starting not only amedia revolution, but a new business model as well. Theother misconception is that media technology can devel-op without experimentation and validation of new ideasand conventions. Historically, new technology enters themarketplaces via the new content that drives adoptionand validates new business models, delivering new prod-ucts that are cheaper, faster, and better than the old.

Strategic media innovationinfrastructure

Each of our research projects adapt a repeatable pat-tern of media innovation abstracted from hundreds ofyears of media revolutions.

Our model (see Figure 6) follows a pattern found inmost media innovation that is obvious in hindsight, yetcontinually ignored. A new science or technology ideasparked almost every disruptive media innovation,transforming old limitations into new capabilities.Within the human experience model, we push these

new capabilities to the limit with content creators whoformulate ideas to transform technological capabilitiesinto creative possibilities. We realize these possibilitiesthrough new media conventions that tap the market-place’s imagination in the form of promotypes (proof-of-concept for the marketplace), creating both a demandfor the product and the human impact. Even after com-pleting these steps, we define a new product’s life cycleby how we can effectively get the product to market lessexpensively and more quickly than ever before. Thus,the process continues with

■ the development of new production and businessmethods that emanate from an analysis and evalua-tion of current standards and practices as well as prob-lems and challenges;

■ the invention and refinement of new prototypes thatserve as models (cost, operation, productionpipelines, and so on) within the laboratory environ-ment to discover failure points and invent refine-ments; and

■ the creation of a process to transfer ownership afterdemonstrating successful models.

Even though the transfer is the last step, the transferprocess must start from stage one, with industry con-tributing to the recognition of limitations that informthe entire process.

Industry then transforms laboratory models intoproducts, and markets and businesses incorporate theintegration of mind, machine, and magic. To do so, engi-neers apply the science and technology for productdevelopment; market development strategically pre-pares new markets to create demand; and operationsand production change the business development toaccommodate the new products and processes.

We map each phase of research collaboration frombasic research on university campuses, to appliedresearch in commercial and academic laboratories, totechnology transfer for industry adoption. Key princi-

Graphically Speaking

92 January/February 2006

Sustaininginnovations

IdeaOpportunityrecogition

InventionOpportunity

validation

InnovationOpportunity

capture

University Laboratory Industry

MachineScience andtechnology

MindHuman impactcognitive andimaginative

MagicMedia and

money(pipeline)

Radicalinnovation

Limitations

Possibilities

Analysis andevaluation

Productdevelopment

Marketdevelopment

Businessdevelopment

Capabilities(human experience

modeler)

Models(prototype)

Conventions(promotype)

6 Transferringinnovationmodels.

pal investigators and sponsors guide and monitor thisprocess. Each cell presents a unique potential failurepoint, tradeoff, and opportunity. Radical innovation notonly creates new products, but whole new industriesthat need not only product models, but also new busi-ness and market models.

ConclusionWith their unique and vast intellectual capital and

facilities, universities are in the ideal position to lead thisventure, but are unfortunately the least preparedbecause of the bureaucratic inertia commonly associat-ed with academia. They must lead and not follow indus-try and invest, nurture, and be accountable for wild ideasand nurture new successful models that can decipherhype from measurable outcomes. As a knowledge net-work and honest broker, academia needs to reinventitself, taking some of the professors and administratorsout of the ivory towers and some of the industrial andpolitical professionals out of the trenches, bringing thesediverse people together to play venture catalysts to thefuture. The creative individuals need a technologicalplayground of ideas and dreams to discover the alche-my of the mind, machine, and magic without the pres-sure of making or losing funding. This way, academiacan more rapidly capture the radical ideas that canchange society’s future for the better. ■

AcknowledgmentsMany generous collaborations with industry, govern-

ment, and academic institutions have made our efforts

possible. The Institute for Simulation houses our play-ground; Canon trusted our instincts; the USGovernment invested in us; and the Florida HighTechnology Corridor Council saw and helped nurtureour vision. We owe a great deal to our other foundingpartners Michael Moshell and Eileen Smith.

References1. C.E. Hughes et al., “Mixed Reality in Education, Enter-

tainment and Training: An Interdisciplinary Approach,”IEEE Computer Graphics and Applications, vol. 25, no. 6,2005, pp. 24-30.

2. S. Malo, C.B. Stapleton, and C.E. Hughes, “Going BeyondReality: Creating Extreme Multi-Modal Mixed Reality forTraining Simulation,” Proc. Interservice/Industry Training,Simulation, and Education Conf. (I/ITSEC), Nat’l TrainingSystems Assoc., 2004.

3. C.M. Fidopiastis et al., “Human Experience Modeler: Con-text-Driven Cognitive Retraining to Facilitate Transfer ofLearning,” Proc. 4th Int’l Workshop Virtual Rehabilitation(IWVR), 2005.

Readers may contact Christopher Stapleton at cstaplet@ist.ucf.edu.

Readers may contact Miguel Encarnação at me@ imedia-labs.com.

IEEE Computer Graphics and Applications 93

By Jim X. ChenGeorge Mason University

A shortcut to graphics theory, tools, and applications, this ReadyNote provides an exhaustive list of currently available graphics tools, categorizing them by their major applications and functions. Find out which tools are applicable to visualization, modeling, animation, and more. $9 www.computer.org/ReadyNotes

Here Now! A Survey of 3D Graphics Software Tools

IEEE ReadyNotes