Multimedia Presentation Design
description
Transcript of Multimedia Presentation Design
![Page 2: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/2.jpg)
2
Papers Presented Today• Automating the Generation of Coordinated Multimedia
Explanations (S. K. Feiner and K. R.McKeown)– COMET: content selection, media selection, media generation,
coordinated layout.• Plan-based Integration of Natural Language and
Graphics Generation (W. Wahlster et al.)– IBIS: generation of 3-D illustrations.
• Automated Generation of Intent-Based 3D Illustrations (D.D. Seligmann and S. Feiner)– Allocation of information to particular media.
• Presentation Design Using an Integrated Knowledge Base (Y. Arens et al.)– WIP: generalization of text-linguistic notions relations to multimedia
presentation• The Knowledge Underlying Multimedia Presentations
(Y. Arens, Y.)– Integrated Interfaces: dynamic construction of multimedia displays
using rules.
![Page 3: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/3.jpg)
Ariffin Yahaya 3
Introduction
• Typical Steps in multimedia presentations:– Determination of communicative intent.– Selection of content from a base of
knowledge.– Grouping/structuring and ordering.– Allocation to particular media.– Layout.
![Page 4: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/4.jpg)
Ariffin Yahaya 4
Automating the generation of Coordinated Multimedia Explanation• Pictures and language complement each other
to enable highly effective communication.
– First generation authoring facility
• BUT how does one go about putting together a system that does it dynamically?
– COMET (coordinated multimedia explanation testbed) created to overcome these problems.
![Page 5: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/5.jpg)
Ariffin Yahaya 5
First generation authoring facilities
• Basic facilities– Create presentations
• Text, Graphics, Animation, Video
• Problems– Requires skills
• Medium conventions (i.e. what people expect)
• Coherent mix of mediums.
– Must be authored in advance• Limits the presentation to a
known audience set.
Figure 1 First Generation Authoring
![Page 6: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/6.jpg)
Ariffin Yahaya 6
Comet
• Goal :– Coordinated, interactive generation of explanations
that combine text and graphics, all generated by the system on the fly.
• Example:– How to repair a radio receiver / transmitter
• Select symptoms from a menu.• System consults a rule database.• System can request user to perform actions.• System explains actions step by step using graphics and
text.
![Page 7: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/7.jpg)
Ariffin Yahaya 7
COMET is used in radio repair
Step 1: Remove the holding battery cover plate, highlighted in the right picture: Loosen the captive screws and pull the holding battery cover plate off the radio.
Remove the old holding Battery. Step 1 of 2.
Figure 2 Sample of COMET output.
![Page 8: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/8.jpg)
8
Comet Overview• Basic Components
– Knowledge Source• Contains databases for
information used in all the components.
– Content planner• “Answers” the user’s
request
– Media coordinator• Associates “answers” with
the best method of presentation.
– Presentation generator• Text generator.
• Graphics generator.
• Media layout
• Render & typeset
User request
“What” to say
“How” to say it.
Coordinateexplanationto medium.
Expert sys.determines
explanation’scontent.
Previousdiscourse
Usermodel
Diagnosticrule base
Staticobjects
GeometricKnowledge
Domainknowledge
Presentationgenerated.
Figure 3 : Comet Architecture & State Diagram
Content Planner
Media Coordinator Presentation generator
Knowledge Source
idle
![Page 9: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/9.jpg)
Ariffin Yahaya 9
Knowledge Source
• Static representation of the objects and actions.– Loom language.
• Declarative knowledge in Loom consists of definitions, rules, facts, and default rules.
• A deductive engine called a classifier utilizes forward-chaining, semantic unification and object-oriented truth maintenance technologies in order to compile the declarative knowledge into a network designed to efficiently support on-line deductive query processing
• Diagnostic Rule Base– Rules pertaining to the application.
• Detailed Geometric knowledgebase for graphics generation
![Page 10: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/10.jpg)
Ariffin Yahaya 10
Content Planner
• Produces full content for the explanations.– Represented as a hierarchy of logical forms.
• Logical forms (LF) are required as input to the next stage which uses functional unification grammar (FUF)*.
– Text Plans or Schemas• “Blackboard” capability.
– Intermediate results can be stored to see other results before committing.
– Previous discourse
* FUF not covered in this presentation
![Page 11: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/11.jpg)
Ariffin Yahaya 11
Media coordinator
• Fine grained analysis of an input LF to decide whether each portion should be realized in images, text or both.
• Uses FUF grammar that maps 6 different types of information to types of media.
• Passes the output to the presentation generators.– Graphics generator.– Text generator.
![Page 12: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/12.jpg)
Ariffin Yahaya 12
6 Different Types of Information
• Locations attributes.– Graphics alone
• Physical attributes.– Graphics alone
• Abstract actions.– Text alone
• Expressive connectives that indicate relationships among actions.– Text alone
• Simple actions.– Both Text and Graphics.
• Compound actions.– Both Text and Graphics.
![Page 13: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/13.jpg)
Ariffin Yahaya 13
Text Generator
• Type and Number of sentences needed.
• Lexical chooser.– LF actions verbs.– LF objects nouns.– Chooses words based on multiple constraints
• Wider variety and more appropriate output. Example:– Previous discourse: install reinstall– Use words user knows: technical term explain procedure
• Sentence generator.– Construct the syntactic structure.– Linearize the resulting tree as a sentence
![Page 14: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/14.jpg)
Ariffin Yahaya 14
Graphics Generator
• Uses IBIS (Intent based Illustration System) from the paper “Automated Generation of Intent-Based 3D Illustrations”.
• This paper is covered within this presentation set, so we will revisit IBIS later.
• Suffices to say that the graphics generator takes in annotated LFs and outputs graphics.
![Page 15: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/15.jpg)
Ariffin Yahaya 15
Media Coordination (1)
• Common content description language.– Text and Graphics influence each other.
• Generators display “cohesive” presentations.
– Communicative goal separated from resources.• LFs only specify goals and what is needed to achieve the
goals, generator defines resources.
– Provides mechanism for Text and Graphic generators to communicate.
• Content description (blackboard) is used to coordinate internal text structures with pictures.
![Page 16: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/16.jpg)
Ariffin Yahaya 16
Media Coordination (2)
• Bidirectional interaction– Certain types of coordination between media
can only be provided by incorporating interactive constrains between text and graphics.
• Coordinating sentence breaks with picture breaks.– Both Text and Graphics generators annotate their
current process and can refer to each others progress and decisions so that it can compensate.
• Cross-referencing text and graphics.– Text generator queries an IBIS database that is indexed
by LF so that it can cross-reference specific objects and refer to the graphical locations within the generated text.
![Page 17: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/17.jpg)
Ariffin Yahaya 17
COMET
Figure 4 COMET architecture from the Paper.
![Page 18: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/18.jpg)
Ariffin Yahaya 18
Automated Generation of Intent-Based 3D Illustrations
• Exact presentation of a message is available to us through technology, but many people viewing the same presentation may not lead to all the people having the same interpretation.
• The intention of the author, and the viewing context (i.e. who is viewing) must be taken into consideration.
• SO, how do we make sure that the presentation is appropriate?– IBIS (Intent Based Illustration System)
![Page 19: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/19.jpg)
Ariffin Yahaya 19
IBIS
• Goals:– Automate the creation of illustrations based on a
specific communicative intent.• Used by COMET
• Method– Formalize the intent and create an illustration that fulfills
the goals set by the intent.
• Example:– Communicative Intent: Show a dice
• Generated illustration: Whole dice is shown
– Communicative Intent: Show the weight in loaded dice• Generated illustration: Transparent dice with weight visible.
![Page 20: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/20.jpg)
Ariffin Yahaya 20
Communication Intent makes a difference!
Intent: Show the dice. Intent: Show the location of weights in loaded dice.
weight
Figure 5 Sample of IBIS output
![Page 21: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/21.jpg)
Ariffin Yahaya 21
Generate and Test• Every time IBIS generates a
stylistic choice, it also associates a set of criteria to which the results are compared to.– System of ratings (criteria).– Thresholds (minimum degree of
success).
• Stylistic choice are associated with methods, and as each method is “tried”, the results are tested against the criteria.
• If the criteria is not fulfilled, a new stylistic choice with a new method is requested. Figure 6 IBIS Generate and Test Cycles
![Page 22: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/22.jpg)
22
IBIS Overview• Basic components
– Communicative Goals• Tightly coupled with
COMET.
– Generate & Test cycle• Method used by IBIS to
evaluate appropriateness.
– Illustrator• Maps intent to stylistic
choice with “Design Rules”.
– Knowledgebase• Superset of graphics info.
– Drafter• Maps stylistic choice to
visual effect with “Style Rules.”
Communicative goals
Design R
ules
idle
Match designrules to
style rules.
Match goalsto design
rules.
Figure 7 : IBIS Architecture & State Diagram
Illustration
Style Strategies
Ineffecti
ve
Inef
fect
ive
Illustrator
Drafter
AbstractProperties
PhysicalProperties
Features
MaterialInformation
GeometricInformation
Knowledgebase
Style rules
Drafter
Gen
erat
e &
Tes
t Cyc
le
![Page 23: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/23.jpg)
Ariffin Yahaya 23
Communicative Goals• Location
– Show the location of an object in a context.• Relative Location
– Show the relative location of 2 or more objects in terms of a specified/derived context.
• Property– Show objects physical properties of material, color,
size or shape.• State*
– Show an object’s state• Change*
– Show the difference between a set of states.* State and change may further be qualified by concepts that refer to how the object is manipulated or has changed.
![Page 24: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/24.jpg)
Ariffin Yahaya 24
Illustrator
• Designs illustrations.– Map communicative goals to style strategies
with design methods.– Evaluate the success of communicative goals
with design evaluators.
• An illustrator may split jobs according to need and assign them to subordinate illustrators.
• Share design rules database.
![Page 25: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/25.jpg)
Ariffin Yahaya 25
Design Rules• Describes on a high level how illustrations
should be put together.– Communicative Goal– Set of Style Strategies.
• Visual effect• Style rules.
• Design Methods specify how to accomplish communication goals.– Specifies what style strategies must be achieved.
• Design Evaluators determine how well communicative goals have been accomplished.– Achievement ratings of a collection of style strategies.
![Page 26: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/26.jpg)
Ariffin Yahaya 26
Knowledgebase• Concerned with physical objects to be illustrated.• Superset of typical graphics databases
– Geometric Information– Material Information
• Also includes– Object’s Features
• What are the object’s capabilities
– Physical Properties• How does an object move (i.e. a hinge).
– Abstract Properties• How things fit together (?)
![Page 27: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/27.jpg)
Ariffin Yahaya 27
Drafter
• Knows nothing about communicative intent.• Translates the illustrator's plans into reality.• Tightly coupled with the hardware they utilize.• Shares a database of style rules.• Report back to the illustrators with the
achievement rating of the various style strategies they implement
• Render the illustrations.
![Page 28: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/28.jpg)
Ariffin Yahaya 28
Style Rules
• 2 types of Style Rules that specify either– Style Methods
• Accomplish visual effects specified by style strategies.
• Illustration methods.
– Style Evaluators• Determine the success of style strategies in a
given illustration.• Illustration evaluator
![Page 29: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/29.jpg)
Ariffin Yahaya 29
Illustrative Style
• Represented by an ordering of the rules such that the preferred methods are always attempted first.
• Illustrators and drafters can be specified with different illustrative styles.
• Illustrations can combine different illustrative styles.
![Page 30: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/30.jpg)
Ariffin Yahaya 30
Other IBIS Features
• Interactive Illustrations– User can change view specifications with IBIS
continuously monitoring to make sure that the communicative goal is maintained.
• Written in C++ and CLIPS.
![Page 31: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/31.jpg)
Ariffin Yahaya 31
Intent based approach to Authoring
Figure 8 Typical intent based authoring architecture.
![Page 32: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/32.jpg)
Ariffin Yahaya 32
Plan-based integration of natural language and graphics generation.
• Multimodal presentations should be generated from a common representation of what is to be conveyed.
• BUT– How do we decompose the communicative goal into
sub goals.– How do we Integrate multiple AI components to
create the presentation.• WIP multimodal presentation system was
created to be a prototype to solve these problems.– Computer as a “desktop publisher”.
![Page 33: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/33.jpg)
Ariffin Yahaya 33
WIP Design Goals
• Generate coordinated multimodal presentations from a common representation.– What should be in text / graphics.– Which kinds of links verbal / non-verbal is necessary.
• Adaptation of these presentations to intended audience and situations.– All presentation decisions are postponed until runtime
• Incrementality of all processes constituting the design and realization of the multimodal output.– computations for an object are performed not long
before the object is output.
![Page 34: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/34.jpg)
Ariffin Yahaya 34
WIP• Goal :
– Allows the generation of alternate presentations of the same content taking into account contextual factors such as the user’s degree of expertise and preferences for a particular output medium or mode.
– Specify information once, but view in infinite ways.
• Example:– How to use an espresso machine.– How to assemble a lawnmower.– How to install a modem
• Inputs– Stereotypes, Target language, Layout format, and
Output modes.
![Page 35: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/35.jpg)
Ariffin Yahaya 35
Interleaved Content Planning
• Processing is done non linearly.• Cascades are used based on some task/results
queues.• Cascade:
– Presentation planner and Layout Manager– Design module– Realization module
• Purpose:– Leave presentation decisions to the last possible
moment to refine the presentation.
![Page 36: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/36.jpg)
36
WIP Overview• Basic Components
– Presentation Planner• Decides on content and
mode combination
– Layout Manager• Screen/Output Manager• Last step to rendering
presentation
– Text & Graphics Cascade• Micro planner.
– Application Knowledge• Application specific data.
– Knowledgebase• Used internally.
Presentation goals
idle
Figure 9 : WIP Architecture & State Diagram
Presentation
Planner
Knowledgebase
Determinecontents
andmode
Determinelayoutand
Generate.
TextDesign
TextRealization
GraphicsRealization
GraphicsDesign
Layout
Manager
Generation Parameters
Incremental
Design Cascade
bus / queue
Graphics D.Strategies
TAG
BasicOntology
UserModel
SelectionRules
RevisionStrategies
PresentationStrategies
ApplicationKnowledge
![Page 37: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/37.jpg)
Ariffin Yahaya 37
Presentation Planner
• Tries to find a presentation strategy whose effect (or header) match the presentation goal.
• Keeps revising plan until some basic elements of the presentation are formed.
• Elements sent to the task queue.
• Design modules take task from the queue and begin processing.
![Page 38: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/38.jpg)
Ariffin Yahaya 38
Layout Manager
• In charge of the screen real estate.
• Positions design components.
• Interacts with the realization module.
![Page 39: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/39.jpg)
Ariffin Yahaya 39
Text & Graphics Cascade
• Design Module– Elementary Speech/Pictoral acts.– What to say (intent) micro-planner
• Which view of the espresso machine.• What is the micro-message (instruction).
• Realization Module
– How to say it micro-planner• Natural Language• Geometric shapes
![Page 40: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/40.jpg)
Ariffin Yahaya 40
Application Knowledge
• Externally coded in RAT
• Main source of knowledge– Domain terminology
• Used in– Presentation Planner– Generation of text– Generation of graphics
![Page 41: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/41.jpg)
Ariffin Yahaya 41
Knowledgebase
• Application Knowledge– Unique domain knowledge
• Strategies– Used to design/revise
• Presentation• Graphics
• User Model– Matches the generation parameters.
![Page 42: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/42.jpg)
Ariffin Yahaya 42
Comparison to COMET
• WIP– Operator based
approach to planning.– Supports
incrementality.– Bidirectional
communication between Presentation Planner and Layout Manager.
• COMET– Schema based
content planner– No increments
– Layout component combines text and graphics fragments during final steps.
![Page 43: Multimedia Presentation Design](https://reader035.fdocuments.in/reader035/viewer/2022062408/56813aad550346895da2a801/html5/thumbnails/43.jpg)
Ariffin Yahaya 43
Coming up next: Anders with the second half of this talk.
Figure 10 Ariffin is smiling ‘cos he’s done!