Dimensions of Media Object Compehensibility Lawrie Hunter Kochi University of Technology

Dimensions of Media Object Compehensibility

Lawrie HunterKochi University of Technologyhttp://www.core.kochi-tech.ac.jp/hunter/

Dimensions of Media Object Compehensibility

Lawrie HunterKochi University of Technologyhttp://www.core.kochi-tech.ac.jp/hunter/

KUT

Island of Shikoku

Kochi

Niigata

Osaka

A pattern language for MMC

Source of insight: language / language learning

Second language (L2) learning: a cognitive process?

Comprehension of partially acquired L2: revealing of the nature of text/media.

Language learning issues are germane to MMC.

Tempering: questions of significance and applicability for machine automation.

A pattern language for MMCInterventionauthor's structural model of content information (for second language learning materials)

Discussion of parameters of difficulty

Ground: related issues in second language learning materials

Exemplars‘considerate text’ ‘considerate multimedia’

Generating parameters of difficulty in media object comprehension

Backgroundwork towards a human-communication paradigm for the guidance of machines

Framethe new multidisciplinary approach of machine-mediated communication

Objectivedevelopment of a pattern language for that multidisciplinary approach to MMC

Focusfactors influencing the difficulty of comprehension of media objects

Question how media objects carry information.

L2 learning materials

must all be more immediately apparent to the learner than in the case of materials for L1 medium learning scenarios

The creation of second language (L2) learning materials demands document transparency:

1. document purpose2. document content3. target behavior4. target lexical items

The creation of second language (L2) learning materials demands document transparency:

L2 learning materials

1. document purpose2. document content3. target behavior4. target lexical items

work towards transparency is informed by

difficulty-related issues

difficulty-related issues inform

human interaction with info media

Earlier work: an EAP tool

*in a talk to KMI at the Open University

David Kolb* re using hypertext to present scholarly text:

"...the easiest ways of making a complex argument available in HTtend to move the text toward linear structures that do not take full advantage of the possibilities of linked text."


*in a talk to KMI at the Open University

David Kolb* re using hypertext to present scholarly text:

"...the easiest ways of making a complex argument available in HTtend to move the text toward linear structures that do not take full advantage of the possibilities of linked text."

"...what the HT can do is present the argument,but also use linkage and juxtaposition to make the reader’s engagement with the argumentmore creative, self-conscious, and self-critical."


Lawrie Hunter re using hypertext to present technical L2 text:

For the L2 reader, engagement can only be enhancedif the rhetorical and information structures are articulated.




What the HT can do for the NNR/W is tp present simultaneously the various faces of a research paper:

the rhetorical moves;the bits of structured information; the text; necessary glosses.

* NNR/W EAP = non-native reader/writer of English for Academic Purposes




What the HT can do for the NNR/W is to present simultaneously the various faces of a research paper:

the rhetorical moves;the bits of structured information; the text; necessary glosses.

And if the NNR/Ws design their personal interface,a negotiated pattern language of NNR/W EAP* will emerge.

* NNR/W EAP = non-native reader/writer of English for Academic Purposes

Arguably important direction

"Tomorrow's literacies... need to be process and systems literacies.”

-John Thackara,

In the Bubble: Designing in a complex world.MIT Press 2005.

Rhetoricalstructures

Knowledgestructures

Cohesiondevices

Grammar(sentence surface structure)

Background

Extension

DiversionsTrain of argument

This is thedomain oftexturedown here.

This is thedomain ofstructuresup here.

Rhetorical structure theory,systemic functional linguistics and knowledge structure mapping form a hierarchy of structures, whereas grammar and sentence diagrams reflect rules for texture management.

Functionalstructures

Structural view of writing


Knowledgestructures

Cohesiondevices


Background

Extension


Falsehierarchy:the trainstops here.






L2 reader needs analysis

Knowledge

Niche grammar structures

Niche rhetorical structures

General register repertoires

(distinguishing formal academic from

informal academic)

Research Paper text structure and

information structure

Language skills

Argument sequencing

Info-structuredsentence generation

Mimicry of model language

Facilities

Concordance & collocation resource

Bank of modelresearch papers

(annotated*)

*c.f. Brown and Brown’s ‘annotation’

L2 reader wants analysis

In a technical hypertext, L2 reader/writers want*:

1. Glossing (of 'difficult' terms and phrases)

2. Moves indicator

3. Lexia position indicator

4. PDF-drawer-like phrase recurrence tab

5. Register converter

(e.g. research paper <=> presentation script)

6. Information structure maps for atomic utterances

7. Overall argument map on every lexia

(similar to Horn's argument maps

or Rhetorical Structure Analysis?)

*Based on a survey of 22 PhD engineering students

Technical hypertext design:

WANTS

NEEDS

A pattern language?www.patternlanguage.com

Technical hypertext design:

WANTS

NEEDS

…The language, and the processes which stem from it, merely release the fundamental order which is native to us. They do not teach us, they only remind us of what we know already, and of what we shall discover time and time again, when we give up our ideas and opinions, and do exactly what emerges from ourselves.

-Christopher Alexander, The Timeless Way of Building

A pattern language?www.patternlanguage.com

Do humans have aGRAPHIC THOUGHT FACILITY?

The knowledge structure map is a matrix (confluence) for the situated learner* and the situated mentor to confirm context and the nature of "stolen property."**

*Jean Lave**Duguid and Brown

<$$$

!

Hunter’s knowledge structure map links

<big

Description Classification

Degreecomparison

Attributecomparison

Sequence Cause-effect

Contrast

!

2005 project: design level

EEAP* students: HT designsfor the analysis of technical academic papers.

*EEAP = Engineering English for Academic Purposes,a subset of EAP,which is a subset of ESP (English for Specific Purposes)

Hunter L. (2005) Technical Hypertext Accessibility: Information Structures and Rhetorical Framing. Presentation at HyperText 2005, Salzburg. http://www.lawriehunter.com/presns/%20HT05poster0818.htm

http://www.lawriehunter.com/presns/%20HT05poster0818.htm

TEXT STRUCTURE

Introduction

Background

Question

Methods andmaterials

Results

Observations

Conclusion

INFOMAP(s) INFOSTRUCTURE

Describe

Classify

Compare

Sequence

Cause-effect

Contrast

UTTERANCE(s)

In general, power plants boil some liquid to make steam, which rotates turbines, which generate electricity.

Power plants boil a liquid to produce steam, which is used to rotate turbines, which in turn generate electricity.

RHETORICALMOVES

Commonknowledge

CiteReportExplain

Claim

Question

Qualify

Evaluate

DecideInfer

Project

TEXT STRUCTURE

Introduction

Background

Question


Results

Observations

Conclusion


Describe

Classify

Compare

Sequence

Cause-effect

Contrast

UTTERANCE(s)

Traditional power plants use fossil fuel heat or heat from nuclear fission to boil water and produce steam at 500°C.

Older type power plants boil water with heat from fossil fuel combustion or nuclear fission to produce steam with a temperature of 500°C.

RHETORICALMOVES

Commonknowledge

CiteReportExplain

Claim

Question

Qualify

Evaluate

DecideInfer

Project

TEXT STRUCTURE

Introduction

Background

Question


Results

Observations

Conclusion


Describe

Classify

Compare

Sequence

Cause-effect

Contrast

UTTERANCE(s)

OTEC power plants use seawater heat to boil ammonia and produce steam at 20°C.

OTEC type power plants boil ammonia with the heat of the sea to produce steam with a temperature of 20°C.

RHETORICALMOVES

Commonknowledge

CiteReportExplain

Claim

Question

Qualify

Evaluate

DecideInfer

Project

RHETORICALMOVES

Commonknowledge

CiteReportExplain

Claim

Question

Qualify

Evaluate

DecideInfer

Project

TEXT STRUCTURE

Introduction

Background

Question


Results

Observations

Conclusion


Describe

Classify

Compare

Sequence

Cause-effect

Contrast

UTTERANCE(s)

Traditional power plants use fossil fuel heat or heat from nuclear fission to boil water and produce steam at 500°C, whereas OTEC type power plants boil ammonia using the heat of the sea to produce steam with a temperature of 20°C.

Older type power plants boil water with heat from fossil fuel combustion or nuclear fission to produce steam with a temperature of 500°C, while OTEC power plants use seawater heat to boil ammonia and produce steam at 20°C.

Obstacle in 2005 projectMassive diversity in learner perception of knowledge structures.


Knowledgestructures

Cohesiondevices


Background

Extension


Falsehierarchy:the trainstops here.







Grammar

stagingInformation orchestration

Rhetoric, flow

Sentence levelPrescriptive order charts (linear);

sentence diagrams

Knowledge structure maps

Topic/stress and subject-verb distance

gizmos

Paragraph level Readability chartsKnowledge structure maps

Old/new and topic/stress

gizmos

Document levelReadability outlines

Knowledge structure maps

Old/new and topic/stress

gizmos

2006~ new layer: READABILITY

The missing link in technical academic writing:

Gopen’s readability-subject-verb distance-topic position / stress position-old/new information placement

Background: readability work

In the design of traditional high-text language learning materials, readability is a prominent concern.Reading difficulty has for some time been seen as depending on

-word length-sentence length-text length-number of sentences per paragraph-vocabulary ‘difficulty’

More recent work has extended this list to include -subject-verb distance -adherence to old/new position conventions-topic position/stress position conventions

Treated extensively inHunter L. (1998) Text Nouveau: Visible Structure in Text Presentation. Computer Assisted Language Learning 11(4) pp. 363-379.

Background: MM readability

Treated extensively inHunter L. (1998) Text Nouveau: Visible Structure in Text Presentation. Computer Assisted Language Learning 11(4) pp. 363-379.

Chun, D. M. and Plass, J. L. 1997.

Research on text comprehension in multimedia environments.

Language learning and technology 1(1): 60-81.

2006~ new layer: READABILITY

Hunter’s newTAW syllabus:assume grammar

Page

1 Readability and cohesion

Topic / stress positions Old / new information Subject-verb separation Logic gaps Ambiguity

2 Usage Dictionaries, guides, corpus and concordance

3 Registers Formal academic Informal academic Casual

4 Abstracts and introductions

The structure of a paper Outlining Summarizing

5 Organization of information

Situation-problem-solution-evaluation General-Specific

6

Information structures, information mapping

Description Classification Comparison, including pie and bar graphs Sequence, including line and bar graphs Cause-Effect Inference (deduction/induction) Pro and Con

7 Rhetoric vs. information

Background information / new content

8 English models

The Style Dossier: model language selection / evaluation Mimicry skills Plagiarism avoidance

9 Data commentaries

10 Appendix: language features

TAW -related grammar points Usage points

2006~ new layer: READABILITY Page

1 Readability and cohesion

Topic / stress positions Old / new information Subject-verb separation Logic gaps Ambiguity

2 Usage Dictionaries, guides, corpus and concordance

3 Registers Formal academic Informal academic Casual

4 Abstracts and introductions

The structure of a paper Outlining Summarizing

5 Organization of information

Situation-problem-solution-evaluation General-Specific

6

Information structures, information mapping

Description Classification Comparison, including pie and bar graphs Sequence, including line and bar graphs Cause-Effect Inference (deduction/induction) Pro and Con

7 Rhetoric vs. information

Background information / new content

8 English models

The Style Dossier: model language selection / evaluation Mimicry skills Plagiarism avoidance

9 Data commentaries

10 Appendix: language features

TAW -related grammar points Usage points

Textural Structural

Grammar Lexical patterns

Register Knowledge structures

Cohesion Coherence/readability

Functional grammar Information organization

Rhetorical device Rhetorical structure

Readability

The creation of second language (L2) learning materials demands appropriate readability.

1. understandable by the learner2. ‘stretching’ learner knowledge/skill3. contextualized to support stretching4. orchestrated with degrees of scaffolding

Considerate text

Original framing:･ well-written,･ well-organized, and･ signals the organization of its thought to the reader

One inroad to readability is considerate text:

Considerate text

Original framing:･ well-written,･ well-organized, and･ signals the organization of its thought to the reader

One inroad to readability is considerate text:

More recent takes:-glossing-phrase boundary marking-de-idiomatizing-the Plain English movement-graphic organizers -text nouveau

Text nouveau is still text

Text comprehension in multimedia environments is a rich variant, BUT :

Chun, D. M. and Plass, J. L. 1997.

Research on text comprehension in multimedia environments.

Language learning and technology 1(1): 60-81.

Sharing considerate text

Appropriateness of learning materials/tasks is very complex.Tagging of these materials & tasks is daunting.

L2 learning objects: welcome to the TagTower of Babel!

KUT English is a Moodle department.

Sharing considerate text

Appropriateness of learning materials/tasks is very complex.Tagging of these materials & tasks is daunting.

Fortunately, as David Weinberger points out*,there is a huge amount of metadata out there,but this allows multiple simultaneous organizations of content.

*June 12, 2007 interview with IT Conversationshttp://www.itconversations.com/shows/detail1838.html

Weinberger booksThe Cluetrain ManifestoSmall Pieces Loosely JoinedEverything is Miscellaneous

http://users.aber.ac.uk/lss/Soldatova.pdf

Considerate multimedia?

Tentative definition:considerate multimodal objects are those which contain few non-essential obstacles to their comprehension.

Considerate text in the context of M3C suggeststhe notion of considerate multimedia


Tentative definition:considerate multimodal objects are those which contain few non-essential obstacles to their comprehension.

Considerate text in the context of M3C suggeststhe notion of considerate multimedia

Tentative definition 2:considerate multimodal objects are those which are tagged for various forms of comprehension difficulty.

Multimedia comprehensibility?

Alternative approach: create a set of parameters for multimedia comprehensibility

“Considerate multimedia” confronts vastly more complexitythan considerate text

One approach to comprehensibility:explore obstacles to comprehensibility,as has been done in readability work.

In the domain of multimodal computer-mediated communication, the question of readability translates as ease of comprehension:

Multimodal equivalent of readability

How easy is it for a humanto extract all the information contained in a multimodal media object (MMO)?

To measure the ease of extraction of all the INTENDED information contained in a MMO, we need a characterization of the difficulty of extraction:

Tentativelist of sources of information extraction difficulty, for simplicity’s sake limited here to

text objectsgraphic objects speech objectsvideo objects and combinations thereof.

Parameters of media object function

This is a tentative, exploratory framing of MMO comprehensibility,

Parameter Instance/unit

concept density exophoric references per paragraph/page/frame

metaphor density metaphors per scene/argument/minute

phoneme density phonemes per unbroken utterance* (e.g. Italian speech)

phonemes per inhalation

phonemes per word

phonemes per minute

mathematical symbol density numerals per page

numerals per sentence

formulae per sentence/paragraph

formulae per argument

noise density superfluous signals per utterance, e.g. "...in 1960, oh, sorry, I meant to say in 1960...)

readability stoppages** per sentence

asides per sentence/message

cognitive dissonances per utterance

facial expression/statement conflicts

reference transparency anchoring devices per lexia

anchoring devices per reference

channel-channel synchronicity number of channel-channel synchronicities

number of channel-channel asynchronicities

message-message agreement number of message-message agreements

number of message-message dischords

*utterance: minimal spoken, written or graphical communication unit

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on Learning object tag ‘shopping list’

Concept density* in text spaceConcept density* in aural timeConcept density* in video space

*concept density = number of exophoric referencesper sentence/minute/frame

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on Concept density

Metaphors per sentence.Metaphors per argument.Metaphors per minute.

Idioms per sentence.Idioms per argument.Idioms per minute.

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on Metaphor density

Phonemes per unbroken utterance*.Phonemes per exhalation.Phonemes per word.Phonemes per minute.

*e.g. Italian speech.

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on Phoneme density

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on

Superfluous signals/utteranceReadability ‘stoppages’ per sentenceAsides per sentence/messageCognitive dissonances per utteranceFacial expression-statement conflicts

‘Noise’ density

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on

Numerals per page.Numerals per sentence.Formulae per sentence/paragraph.Formulae per rhetorical move.

Symbol density

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on

Imperfect audio channelImperfect text channelImperfect visual channel

Channel imperfections

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on Channel imperfections

Nass and Brave, Wired for speechReeves and Nass, The media equation

Finding:humans retain more infofrom video with animperfect audio channel

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on Channel-channel discord

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on Channel-channel synchronicity

Para

mete

rs o

f m

ed

ia o

bje

ct

fun

cti

on

Message-message agreementsMessage-message discords

Message-message harmony

Both involve reduced text density and spatial highlighting of text,

and suggest the question of a 'graphic thought facility' in humans.

manga knowledge structure maps

Low phoneme density Low phoneme density

Isolated conversational text chunks: X idioms per sentence.

Isolated descriptive text chunks:

0 idioms per sentence

X metaphors per utterance 0 metaphors per utterance

X idioms per utterance 0 idioms per utterance

Graphical situating: narrative/moodGraphical situating:

symbolized relations to other text chunks

manga vs. knowledge structure maps

To illustrate the use of the parameter approach, here is a comparison of two relatively similar types of media objects, manga and knowledge structure maps. Both involve reduced text density and spatial highlighting of text, and suggest the question of a 'graphic thought facility' in humans.

Once a comprehensive set of parameters of MMO comprehensibility has been developed, questions of application will arise.

How can (should?) these parameters be situated among larger semantic frameworks?

Which of these parameters are relevant to the development of machine-mediated communication?

How can they be operationalized in computable form?

Tempering: questions of significance and applicability for machine automation

Generating parameters of difficulty in media object comprehension

Work on ontology-based research writing * :reforming how scientific research is written/read.

EXPO* and the Robot Scientist

Does the ontology EXPO feed backfrom a machine interface with a body of knowledge/practiceto a solidification of human interface with that body of knowledge/practice?

Daunting: ontology-based readability

EXPO: An Ontology of Scientific Research. Ross D. King & Larisa N. Soldatovahttp://www-tsujii.is.s.u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf

http://www-tsujii.is.s.u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf


“Use of Natural Language is a great hindrance when using computers to store and analyse data hence the growing importance of text-mining. We argue that the content of scientific papers should increasingly be expressed in formal languages. Is writing a scientific paper closer to writing poetry or a computer program?”



http://www.itconversations.com/shows/detail1838.html


Can humans now experience knowledge differently, thanks to machine interface work,i.e. through a formal language imposed for the machine’s sake?

Will this reform how we read? how we think?



http://www-tsujii.is.s.u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf

References[1] Elsayed, A. (2007) Machine-mediated communication: the technology. 6th IEEE International

Conference on Advanced Learning Technologies, ICALT 2006, 5-7 July 2006, Kerkrade, The Netherlands.

[2] Hunter, L. (2005) Technical hypertext accessibility: information structures and rhetorical framing. Proceedings of the sixteenth ACM conference on Hypertext and hypermedia, Salzburg, Austria.

[3] Kalyuga, S. (2006) Instructing and testing advanced learners: A cognitive approach. Nova Science Publishers.

[4] Mann, B. (1999) An introduction to rhetorical structure theory (RST).

http://www.sil.org/mannb/rst/rintro99.htm

[5] Mohan, B.A.M. (1986) Language and content. Reading, MASS: Addison-Wesley.

[6] Nass, C. and S. Brave. (2005) Wired for speech: How voice activates and advances the human-computer relationship. MIT Press.

Chun, D. M. and Plass, J. L. 1997. Research on text comprehension in multimedia environments. Language learning and technology 1(1): 60-81.

Grow, G. (1996) Serving the strategic reader: cognitive reading theoryand its implications for the teaching of writing. Viewed June 30, 2007 at http://www.longleaf.net/ggrow/StrategicReader/index.html

Goldman, S.R., & Rakestraw, J.A. (2000). Structural aspects of constructing meaning from text. In M.L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. II, pp. 311-335). Mahwah, NJ: Erlbaum.

The Plain English movement http://www.plainenglish.co.uk/index.htm (de-idiomatizing)

http://www.itconversations.com/shows/detail1838.html

References 2Research via ontologies

Ian Horrocks http://www.cs.man.ac.uk/~horrocks/

EXPO Ontology of scientific experiments http://expo.sourceforge.net/

Soldatova L.N., Clare A., Sparkes A. and King, R.D. (2006) An ontology for a Robot Scientist. Bioinformatics (Special issue ISMB) (in press).

Soldatova, LN & King, RD. (2006) An Ontology of Scientific Experiments. Journal of the Royal Society Interface (in press).

EXPO: An Ontology of Scientific Research by Ross D. King & Larisa N. Soldatova, Department of Computer Science, University of Wales, Aberystwyth.

Hunter

Hunter L. (2005) Technical Hypertext Accessibility: Information Structures and Rhetorical Framing. Presentation at HyperText 2005, Salzburg. http://www.lawriehunter.com/presns/%20HT05poster0818.htm

Text Nouveau: Visible Structure in Text Presentation. Computer Assisted Language Learning 11(4) pp. 363-379. (text nouveau)

WordbyWord http://www.core.kochi-tech.ac.jp/hunter/WordByWord/index.html (text nouveau)

Text usability for non-native readers of English. Ueta, R, Hunter, L. & Ren, X.Proceedings, Information Processing Society of Japan, Vol. 2003.7. Pp. 199-200. (phrase boundary marking)

Thank you for your kind attention.

Don’t hesitate to write to me.

Lawrie HunterKochi University of Technology

http://www.core.kochi-tech.ac.jp/hunter

Dimensions of Media Object Compehensibility Lawrie Hunter Kochi University of Technology

Documents

Transcript of Dimensions of Media Object Compehensibility Lawrie Hunter Kochi University of Technology