Warren Hayes Ai Symposium - working draft
-
Upload
margaret-warren -
Category
Technology
-
view
1.051 -
download
1
description
Transcript of Warren Hayes Ai Symposium - working draft
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Pat Hayes, Margaret Warren
A Lightweight Ontology for Linking Images
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
what we are/and are not doing
• images themselves are our main topic, rather than as illustrations of other topics, linking information to and from them and linking the images themselves
• Interested in using linked data to represent the kinds of things artists themselves say about their work (or the person doing the markup)
• What we don’t do: expect people to agree on artistic interpretations or emotional content of images
• ignore our paper in the proceedings – it was a working draft and we have made significant changes.
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
foaf:depiction
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Snapshot, line drawings, news photos, paintings, fine art
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
• By Image, we mean a digital image, typically one that can have associated EXIF data.
• So, other kinds of "images" (drawings, paintings, non-digital photographs, stained glass, etc..) have to be classified as Works.
• A work may be a physical thing with dimensions, a location, a time and place of creation, etc. that is separate from those same properties in an image; a digital image can be a work.
• Basic relation between Work and Image is facsimile, a kind of 'visually same As' relation.
Image identity
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Broader notion of depicts
All images depict the person: McKenzie Oerting
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Depiction can include imaginary, biblical, mythological notions
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
What is the subject?
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
shows, depicts, subject, conveys
Shows: trees, grass, house,
window, ventilator
Depicts: particular type of car,
each of the people
Subject: group looking at a car
Shows: head
Depicts: hands
Conveys: grief, sadness
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Do all images depict something?
Depict diamonds? Shapes? Or Nothing…
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Pictorial elements
Pictorial elements are related by inclusion:
Basic graphical 'unit' of depiction is a pictorial element.
This includes 'boxes' but also lines, regions, areas,
patches of color, texture, etc.. “
elements of art - The basic components used by the artist
when producing works of art. Those elements are color,
value, line, shape, form, texture, and space. The elements
of art are among the literal qualities found in any artwork.
(Artlex, 1996-2010)
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Pictorial Elements:Shape of thing that
looks like a
female torso
Shape of thing that looks like a
representation of a brain
area that looks likethe torso is
sitting astride a seat of some sort
Depicts vs. looksLike
Depicts: egg, a test tube, a photographic transparent layer, a piece of jewelry
Title: Surfing With Psyche
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Pictorial Elements can be 'empty'
A reclining nude is depicted by white space defined by an outline. The white space is a pictorial element.
Cataloging pictorial elements is a large task. As a beginning, we distinguish visible/implied and line/area. Here, the nude is depicted by an implied area. All the visible elements are red lines.
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
backgrounds
hasPictorialBackground
all white, not necessarily
able to define with a region
hasDepictedBackground
sky, clouds, not necessary
to define with a region
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Angels in the clouds
depicts:
Blue Angels ,jets,
planes, formation
conveys:
military power?
subject: specific formation?
hasDepictedBackground:
sky, clouds
region contains a shape
(PictorialElement) that
looksLike an Angel
(iconic)
depicts: clouds
depicts: angel
shows: clouds
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
The ontology so far
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
future work
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
http://www.purl.org/net/mw
http://www.carmapro.com/mw/mw.rdf
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Extra slides follow
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Looking like an archetype• Some shapes are best described by what they look like iconically.
• If we can use RDFS or OWL Full, this could be described as 'looking like' a class; but this will break OWL-DL reasoners.
• Or, we could treat a SKOS concept as an archetype, but this is not (yet) a widely accepted convention.
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
visiblePictorialElements
looksLike iconic heart
hasColor Red
looksLike iconic dancing figures
hasColor yellow orange red
hasPictorialBackground: white
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
What's in an image?
• We conventionally say that depicted things are "in" an image.
• Is this a philosophical mistake? Maybe. But rather than try to correct it, the ontology tries to avoid it.
• Basic methodological rule for linked data: don't try to repair "bad" intuitions.
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
shows: person, depicts: face in hands
subject: crying conveys: grief or despair, etc.
shows: broken headlight, turn signal
depicts: front fender of school bus
subject:
conveys: beauty in decay, decline of
education
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Can any Thing be depicted? Do we need a class of 'depictable things'?
Things depicted include
(keywords):
• the car (specific type)
• people (specific people)
• a group of 7 people
• a building
• trees
• brown grass
Subject: group studying a car?
one of the people?
the car itself?
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Adult, Adults Only, Agriculture, Beautiful People, Beauty, Beauty In Nature,
Biting, Branch, Color Image, Concepts, Custard Apple, Desire, Enjoyment,
Food, Freshness, Fruit, Green, Gripping, Growth, Harvesting, Healthy Eating,
Healthy Lifestyle, Horizontal, Human Face, Human Hand, Human Mouth,
Hungry, Leaf, Lifestyles, Meal, Nutrient, One Person, One Woman Only, One
Young Woman Only, Only Women, Open, Opening, Organic, People,
Photography, Picking, Plant, Southeast Asian Ethnicity, The Human Body,
Tree, Vegetarian Food, Women, Young Adult
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
What can be depicted?
• FOAF: People.
• DBPedia: People, places, events,…
• Umbel (Cyc): VisibleThings
• Visual Resources Association: myths, legends, intentions, …
• We provide a different property conveys for the relationship between an image and "un-depictable" things such as mood, idea, intention, emotion, etc..
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
http://www.gettyimages.com/Search/Search.aspx?query=z.i.H4
sIAAAAAAAEAO29B2AcSZYlJi9tynt_SvVK1-
B0oQiAYBMk2JBAEOzBiM3mkuwdaUcjKasqgcplVmVdZhZAz
O2dvPfee--
999577733ujudTif33_8_XGZkAWz2zkrayZ4hgKrIHz9-
fB8_In7dfLn91etf49f4NX6PX_dskV3kvyb9mtD_f9N5ns3S9fKqz
larYnnxa_7f9PwaG59f0_xM6qpqX2Z1tmjMZ78W_r-
78xvSj1_v3evr5vSdeePX9P7mltm5Bai_4-
VfOzuf2g78P35tfLDD8Bete3PR_rr6-2_xa3DX9tc99-s99-t99-s-
__rrUB-lhWf_Qne_Lv1RLyw24Z-_Pj7axV-_Hn8-
sTA6f_Nr5yEU-
ydD2bFQzjtQ3N_8Wm1e6_zJUPYslLoDxf1NBM0tDPMH_s_U
3QV1iQL5yr4f_PVr55PGfmP-
4BnJXY_e779W7pp7v__azWplP8cfBqPfyr54OXUvXk5_bf39t8
TfO_a3ffvbLn77dara4zb7F381yf2vzF94fu3Z1dL8_muaP8BSv0
6z9kYb_PVr55nDz_-Dfl_5X9g_ft0mgBb--etM83futeCvX_vds-
dOGugP-
0VWe2h7f_xGzbRa5U_Wy1npUT_41AD8ccB5sHMfv_86xGpr2
zz46_8Bs9pv0zoEAAA.
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Linked Image Data
• Linked data…
• Newest manifestation of the semantic web: use RDF to create a web of semantic links between data items.
• Meanings of links are described by lightweight (small, intuitive, useful) ontologies.
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Linked data, not data silos
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Linked data is link + description
• RDF descriptions use URIs as names to refer to things; linking means using those URIs to also connect to the described thing, or a source of more information about it.
• This can be smoothly integrated into HTML using RDFa.<foaf:knows> <foaf:Person rdf:about="http://danbri.org/foaf.rdf#danbri">
<foaf:mbox_sha1sum>362ce75324396f0aa2d3e5f1246f40bf3bb44401</foaf:mbox_sha1sum>
<foaf:name>Dan Brickley</foaf:name>
<rdfs:seeAlso rdf:resource="http://danbri.org/foaf.rdf"/>
</foaf:Person>
</foaf:knows>
<foaf:based_near rdf:resource="http://dbpedia.org/resource/Milton_Keynes"/>
<PersonalProfileDocument rdf:about="">
<cc:license rdf:resource="http://creativecommons.org/licenses/by-nc/3.0/"/>
<dc:title>Tim Berners-Lee's FOAF file</dc:title>
<maker rdf:resource="http://www.w3.org/People/Berners-Lee/card#i"/>
<primaryTopic rdf:resource="http://www.w3.org/People/Berners-Lee/card#i"/>
</PersonalProfileDocument>
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Linked data is link + description
• Description terms defined by lightweight ontologies.
• Designing lightweight ontologies is an art, not a science.
• Often, the formal ontologies are quite minimal, eg FOAF refuses to say anything more than the class People is all people, and Agent is "things that make stuff happen". Still, it does specify that People is an rdfs:subClass of Agent. In practice, meaning is specified by use (in the actual links).
• It is considered good form to re-use existing relations, classes and properties whenever possible. But this can set up some tension with the meaning-as-use observation.
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Linked data is here, now
Collectively, the data sets consist of over 13.1 billion RDF triples (dataset statistics),
which are interlinked by around 142 million RDF links (linksset statistics) (November 2009).
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Linked Image data?• There are many images in existing linked data, but almost always as illustrations of things
(people, places, buildings..), as in FOAF.
• We want to describe images as objects in their own right. Images are our topic. So we need an image-describing vocabulary with its own lightweight ontology.
Earlier work on image description
all focuses either on the things
depicted, or on the cultural setting
of the images, or on the curation
and cataloging of museum
collections.
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Very lightweight image ontology
Questions one might ask….
1. Can any Thing be depicted? Do we need a class of 'depictable things'?
2. Do all images depict something?
3. What exactly is depiction anyway?
4. What exactly is an Image? (Is it a file type, for example? Are all images
digital images?)
etc…
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
• Dense lines of particular colors, splatters, all pictorial elements
• No distinct background, nothing depicted
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Pictorial elements
Pictorial elements are related by inclusion:
Basic graphical 'unit' of depiction is a pictorial element.
This includes 'boxes' but also lines, regions, areas,
patches of color, texture, etc.. “
elements of art - The basic components used by the artist
when producing works of art. Those elements are color,
value, line, shape, form, texture, and space. The elements
of art are among the literal qualities found in any artwork.
(Artlex, 1996-2010)
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Ontology can be molded to fit
A depiction of the
torso of
a female mannequin
AND/OR
A Pictorial Element
that looks like a
female torso
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Depicting vs. looking like
• Simple depiction (as in FOAF) means something like "is a picture of". Our depiction is wider in scope, and does not imply visual similarity. An Xray image is a depiction, for example. So we also need the notion of looking like.
• This is also important for characterizing pictorial elements.
Depicted female
torso, x-ray
Depicted female
torso
looksLike
female torso*
looksLike
iconic female
torso
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
If an image can depict, then a PART of an image can depict. if a whole image can depict, so can part of an image.
Images are pictorial elements.
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Entire image is a pictorial element
that looks like a female torso
Entire image is a pictorial element
that depicts a female torso
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Depiction is in the eye of the …
depicts Marie-Therese Walter depicts female torso
3/23/2010© 2009 - 2010 Margaret Warren, Pat Hayes, CARMA Productions
Defining classes of image
• Can now define for example 'line drawing': an image all of whose component visible pictorial elements is a line.
<owl:Class rdf:ID="LineDrawing">
<owl:intersectionOf rdf:parseType="Collection">
<owl:Class rdf:about="#Image" />
<owl:Restriction>
<owl:onProperty rdf:resource="#hasVisualPart" />
<owl:allValuesFrom>
<owl:unionOf rdf:parseType="Collection">
<owl:Class rdf:about="#ImplicitPictorialElement" />
<owl:Class rdf:about="#LinearPictorialElement" />
</owl:unionOf>
</owl:allValuesFrom>
</owl:Restriction>
</owl:intersectionOf>
</owl:Class>