The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating...
Transcript of The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating...
![Page 1: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/1.jpg)
Marc Schröder, DFKI
eNTERFACE 2010, Amsterdam
The SEMAINE API:The SEMAINE API:An open-source research platform for An open-source research platform for
multimodal, emotion-oriented multimodal, emotion-oriented interactive systemsinteractive systems
![Page 2: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/2.jpg)
2Marc Schröder, DFKI
OutlineOutline
The SEMAINE project
The SEMAINE API
Motivation
A component integration framework
API architecture
Markup interfaces
Building new emotion-oriented systems with the SEMAINE API
combining existing and new modules
Overview of Tutorial
![Page 3: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/3.jpg)
3Marc Schröder, DFKI
The SEMAINE projectThe SEMAINE project
“Sustained Emotionally coloured Machine-human Interaction using Nonverbal Expression”
http://www.semaine-project.eu
FP7 project, Jan. 2008 – Dec. 2010
Aim: build an autonomous, responsiveSensitive Artificial Listener (SAL) system
real-time multimodal interaction
strong on non-verbal skills
cut down on verbal skills for now
conversation: chat, not task-oriented
![Page 4: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/4.jpg)
4Marc Schröder, DFKI
The SEMAINE project: PartnersThe SEMAINE project: Partners
Coordinator: DFKI Saarbrücken (M. Schröder)system integration, speech synthesis
Queen's University Belfast (R. Cowie)data collection, evaluation
Imperial College, London (M. Pantic)facial analysis
University of Twente (D. Heylen)dialogue modelling
CNRS (C. Pelachaud)ECA system
TU München (B. Schuller)ASR, vocal emotion recognition
![Page 5: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/5.jpg)
5Marc Schröder, DFKI
The SEMAINE systemThe SEMAINE system
![Page 6: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/6.jpg)
6Marc Schröder, DFKI
The SEMAINE systemThe SEMAINE system
![Page 7: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/7.jpg)
7Marc Schröder, DFKI
The SEMAINE system: ComponentsThe SEMAINE system: Components
Facial analysis (Windows / Mac, C++)
face detector, nod/shake detector
Voice analysis (Linux / Mac, C++)
feature extraction, keyword spotting, emotion rec.
Verbal utterance planning (Java)
interpret user behaviour, plan agent behaviour
Listener behaviour (Windows, C++)
generate backchannels + feedback
Speech synthesis (Java)
Visual behaviour realisation (Windows, C++)
![Page 8: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/8.jpg)
8Marc Schröder, DFKI
Emotion-oriented interactive systemsEmotion-oriented interactive systems
Emotion-enabled interactive characters
MAX
FearNot: learning anti-bullying strategies
IDEAS4Games
…
Other emotion-related functionality
AffectiveDiary
eMoto: Pen for affective text messages
...
![Page 9: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/9.jpg)
9Marc Schröder, DFKI
Current state: island solutionsCurrent state: island solutions
Research systems tend to create ad hoc systems with domain-specific representations
Difficult to reuse
Difficult to combine parts of existing systems to implement new functionality
![Page 10: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/10.jpg)
10Marc Schröder, DFKI
The benefit of standardsThe benefit of standards
Standards are taken for granted in many parts of our life
electric voltage (220 V throughout Europe)
fuel grade (95 Octane)
screw threads
...
more recently: Document formats (ODF)
Users can rely on guaranteed properties without worrying who provides the serviceand how it is implemented
supports interoperability and reuse
![Page 11: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/11.jpg)
11Marc Schröder, DFKI
The SEMAINE APIThe SEMAINE API
Integrate components across languages and operating systems: Middleware!
Message-oriented middleware (MOM)
asynchronoussender doesn't need to wait for receiver
publish-subscribe to “Topics”flexible n-to-m message passing
currently using Apache ActiveMQ, a JMS serveropen source, Java/C++, reasonably fast
SEMAINE API is a component integration layer on top of the MOM
![Page 12: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/12.jpg)
12Marc Schröder, DFKI
““Reasonably fast”: ActiveMQ vs. PsycloneReasonably fast”: ActiveMQ vs. Psyclone
ActiveMQ
0.3 – 55 ms
Psyclone
16 – 408 ms
ActiveMQ between 10 and 50 times faster than Psyclone
10 100
1,000
10,00
0
100,0
00
1,000
,000
0
20
40
60
80
100
PsycloneActiveMQ
String length
Mill
ise
cond
s
Round-trip message routing times on localhost
![Page 13: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/13.jpg)
13Marc Schröder, DFKI
The SEMAINE API: System architectureThe SEMAINE API: System architecture
message-orientedmiddleware
systemmanager
semaine.meta
semaine.log.*
semaine.data.* metamessenger
component
metamessenger
componentmessagelogger
log reader
...
system monitorGUI
system monitorGUI
semaine.callback.*
![Page 14: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/14.jpg)
14Marc Schröder, DFKI
The SEMAINE API: System monitorThe SEMAINE API: System monitor
![Page 15: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/15.jpg)
15Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
SemaineML
feature vectors
FML
BML w/SSML and MaryXML
audio data, FAP, BAP
Action selectionAction proposersAction proposersAction proposers
user state agent statedialog state
Action proposersAction proposersInterpreters
features
Feature extractors
Analysers
candidateaction
Behaviour planner
action
Behaviour realiser
behaviourplan
Player
behaviourdata
EMMA w/EmotionML
or BML
![Page 16: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/16.jpg)
16Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
EMMA: W3C Extensible Multimodal Annotation markup language
container format for user behaviour analysis
EMMA transporting EmotionML:<emma:emma xmlns:emma="http://www.w3.org/2003/04/emma" version="1.0"> <emma:interpretation emma:start="123456789"> <emotion xmlns="http://www.w3.org/2005/Incubator/emotion"> <dimensions set="valenceArousalPotency"> <arousal value="-0.29"/> <valence value="-0.22"/> </dimensions> </emotion> </emma:interpretation></emma:emma>
![Page 17: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/17.jpg)
17Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
EmotionML
W3C working draft
Representation of emotions informed by affective sciences
categories, dimensions, appraisals, action tendencies
links to triggers, objects, expressive behaviour
customizable: choose vocabulary to use
![Page 18: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/18.jpg)
18Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
EMMA: W3C Extensible Multimodal Annotation markup language
container format for user behaviour analysis
EMMA transporting BML:<emma:emma xmlns:emma="http://www.w3.org/2003/04/emma" version="1.0"> <emma:interpretation emma:start="123456789"> <bml:head type=”NOD” xmlns:bml=”http://www.mindmakers.org/projects/BML”/> </emma:interpretation></emma:emma>
![Page 19: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/19.jpg)
19Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
SemaineML
custom format for domain-specific annotations
SemaineML representing dialogue state:<dialog-state xmlns="http://www.semaine-project.eu/semaineml" version="0.0.1"> <speaker who="agent"/> <listener who="user"/></dialog-state>
![Page 20: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/20.jpg)
20Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
BML: Behaviour Markup Language
drive ECA behaviour
speech, face, and gesture
first, symbolic representation of AV synchronization
TTS adds detailed timings
![Page 21: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/21.jpg)
21Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
BML with symbolic time markers:<bml xmlns="http://www.mindmakers.org/projects/BML" id="bml1"> <speech id="s1" language="en-US" text="Hi, I'm Poppy." ssml:xmlns="http://www.w3.org/2001/10/synthesis"> <ssml:mark name="s1:tm1"/> Hi, <ssml:mark name="s1:tm2"/> I'm <ssml:mark name="s1:tm3"/> Poppy. <ssml:mark name="s1:tm4"/> <pitchaccent id="xpa1" start="s1:tm1" end="s1:tm2"/> <pitchaccent id="xpa2" start="s1:tm3" end="s1:tm4"/> <boundary id="b1" time="s1:tm4"/> </speech> <gaze id="g1" start="s1:tm1" end="s1:tm4"> ... </gaze> <head id="h1" start="s1:tm3" end="s1:tm4" type="NOD"> ... </head></bml>
![Page 22: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/22.jpg)
22Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
BML with symbolic time markers:<bml xmlns="http://www.mindmakers.org/projects/BML" id="bml1"> <speech id="s1" language="en-US" text="Hi, I'm Poppy." ssml:xmlns="http://www.w3.org/2001/10/synthesis"> <ssml:mark name="s1:tm1"/> Hi, <ssml:mark name="s1:tm2"/> I'm <ssml:mark name="s1:tm3"/> Poppy. <ssml:mark name="s1:tm4"/> <pitchaccent id="xpa1" start="s1:tm1" end="s1:tm2"/> <pitchaccent id="xpa2" start="s1:tm3" end="s1:tm4"/> <boundary id="b1" time="s1:tm4"/> </speech> <gaze id="g1" start="s1:tm1" end="s1:tm4"> ... </gaze> <head id="h1" start="s1:tm3" end="s1:tm4" type="NOD"> ... </head></bml>
issue with XML namespacesin current BML draft
![Page 23: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/23.jpg)
23Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
BML with symbolic time markers:<bml xmlns="http://www.mindmakers.org/projects/BML" id="bml1"> <speech id="s1" language="en-US" text="Hi, I'm Poppy." ssml:xmlns="http://www.w3.org/2001/10/synthesis"> <ssml:mark name="s1:tm1"/> Hi, <ssml:mark name="s1:tm2"/> I'm <ssml:mark name="s1:tm3"/> Poppy. <ssml:mark name="s1:tm4"/> <pitchaccent id="xpa1" start="s1:tm1" end="s1:tm2"/> <pitchaccent id="xpa2" start="s1:tm3" end="s1:tm4"/> <boundary id="b1" time="s1:tm4"/> </speech> <gaze id="g1" start="s1:tm1" end="s1:tm4"> ... </gaze> <head id="h1" start="s1:tm3" end="s1:tm4" type="NOD"> ... </head></bml>
non-standard extension
![Page 24: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/24.jpg)
24Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
BML with phone timing for lip synchronization:<bml xmlns="http://www.mindmakers.org/projects/BML" id="bml1"> <speech id="s1" language="en_US" text="Hi, I'm Poppy." ssml:xmlns="http://www.w3.org/2001/10/synthesis" mary:xmlns="http://mary.dfki.de/2002/MaryXML"> ... <ssml:mark name="s1:tm3"/> Poppy. <mary:syllable stress="1"> <mary:ph d="0.092" end="1.011" p="p"/> <mary:ph d="0.112" end="1.123" p="A"/> <mary:ph d="0.093" end="1.216" p="p"/> </mary:syllable> <mary:syllable> <mary:ph d="0.141" end="1.357" p="i"/> </mary:syllable> <ssml:mark name="s1:tm4"/> ...</bml>
![Page 25: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/25.jpg)
25Marc Schröder, DFKI
Representation formats in the SEMAINE APIRepresentation formats in the SEMAINE API
Some representations available from standards or standards-in-the-making
EMMA, EmotionML, BML, SSML
Others still missing
FML, dialogue state, …
Implementation provides “reality check” for a specification
feedback from implementation can improve spec
![Page 26: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/26.jpg)
26Marc Schröder, DFKI
Building new emotion-oriented systemsBuilding new emotion-oriented systems
SEMAINE API simplifies the integration task
abstract away from operating system,programming language and communication issues
use standard representation formats where possible
provide suitable support for XML handling
Three example systems:
Hello world
Emotion mirror
The swimmer's game
![Page 27: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/27.jpg)
27Marc Schröder, DFKI
1. Emotional Hello world1. Emotional Hello world
Minimal system:
text input component
dummy text analysis to infer emotional state
emoticon output
Code is short – about 20 lines each
![Page 28: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/28.jpg)
28Marc Schröder, DFKI
1. Emotional Hello world1. Emotional Hello world
Valence- 0 +
Arousal + 8-( 8-| 8-)
0 :-( :-| :-)- *-( *-| *-)
EmotionML
![Page 29: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/29.jpg)
29Marc Schröder, DFKI
1. Emotional Hello world1. Emotional Hello world
1 public class HelloAnalyser extends Component { 2 3 private XMLSender emotionSender = new XMLSender("semaine.data.hello.emotion", "EmotionML", getName()); 4 5 public HelloAnalyser() throws JMSException { 6 super("HelloAnalyser"); 7 receivers.add(new Receiver("semaine.data.hello.text")); 8 senders.add(emotionSender); 9 }10 11 @Override protected void react(SEMAINEMessage m) throws JMSException {12 int arousalValue = 0, valenceValue = 0;13 String input = m.getText();14 if (input.contains("very")) arousalValue = 1;15 else if (input.contains("a bit")) arousalValue = -1;16 if (input.contains("happy")) valenceValue = 1;17 else if (input.contains("sad")) valenceValue = -1;18 Document emotionML = createEmotionML(arousalValue, valenceValue);19 emotionSender.sendXML(emotionML, meta.getTime());20 }
![Page 30: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/30.jpg)
30Marc Schröder, DFKI
2. Emotion mirror2. Emotion mirror
Mimick user emotion
Infer user emotion from user speech
Represent inferred emotion as an emoticon
Benefit of reuse: need to implement only one new component
Reuse openSMILE emotion recognition from SEMAINE system
Reuse EmoticonOutput from Hello World example
Implement simple “decision” component that extracts emotion judgements from recognition output
![Page 31: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/31.jpg)
31Marc Schröder, DFKI
2. Emotion mirror2. Emotion mirror
![Page 32: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/32.jpg)
32Marc Schröder, DFKI
2. Emotion mirror2. Emotion mirror
1 public class EmotionExtractor extends Component { 2 private XMLSender emotionSender =
new XMLSender("semaine.data.hello.emotion", "EmotionML", getName()); 3 4 public EmotionExtractor() throws JMSException { 5 super("EmotionExtractor"); 6 receivers.add(new EmmaReceiver("semaine.data.state.user.emma")); 7 senders.add(emotionSender); 8 } 9 10 @Override protected void react(SEMAINEMessage m) throws JMSException {11 SEMAINEEmmaMessage emmaMessage = (SEMAINEEmmaMessage) m;12 Element interpretation = emmaMessage.getTopLevelInterpretation();13 List<Element> emotionElements = emmaMessage.getEmotionElements(interpretation);14 if (emotionElements.size() > 0) {15 Element emotion = emotionElements.get(0);16 Document emotionML = XMLTool.newDocument(EmotionML.ROOT_ELEMENT,
EmotionML.namespaceURI);17 emotionML.adoptNode(emotion);18 emotionML.getDocumentElement().appendChild(emotion);19 emotionSender.sendXML(emotionML, meta.getTime());20 }21 }22 }
![Page 33: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/33.jpg)
33Marc Schröder, DFKI
3. The Swimmer's Game3. The Swimmer's Game
Simple emotional speech-driven game
a swimmer needs to reach the river bank, but is pulled by the water towards the waterfall
user can cheer up the swimmer with aroused speech
Components:
emotion detection from speech
position computer for swimmer
GUI display of swimmer position
commentator using TTS output
![Page 34: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/34.jpg)
34Marc Schröder, DFKI
3. The Swimmer's Game3. The Swimmer's Game
![Page 35: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/35.jpg)
35Marc Schröder, DFKI
3. The Swimmer's Game3. The Swimmer's Game
![Page 36: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/36.jpg)
36Marc Schröder, DFKI
SEMAINE API: SummarySEMAINE API: Summary
SEMAINE API makes it easy to write new components in Java / C++
Integration simplified by standard representation formats
![Page 37: The SEMAINE API - UvA · The SEMAINE API Integrate components across languages and operating systems: Middleware! Message-oriented middleware (MOM) asynchronous sender doesn't need](https://reader030.fdocuments.in/reader030/viewer/2022041111/5f135a60fd50066f7c4a0885/html5/thumbnails/37.jpg)
37Marc Schröder, DFKI
What you will learn in the TutorialWhat you will learn in the Tutorial
Install the SEMAINE 2.0 system
Understand parts involved:
MOM ActiveMQ
Java components
native components
distributed system
Write a SEMAINE component in Java/C++
Create a new emotion-oriented system from new and existing components