Real-Time Feedback System for Monitoring and Facilitating...
Transcript of Real-Time Feedback System for Monitoring and Facilitating...
![Page 1: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/1.jpg)
Real-Time Feedback System for Monitoring and Facilitating Discussions
Supervisor: Prof Justin DAUWELS School of Electrical and Electronic Engineering, NTU
Co-Supervisor: Prof Daniel THALMANN Institute for Media Innovation (IMI), NTU
1
PhD Student: Yasir Tahir Institute for Media Innovation (IMI), NTU
![Page 2: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/2.jpg)
Introduction
2
• Social Signal Processing
• Is the research and technological domain that aims at providing computers with the ability to sense and understand human social signals.
• Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life.
• Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis of relevant behavioural cues development of automated systems for Social Signal Processing (SSP) is rather challenging.
Vinciarelli, Alessandro, Maja Pantic, and Hervé Bourlard. "Social signal processing: Survey of an emerging domain." Image and Vision Computing27.12 (2009): 1743-1759.
![Page 3: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/3.jpg)
Outline
1. General Objective 2. Speech Analysis 3. Video Analysis 4. Data Collection 5. Feedback
3
![Page 4: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/4.jpg)
Enhancing Social Cognition
4
Posture
Gesture
Appearance
Speech
Distance
Facial expression
Behavioral cues (e.g. pitch, volume, turn-taking , posture)
Useful feedback for more effective conversations
Applications • Mentoring for leadership development • Team meetings • Interviewing systems
![Page 5: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/5.jpg)
Functional Blocks of the System
5
![Page 6: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/6.jpg)
Outline
1. General Objective 2. Speech Analysis 3. Video Analysis 4. Data Collection 5. Feedback
5
![Page 7: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/7.jpg)
Speech Analysis
6
• Four types of vocal social signalling :
• Activity level • Engagement • Stress • Mirroring
Pentland, Alex. "Social dynamics: Signals and behaviour." International Conference on Developmental Learning. Vol. 5. 2004.
![Page 8: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/8.jpg)
Speech Features
6
![Page 9: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/9.jpg)
Speech Analysis: Setup
7
Recording setup using the Zoom H4n voice recorder with Sennheiser e845s microphones for two people conversation.
![Page 10: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/10.jpg)
8
Speech Analysis: GUI
GUI for real-time analysis in MATLAB
![Page 11: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/11.jpg)
Outline
9
1. General Objective 2. Speech Analysis 3. Video Analysis 4. Data Collection 5. Feedback
![Page 12: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/12.jpg)
Video Analysis: Setup
10
• For video analysis we are using Kinect sensor. It provides a wide range of sensors which can be used to extract many useful features.
• Physical Capabilities – Angles of Kinect vision (Depth and RGB)
• Horizontal: 57.5 degrees • Vertical: 43.5 degrees with -27 to +27 degree tilt range up and down • 1.2 to 3.5m distance range for depth • Microphone array
![Page 13: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/13.jpg)
Video Analysis: Features
11
• We are extracting following features from the video data :-
– Posture – Gesture usage – Nodding – Audio-visual speech detection
![Page 14: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/14.jpg)
Video Analysis: Detection
12
• Nodding detection
– We are detecting consecutive vertical head movement. Such movement is classified as YES or vertical nodding. – Similarly consecutive horizontal head movement is detected and such motion is classified as NO or horizontal nodding.
![Page 15: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/15.jpg)
Video Analysis: Detection
13
• Posture detection
– Posture detection is done using the angle between head and shoulders of a person. From this information we can detect the posture. – Right now we have implemented three basic sitting postures i.e. upright, leaned back and hunched forward.
![Page 16: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/16.jpg)
Video Analysis: Detection
15
• Speech detection
– Speech detection is done using lip motion in the detected face. – Fusing the audio data for speech detection really improves the speech detection and reduces false detections.
![Page 17: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/17.jpg)
Video Analysis: Detection
15
• Gesture Usage
– Instead of looking for certain gestures, we are calculating the overall hand movement. Higher value of this measure represent greater use of gestures, whereas low value represents less gesture use.
![Page 18: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/18.jpg)
Outline
16
1. General Objective 2. Speech Analysis 3. Video Analysis 4. Data Collection 5. Feedback
![Page 19: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/19.jpg)
Data Collection
17
• We have recorded 50 sessions of two person discussions. The duration of each session is five minutes. The topics are everyday issues ranging from social issues to movies.
• Four people two males and two females participated in the data collection.
• Audio data was recorded using lapel microphones. • Video and depth data was recorded using kinect sensor, one for each
person. • We focused on meeting scenarios :
– Participants were sitting – we did not ask the participants to restrict their movements to keep the social
aspect intact.
![Page 20: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/20.jpg)
Data Collection: Example
18
![Page 21: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/21.jpg)
Data Collection: Results
16
Distance of Speakers
Seating Arrangement
Ambient Noise
No of Samples
Undetected Speech
False Detection Interference Overall Accuracy
0.4-0.5m Side Low 19200k 6.5% 4% 9% 80.5%
0.7-0.8m Side Low 19200k 2.0% 2.5% 3.5% 92%
1.0m Front Low 19200k 2.5% 2% 12% 84.5%
1.2m Front Low 19200k 2.0% 0.0% 1% 97%
1.5m Front Low 19200k 0.5% 0.5% 0.5% 98.5%
1.8m Front Low 19200k 2.0% 0.0% 0.0% 98.0%
1.5-1.8m Front High 19200k 0.5% 1.5% 1.0% 97%
Visual Cue Video duration False detection Accuracy
Posture 500 min 5% 95%
Gesture 500 min 13% 87%
• For the case of visual cues. We analyzed the video recorded in 50 sessions
![Page 22: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/22.jpg)
Outline
20
1. General Objective 2. Speech Analysis 3. Video Analysis 4. Data Collection 5. Feedback
![Page 23: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/23.jpg)
Methodology
Behavior Detection: Dominance, Interest, Discord, Consistency, Mirroring via
Support Vector Machines (SVM)
Speech Cues Visual Cues Feature Extraction
Feedback
Audio and Video Data Acquisition
Pre-processing Pre-processing Step for Audio: Speech Detection
Pre-processing Step for Video: Face and Skeleton Detection
![Page 24: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/24.jpg)
Social Roles
• In real life situations, a combination of the honest signals can be observed in individuals. Four core groups of social roles - Listening, teaming, exploring and leading have been identified. • Listening is said to signal a combination of attentive interest and
openness to ideas - Variable emphasis and suppressed activity. • Teaming involves a combination of attention, empathetic understanding
and focused thought and purpose - High influence, mimicry and consistent emphasis.
• Exploring represents possibility of establishing a meaningful relationship with someone for which high levels of interest and openness to influence is necessary - High activity, variable emphasis and rhythm.
• Leading displays attention, interest and great focus in thought and purpose - High influence and activity levels and consistent emphasis.
Pentland, Alex Sandy. Honest signals: how they shape our world. MIT Press, 2008.
22
![Page 25: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/25.jpg)
SVM output Dominance
High Dominance 23
![Page 26: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/26.jpg)
SVM output Discord
High Discord
24
![Page 27: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/27.jpg)
SVM output Interest
Low Interest
High Interest
25
![Page 28: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/28.jpg)
Real-Time Feedback
• The concept is to analyse acquired audio and video data and
provide feedback during an ongoing conversation. • We use certain time window, after each window the data is
analysed and a feedback is generated if required. • We also plan on observing the effect of this feedback on the
speakers. • The medium to provide feedback is also very important. It
should not be too abstract or distracting that the user gets disturbed.
26
![Page 29: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/29.jpg)
Feedback Platforms
• Social Mediator via Nao • Retrospective Feedback via Avatar Animation • Socio-Feedback via Skype Application • Socio-Feedback via Android Application • Socio-Feedback via Vuzix
27
![Page 30: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/30.jpg)
Applications
Speech Coach
Interview Analysis + Guidance Team Facilitation
Monitoring of mental state (e.g., stress, alertness, concentration)
28
![Page 31: Real-Time Feedback System for Monitoring and Facilitating ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/Yasir_T… · Real-Time Feedback System for Monitoring and Facilitating](https://reader035.fdocuments.in/reader035/viewer/2022071012/5fca22c0ea95cf52522fd29a/html5/thumbnails/31.jpg)
Q & A
Q&A
29