C OMP 110 M AIN & C ONSOLE I NPUT Instructor: Sasa Junuzovic.
Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan...
-
Upload
kristina-charles -
Category
Documents
-
view
213 -
download
0
Transcript of Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan...
![Page 1: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/1.jpg)
What Did I Miss?
Kori Inkpen John C. TangRajesh Hegde
Zhengyou ZhangSasa JunuzovicChris BrooksUniv. Saskatchewan
In-Meeting Review using Multimodal Accelerated Instant Replay (AIR) Conferencing
![Page 2: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/2.jpg)
GoalMake meetings (and play) better with telepresence technology
![Page 3: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/3.jpg)
Meetings are important
![Page 4: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/4.jpg)
Do you ever ….
![Page 5: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/5.jpg)
Partial Meeting Review95% 91%
73%
30%
Verizon Conferencing Report - 2003
![Page 6: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/6.jpg)
What did I just miss?
![Page 7: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/7.jpg)
DVR Functionality -> Live Conference
![Page 8: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/8.jpg)
What makes it challenging?
![Page 9: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/9.jpg)
Previous ResearchMeeting Capture
Cutler et al., 2002; Jian, Kim & Li, 2003; Ranjan, Birnholtz, & Balakrishnan, 2008
SummariesVideo Skimming & Summarization - Smith & Kanade, 1995; He, Gupta & Grudin, 1999; Christel, 2006; Money & Agius, 2008;
Indexing Recorded MeetingsMoran et al., 1997
Time-CompressionOmoigui et al., 1999; Orr, 1971; Wildemuth et al., 2003
Real-time CatchupTucker et al., 2010
![Page 10: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/10.jpg)
Prototype System
AIR Conferencing
![Page 11: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/11.jpg)
AIR Conferencing System 1/2
LIVE
![Page 12: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/12.jpg)
AIR Conferencing System 2/2
PLAYBACK
![Page 13: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/13.jpg)
AIR Conferencing - User Study #1
Explore the benefits (and challenges) of Accelerated Instant Replay for meetings
![Page 14: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/14.jpg)
Experimental Design
18 Participants (1 female)Three conditions
Background InterviewsTask: Live Status Meeting
3-way VideoconferenceEach person gets a chance to presentEach person gets interrupted 4 times and has to catch up
Recall assessment (quiz on facts + explanations)
Transcript-Only Muted Replay Full Replay
![Page 15: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/15.jpg)
AIR – Interview Results
All occasionally miss parts of meetings
Current catch-up strategies (most indicated they do not want to disrupt the meeting)
often just listen and try to deduce what they missed, possibly interrupt with a question if they are confusedwait for a recap by the presenterbrowse meeting material to figure out what was misseddiscreetly ask someone what was missed (whisper or IM)
![Page 16: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/16.jpg)
AIR – Interview ResultsUseful to replay what was missed during a videoconference?
Yes (10), maybe (7), no (1). “Often you miss critical conversations when you step out or are interrupted during a meeting and then you try to play catch-up during the rest of the meeting. Getting to know what was covered and who said it and the body language would put me back into the meeting very quickly.”
“It would depend mostly upon the importance of the meeting, followed by the duration of how much I missed, and finally, on how discreetly I could review the video”
![Page 17: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/17.jpg)
AIR – Recall Results
Full Replay enabled users to recall as much as they did when they were not interrupted.
Recall TestFacts Explanations
Baseline 78% 43%
Transcript only 45% 16%
Muted Replay 40% 13%
Full Replay 80% 49%
![Page 18: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/18.jpg)
AIR – Preference Results
Current System
Perfect Speech-to-
TextTranscript Only 2 7
Muted Replay 0 2
Full Replay 16 6
–
Full Replay:“it was fast, easy to concentrate and auto catch-up”“easiest to follow; leads most smoothly into rejoining live”“seems like the only way to catch up in a focused way”“fast audio was really cool …. I feel like I can cheat in time with fast forward”
![Page 19: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/19.jpg)
AIR – Divided Attention Results
Very Difficult
Somewhat Difficult
Neither Easy nor Difficult
Somewhat Easy
Very Easy
6 8 2 2 0
Past + Present:“I can listen to audio while watching the live slide show and transcript”“you can listen to current conversations and read transcripts”
![Page 20: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/20.jpg)
New Questions
1. How important were the additional modalities (over just audio alone)?
2. How successful would the system be with perfect speech-to-text transcription?
3. How can we better support multitasking?
![Page 21: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/21.jpg)
AIR Conferencing - User Study #2
Examining individual modalities and impact of perfect speech-to-text
transcription
![Page 22: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/22.jpg)
Prototype System
Videos
SharedDesktop
Speech-to-text Transcript
![Page 23: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/23.jpg)
Experimental Design
58 Participants (25 female)Pre-recorded status meeting Mixed Design
Everyone did audio only + 1 other condition
Focused review (no divided attention)
![Page 24: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/24.jpg)
Modalities
Transcript only
Audio only
Audio + Transcrip
t
Audio + Video
Audio + Shared
Workspace
Audio + All
(full replay)
![Page 25: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/25.jpg)
User study #2 – Recall resultsRECALL Fact Explanation Identification
Live 93% 77% 84%
Audio Only 74% 50% 61%
Enhanced Audio 83%1 62%1 72%1
1 Significantly better than audio only but worse than live
![Page 26: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/26.jpg)
User study #2 – Recall results
RECALL Fact Explanation Identification
Live 93% 77% 84%
Audio Only 74% 50% 61%
Audio + All 89%1 70%1 81%1
Audio + Workspace 90%1 65% 72%
Audio + Transcript 83% 67%2 66%
Audio + Video 70% 48% 68%
1 Significantly better than audio only and no difference with live
2 Significantly better than audio only but worse than live
![Page 27: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/27.jpg)
User study #2 – Recall results
RECALL Fact Explanation Identification
Live 93% 77% 84%
Audio Only 74% 50% 61%
Audio + All 89%1 70%1 81%1
Audio + Workspace 90%1 65% 72%
Audio + Transcript 83% 67%2 66%
Audio + Video 70% 48% 68%
1 Significantly better than audio only and no difference with live
2 Significantly better than audio only but worse than live
Users were significantly more confident with their answers in
the Audio + All condition (p<.05).
![Page 28: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/28.jpg)
User study #2 – Recall results
RECALL Fact Explanation Identification
Live 93% 77% 84%
Audio Only 74% 50% 61%
Audio + All 89%1 70%1 81%1
Audio + Workspace 90%1 65% 72%
Audio + Transcript 83% 67%2 66%
Audio + Video 70% 48% 68%
1 Significantly better than audio only and no difference with live
2 Significantly better than audio only but worse than live
“I used the audio to see who was talking, and every time I missed something, I had the transcript which kept a recording of everything and I could just look back at it.”
“Being able to see who was talking during catch-up helped to associate a face, name, and voice with the answers given.”
![Page 29: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/29.jpg)
User study #2 – Transcript Only
RECALL Fact Explanation Identification
Audio Only 71% 50% 56%
Transcript Only 77%1 52% 43%2
1 Significantly better than audio only
2Significantly worse than audio only
![Page 30: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/30.jpg)
User study #2 – Transcript Only
RECALL Fact Explanation Identification
Audio Only 71% 50% 56%
Transcript Only 77%1 52% 43%2
1 Significantly better than audio only
2Significantly worse than audio only
Transcript alone was way too fast to really understand what was going on. I could kind of skim, and I got some information but I felt like I was just bouncing along.
![Page 31: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/31.jpg)
AIR Conferencing Conclusions
Enhanced audio catch-up is superior to audio-only review.
Users preferred, felt more confident with, and performed better with enhanced-audio than audio-only review. As good as live!Results consistent between two studies
Audio + All showed the strongest benefitAudio + Shared Workspace improved recall of factsAudio + Transcript improved recall of explanations
Speech-to-Text transcript not as beneficial as expected
![Page 32: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/32.jpg)
AIR Conferencing Conclusions
Cost/benefit tradeoff: importance of information vs. overhead Ensure seamless use so as to not detract from the meeting
Next Steps:Enhanced timeline indexMulti-taskingAIR for face-to-face meetings
(with mobile phones/laptops & headsets)
![Page 33: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/33.jpg)
AIR Vision
Replaying life should be “a given”But sometimes participation in real-time is important, so we need to enable catch-up
![Page 34: Kori Inkpen John C. Tang Rajesh Hegde Zhengyou Zhang Sasa Junuzovic Chris Brooks Univ. Saskatchewan In-Meeting Review using Multimodal Accelerated Instant.](https://reader035.fdocuments.in/reader035/viewer/2022070401/56649f1d5503460f94c34d4b/html5/thumbnails/34.jpg)
Thank you!
Questions?