Post on 25-Jul-2020
Bongshin Lee
Microsoft Research
Harnessing the Power of Visualizationfor Human-Data Interaction
The Rise of Self-Tracking
• 259,000 mHealth apps listed on major app stores (2016)1
• Nearly 25% of Americans own a wearable (2016)2
1 Research 2 Guidance, mHealth App Developer Economics 2016.2 Rock Health, 50 things we now know about digital health consumers
Promises
Personal Data Self-Knowledge Self-Improvement
?
구슬이 서말이라도 꿰어야 보배
It takes more than pearls to make a necklace.
Nothing is complete unless you put it in a final shape.
Information Visualizationthe use of computer-supported, interactive, visual representations of abstract data to amplify cognition [Card et al., 1996]
I explore innovative ways to help people understand and communicate their data
leveraging visualization.
Why visualize data?
Anscombe's Quartet
• Mean of the x values = 9.0
• Mean of the y values = 7.5
• Equation of the least-squared regression line: y = 3 + 0.5x
• Sums of squared errors (about the mean) = 110.0
• Regression sums of squared errors (variance accounted for by x) = 27.5
• Residual sums of squared errors (about the regression line) = 13.75
• Correlation coefficient = 0.82
• Coefficient of determination = 0.67
Anscombe's Quartet
Anscombe's Quartet
• X Mean: 54.26
• Y Mean: 47.83
• X SD: 16.76
• Y SD: 26.93
• Correlation: -0.06
[Matejka & Fitzmaurice, CHI 2017]
Datasaurus Dozen
128176875613897654698450698560498289809858458224509856458945098450980990910302099059595957725646750506789884578980982167765487636490856091299856049828267629809858458224509856484582245098564589450984509809435859
How many 3’s?
128176875613897654698450698560498289809858458224509856458945098450980990910302099059595957725646750506789884578980982167765487636490856091299856049828267629809858458224509856484582245098564589450984509809435859
How many 3’s?
Is a red circle present?
Is a red circle present?
Is a red circle present?
My Publications
http://www.wordle.net
My Publications
My Publications
HCIHCI
Visualization
Visualization for Machine Teaching
Natural Interactionfor Visualization
Data-DrivenStorytelling
Personal Vis &Self-Monitoring
Research Goal
https://recoveringengineer.com/leadership-skills/new-supervisor-skills-people-must-feel-empowered
Analyze ShareCollect
Human-Data Interaction
Collect ShareAnalyze
TouchPivot [CHI 2017] Visualized Self [PervasiveHealth 2017]
How to help novice users visually explore data?
Powerful way to help people gain meaningful insights from their data
Visual Data Exploration
Visual Data Exploration
Data Insight
But, it is difficult for novices to perform visual data exploration.
Grammel et. al, How information visualization novices construct visualizations. [IEEE TVCG, 2010]
TouchPivotBlending WIMP & Post-WIMP Interfaces for Data Exploration on Tablet Devices [CHI 2017]
Bongshin Lee
Microsoft Research
Sehi L’Yi
SNU
Jaemin Jo
SNU
Jinwook Seo
SNU
Data Transformation & Visual Mapping
State Gender Population
Colorado Male 700
Colorado Female 900
Utah Male 400
Utah Female 300
Data Transformation
Pivot by Gender
GenderSUM
(Population)
Male 1,100
Female 1,200
0
500
1000
1500
Male Female
Population
Visual Mapping
Shelf Configuration Interfaces
PivotTable(Microsoft Excel)
Tableau
Where should I put this column?
To alleviate novices’ hurdles in visual data exploration
leveraging pen and touch interactions
Research Goal
TouchPivot Demo Video
https://www.youtube.com/watch?v=Q6quofDiO7I
Favor simplicity over flexibility
Provide tight coupling between data and visualization
Suggest appropriate visualizations
Combine pen & touch with WIMP interfaces
Design Rationale
How do people reflect on their own self-tracking data?
Eun Kyoung Choe
UMCP
Understanding Self-Reflection:How People Reflect on Personal Data through Visual Data Exploration [PervasiveHealth 2017]
Bongshin Lee
Microsoft Research
Haining Zhu
PSU
Dominikus Baur
Nathalie Henry Riche
Microsoft Research
Challenges with Personal Data ExplorationData is scattered across multiple platforms [Li et al., 2011; Choe et al., 2014.]
People don’t know what to do with the data [Choe et al., 2014; Epstein et al., 2015; Lazar et al., 2015.]
Analyzed 30 video recordings of QS presentations
[IEEE CG&A 2015]
Eun Kyoung Choe m.c. schraefelBongshin Lee
Types of Personal Insights
1. What I did
2. How I did it
3. What I learned
Detail
Self-Reflection
Trend
Comparison
Correlation
Data Summary
Distribution
Outlier
Visualization Insights
(74%)
(51%)
(36%)
(35%)
(11%)
(9%)
(6%)
(2%)
Data Integration from Multiple Sources
Data Summary
Trend Comparison
Interactive Data Exploration
Study Session
“I think that was soon after my surgery and that maybewould make sense cause I’d have to get up to takemedicine and maybe being restless or something.” [P8]
Visual data exploration Contextual information
Insight Gaining Pattern #1
External context
Visual data exploration Contextual information
Insight Gaining Pattern #2
External context
Question (Hypothesis)
Visual Data Exploration: Comparison by time segmentation
Question Did changing jobs affect my weight?
P1: [entering Sept 15, 2015 to compare his weight before and after this date]
Researcher: “Why Sept 15?”
P1: “That's kind of around the time I changed jobs. I was wondering if there wasanything interesting there.”
Support personal data exploration on mobile environment
Incorporate system-driven insights
Capture and share interesting questions and insights
Going Forward …
Collect Analyze Share
How to help people communicate their insights?
ChartAccent [PacificVis 2017]
Bongshin Lee
Microsoft Research
Matthew Brehmer
Microsoft Research
Donghao Ren
UCSB
Eun Kyoung Choe
UMCP
ChartAccentAnnotation for Data-Driven Storytelling [PacificVis 2017]
Tobias Höllerer
UCSB
“the annotation layer is the most important thing we do . . . otherwise it’s a case of here it is, you go figure it out.”
—Amanda Cox, The New York Times Graphics Editor
Current annotation support is limited
ChartAccent Demo Video
https://www.youtube.com/watch?v=8ogkSBlE2f0
Annotation Target Annotation Form
Annotation Design Space
Annotation Form
Text Shape Highlight Image Combined
Annotation Target Type
Annotation Target Type
chartaccent@gmail.com | @chartaccent
Examples, Tutorial, and Survey
Open-source project -- https://github.com/chartaccent
ChartAccent.github.io
ShareAnalyzeCollect
How to support people’s diverse tracking needs?
OmniTrack [UbiComp 2017]
A Flexible Self-Tracking Approach Leveraging Semi-Automated Tracking[UbiComp 2017]
Bongshin Lee
Microsoft Research
Jae Ho Jeon
Kakao Corp.
Young-Ho Kim
SNU
Jinwook Seo
SNU
Eun Kyoung Choe
UMCP
The Rise of Self-Tracking
• 259,000 mHealth apps listed on major app stores (2016)1
• Nearly 25% of Americans own a wearable (2016)2
1 Research 2 Guidance, mHealth App Developer Economics 2016.2 Rock Health, 50 things we now know about digital health consumers
Most self-tracking apps provide little or no flexibility.
Imagine how you might record your reading activities
Diverse Tracking Needs
texttextstar
Book Reviews
TitleAuthorRating
Book Reviews
texttextnumbertext
TitleAuthorPagesReview
Reading Logs
TitlePage FromPage ToDate
textnumbernumberdate
Challenging to find an existing app that perfectly suits one’s tracking needs
Research Goal
To support people’s diverse tracking needs through
a flexible self-tracking system
Semi-Automated Tracking
Reduced mental load
Better accuracy (depending on the data)
Cumbersome to wear (wearable sensing)
Reduce engagement with data
+
+
--
Engagement with data
Increased self-awareness
Flexibility of choosing target behaviors
Some data can only be tracked manually
High capture burden
Compromised data accuracy
+
+
+
+
--
Fully manualtracking
Fully automatedtracking
Choe et al., Semi-Automated Tracking: A Balanced Approach for Self-Monitoring Applications [IEEE Pervasive Computing 2017]
Semi-Automated Tracking
Fully manualtracking
Fully automatedtracking
Paper DiaryEmbedded Sensing
Semi-automated Tracking Spectrum for Sleep
Balancing Burden with OmniTrack
Automated Capture of Sleep Duration
Manual Capture ofSubjective Sleep Quality
Fully manualtracking
Fully automatedtracking
Semi-automated Tracking Spectrum for Sleep
OmniTrack Demo Video
https://www.youtube.com/watch?v=zRIuENNRjEM
Experiment Design
Tracker Design 1
Tracker Design 2
Group A
Group B
Data & Experiment Management
Progress Monitoring Intervening
Create data collection tools without programming
OmniTrack ResearchKit
Researcher Participants
Data
Trackers
Interested in conducting a diary study/ESM using OmniTrack ResearchKit?
Contact
Personalized Self-Reports
Self-Report + Fitness Device Tracking
Multi-Device Data Collection
…
omnitrackapp@gmail.com
.github.io
omnitrackapp@gmail.com | @omnitrack_app
Detailed Information, Supplementary Materials
OmniTrack will be open-sourced soon!
Analyze ShareCollect
Human-Data Interaction
Research Avenues
Mobile Data Visualization
Patient-Clinician Communication
Visualization Literacy
Analyze ShareCollect
One Last Thing
Work with visualization researchers
Thank You!
Analyze ShareCollect
bongshin@microsoft.com | aka.ms/bongshin | @bongshin