Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland...
Transcript of Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland...
![Page 1: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/1.jpg)
1
Modifying the MARS Scale to Evaluate Health Applications
Session #107, February 13, 2019
Speaker: Ryan Romero, MPH Student, University of Florida
Rick Kates, Clinical Assistant Professor, University of Florida
![Page 2: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/2.jpg)
2
Ryan Romero, MPH Student
Has no real or apparent conflicts of interest to report.
Frederick Kates, PhDHas no real or apparent conflicts of interest to report.
Conflict of Interest
![Page 3: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/3.jpg)
3
1. Introduction to MARS
– A. Sections of the MARS Scale
– B. Application Specific Measures
– C. Validation of Content Specific Measures
2. Study Specific Populations and Our Application of MARS
– A. The deaf and Hard of Hearing Community
– B. Use of Hard of Hearing Applications
– C. App Specific Measures Developed for Target Population
– D. Study Characteristics
3. New Criteria For Application Specific Measures Using MARS
– A. Identification of a content expert in subject field (with example)
– B. Use of content expert or population member to help develop content specific measures
– C. Use of content expert to gauge specific interests and needs of population
– D. Short wrap up discussion
Agenda
![Page 4: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/4.jpg)
4
• Design content-specific criteria to evaluate mobile
health applications
• Identify and engage an appropriate content
expert
• Identify population-specific needs and implement
criteria into rating scale
Learning Objectives
![Page 5: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/5.jpg)
5
• Content Expert: Knowledge expert in a particular field of study that has some professional involvement in the subject [1]
• Application Specific Measures (Content Specific): Evaluation metrics designed to gauge the value of specific features towards an intended population/ function [2]
• Appropriateness: Relevance of features to intended users
• MARS (Mobile Application Rating Scale): Quantitative (1-5) scale used to evaluate mHealth applications [2]
Key Terms
![Page 6: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/6.jpg)
6
Join at kahoot.it
1. Health Infinity 2. Carb Manager 3. MyNetDiary
Which mHealth App is the Best?
![Page 7: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/7.jpg)
7
1. Health Infinity
![Page 8: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/8.jpg)
8
2. Carb Manager
![Page 9: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/9.jpg)
9
3. MyNetDiary
![Page 10: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/10.jpg)
10
• The Mobile Application Rating Scale (MARS) [2]
• Developed at Queensland University of Technology
– Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko, Dian Tjondronegoro, MadhavanMani
• Provides a standardized approach to comparing and rating health applications for mobile devices (iPhone, iPad, Android, Android Tablet)
• Allows for modularity as key part of design
• Frequently used to evaluate mHealth applications
Introduction to MARS
![Page 11: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/11.jpg)
11
Sections of the MARS Scale[2]
Engagement
5 subsections
Engagement
5 subsections
Functionality4 subsections
Functionality4 subsections
Aesthetics
3 subsections
Aesthetics
3 subsections
Information
7 subsections
Information
7 subsections
Subjective
Quality*
Subjective
Quality*
![Page 12: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/12.jpg)
12
Alpha test of interrater agreement is preformed to ensure acceptable levels of agreement [3]. Sections where there is a lack of agreement or uncertainty are discussed and resolved by consultation of the content expert and reevaluation.
Training and Validation of Raters
![Page 13: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/13.jpg)
13
• MARS allows for custom measures intrinsically
– Tells us about app quality
– Tells us about appropriateness and intended fit
• Use of section as indicator of status vs indicator for change*
– Section can be used to both address quality and declare the need or desire for specific feature
• Content development and developer feedback
– Use of content expert to guide study’s intent
– Feedback to developers
– Empirical data to support stance
Content Specific Measures
![Page 14: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/14.jpg)
14
Study Specific Populations and Our Application of the
MARS
![Page 15: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/15.jpg)
15
New Translation Technologies
![Page 16: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/16.jpg)
16
• The Deaf and Hard-of-Hearing community- 7 million members of HOH community[6]
• Electronics as part of social interface capability
• Electronics used to communicate with family and friends
• Necessity for evaluation of hard-of-hearing applications
Hard of Hearing Apps & Our Study
![Page 17: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/17.jpg)
17
• Preliminary Literature Review
– 37 million with hearing difficulties[7]
– Needs of population over 60 years old[8]
• Decision to Include Content Expert
• Inclusion and Exclusion Criteria
– Need to determine appropriateness[9]
– Mitigate risk
• Results
Study Characteristics
![Page 18: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/18.jpg)
18
• Application features
• Is technology likely to be used?
• Inclusion and Exclusion Criteria
• Multi category approach to population specific applications
– Text to Speech
– ASL Translator
– Hard-of-Hearing Assistant
Hard of Hearing Applications
![Page 19: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/19.jpg)
19
Content Expert
Stephen J. Hardy, IIM.Ed.
Lecturer, University of Florida
College of Public Health and Health
Professions
Fluent in Deaf Culture, ASL, and Deaf
Studies
![Page 20: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/20.jpg)
20
App-Specific Measure #1
![Page 21: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/21.jpg)
21
App-Specific Measure #2
![Page 22: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/22.jpg)
22
App-Specific Measure #3
[4]
![Page 23: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/23.jpg)
23
App-Specific Measure #4
![Page 24: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/24.jpg)
24
Related Study
[5]
![Page 25: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/25.jpg)
25
• 5 / 21 applications no longer available for evaluation after follow up
– Simply disappeared
– Indicator of status of application field
– Potential lack of content validity
– Niche product does not necessarily entail less competition
Application Turnover
![Page 26: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/26.jpg)
26
• Please take out your smartphone or device and enter the activity at: Kahoot.it
• Winner will receive small non-monetary prize to keep it interesting
Time to Get Involved!
![Page 27: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/27.jpg)
27
• Three important topics:
1. Use of content experts as guide for project
2. Use of content experts to achieve validity of experiment design
3. Approaching a content expert for consultation
Content Experts and Design Validity
![Page 28: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/28.jpg)
28
• What qualifications should my content expert have?
– How should they relate to the topic we are studying?
• How can they be used to the advantage of the study?
– Guide for purpose
– Who does study effect?
– Discussion section
• Use of multiple content experts
Identification of a Content Expert
![Page 29: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/29.jpg)
29
• Knowledge on use of content experts in experimental design and analysis
• Recognition of importance of content validity and designing a study to help the subject it focuses on
• Recommendation of quality apps based on MARS
How Can This Information Help YOU?
![Page 30: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/30.jpg)
30
• Discussion on how this information relates to your focus
• Using this information to develop a study
• How health information professionals can use this
information to design better applications
Discussion and Q&A
![Page 31: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/31.jpg)
31
• Ryan Romero, MPH Student
• Rick Kates, PhD
Please feel free to come up and speak with us after the presentation!
Speaker Contact
![Page 32: Modifying the MARS Scale to Evaluate Health Applications€¦ · •Developed at Queensland University of Technology –Stoyan Stoyanov, Leanne Hides, David Kavanagh, Oksana Zelenko,](https://reader035.fdocuments.in/reader035/viewer/2022062917/5ed9bcb924797b75744b16c6/html5/thumbnails/32.jpg)
32
1. Grant, J.S., & Davis, L.T. (1997). Selection and use of
content experts in instrument development. Research
in Nursing & Health, 20, 269–274.
2. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M
Mobile App Rating Scale: A New Tool for Assessing the Quality of Health Mobile Apps JMIR
Mhealth Uhealth 2015;3(1):e27. URL: https://mhealth.jmir.org/2015/1/e27
3. Shrout PE, Fleiss JL. Intraclass correlations: Uses in assessing rater reliability. Psychol Bull
19790101;86(2):420. [doi: 10.1037/0033-2909.86.2.420]
4. SkoogMusic. The Skoogmusic Skwitch. Digital Image. Accessed from
http://skoogmusic.com/skwitch/
5. Arellano, Patricia, Bochinski, Janet, Elias, Beth, Houser, Shannon, Martin, Thomas, Head, Hank.
2012. Selecting a Mobile App: Evaluating the Usability of Medical Applications. mHIMSS App
Usability Work Group.
6. Carroll DD, Courtney-Long EA, Stevens AC, Sloan ML, Lullo C, Visser SN, Fox MH, Armour BS,
Campbell VA, Brown DR, Dorn JM. Vital signs: disability and physical activity--United States,
2009-2012. MMWR Morb Mortal Wkly Rep 2014 May 9;63(18):407–413. PMID:24807240
7. Ross, Mitchell, Karchmer, Michael. (2004). Chasing the Mythical Ten Percent: Parental Hearing Status of Deaf and Hard of Hearing Students in the
United States. Sign Language Studies, (4)2; 138-163.
8. Hoffman, H. J., Dobie, R. A., Losonczy, K. G., Themann, C. L., & Flamme, G. A. (2017). Declining Prevalence of Hearing Loss in US Adults Aged 20
to 69 Years. JAMA otolaryngology-- head & neck surgery, 143(3), 274-285.
9. Lewis, T. L., & Wyatt, J. C. (2014). mHealth and mobile medical Apps: a framework to assess risk and promote safer use. Journal of medical Internet research, 16(9), e210.
doi:10.2196/jmir.3133
References