Themes that Cut Across Disciplines: Quality Control and Research Methods
description
Transcript of Themes that Cut Across Disciplines: Quality Control and Research Methods
![Page 1: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/1.jpg)
Themes that Cut Across Disciplines: Quality Control and
Research MethodsFaina Linkov, PhD
Research Assistant ProfessorUniversity of Pittsburgh Cancer Institute
The journey of quality control in the SupercourseThe journey of a thousand miles starts with a single step
![Page 2: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/2.jpg)
Overview
• Overview the history of quality control and peer review
• Quality control on Amazon.com• Quality control in the Supercourse: QC on
PPT lectures is at the core of what we do• Supercourse studies of quality control• Our conclusions and future directions
![Page 3: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/3.jpg)
What do scientists have in common?
• Quest for discovery• Common research methods, such as
statistics• Journal article format• Quality control mechanisms• Peer review: is it the same as quality?
![Page 4: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/4.jpg)
History of Quality: Early times
• 384-322 BCE-Aristotle and the birth of peer review
• 854-931 “Ethics of the Physician” by Ishap bin Ali Al Rahwi-first documented description of peer review in Medicine
![Page 5: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/5.jpg)
Royal Society of London
Philosophical Transactions- Originally edited by Henry Oldenburg
1665
First peer reviewed journal
![Page 6: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/6.jpg)
Typewriter: the tool that fueled peer review process
1890’s
![Page 7: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/7.jpg)
Quality control criteria of the past
Albert Einstein, 1905His famous work, Annus Mirabilis, was not peer reviewed!
![Page 8: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/8.jpg)
History of quality continued
19th century – birth of quality control discipline
1950 - W. Edwards Deming and statistical quality control
1970’s- Total Quality Management (TQM)
![Page 9: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/9.jpg)
What is Quality
“ Doing the right thing right, right away” W. Edwards Deming, 1982
“a measure of goodness that relates to the intended use of a product and the expectations customers have concerning this product”
Barkman, 1989
![Page 10: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/10.jpg)
Traditional Quality control of Scientific Materials: Peer Review
“A peer-reviewed journal is one that has submitted most of its published articles for review by experts who are not part of the editorial staff”
Vancouver Group of Editors
![Page 11: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/11.jpg)
Is peer review the same as quality?
South Korean cloning scandal, alongside with other examples, demonstrate that peer review is not perfect
![Page 12: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/12.jpg)
Peer review congress: finding new ways or fixing the old system?
![Page 13: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/13.jpg)
The State of QC for PowerPoint on the Web
• Over 7,600,000 PowerPoint files on the Internet give us unique new ways to do QC
• Only 3383 lectures tested for QC-Supercourse
• We published almost all articles concerning the state QC of PowerPoint on the web
![Page 14: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/14.jpg)
Amazon.com: the first website that allowed to post a book review
![Page 15: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/15.jpg)
PowerPoint lectures
• No information on quality control for PowerPoint lectures on-line
• There is a growing need to establish quality control for Scientific PPT lectures on Line
• Migration of 7 million lectures to the web by 2008
• Establishment of “Quality Circles” in the Supercourse: John Last, Vint Cerf, and many others
![Page 16: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/16.jpg)
Progress Report in the Supercourse
Over 14000 lecture reviews were
obtained in the last 6 years
Lecture review forms utilize Likert like scales, regularly used for the evaluation of educational programs and amazon, ebay, consumers reports..
![Page 17: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/17.jpg)
Space for written comments
Some of the URLs have changed since the lecture was developed. It would be helpful if that was corrected, so you could actually see the informationreferred to.
Is there any text(s) that you could recommend for epidemiology, i.e., a basic text, an intermediate text, and an advanced text?
![Page 18: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/18.jpg)
My dissertation, 2005First evaluation of peer review mechanisms for
PPT lectures on-line
![Page 19: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/19.jpg)
7 expert reviewers from 6 countries
![Page 20: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/20.jpg)
Results and conclusions of this research
• Lectures were rated highly by experts and non-experts
• People are primarily rating at the upper tail of the distribution
• There is a poor agreement among reviewers (very low Kappa statistic)
• The role of personal background in the quality of the peer review is not clear and needs to be further investigated
• There is a pressing need for scientific evaluation of PPT lectures on the web
![Page 21: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/21.jpg)
Implications for the Supercourse of Science?
• There is no consistency about the use of peer review on the web
• Utility of peer review strategies for PPT lectures is not clear
• New approaches will be developed for the Supercourse of Science.
• Shall we allow disciplines to decide the QC techniques which are most applicable for their fields?
• There is a need to build a science of QC
![Page 22: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/22.jpg)
Where do we go from here for the development of quality control
mechanisms in the Supercourse?
Continuous quality improvement (CQI): not a destination, but a journey
![Page 23: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/23.jpg)
Supercourse aim:To scientifically develop a system of
QC on the web by making multiple measures available
Power of the Internet makes the development of the Science of QC
possible.
![Page 24: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/24.jpg)
Moving beyond peer review
“… We had great difficulty in finding any real hard evidence of the system’s effectiveness, which is disappointing, as peer review is the cornerstone of editorial policies worldwide”
Tom Jeffersen
![Page 25: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/25.jpg)
Suggested Quality Control Mechanisms for the Supercourse of
Science: Multiple Measuresa) Expert editorial boardb) Screening to identify non-appropriate lectures, as done
in nature and BMJc) Five star system (or Priority Score) similar to that in
Amazon.comd) Opinion of experts (professors)e) Personal Characteristics of authors (rank, university,
citations)f) Web statistics for utilization of lectures: hits, links, Page
Rankings, etc.g) Key note speechesh) Publications and citations from Google Scholari) Model similar to NIH style review
![Page 26: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/26.jpg)
Editorial Board (in formation)• John Last, MD
– father of modern environmental epidemiology
• Peter Greenwald, MD, PhD- leading cancer control specialist, NCI
• Paul Zimmet, MD-leading diabetes research in Australia
• Dr. Steven N. Blair-leader in exercise science
Plus 8 others
![Page 27: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/27.jpg)
Screening to eliminate poor quality lecture
Key criteria for lecture rejection in the parent Supercourse project
• Not related to prevention• Inaccurate• Out of date• Incomplete
![Page 28: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/28.jpg)
Amazon.com like systemCustomer Reviews <Book title>
42 Reviews
Average Customer Review (42 customer reviews) Share your thoughts with other customers
5 star:
(42)
4 star: (0)
3 star: (0)
2 star: (0)
1 star: (0)
![Page 29: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/29.jpg)
Mean Lecture Scores in the Supercourse
First 2000 surveys
Mean Score: 4.12
![Page 30: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/30.jpg)
Personal Characteristics of Authors(rank, university, citations)
1288 authors in the Supercourse• 30 authors from Harvard• 8 authors from Johns Hopkins• 100 authors from Pitt• 7 authors from Columbia
![Page 31: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/31.jpg)
Web statistics
•Total number of page views 41,406 (since 2003)•Viewed in 145 countries•Translated into 13 Languages•First Google Page Rank for “lecture on prevention”•Over 70 websites linking to the lecture
•
![Page 32: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/32.jpg)
Keynote speeches
Someone chosen for keynote address is expected to give a good lecture
![Page 33: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/33.jpg)
NIH Model and Continuous Quality Monitoring
Triage Priority Scores
ResubmissionSubmission
Supercourse Model: Continuous Quality Control
![Page 34: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/34.jpg)
Google model for QC: Opportunities for exploration?
• Do not exclude anything, but give it a rank• Development of search mechanisms where
you can find anything you like
![Page 35: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/35.jpg)
Why do this research?
“I believe this lecture provides simple, understandable, thorough and consolidated information of the subject. This is actual service to the humanity”
From the evaluator of the Supercourse lectures
![Page 36: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/36.jpg)
Conclusions
• There is a need to define the science of QC for online lectures
• Scientists need to be able to choose what they want in terms of QC mechanisms, including peer review and other systems
• Science needs to replicate success of industry in the area of QC
• Traditional approaches (such as peer review) may not be bad, but with Internet technologies much more can be done
![Page 37: Themes that Cut Across Disciplines: Quality Control and Research Methods](https://reader036.fdocuments.in/reader036/viewer/2022062501/56815e00550346895dcc4450/html5/thumbnails/37.jpg)
Vision for the Supercourse of Science
• Development of novel QC systems for the Supercourse of science.
• Need to systematically and continuously evaluate QC systems
• Upfront QC systems and information collection