Soziologisches Institut–Prof. Dr. Katja Rost ...
Transcript of Soziologisches Institut–Prof. Dr. Katja Rost ...
![Page 1: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/1.jpg)
TITELMASTERFORMAT DURCH
KLICKEN BEARBEITEN
Soziologisches Institut – Prof. Dr. Katja Rost
PROBLEM SPECIFICATION, STUDY RETRIEVAL AND CODING
MAER-Net Colloquium 2013
5 – 7 September 2013
![Page 2: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/2.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
1. Problem Specification
2. Study Retrieval
3. Coding
Content
2
![Page 3: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/3.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
More differentiated and sophisticated than qualitative “vote-counting”
Hidden effects
Large amount of information
Moderator analysis
Effort and expertise
Social context
Apples and oranges
Study population
Strenghts and Weaknesses
![Page 4: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/4.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Needs to be straightforward and complete BUT, at this stage, not be highly detailed
o Guides
– the selection of studies
– the coding of information
– the analysis of the resulting data
o Example: (Under which conditions) Does pay-for-performance reduce work effort?
Problem Statement
![Page 5: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/5.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Assumptions underlying existing literature remain unchallenged
– “extend(ing) this literature”
– to “address this gap in the literature”
– to “fill this gap”
– to point at themes that others “have not paid particular attention to”
– or to “call for more empirical research”
o under-problematize existing literature o incremental, footnote-on-footnote research
o reinforces already influential theories
The positive research agenda Gap spotting
Alvesson, Sandberg (2010)
![Page 6: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/6.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o How do theories which are generally considered interesting differ from theories which are generally considered uninteresting?
o Interesting theories are those which deny certain assumptions of their audience, while non-interesting theories are those which affirm certain assumptions of their audience.Attack on the taken-for-granted world of their audience.A theorist is considered great, not because his theories are true, but because they are interesting.
That's Interesting!
Davis, M. S. (1971)
![Page 7: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/7.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o What seems to be …– disorganized is organized – individual is holistic – local is general – ineffectively is effectively– bad is good – unrelated is correlated – exist together cannot exist together – heterogeneous is composed of a single element – stable/unchanging is unstable/changing – a positive co-variation is a negative co-variation– similar is opposite – independent is dependent
The Index of theInteresting
Davis, M. S. (1971)
![Page 8: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/8.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
Examples for Abstraction
Davis, M. S. (1971: 316)
![Page 9: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/9.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o highlights what is ‘wrong’ with existing knowledge
– Social constructionism
– Postmodernism
– Feminism
– Critical theory
o over-problematizes existing literature
o disrupts, rather than builds upon, and extends an established body of literature
The negative research agenda Over-problematization
Alvesson, Sandberg (2010)
![Page 10: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/10.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Evaluate you own work.
– That’s obvious!
– It’s absurd!
– That’s interesting!
Principles for identifying and challenging assumptions
Alvesson, Sandberg (2010)
![Page 11: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/11.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
1. Problem Specification
2. Study Retrieval
3. Coding
Content
11
![Page 12: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/12.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
Identification of the Study Population
![Page 13: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/13.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Distinguishing features, e.g. topic, constructs
o Research respondents, e.g. SMEs, adults
o Key variables, e.g. control variables
o Research method, e.g. experimental design, survey
o Cultural and linguistic range, e.g. countries, only English reports
o Time frame, e.g. only recent studies
o Publication type, e.g. articles, books, dissertations, technical reports, unpublished manuscripts, conference presentations
Predefinition of Study Eligibility Criteria
![Page 14: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/14.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o More! than one computerized bibliographic database (Web of Science, SSRN, Google Scholar, Research Gate)
1. Effective keyword search
2. Review articles
3. Backward- and forward citation analyses
4. Manual search of relevant publication outlets
5. Experts
o Retrieve everything feasible; any omissions may create a selection bias!
Identification ofStudies
![Page 15: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/15.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Identification numbers
o Bibliography
o Abstract
o Publication type
o Availability (ordered from…),
o Eligibility (dropped, abandoned…)
Recording of identified Studies
![Page 16: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/16.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Methodological quality exists in the eye of the beholder
– Research findings are not robust to methodological differences among studies
– Few schemes for assessing methodological quality
o Full representation of available research on a topic
Study selection according to methodological criteria?
![Page 17: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/17.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Bivariate correlations
o Measurement research (reliability)
o Std. vs. unstd. regression coefficients
o Proportion of observations
o Pre-Post contrasts
o Group contrast , mean differences
You can mix different forms but the same effect size statistics should be used for analyzing all the findings in a given meta-analysis and you have to control for the form of research in the meta-regressions
Study selection according to form of research findings?
![Page 18: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/18.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
1. Problem Specification
2. Study Retrieval
3. Coding
Content
18
![Page 19: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/19.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Study descriptors (x)
– Study-level: time, sample and source descriptors, method, theory, researcher etc.
– Sub-effect-level: measurements, type of statistical test, control variables, construct reliability, missing data etc.
o Effect sizes (y)
– Std. size of the effect
– Sample size
– T-statistic
Developing a Coding Protocol
![Page 20: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/20.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
Example of a Data Base
![Page 21: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/21.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
Example of Study Coding
![Page 22: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/22.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Use theoretical consideration to decide which variable should be included and which not
o Use close-ended items (open-ended only if categories for variables cannot predetermined)
o Cluster items that deal with similar topics
o Include not too many variables (time!)
o Include only variables which are available among all studies
o Develop coder guidelines, i.e. full definition of each item and coding of ambiguous, unusual, or borderline cases
Important Aspects of Coding Protocols
![Page 23: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/23.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Must understand the coding protocol
o Must have knowledge and skills to read and interpret research
o Must understand complex statistical reporting
o Mostly at least Master student!
Tip: Do the coding on your own and involve a master student for reliability checks
Training of Coders
![Page 24: Soziologisches Institut–Prof. Dr. Katja Rost ...](https://reader031.fdocuments.in/reader031/viewer/2022013009/61ce26fe0869dd63097ed7b7/html5/thumbnails/24.jpg)
MAER 201310.08.2013
Soziologisches Institut – Prof. Dr. Katja Rost
o Internal validity
– Consistency of a single coder: Draw a subsample of 20 or more studies and code it again (time lag!)
– Consistency of different coders: Compare the coding of two independent coders for a subsample of 20 or more studies
– You may include a confident scale for difficult items
o External validity
– Include missing data studies and analyze them separately for systematic differences
– Differentiate between „missing“ and „not applicable“
Reliability Checks