DONE ADU Slides 2014--2-EBM & Research Basics
17
Human Resources, Law and Management Department - September 2009 A Focus on Evidence-Based Management Leading scholars have promoted EBM as an integrated and comprehensive approach to help bridge the rigor-relevance gap and improve the management of organizations Denise M. Rousseau’s 2005 Presidential Address to the Academy of Management Jeffrey Pfeffer’s 2007 Keynote Address to the Society for Industrial and Organizational Psychology 2007 Conference Board Report, Evidence-Based Human Resources: A Primer and Summary of Current Literature.
description
ebm
Transcript of DONE ADU Slides 2014--2-EBM & Research Basics
Présentation PowerPointA Focus on Evidence-Based Management
Leading scholars have promoted EBM as an integrated and comprehensive approach to help bridge the rigor-relevance gap and improve the management of organizations
Denise M. Rousseau’s 2005 Presidential Address to the
Academy of Management
Jeffrey Pfeffer’s 2007 Keynote Address to the Society for
Industrial and Organizational Psychology
Resources: A Primer and Summary of Current Literature.
Human Resources, Law and Management Department - September 2009
The Origins: Evidence-Based Medicine
“the conscientious, explicit and judicious use of current best medical/clinical research evidence in making decisions about the care of individual patients.”
Dr. David Sackett (McMaster University)
Human Resources, Law and Management Department - September 2009
Evidence-Based Management
Ignorance is arguably higher in management than in medicine…
“If doctors practiced medicine like many companies practice management, there would be more unnecessarily sick or dead patients and many more doctors in jail or suffering other penalties for malpractice”
(Pfeffer & Sutton, 2006, p1)
Pfeffer, J. & Sutton, R. I. (2006). Evidence-based management. Harvard Business Review, January, pp1-12.
Confer also Rousseau, D. M. (2006). Is there such a thing as "evidence-based management"? Academy of Management Review. 31, pp 256-269
Human Resources, Law and Management Department - September 2009
What is Evidence-Based Management?
A “way of thinking” about organizational decisions in a systematic manner
Being committed to doing things based critically on the “best” available scientific evidence
Not solely based upon unsystematic experience, instincts, intuition, tradition, untested beliefs, trends and fashions etc.
Translating the best scientific evidence into organizational practices (“Big E Evidence”)
Systematically gathering local data to inform decision-making (“little e evidence”)
Learning about human behavior within organizations and using that knowledge to enhance managerial practice.
Human Resources, Law and Management Department - September 2009
What is Evidence-Based Management?
practitioner expertise and judgment
a critical evaluation of the best available research evidence
the perspectives of those people who might be affected by the decision
Briner, R. B., Denyer, D., & Rousseau, D. M. (2009). Evidence-Based Management: Concept Cleanup Time? Academy of Management Perspectives. 23, pp19-32.
“Using evidence does not mean slavishly following it, acting only when there is good evidence, or doing nothing if there is none.”
Briner, R. B. and Rousseau, D. M. (2011). Evidence-Based I–O Psychology: Not There Yet. Industrial and Organizational Psychology.4, pp3–22.
Human Resources, Law and Management Department - September 2009
What EBM is !
Something practitioners already do to some extent
About the practice of management
A family of related approaches to decision making
A way of thinking about how to make decisions
About using a wide range of different kinds of research evidence depending on the problem
Practitioners using research as only one source of information to inform decisions
A means of getting existing management research out to practitioners
Likely to help both the process and outcome of practitioner decision-making
About questioning ideas such as “best practice”
Human Resources, Law and Management Department - September 2009
What EBM is NOT!
NOT A brand-new way of making decisions
NOT About conducting particular types of research
NOT A single decision-making method
NOT A rigid, one-size-fits all decision-making formula
NOT About using only certain types of research evidence irrespective of the problem
NOT Scholars or researchers telling practitioners what to do
NOT About conducting research only about management practice
NOT The solution to all management problems
NOT About identifying and promoting “best practice”
Human Resources, Law and Management Department - September 2009
EBM QUIZ
Rynes, S. L., Colbert, A. E., & Brown, K. G. (2002). HR professionals‘ beliefs about effective human resource practices: Correspondence between research and practice. Human Resources Management, 41(2), pp149-174.
Rynes, S. L., Giluk, T. L., & Brown, K. G. (2007). The very separate worlds of academic and practitioner periodicals in human resource management: Implications for evidence-based management. Academy of Management Journal, 50(5), pp987-1008.
TRUE or FALSE?
General mental ability is the strongest, or one of the strongest, predictors of job performance
TRUE
Setting goals and providing feedback is a highly effective motivational practice
TRUE
FALSE
TRUE
TRUE
EBM QUIZ
On average, encouraging employees to participate in decision making is more effective for improving organizational performance than is setting performance goals
FALSE
Most people under-evaluate how well they perform on the job
FALSE
Training for simple skills will be more effective if it is presented in one concentrated session than if it is presented in several sessions over time
FALSE
“Integrity tests” designed to predict whether someone will steal, be absent, etc. can predict those types of counterproductive behaviors
TRUE
Graphology (handwriting analysis) is able to predict job performance effectively because it is difficult to “fake”
FALSE
Theory?
An organized framework of concepts aimed at illustrating, interpreting and/or predicting reality
“It is the theory that decides (determines) what we can observe”
→ A Frame or an Abstraction of Reality Albert Einstein (via Gregory, 1990)
“There is nothing more practical than a good theory” Kurt Lewin (1951,p169)
*
Independent Variable (Predictor) & Dependent Variable (Criterion)
a theoretical model establishes a causal relationship between 2 variables which is then tested in research studies
the independent variable (IV) can be either qualitative or quantitative and is typically the variable being manipulated or changed in the study→ What is being changed
the dependent variable (DP) can be either qualitative or quantitative and is the observed result of the independent variable being manipulated or changed → What is being observed?
Be aware that finding a relation between the IV & the DP doesn't establish the direction of causality
Causality or which is the cause and which is the effect can only be established by the type of the research design
Human Resources, Law and Management Department - September 2009
2 Types of Research Design…
Concurrent or Cross-Sectional Research Design: both the independent and the dependent variable are measured in the same point in time (roughly simultaneously)→ A Cross-Section in the Time-Continuum
Predictive and/or Longitudinal Research Design: the independent variable is measured at an earlier point in time than the dependent variable and/or repeatedly for a long time period
Only Predictive and/or Longitudinal Research Designs can establish Causality and consequently they are considered scientifically superior
However in the position of IV you ought to find that variable which is at least theoretically assumed to be the cause of an effect in the DP, with the former being treated as the predictor and the later as the criterion
Human Resources, Law and Management Department - September 2009
Reliability
reliability: the estimated consistency of the measurement for a given instrument/method (e.g. test) employed with itself (internal validity), across time (test-retest) and within/across raters (intra/inter-rater)→ the repeatability of the measurement
the level of reliability puts an upper limit to the possible level of validity
high reliability is necessary but not sufficient for high validity
internal reliability: coefficient alpha 0.80(at least 0.70) for a measure to have an acceptable degree of reliability
Inter-rater agreement: minimum level 75% or higher
test-retest reliability: concerned with time stability of measurement; level of r should range between r = .50 to .90
Intra-rater agreement: for short time intervals between measures, a fairly high relationship is expected; r = .80 to .90
Human Resources, Law and Management Department - September 2009
Validity
conclusion or criterion related validity: the degree to which the independent variable measures or is correlated to the dependent variable (correlation coefficient: r)
concurrent validity under a concurrent or cross-sectional research design
predictive validity under a predictive research design
the square of the correlation coefficient states the amount of variance in the DV explained by the IV
internal validity: an overall inductive estimation of the degree of established causality between IV & DV
external validity: an estimation of the degree of generalizability of our conclusions at the population level
Human Resources, Law and Management Department - September 2009
Intervening Variables: Moderator
intervening variables: variables which interact with the IV affecting its relationship with the DV and which when incorporated in our models upgrade theoretical rigor
moderator: a variable that interacts with the IV affecting the direction (positive or negative) and/or strength of its relation with the DV
a type of multiplicative statistical interaction
the moderator can be either a qualitative (e.g. gender, race…) or a quantitative (e.g. amount of pay…) variable
be aware that the moderator stands at the same point of causal precedence or supremacy with the IV
Human Resources, Law and Management Department - September 2009
Intervening Variables: Mediator
mediator: a variable through which the IV relates with the DV either partially or fully
a mediation is not a statistical interaction
a mediator can be either a qualitative or a quantitative variable
when the independent variable is related to the DV both indirectly through the mediator and directly we have a partial mediation
when the IV is related to the DV only through the mediator then we have a full mediation
the mediator is causally inferior to the IV but causally superior to the DV
Human Resources, Law and Management Department - September 2009
Control Variables
control variables: variables known from research to be having an effect on the DV and for that reason are kept constant to prevent their influence on the effect of the IV on the DV→ alternative causes (predictors) of the dependent variable (criterion)
→ What other variables have to be held constant in order to isolate the net effect of the IV to the DV?
Leading scholars have promoted EBM as an integrated and comprehensive approach to help bridge the rigor-relevance gap and improve the management of organizations
Denise M. Rousseau’s 2005 Presidential Address to the
Academy of Management
Jeffrey Pfeffer’s 2007 Keynote Address to the Society for
Industrial and Organizational Psychology
Resources: A Primer and Summary of Current Literature.
Human Resources, Law and Management Department - September 2009
The Origins: Evidence-Based Medicine
“the conscientious, explicit and judicious use of current best medical/clinical research evidence in making decisions about the care of individual patients.”
Dr. David Sackett (McMaster University)
Human Resources, Law and Management Department - September 2009
Evidence-Based Management
Ignorance is arguably higher in management than in medicine…
“If doctors practiced medicine like many companies practice management, there would be more unnecessarily sick or dead patients and many more doctors in jail or suffering other penalties for malpractice”
(Pfeffer & Sutton, 2006, p1)
Pfeffer, J. & Sutton, R. I. (2006). Evidence-based management. Harvard Business Review, January, pp1-12.
Confer also Rousseau, D. M. (2006). Is there such a thing as "evidence-based management"? Academy of Management Review. 31, pp 256-269
Human Resources, Law and Management Department - September 2009
What is Evidence-Based Management?
A “way of thinking” about organizational decisions in a systematic manner
Being committed to doing things based critically on the “best” available scientific evidence
Not solely based upon unsystematic experience, instincts, intuition, tradition, untested beliefs, trends and fashions etc.
Translating the best scientific evidence into organizational practices (“Big E Evidence”)
Systematically gathering local data to inform decision-making (“little e evidence”)
Learning about human behavior within organizations and using that knowledge to enhance managerial practice.
Human Resources, Law and Management Department - September 2009
What is Evidence-Based Management?
practitioner expertise and judgment
a critical evaluation of the best available research evidence
the perspectives of those people who might be affected by the decision
Briner, R. B., Denyer, D., & Rousseau, D. M. (2009). Evidence-Based Management: Concept Cleanup Time? Academy of Management Perspectives. 23, pp19-32.
“Using evidence does not mean slavishly following it, acting only when there is good evidence, or doing nothing if there is none.”
Briner, R. B. and Rousseau, D. M. (2011). Evidence-Based I–O Psychology: Not There Yet. Industrial and Organizational Psychology.4, pp3–22.
Human Resources, Law and Management Department - September 2009
What EBM is !
Something practitioners already do to some extent
About the practice of management
A family of related approaches to decision making
A way of thinking about how to make decisions
About using a wide range of different kinds of research evidence depending on the problem
Practitioners using research as only one source of information to inform decisions
A means of getting existing management research out to practitioners
Likely to help both the process and outcome of practitioner decision-making
About questioning ideas such as “best practice”
Human Resources, Law and Management Department - September 2009
What EBM is NOT!
NOT A brand-new way of making decisions
NOT About conducting particular types of research
NOT A single decision-making method
NOT A rigid, one-size-fits all decision-making formula
NOT About using only certain types of research evidence irrespective of the problem
NOT Scholars or researchers telling practitioners what to do
NOT About conducting research only about management practice
NOT The solution to all management problems
NOT About identifying and promoting “best practice”
Human Resources, Law and Management Department - September 2009
EBM QUIZ
Rynes, S. L., Colbert, A. E., & Brown, K. G. (2002). HR professionals‘ beliefs about effective human resource practices: Correspondence between research and practice. Human Resources Management, 41(2), pp149-174.
Rynes, S. L., Giluk, T. L., & Brown, K. G. (2007). The very separate worlds of academic and practitioner periodicals in human resource management: Implications for evidence-based management. Academy of Management Journal, 50(5), pp987-1008.
TRUE or FALSE?
General mental ability is the strongest, or one of the strongest, predictors of job performance
TRUE
Setting goals and providing feedback is a highly effective motivational practice
TRUE
FALSE
TRUE
TRUE
EBM QUIZ
On average, encouraging employees to participate in decision making is more effective for improving organizational performance than is setting performance goals
FALSE
Most people under-evaluate how well they perform on the job
FALSE
Training for simple skills will be more effective if it is presented in one concentrated session than if it is presented in several sessions over time
FALSE
“Integrity tests” designed to predict whether someone will steal, be absent, etc. can predict those types of counterproductive behaviors
TRUE
Graphology (handwriting analysis) is able to predict job performance effectively because it is difficult to “fake”
FALSE
Theory?
An organized framework of concepts aimed at illustrating, interpreting and/or predicting reality
“It is the theory that decides (determines) what we can observe”
→ A Frame or an Abstraction of Reality Albert Einstein (via Gregory, 1990)
“There is nothing more practical than a good theory” Kurt Lewin (1951,p169)
*
Independent Variable (Predictor) & Dependent Variable (Criterion)
a theoretical model establishes a causal relationship between 2 variables which is then tested in research studies
the independent variable (IV) can be either qualitative or quantitative and is typically the variable being manipulated or changed in the study→ What is being changed
the dependent variable (DP) can be either qualitative or quantitative and is the observed result of the independent variable being manipulated or changed → What is being observed?
Be aware that finding a relation between the IV & the DP doesn't establish the direction of causality
Causality or which is the cause and which is the effect can only be established by the type of the research design
Human Resources, Law and Management Department - September 2009
2 Types of Research Design…
Concurrent or Cross-Sectional Research Design: both the independent and the dependent variable are measured in the same point in time (roughly simultaneously)→ A Cross-Section in the Time-Continuum
Predictive and/or Longitudinal Research Design: the independent variable is measured at an earlier point in time than the dependent variable and/or repeatedly for a long time period
Only Predictive and/or Longitudinal Research Designs can establish Causality and consequently they are considered scientifically superior
However in the position of IV you ought to find that variable which is at least theoretically assumed to be the cause of an effect in the DP, with the former being treated as the predictor and the later as the criterion
Human Resources, Law and Management Department - September 2009
Reliability
reliability: the estimated consistency of the measurement for a given instrument/method (e.g. test) employed with itself (internal validity), across time (test-retest) and within/across raters (intra/inter-rater)→ the repeatability of the measurement
the level of reliability puts an upper limit to the possible level of validity
high reliability is necessary but not sufficient for high validity
internal reliability: coefficient alpha 0.80(at least 0.70) for a measure to have an acceptable degree of reliability
Inter-rater agreement: minimum level 75% or higher
test-retest reliability: concerned with time stability of measurement; level of r should range between r = .50 to .90
Intra-rater agreement: for short time intervals between measures, a fairly high relationship is expected; r = .80 to .90
Human Resources, Law and Management Department - September 2009
Validity
conclusion or criterion related validity: the degree to which the independent variable measures or is correlated to the dependent variable (correlation coefficient: r)
concurrent validity under a concurrent or cross-sectional research design
predictive validity under a predictive research design
the square of the correlation coefficient states the amount of variance in the DV explained by the IV
internal validity: an overall inductive estimation of the degree of established causality between IV & DV
external validity: an estimation of the degree of generalizability of our conclusions at the population level
Human Resources, Law and Management Department - September 2009
Intervening Variables: Moderator
intervening variables: variables which interact with the IV affecting its relationship with the DV and which when incorporated in our models upgrade theoretical rigor
moderator: a variable that interacts with the IV affecting the direction (positive or negative) and/or strength of its relation with the DV
a type of multiplicative statistical interaction
the moderator can be either a qualitative (e.g. gender, race…) or a quantitative (e.g. amount of pay…) variable
be aware that the moderator stands at the same point of causal precedence or supremacy with the IV
Human Resources, Law and Management Department - September 2009
Intervening Variables: Mediator
mediator: a variable through which the IV relates with the DV either partially or fully
a mediation is not a statistical interaction
a mediator can be either a qualitative or a quantitative variable
when the independent variable is related to the DV both indirectly through the mediator and directly we have a partial mediation
when the IV is related to the DV only through the mediator then we have a full mediation
the mediator is causally inferior to the IV but causally superior to the DV
Human Resources, Law and Management Department - September 2009
Control Variables
control variables: variables known from research to be having an effect on the DV and for that reason are kept constant to prevent their influence on the effect of the IV on the DV→ alternative causes (predictors) of the dependent variable (criterion)
→ What other variables have to be held constant in order to isolate the net effect of the IV to the DV?
