PowerPoint Presentation

Evidence-Based Management(Searching & Finding Scientific Evidence)Pitstop Evidence-Based HR, VOV lerend netwerk,Antwerpen, 11 januari 2013

Pit stop EBMgt Evidence-Based ManagementA (very) Short Introduction Pit stop EBMgtMcMaster University Medical School, CanadaMedicine: Founding fathers

David SackettGordon GuyattPit stop EBMgt

How it all startedPit stop EBMgtMore than 1 million articles in 40,000 medical journals per year (= 1995; now probably more than 2 million). For a specialist to keep up this means reading 25 articles every day (for a GP more than 100!)

Most of the new insights and treatment methods dont reach the target group

Problem I: too much informationPit stop EBMgtProblem I: too much informationHRM: 1,350 articles in 2010 (ABI/INFORM). For an HR manager to keep up this means reading 3 to 4 articles every day (for a general manager more than 50!)

Pit stop EBMgtProblem II: persistent convictionsif youre hyperventilatingbreathe into a bag

Pit stop EBMgtProblem III: jumping to conclusionselderly people who have an irregular heartbeat are much more likely to die of coronary diseasegive them a drug that reduces the number of irregular beats

Pit stop EBMgtHow 40,000 cardiologists can be wrongIn the early 1980s newly introduced anti-arrhythmic drugs were found to be highly successful at suppressing arrhythmias.

Not until a RCT was performed was it realized that, although these drugs suppressed arrhythmias, they actually increased mortality.

The CAST trial revealed Excess mortality of 56/1000.

By the time the results of this trial were published, at least 100,000 such patients had been taking these drugs.

Pit stop EBMgtDavid SackettHalf of what you learn in medical school will be shown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you which half.The most important thing to learn is how to learn on your own.(Remember that your teachers are as full of bullshit as your parents)Pit stop EBMgtEvidence-Based Practice1991Medicine1998Education1999Social care2000Nursing2000Criminal justice????Management?

Pit stop EBMgt2. Evidence based management:What is it?

Pit stop EBMgtEvidence based practice:Improve information to support decision making DefinitionPit stop EBMgtDefinitionEvidence-based management means making decisions about the management of employees, teams or organizations through the conscientious, explicit and judicious use of four sources of information:1. The best available scientific evidence2. Organizational facts, metrics and characteristics3. Stakeholders values and concerns4. Practitioner expertise and judgmentPit stop EBMgt

Four sourcesPit stop EBMgt15Evidence-based practice:

Focuses on the decision making processThinks in terms of probability (instead of golden bullets).Pit stop EBMgt163. Evidence-based management:Why do we need it?

Pit stop EBMgt

Four sourcesPit stop EBMgt18

Trust me: 20 years of management experience!Pit stop EBMgtBounded rationality

Pit stop EBMgtBounded rationality

System 1

FastIntuitive, associativeheuristics & biasesSystem 2

Slow (lazy)Deliberate, reasoningRational

Pit stop EBMgtSeeing order in randomnessMental corner cuttingMisinterpretation of incomplete dataHalo effectFalse consensus effectGroup thinkSelf serving biasSunk cost fallacyCognitive dissonance reductionSystem 1: prone to biasesConfirmation biasAuthority biasSmall numbers fallacyIn-group bias Recall biasAnchoring biasInaccurate covariation detectionDistortions due to plausibilityPit stop EBMgt2. Searching & Finding The Best Available Scientific Evidence Pit stop EBMgtStep 1: Formulate a focused question

Pit stop EBMgt5-step approachEBMgt is a 5-step approach

Formulate an answerable question (PICOC)Search for the best available evidenceCritical appraise the quality of the found evidenceIntegrate the evidence with managerial expertise and organizational concerns and applyMonitor and evaluate the resultsPit stop EBMgtAsking the right question?Effect vs Non-effect

Pit stop EBMgtTypes of questions: effectDoes it work?Does it work better than ....?Does it have an effect on ....?What is the success factor for ....?What is required to make it work ...?Will it do more good than harm?

EffectPit stop EBMgtTypes of questions: non-effectNeeds:What do people want or need?Attitude:What do people think or feel? Experience: What are peoples experiences? Prevalence:How many / often do people / organizations ...?Procedure: How can we implement ...?Process:How does it work?Explanation:Why / how does it work?Economics:How much does it cost?Pit stop EBMgtFocused question?Does team-building work?What are the costs and benefits of self-steering teams?What are the success factors for culture change? Does management development improve the performance of managers?Does employee participation prevent resistance to change?How do employees feel about 360 degree feedback?Pit stop EBMgtWhat is a team?What kind of teams?In what contexts/settings?What counts as team-building?What does work mean?What outcomes are relevant?Over what time periods?Foreground question?Does team-building work?Pit stop EBMgtP =PopulationI = Intervention (or success factor)C = Comparison O = OutcomeC = Context

Answerable question: PICOCPit stop EBMgtExample: merger

Pit stop EBMgtAnswerable question: PICOCImagine you are a consultant, your client is the board of directors of a large Canadian health-care organization. The board of directors has plans for a merger with a smaller healthcare organization in a nearby town. However, its been said that the organizational culture differs widely between the two organizations. The board of directors asks you if this culture-difference can impede a successful outcome of a merger. Most of them intuitively sense that cultural differences matter, but they want an evidence-based advice. Pit stop EBMgtAnswerable question: PICOCWhat else would you like to know?Pit stop EBMgt

Answerable question: PICOCP: What kind of Population are we talking about? Middle managers, back-office employees, medical staff, clerical staff? O: What kind of Outcome are we aiming for? Employee productivity, return on investment, profit margin, competitive position, innovation power, market share, customer satisfaction? P/C: And how is the assumed cultural difference assessed? Is it the personal view of some managers or is it measured by a validated instrument? Pit stop EBMgtAccording to the board the objective of the merger is to integrate the back-office of the two organizations (ICT, finance, purchasing, facilities, personnel administration, etc.) in order to create economy of scale. The front offices and primary process of the two organizations will remain separate. The cultural difference is not objectively assessed (it is the perception of the senior managers of both organizations).Answerable question: PICOCPit stop EBMgtP =back office employees in a healthcare organisationI = merger, integration back office C = status quoO = economy of scale C = different organizational culture, unequal

Answerable question: PICOCPit stop EBMgtRead the following five scenarios Formulate on the basis of each scenario a focused question (use the PICOC format).Exercise

Pit stop EBMgt38Scenario 2Ik werk als manager bij een grote financile dienstverlener. Het concern is een holdingmaatschappij met daaronder een aantal divisies met een eigen P&L verantwoordelijkheid. Het accountmanagement van de divisies wordt aangestuurd op basis van een eigen set kritische prestatie indicatoren (kpis) die niet onderling is afgestemd.

Het gevolg is dat de accountmanagers elkaar regelmatig in de wielen rijden bij gezamenlijke klanten en de omzet wordt weggekaapt bij de ander. Kortom, er wordt onvoldoende samengewerkt. Nu wordt door de Executive Board van de holding een beroep gedaan op de divisiedirecties om meer met elkaar te gaan samenwerken.

Mijn vraag is: staat het feit dat de divisies geen gedeelde kpis hebben een goede interne samenwerking in de weg of hoeft dit geen probleem te zijn?Pit stop EBMgtScenario 3I am a Dutch quality manager at a large international brewery. In the past 4 years I facilitated a lean management program that is running for more than 10 years in this organization. Since a couple of years, however, we see a decline in results, such as:

The number of problem analysis and improvement teams has been increased, but the amount of breakdowns and short stops in the production lines has not decreased.There is slow progress for the total program for the Netherlands at this moment.

According to the middle managers the improvement program does not address really important topics. I would therefore like to know how our employees feel about lean management.Pit stop EBMgt40Neem een actueel probleem / vraag / issue dat speelt in de organisatie waar je stage hebt gelopen (of een vraagstuk dat je persoonlijke interesse heeft).Formuleer op basis daarvan een onderzoeksvraagGeef aan wat voor type vraag het is (effect of non-effect)Benoem de PICOC elementen. Pit stop EBMgt41Step 2: Search for the best available evidence

Pit stop EBMgt5-step approachEBMgt is a 5-step approach

Formulate an answerable question (PICOC)Search for the best available evidenceCritical appraise the quality of the found evidenceIntegrate the evidence with managerial expertise and organizational concerns and applyMonitor and evaluate the resultsPit stop EBMgt Searching evidenceWhat do we search?

Pit stop EBMgtType of InformationSourceCurrent InformationWall street Journal, Financial Times, Business week, Financieel DagbladOverview of a subjectGeneral backgroundAcademic InformationStatistical InformationTextbooks and popular booksEncyclopedias, yearbooks & book reviewsABI/INFORM, Business Source Premier, Emerald, PsychInfo, Science DirectCBS Statline, EurostatTheories about a subjectTextbooks and encyclopediasInformation sourcesCompany informationCompany Annual Reports, Datastream, Factiva.com, AmadeusPit stop EBMgtArticles in peer reviewed journals

Pit stop EBMgtSearching for evidence

Postgraduate CoursePRESENTATION ONE18-01-13Introduction to Evidence-Based Practice47

Databases

Postgraduate CourseABI/INFORMBusiness Source ElitePsycINFOWeb of KnowledgeERICGoogle Scholar

Searching for evidencePostgraduate CoursePRESENTATION ONE18-01-13Introduction to Evidence-Based Practice49 Searching evidenceHow do we search?Search Strategy

Postgraduate CourseTwo types of search strategiesSearch strategyBuilding blocks method

Snowball method

Pit stop EBMgtSnowball methodStarting from one book or article, you search for other literature on the same topic.

Snowballing to older publications by finding out which publications were used by the author (see bibliography of book or article). Snowballing to more recent publications by finding out how often that book or article has been cited by other authors (see Web of Knowledge or Google Scholar).

Pit stop EBMgtSnowball method

Pit stop EBMgtSynonyms or related terms . . . .Synonyms or related terms . . . .Synonyms or related terms . . . .Building blocks methodSynonyms or related terms . . . .Keyword 1Keyword 2Keyword 3Keyword 4ANDANDANDOROROR

Pit stop EBMgtP =back office employees, health care I = merger, integration, back office C = status quoO = economy of scale C = different organizational culture, unequal

Answerable question: PICOCUnderline the keywordsNumber the order of importance from 1-4Pit stop EBMgtP =back office employees 5. healthcareI = 1. merger, 3. integration, back office C = status quoO = 4. economy of scale C = different 2. organizational culture, unequal

Answerable question: PICOCUnderline the keywordsNumber the order of importancePit stop EBMgt Search termsOperationalise your Pico elements!O = long term profitability?

Pit stop EBMgtConstruct validity

The Ambidextrous Organization (??)Pit stop EBMgtcorporate culture: organizational behavior/character, corporate identitymerger: acquisition, take-over, fusion, combination, unificationprofitability: profit, advantage, return on investment, shareholder value The keywords of your PICOC may be enough.If not, select more words by using: Select keywords synonyms alternate spelling, translations related terms / words / subjects narrower or broader termsPit stop EBMgt Search in the following order Thesaurus (SU) Title (TI) Abstract (AB) Alternate spellings, synonymsPit stop EBMgtMergerFusionCombinationTake overAcquisitionUnification1. Merger3. IntegrationHealthcare organizationNon profitNot for profit4. Health care organizationAND Search Query: an exampleIIOORORIntegrationCorporate cultureOrganizational behaviorOrganizational characterCorporate identityCore beliefsShared values2. Corporate cultureCANDORPit stop EBMgtorangesapplesapples AND orangesapples OR orangesorangesapplesBoolean operatorsPit stop EBMgt Start up

Pit stop EBMgt Always search for indexed topics or thesaurus terms

Pit stop EBMgt thesaurus terms

Pit stop EBMgtUse the history function to combine results Combining results

Pit stop EBMgtUse the history function to combine results Combining results

Pit stop EBMgt Narrow your results

Pit stop EBMgt Narrow your results

Pit stop EBMgt Save your results

Pit stop EBMgt If an article is not available in ABI/ Inform or BSE: try Google (advanced search)

Pit stop EBMgtSummaryFormulate a focused question and break it down into keywords (PICOC), Underline the 2 to 4 most relevant keywords, Start with the most relevant keyword: is it listed in the thesaurus? If not: search in the title or abstract. Repeat your search with alternative terms, synonyms or alternate spellings. Combine the outcome of step 4 (thesaurus), 5 (title or abstract) and 6 (alternative terms) with OR (use the history function!)Repeat above steps for keyword 2Combine the outcome of your search with keyword 1 and 2 with AND (use the history function!)Filter the articles with studies (subject > studies)Read the titles and abstracts of your results, select the relevant articles and safe them.No full text available in ABI/Inform or BS Elite? Try Google Advanced.Pit stop EBMgt Learning through play !Try all buttons Make lots of mistakesHave fun !

Pit stop EBMgtAssignment: search strategySearch for research articles on teambuilding

Pit stop EBMgt742. Search in ABI/Inform:How many articles has Stephen Covey published in peer reviewed journals?How many of these articles are based on scientific research?Are there articles (by other authors) that are critical of Coveys 7 Habits?How many of these critical articles are based on scientific research?

Exercise: Search for evidencePit stop EBMgt75Evidence-based management Step 3: Critical appraisal of studies

Pit stop EBMgt5-step approachEBMgt is a 5-step approach

Formulate an answerable question (PICOC)Search for the best available evidenceCritically appraise the quality of the found evidenceIntegrate the evidence with managerial expertise and organizational concerns and applyMonitor and evaluate the resultsPit stop EBMgtEffect vs Non-effect

Pit stop EBMgtInternal validity

Pit stop EBMgtinternal validity = indicates to what extent the results of the research may be biased and is thus a comment on the degree to which alternative explanations for the outcome found are possible.Internal validityPit stop EBMgt

Levels of internal validityPit stop EBMgtWhich design for which question?

ExplanationPit stop EBMgtBest research design?

Pit stop EBMgt83

Critical appraisalPit stop EBMgt84Critical appraisal

Postgraduate Course

Is the study design appropriate to the stated aims?Was a control group used?Was a pretest used?Are the measurements likely to be valid and reliable?Could bias or confounding have occurred?How large was the effect size?Pit stop EBMgtLevels of internal validityWere there enough subjects in the study?Was a control group used?Were the subjects randomly assigned?Was a pretest used?Was the study started prior to the intervention or event?Was the outcome measured in an objective and reliable way?

6x yes = very high (A)5x yes = high (A)4-3x yes = limited (B)2x yes = low (C)1-0x yes = very low (D)Pit stop EBMgt