May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected...
Transcript of May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected...
![Page 1: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/1.jpg)
Reed Early, BC Office of the Auditor General andNormand Péladeau, Provalis Research
May 31, 2009 Ottawa
![Page 2: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/2.jpg)
Part 1 QDA MethodsOutcomes – learners understand the origin and current practice of QDA methods in general, grounded theory in particular, and possible applications of the methods.
![Page 3: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/3.jpg)
1. Methodsa. useful for inductive researchb. useful in naturalistic inquiryc. qualitative methods growing consensusd. collection ↔ analysis ↔ collection
![Page 4: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/4.jpg)
2. Qualitative Data (QD)a. open b. exploratoryc. useful when questions to ask not yet been definedd. allows insights
![Page 5: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/5.jpg)
3. Overview of QDA
![Page 6: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/6.jpg)
4. QDA cycle
![Page 7: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/7.jpg)
5. overview of process
![Page 8: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/8.jpg)
6. Work plan of research process
![Page 9: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/9.jpg)
7. Characteristics of QDAa. constructivist - many meaningsb. context bound i.e. “cast"c. uses inflection i.e. "THIS was good."d. can be sorted in many ways e. QD by itself has meaning i.e “apple”
![Page 10: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/10.jpg)
8. Sources of QDa. interviewsb. focus Groupsc. field observationsd. survey comments e. historical recordsf. secondary datag. photos, paintings, songs
...
![Page 11: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/11.jpg)
9. Types of QDa. structured text writings, stories, survey
comments, news articles, books etc
b. unstructured text transcription, open interviews, focus groups, conversation
c. audio recordings, as above, music
d. visual recordings, graphics, art, pictures
![Page 12: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/12.jpg)
10. Methods Matched to Type of Data a. structured text
content and matrix analysis etc
b. unstructured text i.e. narratives, open interviews
phenomenology, grounded theory etc
c. audio i.e. interviews, anecdotes, “stories”, music
matrix analysis, grounded theory etc d. visual i.e. graphics, art, pictures
matrix analysis, grounded theory etc
![Page 13: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/13.jpg)
11. Methods Matched to Principle Task a. description
content analysis …b. explanation
matrix analysis …c. derive new ideas and insights
phenomenology …d. test significance
matrix analysis, quantitative …e. map theoretical relation
grounded theory, 2D mapping …
![Page 14: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/14.jpg)
12. Principles of QDAa. Data entry (gathering)b. Comprehending (immersion)c. Synthesizing (sifting)d. Theorizing (sorting)e. Re-Contextualizing (emerging theory)
![Page 15: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/15.jpg)
13. Data entry (analogous demo)
a. not easily mechanizedb. important part of processc. often done by analystd. concurrent with analysis e. transcribe thoroughly, ASAP f. write memos (reflect)g. coding (start with few)
![Page 16: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/16.jpg)
14. Comprehending (immersion)a. begin while entering data b. start QDA immediatelyc. “live with it”d. line by line examinatione. create new questions for collection
![Page 17: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/17.jpg)
15. Synthesizing (sifting) “quotes” (decontextualize)
a. use inductive categoriesb. find common threadsc. compare transcriptsd. aggregate stories
![Page 18: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/18.jpg)
16. Theorizing (sorting) “coding”a. ask questions of the datab. find alternative explanationsc. allow sufficient time d. be open to insights
![Page 19: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/19.jpg)
17. Re-Contextualizing a. develop theoretical “elegance”b. apply to other settingsc. examine fit to literature/researchd. describe emerging theory
![Page 20: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/20.jpg)
18. Data Management Principlesa. stay close to the datab. be sensitive to emergent theoryc. allow recontextualizing d. it is a non-linear process
![Page 21: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/21.jpg)
19. Grounded Theory
Primary documents (comprehending)
immerse in the primary documents
begin as data are collected
read/view/listen to the data
![Page 22: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/22.jpg)
b. Quotations (synthesizing)select and mark salient quotations/passages
compare each line to other data
![Page 23: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/23.jpg)
c. Coding (theorizing)assign codes in margin
group, sort, categorize codes into families
collect new data based on emerging theory, memos, codes
![Page 24: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/24.jpg)
d. Memos (aids in all processes)
record insights on memos or post-it notes ie: ideas for emerging theories, thematic ideas, linked memos
(Exercise p7)
![Page 25: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/25.jpg)
e. 2D Network (recontextualize)
create a 2D network (mind map)add and arrange network nodes
• (quotes, memos and codes)
collect more data as needed(Exercise p9)
![Page 26: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/26.jpg)
f. Generate theoryMake a matrix of themes (rows) by roles (cols)Fill in cells with either a selected quote or “*” to indicate missing data.Look for patterns, empty cells, areas of convergence.Generate an explanation and provide a short quote to support your "theory". (exercise p11)
![Page 27: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/27.jpg)
20. Displaying results (computer methods)
display code frequencies and charts (demonstrated in QDA Miner)
Code Count % Codes Cases % Cases
1.1 Defines Mgmt Structure, Roles, Resp 36 1.20% 21 5.30%
1.1 Framework Contents 42 1.40% 23 5.80%
1.1 Key People Involved 119 3.90% 59 14.80%
1.1 Lines of Authority 46 1.50% 26 6.50%
1.2 Approval and Endorsement 25 0.80% 17 4.30%
1.2 Key Elements of Strategic Plan Exist 72 2.40% 36 9.00%
1.2 Plan Communicated 55 1.80% 26 6.50%
1.2 Plan Updated 18 0.60% 14 3.50%
![Page 28: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/28.jpg)
Freq charts
![Page 29: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/29.jpg)
run crosstab (matrix) codes and variables
Proj A Proj B Proj C Time 1 Time 2 Time 3 Chi-square P value
finding 1.1.x 2 1 3 9.924 0.357
finding 1.2.x 1 9 1 2 17.447 0.042
finding 1.3.x 1 5 1 10.624 0.302
finding 1.4.x 1 2 12 4 2 41.016 0
finding 2.1.x 3 1 2 1 20.759 0.014
finding 2.2.x 1 2 2.13 0.989
finding 2.3.x 20 2 1 3 4 2 16.514 0.057
Risks 6 2 2 17 4 4 33.709 0
Best Practices 4 1 1 3 6 4 49.178 0
![Page 30: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/30.jpg)
display code 2D network map (MDS)
![Page 31: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/31.jpg)
export .html web page for multimedia use
![Page 32: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/32.jpg)
Q&A Questions and answers
![Page 33: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/33.jpg)
Exercise: Quotes, Codes & Memos
![Page 34: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/34.jpg)
Exercise: Grounded Theory and 2D Network Mapping
![Page 35: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/35.jpg)
Alternate Exercise: Theorizing Using Matrix Analysis
![Page 36: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/36.jpg)
Some QDA Methods (and Software)1. Content Analysis
Word, QDA Miner Word-SimStat, Excel, Atlas-ti 2. Matrix analysis
QDA Miner Word-SimStat, NVivo3. Grounded Theory Mapping
QDA Miner Word-SimStat, Atlas-ti4. Phenomenology - using mind maps
Inspiration, Visio5. Concept Mapping
QDA Miner Word-SimStat, Concept Systems
![Page 37: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/37.jpg)
QDA softwareQDA Miner Word-SimStat (Provalis, CAN)Atlas-ti (Scientific Software, GER)NVivo (QSR, AU) Inspiration (USA)Concept Systems (USA)Excel, Access, SPSS (USA)
![Page 38: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/38.jpg)
How to choose software - Key QuestionsLewins and Silver 2005
![Page 39: May 31, 2009 Ottawa - evaluationcanada.ca · 5/31/2009 · Fill in cells with either a selected quote or “*” to indicate missing data. Look for patterns, empty cells, areas of](https://reader033.fdocuments.in/reader033/viewer/2022060320/5f0d0ef47e708231d4387909/html5/thumbnails/39.jpg)
HandoutsHandout 1. Summary of Manual and Software Qualitative
Methods Handout 2. Website, Software and Internet Resources