Project Design and Data Collection Methods: A quick and dirty introduction to classroom research...
-
date post
22-Dec-2015 -
Category
Documents
-
view
215 -
download
1
Transcript of Project Design and Data Collection Methods: A quick and dirty introduction to classroom research...
![Page 1: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/1.jpg)
Project Design and Data Collection Methods:
A quick and dirty introduction to classroom research
Margaret Waterman
September 21, 2005
SoTL Fellows 2005-2006
![Page 2: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/2.jpg)
Where are we inthe Action Research Cycle?
Identify Innovation
Gather Data
Analyze DataInterpret Data
Develop Action Plan
☻
Decide when and how to collect data
![Page 3: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/3.jpg)
Design vs. Method Design is about when and from whom you are
going to collect data so that the data can be interpreted.
On Tuesday
With Colonel Mustard
Methods are about where and how you are going to collect that data.
In the Library
With a candlestick
With an interview
![Page 4: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/4.jpg)
Designs: Experimental Experimental: strongest for drawing
inferences about cause and effect. Control vs. experimental groups, randomly
assigned, one variable at a time Diagram as follows:
O1 X1 O2
O1 O2
O = observation, X = treatment; Each row is a group
![Page 5: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/5.jpg)
Experimental designs
Problems: Not always possible in education to have
two or more matched groups,
Random assignment usually not feasible,
May not be suitable for small-scale projects
Usually not for classroom research problems
![Page 6: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/6.jpg)
Designs: Pre-Experimental Weakest designs: Post test only
X1 O1
Slightly stronger: Post-test with comparison group
X1 O1
O1
NOT RECOMMENDED
![Page 7: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/7.jpg)
Pre-Experimental: Post Test OnlyProblems:
Too many other reasons for observed conditions Cannot draw valid inferences about causality
because you don’t know what pre-treatment condition is.
In comparison group: may have differences in incoming ability in the two groups
Good for: preliminary observation and development of hypotheses, not showing cause and effect
![Page 8: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/8.jpg)
Designs: Pre-Experimental One group, Pre-test Post-test
O1 X1 O2
Uses: Can make some inferences about cause and effect, but only when discuss possible intervening variables.
Useful in classrooms when trying new techniques or materials.
![Page 9: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/9.jpg)
Pre-Experimental: Pre-test, Post-test Problems:
Can’t discount all intervening variables: e.g., history, awareness of being in an experiment, selection effects on group membership.
Generalizability is limited May sensitize students with the pre-test
![Page 10: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/10.jpg)
Designs: Quasi-Experimental
Uses an experimental design, but groups are not randomly assigned.
O1 X1 O2
O1 O2 Effect of treatment isolated
X1 O2 Use two more groups to
O2 isolate effect of pre-test.
Can make inferences with more confidence than with one-group designs
Problem: intervening variables again.
![Page 11: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/11.jpg)
Design: Time Series
One group, multiple observations before and after treatment. Can be two groups or two treatments.
O1 O2 O3 O4 X1 O5 O6 O7 O8
OR
O1 O2 O3 O4 X1 O5 O6 O7 X2 O8 O9 O10
Look for BIG differences after treatment as compared to before treatment.
Problem: The intervening variables
![Page 12: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/12.jpg)
Linking Design and Methods
O1 X1 O2
Suppose the design is one-group, pre-test, post-test:
How, where, and exactly when will the data be gathered at the two points of observation?
![Page 13: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/13.jpg)
Action Research Often Uses These Kinds of Methods to Gather Data
Surveys
Observations
Interviews
Artifact Analysis
![Page 14: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/14.jpg)
Methods of Gathering Data Make observations
Structured observations
Unstructured observations
Ask questions
Surveys
Interviews
Examine and Score Artifacts
Tests, portfolios, treatment plans, student comments
Must design scoring guide
![Page 15: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/15.jpg)
Structured Observations Examples:
Use a class map to keep track of who asks questions, answers questions, makes comments.
Have a list of behaviors. Make simple hash marks for each type of behavior when it occurs.
Make a list, from a recorded class discussion, of the kinds of questions you asked.
![Page 16: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/16.jpg)
Methods: Unstructured Observations Examples
A journal that a teacher keeps to record personal impressions of how a class is going.
A written remembrance of interactions with one or a few students that are being tracked over time.
A written set of impressions made while watching a videotape of a class.
![Page 17: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/17.jpg)
Methods: Unstructured Observations Think of these as similar to ethnographic “field
notes,” a place to record reactions, impressions, planned actions, responses and thoughts.
Can be useful to see trends in how YOU are responding to the innovation you are attempting.
Can be useful when studying one or a few students, again, over time to see trends.
![Page 18: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/18.jpg)
Methods of Gathering Data Make observations
Structured observations
Unstructured observations
Ask questions
Surveys
Interviews
Examine and Score Artifacts
Tests, portfolios, treatment plans, student comments
Must design scoring guide
![Page 19: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/19.jpg)
Methods: Questioning (survey/interview)
Uses a prepared set of questions. May have open-ended or close-ended
questions, or a combination. Good for getting background info. Can be used pre and post treatment. Can address things that may not be visible,
such as attitudes, motivation. Use with individuals or groups.
![Page 20: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/20.jpg)
Methods: Questioning with Surveys Examples:
A survey of attitudes toward science
Rankings of importance
Student ratings of instruction
May include open-ended questions, e.g., what element of this course most helped you to learn?
May include some content, but if entirely content, it’s a test and is an artifact of the course.
![Page 21: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/21.jpg)
Sample from an Survey
![Page 22: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/22.jpg)
Methods: Questioning with Interviews Advantage over surveys: can ask follow-ups,
more personal contact. Examples:
Sort readings by usefulness, telling why Solve a genetics problem aloud, explaining
thinking A focus group Interview of team members in small groups
![Page 23: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/23.jpg)
Sample from Interview
![Page 24: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/24.jpg)
Methods of Gathering Data Make observations
Structured observations
Unstructured observations
Ask questions
Surveys
Interviews
Examine and Score Artifacts
Tests, portfolios, treatment plans, student comments
Must design scoring guide
![Page 25: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/25.jpg)
Method: Examine and Score Artifacts Examples of artifacts:
Diagrams of cells before and after instruction Wear on computer keys to see which are hit most Answers to a test question Portfolios Term papers Case analyses Management plans Treatment plans
![Page 26: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/26.jpg)
Methods: Artifact Analysis
Decide what you kinds of materials you want to collect
Justify why the artifact you are choosing is a good choice given your research question
Create a scoring rubric (guide) to assign points
Good for pre-post designs
![Page 27: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/27.jpg)
A KEY TO SUCCESS: Pilot Test Your Instruments
Give a small group of people (not in your class if you can) your survey or interview
Collect sample artifacts to see if your grading scheme works
Try out your observation scheme to see if it needs to be tweaked.
![Page 28: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/28.jpg)
Designs and Methods: Summary Many of these approaches are familiar to we
who teach Much of this is used in research in many
disciplines. Talk about your design and instruments with
Teaching Associates, Fellows, other colleagues.
Keep your data collection focused. Try to keep the project reasonable for the
time you have available.
![Page 29: Project Design and Data Collection Methods: A quick and dirty introduction to classroom research Margaret Waterman September 21, 2005 SoTL Fellows 2005-2006.](https://reader035.fdocuments.in/reader035/viewer/2022062516/56649d7e5503460f94a6130d/html5/thumbnails/29.jpg)
To create your own plans answer the questions below: From whom are you gathering data?
More than one class, subgroups?
When will you gather data?
All one term, over several courses, in three weeks?
How will you gather data?
Questions, artifacts, observations?
Where will you gather data?Classroom, online forum, mailed survey, telephone?