Data Collection Data is your friend. Agenda Action research check-up Measures (aka, ways to collect...
-
Upload
agnes-mae-adams -
Category
Documents
-
view
219 -
download
2
Transcript of Data Collection Data is your friend. Agenda Action research check-up Measures (aka, ways to collect...
Data Collection
Data is your friend
Agenda
• Action research check-up
• Measures (aka, ways to collect data)
• Midterms
Public Service Announcement
APA style guide websites• http://owl.english.purdue.edu/owl/resource/560/01/
• http://citationmachine.net/index.php?reqstyleid=2
Action Research Projects
• Where you should be
• Finding research related to, but not necessarily identical to, your topic
• Research questions may shift
Action Research Projects
• Organizing Sources for literature reviewliterature review
• Example• Topic A (Source A, C, G, H, K)
• Subtopic 1 (Source G, K)
• Subtopic 2 (Source A, G, H)
• Subtopic 3 (Source A, C, H, K)
• Topic B (Source B, D, E, F, G, H)• Subtopic 1 (Source B, F, G, H)
• Subtopic 2 (Source D, E, F, G)
Action Research Projects
• Collecting Data• What types of data should I collect to answer
my research question?• What types of data should I collect to help
answer the why questions in my study?
Measures of Data Collection
• Interviews
• Questionnaires
• Observations
• Tests
Measures (Means of Data Collection)
You must match the instrument to the research question!
Examples to critique
• Measures• Questionnaire – Psychological School
Membership Survey used with middle school students
• Interview protocol – for teachers & counselors regarding professional development issues
• Observation instrument – PDE 430 for student teachers
• What are 2 benefits and 2 limitations of this measure?
Interviews
• What are some important things happening in this video related to interviewing?
http://www.youtube.com/watch?v=eAfTHwVTOrY
Interviews
• Advantages• Establish rapport & enhance motivation• Clarify responses through additional questioning• Capture the depth and richness of responses• Allow for flexibility• Reduce “no response” and/or “neutral” responses
• Disadvantages• Time consuming• Expensive• Small samples• Subjective – interviewer characteristics,
contamination, bias
Questionnaires
• Used to obtain a subject’s perceptions, attitudes, beliefs, values, opinions, or other non-cognitive traits
• Example of psychology questionnaires measuring aspects of happiness, “signature strengths,” well-being
http://www.authentichappiness.sas.upenn.edu/
Questionnaires
• Scales - a continuum that describes subject’s responses to a statement • Likert• Checklists• Ranked items
Questionnaires
• Likert scales• Response options require the subject to
determine the extent to which they agree with a statement
• Debate over odd v. even number responses• Statements must reflect extreme positive or
extreme negative positions
Questionnaires
• Checklists• Choose options
• Ranked items • Sequential order• Avoids marking everything high or low
Questionnaires
• Problems with measuring non-cognitive traits• Difficulty clearly defining what is being measured
• Self-concept or self-esteem
• Response set• Responding same way (Ex - all 4’s on CATS)
• Social desirability /faking• “PC filter”
• Agreeing with statements because of the negative consequences associated with disagreeing
Questionnaires
• Controlling problems• Equal numbers of positively and negatively
worded statements• Alternating positive and negative statements • Providing confidentiality or anonymity to
respondents
Designing Questionnaires
• Online resources• www.surveymonkey.com• http://pareonline.net/getvn.asp?v=5&n=3• http://www.peecworks.org/PEEC/PEEC_Inst/
I0004E536• http://www.statpac.com/surveys/
Observations
• Observations - direct observations of behaviors• Provide first hand account (ameliorates issues
of self-reporting in questionnaires)• Natural or controlled settings
• Ex – classroom vs. lab (child attachment studies)
• Structured or unstructured observations• Ex – frequency counts vs. narrative record
• Detached or involved observers
Observations
• Inference• Low inference - involves little if any inference
on the observers’ part• Student participation
• High inference - involves high levels of inference on the observers’ part
• Teacher effectiveness – PDE form 430
Observations
• Controlling observer effects• Observer bias
• Training• Inter-rater reliability (Cronbach’s alpha)• Multiple observers
• Contamination - knowledge of the study influences the observation
• Training• Targeting specific behaviors• Observers do not know of the expected outcomes• Observers are “blind” to which group is which
Observations
• Observer effects• Halo effect - initial ratings influence subsequent
ratings
• Hawthorne effect - increased performance results from awareness of being part of study
• Leniency - wanting everyone to do well
• Central Tendency - measuring in the middle
• Observer Drift - failing to record pertinent information
Tests
• PSSA Writing Assessment• Purpose• Domains• Scoring
• Questions to Consider
Score Interpretation of Tests
• Know the norming group on a norm-referenced test
• Self-report information is not very reliable• Inferences must be limited to what is being
tested• Grade equivalency scores should not be
interpreted to indicate grade assignment• Comparisons on a norm-referenced test can not
be made to populations outside the given norming group.
Tests
• Standardized tests• Uniform procedures for administration, scoring,
and interpreting test scores• Benefits and limitations?• Validity and reliability?
Benefits
• Forces teachers to get students on track with specific standards; quality control; assures students getting skills
• Red flag for a poor teacher who is negligent
• Consistency across school districts; use scores as consistent measure
• Consistency of subject matter
• Accountability of teachers
• Find areas where students are lacking and teach to their deficits
• Force students to be accountable
Limitations
• “If test scores improve (as I believe they will) and parent and policy makers are pleased with the results, are children receiving a better education?” (Cuban, 1983, p. 696).
• Once we know the expectation we limit what we teach to just what is covered on the test; restricts content taught
• Teaching to the test• Alignment between test and standard; is test valid?• Which came first—the standard or the assessment?• How to provide opportunities for remediation for those not meeting the
standard without cheating the “smart” kids.• Doesn’t measure creativity• Leads to labels of students• Bad test takers penalized
Validity & Reliability
• What are issues of validity and reliability that must be addressed in relation to standardized testing?
• PSSA issues
Standardized Testing Resources
• AERA statement about standardized tests• http://www.aera.net/policyandprograms/?id=378
• AERA “Research Points”• http://www.aera.net/uploadedFiles/Journals_and_Publications/
Research_Points/RP_Spring03.pdf
• Article by James Popham• http://school.familyeducation.com/educational-testing/educational-
philosophy/38778.html
• Dept of Education guide for policy makers and educators
• http://www.ed.gov/offices/OCR/archives/pdf/TestingResource.pdf