Mixing the Boundaries of Mixing the Boundaries of Research: Examining Mixed Research: Examining Mixed

Methods from a Methods from a Participatory/Use PerspectiveParticipatory/Use Perspective

Robert ShumerUniversity of Minnesota

Kim Sabo FloresConsultant

Evolution of Evaluation and Evolution of Evaluation and ResearchResearch

Fourth Generation Evaluation: Guba and Lincoln (1989)

The first generation refers to studies that were concerned with measurement, with a focus on testing and mathematical representations of data. This was referred to as the “measurement generation”

22ndnd Generation Generation

The second generation focused more on description.

Citing the 8 Year Study conducted by Ralph Tyler at Ohio State, the authors discuss how the efforts to measure change led to the development of formative evaluation and research, where there was a focus on the process as well as the outcomes

33rdrd Generation Generation

The third generation added the elements of judgment

Evaluators and researchers had to judge how the standards and outcomes expected (theories included) were to be assessed and how explanations about a study’s value and worth were to be presented.

44thth Generation Generation

• Believed knowledge is a constructed phenomenon

• The evaluator assumes a managerial role; involves stakeholders and others with an interest in the study in both the design and the measurement of the investigation.

• The focus is on interactive, negotiated studies that have participants and others engaged in the investigative process.

Essential ElementsEssential Elements

Fourth Generation Evaluation is based upon:Responsive focusing ---determining what

questions are to be asked and what information is to be collected on basis of stakeholder inputs

Constructivist methodology: carrying out the inquiry process within the ontological and epistemological presuppositions of the constructivist paradigm

Educational FoundationEducational Foundation

Based on constructivist theories of learningPiaget VygotskyBruner DeweyLearning is a function of experience,

reflected upon, placed in learner’s schema, and then enlarged/modified by subsequent experiences

Research Issues Associated with Research Issues Associated with 44thth Generation Generation Evaluation/ResearchEvaluation/Research

Confidentiality

Bias

Violation of trust

Purpose of Constructivist Purpose of Constructivist Learning/EvaluationLearning/Evaluation

Meaning is derived from use and application

Vygotsky – learning is a social endeavor…thus interaction is key

Expansion of understanding is based on dialogic process --- efforts to compare different forms of reality and meaning

Rigorous Research and EvaluationRigorous Research and Evaluation

Based on negotiated planning with stakeholders

Based on process of agreeing upon measures and action

Based upon continuous interaction of various appropriate approaches: can be quantitative, qualitative, or mixed

Mixed appears to be most complete because it allows for most points of view/perspective

Patton FrameworkPatton Framework

Differences between Quantitative, Qualitative, and Mixed Methods

Greatest challenge is mixing methods….planning from the beginning how data will be recorded and for what purpose

Best mixed approaches plan for integration of data at the beginning; failure to do so usually doesn’t allow for good use of multiple sources

ExamplesExamples

Kim Sabo Flores : youth participatory evaluation

Youth participate because they best understand their culture….can construct meaning through their experience

Evaluation and understanding are inseparable – yet assumed young people can’t think, plan, act, and reflect

Use methods and approaches that are part of youth culture – talking, acting, showing, modeling

A Story of Youth ParticipationA Story of Youth Participation

From Beyond ResistanceUCLA: Institute for Democracy, Education, and Access

Ernest MorrellAn example of critical research: critical research is

collaborative; requires proximity/intimacy with peopleTrustworthiness is better criterion than objectivityYouth carry out research projects in the communityDevelop surveys, interviews, focus groupsDevelop Educational Bill of RightsYouth run/win presidency/v.presidency of school…

work for change in community

ShumerShumer

A model of a mixed approach…crossing the boundaries of university and community

Minnesota AmeriCorps evaluation/research: 5 year project Engaged graduate students in course evaluating community –

based programs Conducted interviews, developed surveys, monitored

documents, conducted cost-benefit studies, individual profiles, case studies of Corps programs and of individuals (special needs woman…42 page report

All data designed to tell complete story: % of Corps members who dropped out (Native Am/Af. American), benefits of program; policy implications (affect on SSI for Handicapped),……..cost effectiveness of money spent, etc…..

Focus of evaluation on use….by programs, policy people, agencies

Youth Model in SingaporeYouth Model in Singapore

Research in Service-Learning…Raffles School….elite secondary school…top 2% of students

Conduct year long project in service-learningRequired to develop an evaluation/research

plan that tracks their work, measures their progress, and provide evidence of learning and of impact

Service-learningpartnership.org/research/IFA

Developing the IdeaDeveloping the Idea

On my honour, I promise that I will do my best,

To do my duty to God and to the Republic of Singapore, to help other people and To keep the Scout Law.

The Scout Promise is indeed a powerful driving force. The commitment to self-awareness, discipline, and compassion are timeless qualities that exemplify the scouting movement.

Developing the IdeaDeveloping the Idea

While many scouts in Singapore are actively pursuing progress badges and various awards, the Extension Scouting scene in Singapore is not as vibrant. This is attributed to various factors, such as a lack of public awareness and lack of support from the Singapore Scouts Association (SSA). In fact, most mainstream Scouts are unaware of Extension Scout activities or even of the fact that Extension Scout troops exist.

The Extension Scouts in Singapore form a diversified group. There are three (3) main categories: physical disabilities, mental disabilities and social disabilities.

Stages of ProjectStages of Project

Researching Stage: finding out about topic, programs

Planning Stage: deciding what to do in programActivities: defining the various activities they

would develop over the course of the yearOutcomes: determining what impact they had

on the Extension Scouts and themselvesFuture Plans: what they would do to continue

development of Extension Scouts

Comments on Extension Scout Comments on Extension Scout ProgramProgram

These are 16 year olds conducting research/evaluation!

Learn to use evaluation as an active method to get feedback before, during, and after their activities

Reflective components all along the way…learning about Special Scouts…and about themselves

Produces engaged citizens who make change

Let’s Do It, TooLet’s Do It, Too

Take 5 minutes to develop plan to evaluate this conference: determine the important questions to ask.

Discuss answers…what were your perceptions and did they differ from others?

What recommendations would you make to leadership to change things for the better?

Going Home and….Going Home and….

What ideas/activities will you implement in your home setting after the conference that will focus on 4th Generation Evaluation?

How will you make it participatory and channeled toward use? By whom?

And….And….

Questions and comments