Developmental Evaluation on the

Ground

Brigitte Scott, Ph.D.

Evaluation and Research Specialist

Military Families Learning Network

Military Families Learning Network

Personal Finance

Child Care

Family Advocacy

Network Literacy

Personal Finance

Family Development

Network Literacy

Community Capacity

Building

Nutrition & Wellness

Transition Support

Early Intervention

Special Needs

Life Span Special

Needs

Military Caregiving

MQP: DE Defined

“DE supports innovation development to guide

adaptation to emergent and dynamic realities in

complex environments.” —MQP, p. 1

Goal: support project, program, product, and/or

organizational development with timely

feedback

MQP: Core DE Question

What is getting developed and what are the

implications of what gets developed?

Complexity theory of change:

Bring people together who are knowledgeable and

committed and they will self-organize, take

action, and work together to create movement,

innovation, and change.

MQP: Key Characteristics of DE

Focus on development

Focus on developmentMQP

• NOT the same as continuous improvement (formative eval or accountability)

– Quality improvement is helping programs meet standards that have been set.

Development is when people are changing what they are doing, and the very nature of the standards are also changing. Standards are adapting to changing conditions.

Complex and dynamic environment

Complex and dynamic environmentMQP

Complex environment = lack of

central control

What to do to solve problems is

uncertain and there can be conflict

about how to proceed

Real time feedback

DE becomes part of intervention

Developmental Evaluator

Developmental EvaluatorMQP

• Works collaboratively with innovators to

conceptualize, design, and test new approaches

in a long-term, on-going process of adaptation,

intentional change, and development.

• Ask eval questions

• Apply eval logic

• Gather and report eval data

Developmental EvaluatorMQP

Primary Functions:

Elucidate the innovation and adaptation

processes

Track implications and results

Facilitate data-based decision-making

“There are no best practices. ‘Best’ is

entirely context-free.”—MQP, AEA 2014

Evaluation for MFLN

• Ongoing development

• Innovation in learning

– Internal practice

– External product

DE for MFLN: Why It Fits

Program development

Cooperative agreement environment

+

eXtension Initiative

Cooperative Agreement

“An opportunity to provide responsive and timely educational programming through being actively nimble, flexible, innovative, and creative in true partnership with our funders as they identify organizational priorities and ask us to engage in the construction of appropriate and necessary deliverables that meet the on-time needs of the target audience. The MFLN and funding partners are seen as true and honest equals in expertise, providing valued and accepted feedback bi-directionally. This often results in surpassing expectations for success in delivery.”

—Kyle Kostelecky, National Program Director, MFLN

Constant evaluative feedback

Constant evaluative feedback and

discussion

• Monthly reports (internal, DoD)

• Quarterly reports (NIFA)

• Annual reports (internal, DoD, NIFA)

• Webinar evaluation reports

• Quarterly webinar reports

• Weekly leadership team meetings

• Social Media Specialists meetings

• Net Lit support meetings

Evaluation used for decision-making

Evaluator is on leadership team

DE for MFLN: What It Looks Like

Continuous adaptation

DE for MFLN: What It Looks Like

Continuous adaptationSocial, political, economic, technological, and demographic

patterns

DE for MFLN: What It Looks Like

Continuous adaptationSocial, political, economic, technological, and demographic

patterns

Emergent developments/needs of our target audiences

DE for MFLN: What It Looks Like

Continuous adaptationSocial, political, economic, technological, and demographic

patterns

Emergent developments/needs of our target audiences

Cooperative agreement environment

DE for MFLN: What It Looks Like

Continuous adaptationSocial, political, economic, technological, and demographic

patterns

Emergent developments/needs of our target audiences

Cooperative agreement environment

eXtension

DE for MFLN: What It Looks Like

Continuous adaptationSocial, political, economic, technological, and demographic

patterns

Emergent developments/needs of our target audiences

Cooperative agreement environment

eXtension

DoD

DE for MFLN: What It Looks Like

(not this. . . .)

DE for MFLN: What It Looks Like

(but this. . . .)

DE for MFLN: What It Looks Like

Inductive

DE for MFLN: What It Looks Like

InductiveChaotic and messy

DE for MFLN: What It Looks Like

InductiveChaotic and messy

Embrace forks in the road

DE for MFLN: What It Looks Like

InductiveChaotic and messy

Embrace forks in the road

Developmental moments

Track what’s going on

Understand what’s going on

Adapt to what’s going on

DE for MFLN: What It Looks Like

Everyone is an

evaluator

Inspired by Jeff Conklin, cognexus.org

Time

Complex development situations are ones in which this…

39Michael Quinn Patton

AEA eStudy webinar 2014

Michael Quinn Patton

AEA eStudy webinar 201440

And this…

Turns out to be this…

Time41

Michael Quinn Patton

AEA eStudy webinar 2014

Michael Quinn Patton

AEA eStudy webinar 201442

…looks like this

DE Successes for MFLN

DE Successes for MFLN

• Very responsive to our partners’ programmatic

requests and innovations

DE Successes for MFLN

• Very responsive to our partners’ programmatic

requests and innovations

• Doubling in size in 2015 with competitive award

– New CAs and new opportunities for innovation in

programming

DE Successes for MFLN

• Very responsive to our partners’ programmatic

requests and innovations

• Doubling in size in 2015 with competitive award

– New CAs and new opportunities for innovation in

programming

• Programming

DE Successes for MFLN

• Very responsive to our partners’ programmatic requests and innovations

• Doubling in size in 2015 with competitive award

– New CAs and new opportunities for innovation in programming

• Programming

• “Walk the talk”: model use of social media, collaborative learning, personal learning networks

DE Challenges for MFLN

DE Challenges for MFLN

• Innovation is challenging, slow, and . . . contentious

DE Challenges for MFLN

• Innovation is challenging, slow, and . . . contentious

• Moving from program improvement mode to redesign

based on evaluation findings

DE Challenges for MFLN

• Innovation is challenging, slow, and . . . contentious

• Moving from program improvement mode to redesign

based on evaluation findings

• IRB challenges

DE Challenges for MFLN

• Innovation is challenging, slow, and . . . contentious

• Moving from program improvement mode to redesign

based on evaluation findings

• IRB challenges

• “In your face” evaluation

DE Challenges for MFLN

• Innovation is challenging, slow, and . . . contentious

• Moving from program improvement mode to redesign

based on evaluation findings

• IRB challenges

• “In your face” evaluation

• Daily, evaluative thinking

DE Challenges for MFLN

• Innovation is challenging, slow, and . . . contentious

• Moving from program improvement mode to redesign

based on evaluation findings

• IRB challenges

• “In your face” evaluation

• Daily, evaluative thinking

• Platform delivery limitations

DE Challenges for MFLN

• Innovation is challenging, slow, and . . . contentious

• Moving from program improvement mode to redesign

based on evaluation findings

• IRB challenges

• “In your face” evaluation

• Daily, evaluative thinking

• Platform delivery limitations

• Military culture

DE Challenges for MFLN

• Innovation is challenging, slow, and . . . contentious

• Moving from program improvement mode to redesign

based on evaluation findings

• IRB challenges

• “In your face” evaluation

• Daily, evaluative thinking

• Platform delivery limitations

• Military culture

• Doubling in size in January 2015

DE Features for MFLN

• Reports

• Leadership

• Reflective discussion groups

• Transparency

• Collaboration

DE Features for MFLN

Patton, Michael Quinn. (2014).

“Intermediate Developmental Evaluation.”

AEA eStudy Webinar.

Patton, Michael Quinn. (2011). Developmental

Evaluation: Applying Complexity Concepts to

Enhance Innovation and Use. Guilford: New

York.

Brigitte Scott

@4ed_eval

brigit2@vt.edu

www.blogs.extension.org/militaryfamilies

Icons made by Freepik:

http://www.flaticon.com. Flaticon is

licensed by Creative Commons 3.0.