Northwestern DBIR brown bag

download Northwestern DBIR brown bag

of 25

  • date post

  • Category


  • view

  • download


Embed Size (px)


Brown Bag presentation by Barry Fishman and Bill Penuel at Northwestern University School of Education and Social Policy on Design-Based Implementation Research (DBIR), presented on Thursday, May 23rd, 2013

Transcript of Northwestern DBIR brown bag

  • 1. Design-BasedImplementation Research:Working in Partnership(s) to Transform theRelationship of Research and PracticeBill Penuel (@bpenuel)University of Colorado BoulderBarry Fishman (@barryfishman)University of MichiganMay 23, 2013
  • 2. Acknowledgments National Science Foundation 1054086 Annie Allen, University of ColoradoBritte Haugan Cheng, SRI InternationalNora Sabelli, SRI InternationalAndy Krumm, SRI International
  • 3. Where We Begin Many promising educational interventions have beendeveloped, validated, tested... Then what? The majority fade away as funding ends or attentionturns elsewhere A few are sustained in (or near)the contexts where they weredeveloped A very few are brought to scale andare used across settings/contexts Why is the rate of success so low?
  • 4. A Validity Problem Interventions are usually developed inhothouse environments Researchers seek to reduce sources ofvariation in evaluations to increase internalvalidity Funded research focuses more ondeveloping and validating interventions frombasic research (Type I translation) than onunderstanding or closing gaps betweenresearch and practice (Type II translation)
  • 5. Type I TranslationType IITranslationWhat is beingtranslated?Translating principlesfrom basic learningresearch intointerventionsTranslatinginterventionsdeveloped for one or afew settings intointerventions that arescalable to manysettingsWhat kind ofresearch isinvolved?Design-based researchEfficacy andeffectiveness trialsImplementationresearch(for example, DBIR)
  • 6. Type I TranslationType IITranslationWhat kinds ofquestions doestranslationalresearch answer?What/how do people learnfrom this design?What do problems inlearning or implementationsuggest about redesign?What kinds of capacitiesare required to implementthis design?What supports are neededfor people implementingthe design to adapt it inways congruent with thedesigns core principles?Who is involved?Learning scientists,classroom teachers,subject matter experts,often also softwaredevelopersLearning scientists,organizational researchers,teacher leaders, school anddistrict administrators,often also publishers andenterprise softwareengineers
  • 7. Design-Based Research The objective of DBR is to develop theory, notjust empirically tune what works Design experiments create the conditions fordeveloping theories yet must place thesetheories in harms way. The theory is expected to do real work inpractical educational contexts(Cobb, Confrey, diSessa, Lehrer, Schauble, 2003)
  • 8. Design-BasedImplementation Research Blends learning sciences and policyimplementation research traditions andmethods Learning sciences: iterative, collaborative,guided by and informing theories oflearning/teaching Policy implementation: focus on conditions forimplementation effectiveness, guided by andinforming theories of institutional change andorganizational learning. Focus on the design ofsystems and infrastructure
  • 9. DBIR Principles1. A focus on jointly-defined problems of teaching andlearning practice2. A commitment to iterative, collaborative design3. A concern with developing knowledge and theorythrough disciplined inquiry4. A goal of developing capacity for sustaining changein systemsSource: Penuel, W. R., Fishman, B., Cheng, B. H., & Sabelli, N. (2011).Organizing research and development at the intersection oflearning, implementation, and design. Educational Researcher,40(7), 331337.
  • 10. Example: The Center for LearningTechnologies in Urban Schools
  • 11. Commitment to Iteration &Collaboration All work unfolded over multiple phases of pilot testingand revision, starting with a small core of teachers andthen expanding Design teams formed work circles that includeduniversity and school personnel Project leadership worked to forge new relationshipswithin and across institutions E.g., Bringing curriculum and technology leadershiptogether in Detroit Public Schools E.g., Convening technology leadership across districts
  • 12. Concern for Developing Knowledgeand Theory Through Inquiry The goal throughout was not only to create bettermaterials Goal was to develop new knowledge by developingthe theory of educative curriculum materials, with arange of subgoals, e.g.: Study of teacher learning through professionaldevelopment that linked PD to student outcomes Research on cognition supported by technology Evolving theory of system capacity for supportingreform, and how to design for systemic change
  • 13. A Concern with DevelopingCapacity to Sustain Change
  • 14. Evidence Standards for DBIR
  • 15. Design andDevelopmentEfficacyTrialsEffectivenessStudiesInvolvement of R&D TeamInvolvement of Evaluators & Practitioners
  • 16. Design andDevelopmentEfficacyTrialsEffectivenessStudiesInvolvement of R&D TeamInvolvement of Evaluators & PractitionersSustainability?+0.50+0.00
  • 17. Replication Really? Organizational replication is often thought ofas a process that yields reliable results at theexpense of local and professional control Peurach and Glazer (2011) argue for aknowledge-based logic in which programdevelopers collaborate with schools toproduce, use, improve, and retain practicalknowledge
  • 18. Key Differences from (old) IES Model Teams plan for scaling and sustainability fromthe start Designers maintain their involvementthroughout, because design is ongoing anditerative Powerful innovations are not expected tosurvive in the wild without changes at thesystem level
  • 19. Generating Forms of Evidence in anEra of Fast Science Get it mostly right, fast. How do you know you got it right? Whats a minimum viable innovation to test? Fail early, fail often. How can you put your theory in harms way from thestart? What documentation can benefit your own and otherteams? When do you do the big test?
  • 20. Matching Phase of Developmentto Phase of ResearchPhase of Development Driving Questions Sources of EvidenceProblem Negotiation What problem ofpractice should be thefocus of our joint work?Available data frommultiple sectorsResearch evidencePerspectives and valuesof stakeholders(including nonschoolactors)Co-design What should be thefocus of our work?To what extent do teamsleverage the diverseexpertise ofstakeholders?Design RationalesEthnographic accountsof design processes
  • 21. Matching Phase of Developmentto Phase of ResearchPhase of Development Driving Questions Sources of EvidenceEarly implementation How do implementersadapt the innovation totheir local contexts?How do implementers usethe innovation toreconstruct their practice?What are the appropriatemeasures of impact?Observations ofimplementationInterviewsAssessment designEfficacy What is the potentialimpact of the innovation onteaching and learning?What mediates impacts onlearning?Randomized ControlledTrialsInterrupted Time SeriesDesignsExplanatory CaseStudies
  • 22. Matching Phase of Developmentto Phase of Resea