Intermediary Organizations and Education Innovation · Center for Public Research and Leadership 6...

Intermediary Organizations and Education Innovation

FrameworksandToolsforUnderstandingandEvaluatingIntermediaryOrganizationsandTheirRoleinK-12EducationInnovation

April2017

CenterforPublicResearchandLeadership

Center for Public Research and Leadership

TheCenterforPublicResearchandLeadership(CPRL)isapartnershipoftopU.S.graduateandprofessionalschools.CPRLbringstogetherbusiness,education,law,andpolicystudentstostudyandprovidehigh-qualityresearch,strategicplanning,andevaluationsupporttopublic-andsocial-sectororganizationsseekingtransformationalchangeinK-12education.


3

CONTENTS

Foreward. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Acknowledgements. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Introduction:TheCaseforIntermediaryOrganizations. . . . . . . . . . . . . . . . 6

Part1:DescribingtheRoleofanIntermediaryinDiffusingEducationInnovation. . . . . 10

Part2:AssessingtheRoleofanIntermediaryinDiffusingEducationInnovation. . . . . 14

Closing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

AdditionalResources:MeasurementToolkitforIntermediaryOrganizations. . . . . . . 30

AdditionalResources:PhasedRubricsforAssessingIntermediaryOrganizations. . . . . . .

86


4

FOREWARD In2014,theBill&MelindaGatesFoundationbeganinvestinginpersonalizedlearning—amodelofinstructionsthatishighlylearner-centered—asastrategyforachievingsystemicchangeinK-12education.SincethattimetheFoundationhasinvestedinavarietyofprojectsrangingfromschoolanddistrictredesignstotechnologydevelopmenttoresearchandevaluationallaimedatdiffusing,orspreading,innovativepersonalizedlearningpracticesand,inturn,improvingstudentoutcomesacrossthenation.TheseinvestmentsincludefundingseveralregionalintermediaryorganizationstoactaslocalhubsforinnovationandsystemtransformationthroughtheNextGenerationLearningChallenges(NGLC)RegionalFundsforBreakthroughSchoolsinitiative.

NGLC,anon-profitassociationthatworkstocatalyzeandacceleratethebroadadoptionofeffectiveandinnovativeeducationpractices,launchedtheRegionalFundsforBreakthroughSchoolsinitiativewithadditionalfundingfromtheEliandEdytheBroadFoundationandtheMichaelandSusanDellFoundation.TheRegionalFundinvestmentsaremeanttosupporteducationentrepreneurs,igniteinnovation,encouragecooperationandalignmentwithinregions,andexpandinterestinpersonalized,learner-centeredinstructionalmodels.In2014,NGLCinvestedinsixRegionalPartners:

• CityBridgeEducation(Washington,D.C.)• ColoradoEducationInitiative• LEAPInnovations(Chicago)• NewEnglandSecondarySchoolConsortium• NewSchoolsforNewOrleans• RogersFamilyFoundation(Oakland)

BMGF’sinvestmentinNGLCandtheRegionalPartnersoffersanumberofstrategicbenefitstotheFoundation.ThenetworkhasthepotentialtoacceleratethespeedandreachoftheFoundation’sgrantmaking;itcanofferenhancedlocalexpertiseanddeep,stablesupporttopartners;anditprovidesBMGF,NGLC,andtheRegionalPartnerswithanopportunitytopoollearningfromacrossadiversesetofcontexts.However,workinginnetworkedintermediarystructuresalsoposeschallenges,oneofwhichisestablishingmonitoringandevaluationmechanismsthatareresponsivetothecomplexityofanintermediary’srole.Thiscomplexitystemsfromthefactthatintermediariesworkatmultiple,interrelatedlevelstoimpactchangeandanintermediary’sstrategymustcontinuallyadjusttochangesinanyoneoftheselayers.

TheCenterforPublicResearchLeadership(CPRL)soughttoaddressthischallengebyengagingasubsetoftheNGLCRegionalPartnersinthedevelopmentofframeworkstodescribeandevaluatethevariousstrategiestheyemployastheyworktospreadinnovativeinstructionalpracticesacrosstheirregionsand,inthelongrun,improvestudentoutcomes.


5

ACKNOWLEDGEMENTS ThroughgeneroussupportprovidedbytheBill&MelindaGatesFoundation,CPRLspentthelastyearresearchingtheroleintermediaryorganizationsplayinthediffusionofinnovativeideasandpractices.Alongtheway,wecollaboratedwithinspiringandforward-thinkingorganizationsthatareworkingtoprovidestudentswithrigorous,personalizedlearningexperiences.TheseorganizationincludedNextGenerationLearningChallenges(NLGC)andfourofNGLC’sRegionalFundPartners—CityBridgeEducation,ColoradoEducationInitiative,NewEnglandSecondarySchoolConsortium,andNewSchoolsforNewOrleans.Wearedeeplyappreciativeoftheirengagementinthisproject.

Intotal,wespokewithover30individualsfromacrossthesefiveorganizations,andwearethankfultoeachpersonfortheinsightsshared.Inparticular,wearegratefulforthetimeandenergydevotedbyMargretAngell,CarolineHill,andArthurMcKeefromCityBridgeFoundation;HeatherChikoore,ChristinaJean,EmilyLove,andSamOlsonfromColoradoEducationInitiative;LaurenHinthorneandMarkKostinfromNewEnglandSecondarySchoolConsortium;IndrinaKanthfromNewSchoolsforNewOrleans;andAndyCalkinsandSarahLuchsfromNGLC.TheseindividualsspenthourswithCPRLbothinpersonandonthephonedescribingtheirvisionsforeducation,thestrategiesthatguidetheirday-to-dayoperations,andthelessonstheyhavelearnedalongtheway.

WealsowanttothankLedyardMcFaddenfromSchoolWorksandAmyNowellfromLEAPInnovationsforsharingtheirexpertiseduringourNovember2016conveningatColumbiaUniversity.

Intheend,theworkwouldnothavebeenpossiblewithouttheknowledgewegleanedfromeachoftheseindividuals.


6

INTRODUCTION

Intermediary Organizations Atthemostbasiclevel,anintermediaryorganizationisago-betweenormediator.However,thespecificrolesplayedbyintermediariesarediverse(Blanketal.,2003;Szanton,2003;Wynn,2000).Someintermediariesactasoversightorganizationsthatsetstandardsandmonitorprogress.Othersprovideservicesandresourcesmuchlikeanexternalconsultantortechnicalassistanceprovider.Stillothersfocusonfosteringconnectionsbetweenindividualsororganizationsinordertofacilitatecommunicationandcollaborationlikeanetworkhuborbackboneorganizationtypicallydoes.Finally,manyintermediariesplaytheroleofagrantmakinginstitutionandprovideindividualsandorganizationswithfundingforspecificinitiatives.Toensureacommonlanguage,thisreportdefinesintermediaryorganizationsasindependentlyoperatingentitiesthatworkbetweenmultipleactorstofacilitatecommunicationandcollaboration;buildcapacityandknowledge;andovertimebringaboutchangeintheactors,theiractivities,andtheresultstheyachieve.

Intermediaryorganizationsareoftenpartiallyorcompletelysupportedbylargerphilanthropicfoundationsorgovernmentgrantmakingagencies.Suchstructuresofferbenefitstoboththelargerfunderandtotheindividualororganizationalactorssupportedbytheintermediary.Thesebenefitsarelistedbelow(Szanton,2003;GEO2013,GEO2014).Ingeneral,intermediaryorganizationsallowfunderstoremainagileandleanwhilestillensuringtheirintendedbeneficiariesreceivedhigh-quality,supporttailoredtotheiruniquecontextsandneeds.

“Intermediariesareorganizationsthatoccupythespaceinbetweenatleasttwootherparties…Intermediaryorganizationsoperateindependentlyofthesetwopartiesandprovidedistinctvaluebeyond

whatthepartiesalonewouldbeabletodeveloportoamassbythemselves.Atthesametime,intermediaryorganizationsdependonthosepartiestoperformtheiressentialfunctions.”

—MeredithHonig,“TheNewMiddleManagement:IntermediaryOrganizationsinEducationPolicyImplementation”

------

“Intermediariescomeintobeingtohelpachievemoreefficient,effectiverelationshipsbetweenlayers…Matureintermediariesessentiallytakeonmanyofthefunctionsofsystems—standardssetting,

qualityassurance,training,advocacy,funddevelopment,datacollection.”

—KarenPittman,“TheImportanceofIntermediaryOrganizationstoImplementingCommunityInitiatives”

BenefitstoFunders• Increasedspeedandreachofgrantmaking• Reducedoverhead• Politicalprotections• Easeofprogramexit• Increasedoversightandsupportfor

beneficiaries• Increasedcredibilitywithpractitioners• Increaseddiversificationandmore

opportunitiesforlearning

BenefitstoPartners• Freshperspectives• Specializedskillsandknowledge• Contextualexperienceandexpertise• Durabilityofsupport• Increasedaccesstoresourcesandtechnical

assistanceproviders• Increasedaccesstoprofessionallearning

communities


7

Anotherbenefitofworkingthroughintermediaryorganizationsisthenetworkstructuretheycreate.Thisstructuredevelopsasintermediariesbuildconnectionswithmultiplepartners.AreportfromtheU.S.DepartmentofHealthandHumanServicesdescribestwotypesofintermediariesthatresultfromdifferentdecisionsabouthowtoengagewithpartners—distributiveandconvening(2008).Adistributiveintermediarydisseminatesservicesandresourcestoapartnerthroughaone-to-onerelationship.Theresultisahighlycentralized,hub-and-spokenetworkstructure.Aconveningintermediarydisseminatesservicesandresourcestopartnersdirectlyaswell,butalsoworkstoconnectpartnerstooneanotherformingamoredecentralizednetwork.

Thenetworkstructureallowsintermediariestooperateatmultiplelevelssimultaneously.Atthepartnerlevelanintermediarymayengageindirectfunding,technicalassistance,coaching,orothercapacity-buildingactivities.Atthenetworklevel,intermediariesrecruitnewpartners,facilitatecollaboration,andhostconvenings.Inadditiontothesetwolevelsintermediariesmayalsoworkatathirdlevel—thesystemlevel.Atthislevelanintermediarytargetsthepolitical,economic,orsocialconditionssurroundingandimpactingitsnetwork.

Itisanintermediary’sabilitytosuccessfullyworkateachoftheselevelsthatmakesitwellsuitedtoscalingnewideasandpractices.InDiffusionofInnovation,EverettRogersdefinesdiffusionastheprocessbywhichaninnovationiscommunicatedthroughcertainchannelsovertimeamongmembersofasocialsystem(1962).Thisdefinitionhighlightsthethreelayersatwhichanintermediaryoperates:members,communicationschannels,andthesocialsystem.Bystrengthenandaligningtheselayers,anintermediarycandirectlysupportdiffusion.AfourthpieceinEverett’sdefinition—theinnovation—isalsoanelementthatintermediariescanimpactthroughknowledge-managementactivities.However,thisreportcombinesknowledge-managementactivitieswithpartnercapacitybuilding.

Report Overview Thisreportseekstoclarifytheroleofintermediaryorganizationsinthediffusionofinnovativeeducationpracticesaswellasbuildthecapacitytomonitorandevaluatethisrole.Inordertoachievethesegoalsthereportpresentstwointerrelatedframeworks—theDiffusionLeversandtheDiffusionIntermediaryEvaluationFramework.Theformerisadescriptiveframeworktoassistinunderstandinganddifferentiatingbetweenintermediaries’strategies.Thelatterisamonitoringandevaluationframeworkforassessinganintermediary’sreadinessforandsuccessatimplementingitsstrategy,aswellastheimpactthisstrategyhas.Thereportisorganizedintotwoprimarysectionstocorrespondwiththeseframeworks.

Part 1: Describing the Role of an Intermediary in Diffusing Education Innovation

Part1focusesontheDiffusionLevers,whichprovideacommonlanguagefordescribingthestrategiesofintermediaryorganizations.ThesectiondiscussesthreeLevers—CapacityBuilding,NetworkWeaving,and


8

SystemCultivating—eachofwhichiscommonlyemployedbyanintermediaryinanefforttospreadnewideasandpractices.

Part 2: Assessing the Role of an Intermediary in Diffusing Education Innovation

Part2presentstheDiffusionIntermediaryEvaluationFramework.Thisframeworkoutlinesathree-phaseapproachtoevaluatingintermediaryorganizationsovertime.Eachphaseconsistsofdimensionsforevaluationaswellasspecificcriteria,guidingquestions,andsampleindicators.TheDiffusionLeversareincorporatedintotheEvaluationFrameworktoensurethatitreflectsthespecificrolesintermediariestypicallyplayandtheshort-termoutcomestheycanbeexpectedtoachieve.

Additional Resources

Inadditiontotheprimaryreportsectionsoutlinedabove,thisreportprovidesadditionalresourcestofurthersupportusersinunderstandingandevaluatingintermediaryorganizations.Theseinclude:

• ToolkitsforeachofthethreeDiffusionLeversthatcontainthecriteria,guidingquestions,andsampleindicatorsforevaluationaswellasasmallselectionofdatacollectiontools.

• EvaluationrubricsalignedtothephasesoftheDiffusionIntermediaryEvaluationFramework.

Intended Users CPRLhopesthatthisreportwillbeusefultoanumberofdifferentusersincluding:

v Funderscurrentlyworkingwith,orlookingtoworkwith,intermediaryorganizationsandseekingfurtherinsightintotherolessuchorganizationscanplay,aswellashowtoevaluatetheorganizations’potentialforinfluence,interimprogress,andlong-termimpact.

v Existingintermediaryorganizationsalreadyworkingwithanetworkofpartnerstocatalyzeandaccelerateinnovationineducation,andwhoarelookingforclarityaroundhowtoevaluatetheirrole.

v Non-profits,consortia,foundations,orothergroupsconsideringbecomingintermediariesandsearchingforadditionalinsightintothedevelopmentoftheirvisionandstrategyfordiffusion.

v Individualsandorganizationsconsideringapartnershipwithanintermediaryorganizationandlookingtounderstandtheexperiencesandbenefitstheycanexpect.

Research Methods Abroadrangeofexistingliteratureonthediversestrategiesandgoalsofintermediaryorganizations,aswellastheworkofasampleofcurrentNGLCRegionalPartners,informedthisproject.

Collaboration with NGLC Regional Partners

FourofNGLC’scurrentRegionalPartners—CityBridgeFoundation,ColoradoEducationInitiative,NewEnglandSecondarySchoolConsortium,andNewSchoolsforNewOrleans—participatedinthisprojectalongwithNGLCitself.Theiruniqueandhighlydevelopedstrategiesprovidedthebackboneforthisreport.PriortoworkingwiththeRegionalPartners,CPRLrevieweddocumentsdiscussingtheworkofeachintermediary.CPRLthenconductedinterviewswithmultiplestaffmembersfromeachRegionalPartnerandwithNGLCprogramstaff.Followingthispreliminaryresearch,CPRLengagedtheRegionalPartnersandNGLCinaone-dayconveningatColumbiaUniversityfocusedondocumentingtheorganizations’uniquestrategies.Finally,thegroupconvenedagainlateintheprojecttodiscusstwohigh-priorityareasforevaluation—networkevaluationandevaluatingchangesininstructionalpractice.ThissecondeventwasfurthersupportedbyLEAPInnovationsandSchoolWorks.


9

Review of Related Literature

InadditiontocollaboratingwiththeRegionalPartners,CPRLreviewedvariousstrandsofexistingresearch.Topicsreviewedincludesystems,networks,anddiffusiontheory;networkevaluation;collectiveimpact;scaling;andpriorresearchontheuseofintermediaryorganizationstosupportimprovedsocialoutcomes.Intotal,CPRLconsultedover100books,articles,andwebpagesthroughoutthecourseoftheproject.


10

PART

1 DESCRIBING THE ROLE OF AN INTERMEDIARY IN DIFFUSING EDUCATION INNOVATION

Introduction to the Diffusion Levers TheDiffusionLeversprovideasimplesetoflanguageandconceptsfordescribingthemechanismsintermediaryorganizationsusetospreadnewideasandpractices.ThethreeLevers—CapacityBuilding,NetworkWeaving,andSystemCultivating—highlightthethreelevelsatwhichanintermediarycanwork.TheCapacityBuildingLeverfocusesontheindividualpartnersandtheirknowledgeandresources.Thesepartnersmaybepeopleororganizations.TheNetworkWeavingLeverfocusesonallpartnerstogetherasanetworkandthecommunicationandcollaborationoccurringbetweenthem.TheSystemCultivatingLeverfocusesonthelargerecosysteminwhichthenetworkoperatesincludingthesupportiveorrestrictivesocial,economic,andpoliticalconditionspresentintheecosystem.

Eachlevercanbefurtherspecifiedintoactionsthatintermediaryorganizationstakestoapplytheleveraswellasshort-termoutcomesthatresultfromitsuse.However,theleversthemselvesarenotmutuallyexclusive,andtheapplicationofoneoftenfurtherenablestheapplicationoftheothers.Forexample,thedevelopmentofanextensivenetworkofcloselyconnectedpartnersislikelytofacilitatethegenerationandsharingofnewknowledgewhilealsoincreasingthevisibilityoftheintermediary’smissionanditscampaigningcapacity.Finally,


11

whileintermediariesoftenemployallthreeleverssimultaneously,thedegreetowhicheachisusedandthespecificactionstakenarelikelytodifferbasedonanintermediary’svisionandstrategy,localcontext,andorganizationalcapacity.Together,thethreeLeversdrivetowardthespreadofnewideasandpracticesatboththeindividualandorganizationallevel,and,overtime,thesechangesinpracticedrivetowardimprovedsocialoutcomes.Eachleverisdescribedinmoredetailinthepagesthatfollow.

Capacity Building CapacityBuildingentailsprovidingindividualsandorganizationswiththeknowledgeandresourcesneededtoincreaseunderstandingandimproveperceptionsofinnovation.

Individualsandorganizationspartnerwithintermediaryorganizationsinordertoachievemorethantheycouldachieveontheirown.Thesepartnersrecognizethatintermediaryorganizationsofferaddedcapacityintheformofincreasedfinancialresources,knowledgeofeffectivepractices,toolsforandsupportinimplementingthesepractices,andmore.Commonactionstakenbyintermediaryorganizationtobuildpartnercapacityincludeknowledgemanagement—theprocessofcapturing,synthesizing,organizing,anddisseminatingknowledge—trainingpartnersthroughgroupworkshopsorindividualcoaching,monitoringpartnerprocesses,andbrokeringresourcessuchasfundingandtechnicalassistancesupport.

Theseactionsresultinpartners’increasedreadinesstochangetheirbehaviorsandpractices.Theideaof“readiness”isinformedbycapacitybuildingandchangemanagementliterature,aswellastheinnovation-decisionprocess,whichalldescribevariousstagesindividualsandorganizationsgothroughwhendeterminingwhethertheywillchange(ConnollyandYork,2002;Rogers,2003;KirkpatrickandKirkpatrick,2005).Takentogether,thisbodyofresearchdescribesthatapartnermust(1)buildunderstandingofthenewbehaviorsorpractices,(2)developapositiveperceptionofthem,(3)establishtheintentiontochange,(4)implementthechange,and(5)internalizeorinstitutionalizethechange.Theshort-termoutcomesoftheCapacityBuildingLeverarethefirsttwostagesofthisprocess—increasedunderstandingandimprovedperceptions.Overtime,theseoutcomeswilldrivetowardchangesinpracticeandideallytowardimprovedsocialoutcomes.Apositiverelationshipbetweenchangesinpracticeandimprovedoutcomescreatesavirtuouscycledrivingfurtheradoptionofthenewpracticesacrossthefieldatlarge(Meadows,2008).


12

Network Weaving Connectingindividualsandorganizationstobuildstrongcommunicationschannelsandahealthycommunitythroughwhichinnovationcanspread.

Anintermediaryorganizationcannotbeseparatedfromitsnetworkofpartners,and,asaresult,approachingthedesignandfacilitationofthisnetworkwithintentionalityisakeypartofanintermediary’srole.IntheirbookConnectingtoChangetheWorld,PeterPlastrik,MadelineTaylor,andJohnClevelandidentifyeightdesignissuespertainingtonetworks:

1. Purpose:Network’sreasonforbeing.2. Membership:Eligibilityandparticipationrequirementsformembersaswellasdesiredsizeand

compositionofmembership.3. ValueProposition:Benefitsmemberswillreceive.4. Coordination,Facilitation,andCommunication:Hownetworkmembersworkwitheachother.5. Resources:Network’sfundingmodel.6. Governance:Decision-makingparticipantsandprocesses.7. Assessment:Monitoringandevaluationstrategy.8. OperatingPrinciples:Rulesthatguideculture.

Makingthesedesigndecisionsisakeypartofanintermediary’sNetworkWeavingrole.AdditionNetworkWeavingactionsincluderecruitingmembersintothenetwork,facilitatingongoingcommunicationandcollaborationbetweenthesemembers,andconveningthenetworkface-to-faceorvirtually.

Keyshort-termoutcomesthatresultfromNetworkWeavingactionsincludemeasuresofnetworkformsuchasthesizeandcompositionofthenetwork,thenumberofconnectionsthatexist,andstrengthorpurposeoftheseconnections,aswellasmeasuresofnetworkfunctionssuchasthestabilityofthenetwork’sinfrastructureandoperationsandthedegreetowhichcertainconditionsareinplace,suchastrustandaccountability(Bonbright&Khangram,2010;Plastrik,Taylor,&Cleveland,2014;Taylor,Whatley,&Coffman,2015).Whileresearchindicatesthatdecentralizedandopennetworkstypicalsupportinnovationmoreeffectivelythancentralized,closednetworks,thereisnoonebestdesign(Plastrik,Taylor,&Cleveland,2014).Asaresult,thespecifictargetsfortheseoutcomesshouldaligntotheintermediary’sinitialnetworkdesign.


13

System Cultivating Supportingchangestopolitical,organizational,economic,andsocialconditionstoreducebarrierstoandincreasesupportsforinnovation.

Inordertosupportthespreadofnewideasandpractices,itisalsocrucialforanintermediaryorganizationtoensurethesocial,economic,andpoliticalconditionssurroundingitsnetworkarealignedandsupportive.Anintermediarymaydothisthroughavarietyofactions,suchaspolicyanalysisandpoliticaladvocacy,campaigningandcoalitionbuilding,andreformingorganizationalsystemsandstructures.Inaddition,becausethespecificactionsanintermediarytakeswillbecontingentuponthecurrentandevolvingconditionsinplacewithinthelocalcontextwhereitoperates,itiscrucialforanintermediarytoassesstheseconditionsonaregularbasisthroughactivitieslikesystemmapping.

Short-termoutcomesrelatedtoSystemCultivatingincludeincreasedalignmentofpoliciesandpracticestotheinnovationaswellasexpandedpublicengagement.Morespecifically,SystemCultivatingeffortsarelikelytoleadtothereplacementofformallegislationorinformalorganizationalpracticesthatpreventinnovationwithpoliciesandpracticestheenableorrequireinnovation.Inaddition,theseeffortsarelikelytoleadtotheeliminationofpublicmisconceptionsaboutorproteststonewpractices,andtheremovaloffundingobstaclesthathinderimplementation.

ProofPoints:AnImpactofSuccessfulIntermediaryOrganizations


14

PART

2

ASSESSING THE ROLE OF AN INTERMEDIARY IN DIFFUSING EDUCATION INNOVATION Introduction Whiletheprevioussectionofthisreportoutlinedadescriptiveframeworkforunderstandingthetypesofstrategies,orDiffusionLevers,usedbyintermediaryorganizationsworkingtodiffuseeducationinnovation,thissectionoutlinesanevaluativeframeworkforassessinganintermediary’sreadinessforandsuccessatemployingthesestrategies.Asdescribedinfurtherdetailtotheright,theDiffusionIntermediary’sMonitoringandEvaluationFramework(“Framework”)addressesthecomplexityofevaluatingtherolesofanintermediarythatarecausedbythediversityofstrategiesusedandthetendencyforthesestrategiestochangeovertime.TomaintainalignmentwiththeDiffusionLevers,thisframeworkwasinformedbyliteratureontheevaluationofcapacitybuilding,networks,andsystemchange.Additionalresearchrelatedtoorganizationalcapacity,readinessassessments,andscalingeducationreformalsocontributedtothedevelopmentofthisframeworkandensuresthatitisinclusiveofallstagesofanintermediary’swork.AfulllistofreferencesisinAppendixA.

Framework Organization

TheFramework(Figure1)isorganizedintothreesuggestedassessmentphases:

PhaseI:PotentialforInfluence-Assessingthelikelihoodofanintermediaryimpactingthediffusionofinnovationwithinitslocaleducationcontext.

PhaseII:InterimProgress-Assessinganintermediary’sstrategyimplementationandshort-termoutcomes.

PhaseIII:ImpactsontheField-Assessinganintermediary’sinterimimpactonthediffusionofinnovativeaswellasthelong-termimpactonsocialoutcomes.

Eachphaseconsistsoftwoorthreedimensions,andeachdimensionisfurtherspecifiedintoanumberofcriteriawithguidingquestionsandsampleindicators.

Responding to Diversity and Change

Thediversityofintermediaryorganizations’rolesacrossvariouscontextsandchangestotheserolesovertimerequiresanapproachtomonitoringandevaluationthatisflexibleandfocusesonprocessaswellasoutcomes.Theevaluationframeworkandprocesslaidoutinthisreportseekstodirectlyrespondtotheseneedsasoutlinedbelow.

The Challenges The Solutions

DiversityofStrategy

Intermediaryorganizationsare

diverse.Theyengageinavarietyoftypesof

activitiesinresponsetotheiruniquecontexts

andstrategies.

AFlexibleFramework

à AlignmenttoDiffusionLeversframeworkincreasesapplicabilityacrossdiverseintermediaries.

à Guidingquestionsaregeneralizabletodifferentcontextsandintermediaries.

à Sampleindicatorscanbecustomized.

ChangesinStrategy

Intermediaryorganizationsare

engagedinchangingcomplexsystems,andtheirstrategieschangeinresponsetoshiftsincontextandinterim

results.

AFocusonProcess

à Threephasesassessintermediary’sinitialpotential,interimprogress,andimpact.

à Includesprocesscriteriaaswellasoutcomecriteria.

à Includescontinuouslearningcriteriatoassessdocumentationoflessonslearnedandmidcoursecorrections.


15

DiffusionInterm

ediaryM

onito

ringan

dEvalua

ting

Fram

ework


Phase I: Potential for Influence PotentialforInfluenceassessesthelikelihoodofanintermediaryimpactingthediffusionofnewideasandpracticesacrossitscontext.Thislikelihoodshouldbeassessedattheoutsetofanintermediary’swork.Itcouldbedoneaspartofaninternalself-assessmentbeforeanewintermediarydecidestoformorwhenanexistingintermediaryconsidersmovingintoanewregion.Itcouldalsobepartofanexternalassessmentcompletedbyapotentialfunder.Thephaseconsistsofthreedimensionsforevaluation:

• ReadinessofLocalContext-Theregion’ssupportandneedforinnovativeeducationreformideasandpractices,asindicatedbythepolitical,social,andeconomicconditions.

• Vision&Strategy-Theintermediaryorganization’stheoryofaction—theinnovativeideasorpracticesitseekstospreadandthestrategyitwillusetodoso.

• OrganizationalCapacity-Theresourcesandinfrastructuretheintermediarybringstobearinimplementingitsstrategy.

ThesethreedimensionswereinformedbyProfessorMarkMoore’stheoriespertainingtostrategyinthepublicsectorandspecificallythestrategictriangle(Moore,1995).Thissimpleconceptualframeworkdrawsattentiontothreeissuesthatpublicsectororganizations,includingthetypesofintermediaryorganizationsdiscussedhere,mustconsiderwhenplanning.Theseincludepublicvalueorwhattheorganizationseekstoimpactwithinthesector,legitimacyandsupportortheexternalresources,structures,andconditionstheorganizationwillrelyon,andoperationalcapabilitiesortheinternalresources,structures,andconditionstheorganizationcanemploy.Mooreviewsthesethreeissuesashighlyinterdependent,withchangesinonedirectlyimpactingtheothertwo.AswithMoore’striangle,thethreedimensionsofPotentialforInfluenceareinterdependent.Asaresult,eachshouldbeconsideredinrelationtotheothersduringbothplanningandevaluation.Inotherwords,anintermediary’spotentialshouldbeviewedinrelationtothecurrentconditionintheregionwhereitoperatesandthedegreetowhichtheintermediaryhastheinfrastructureandoperationsitwillneedgivenitslong-termvisionandthestrategyitproposesforachievingthisvisionwithintheregion.

Thefollowtablesoutlinethethreedimensionsinmoredetail.Eachdimensionisfurtherspecifiedincriteriaforevaluationaswellasguidingquestionsandsampleindicators.

1.1 Readiness of Local Context

To what extent does the local context demonstrate both a need for and openness to the intermediary’s vision?

Criteria GuidingQuestions SampleIndicators

SupportforInnovation

Towhatextentisthelocalcontextsupportiveofthe

changesandworkproposedbytheintermediary?

• School-baseddecision-makingpolicies• Policies,orpolicywaivers,thatsupportnon-traditionaluse

oftime,talents,andspacewithinschools• Availabilityofinnovation-focusedfundingfromstateor

localgovernmentorfromprivatephilanthropies• Educationinnovationnon-profitsandincubators• Educationconferencesorprofessionalgatheringsfocused

oninnovativeideasandpractices• Presenceofpractitioners,researchers,funders,and

ReadinessofLocalContext

OrganizationalCapacity

Vision&Strategy


17

1.2 Vision and Strategy

To what extent does the intermediary have a logical and viable vision and strategy that is likely to transform instructional practice and improve student outcomes?

technicalassistanceproviderswhoareinterestedinworkingwiththeintermediary

• Mediacoveragethatdemonstratespositiveattitudestowardseducationreformandinnovationgenerallyorthenewideasandpracticesthattheintermediaryseekstodiffuse

FieldNeed

Towhatextentisthelocaleducationsystem

demonstratinganeedforimprovement?

Towhatextentdoesthelocaleducationsystemneed

additionalsupportofthekindanintermediaryprovides?

• Presenceofindividualsororganizationslikelytoparticipate

• Presenceofpersistentorgrowingracialand/oreconomicachievementgaps

• Persistentlylowachievementcomparedtothenationalaverage

• Localachievementhasstagnatedordeclinedoverthepastfiveyears

• Dearthofsupportorganizationsworkingtowardeducationreformorinnovation

• Lackofdiversityintypesofsupportsprovidedbyschoolsupportorganizations

Description GuidingQuestions SampleIndicators

TransformativeVision

Towhatextentdoestheintermediary’svisionfor

changebreakfromtraditionaleducationalpracticesandsystems?

• Visionincludesnewanduniqueusesoftimeandspace• Visionincludesthecreativeintegrationoftechnology• Visionincludesatypicalstaffingallocationsandrole

definitions• Visionredefinescoresocialinteractionsbetweenteachers

andstudentsthatarecurrentlyinplaceinthelocalcontext• Visionisbasedonunderlyingassumptionsabouthow

studentslearnandwhatgoodinstructionmeansthatbreakfromthenormforthelocalcontext

ViabilityofStudentImpact

Howrobustistheorganization’sbasis(inevidenceorinlogic)forpredictingthatitsvisionwillpositivelyimpact

students?

• Linksbetweentheideasandpracticestheintermediaryseekstospreadandpositivestudentoutcomesaredocumentedandsupportedbyrigorousacademicresearch

• Relationshipsbetweentheideasandpracticestheintermediaryseekstospreadandpositivestudentoutcomesarebasedonalogical,detailedhypothesis

• Evidenceofanintermediary’spastworkpositivelyimpactingstudentoutcomesthatcannotbeeasilydiscreditedorexplainedbyotherchangesinthelocalcontext


18

1.3 Organizational Capacity

To what extent does the intermediary have the resources and infrastructure required to enact its strategy?

Description GuidingQuestions SampleIndicators

InternalResources

Towhatextentdoestheintermediaryhave

sufficientfinancialandhumanresourcestosupport

itsstrategy?

• Sufficientfundsavailabletocompleteworkoutlinedinstrategy

• StaffexpertiseinkeystrategicareassuchasK-12pedagogy,adultlearning,changemanagement,knowledgemanagement,researchandevaluation,etc.

• Sufficientnumberofstaff• Suitabletechnologyinfrastructuretosupportgoals,

especiallyaroundpartnercommunicationandknowledgemanagement

• Sufficientaccesstospace

Connectedness

Towhatextentdoestheintermediaryhave

relationshipswithfield(e.g.,K-12education)andsystem

partners(e.g.,policymakers,funders,technicalassistanceproviders)across

thelocalcontext?

• Regularcommunicationwithfunders,technicalassistanceproviders,policymakers,practitioners,researchers,etc.

• Instancesofpriorcollaborationwithfunders,technicalassistanceproviders,policymakers,practitioners,researchers,etc.

InformedStrategy

Towhatextentdoestheintermediary’sstrategyincorporatethekey

DiffusionLeversofcapacitybuilding,networkweaving,andsystemcultivating?

• Strategyincludesactivitiesaimedatincreasingpartnerknowledge,skills,andresources,suchasgrantmaking,1:1coaching,knowledgedissemination,etc.

• Strategyincludesactivitiesintendedtobuildandcontinuallystrengthenconnectionsbetweenpartners,suchasconveningpartners,managingcommunicationssystems,facilitatingcollaborativeproject,etc.

• Strategyincludesactivitiesintendedtodecreasesystemicbarriersand/orincreasesystemicsupports,suchasdraftinglegislature,identifyingandbuildingchampionsforthework,rigoroussystemmapping,etc.

OperationalAlignmenttoStrategy

Towhatextentaretheintermediary’sday-to-dayactionsandprogramslinkedtoitsstrategy?

• Intermediarycanmapallelementsofprogrammingbacktothevariousprongsofitsstrategy

• Staffcanarticulatehowday-to-daytasksaligntospecificprogrammaticelements


19

LearningOrientation

Towhatextentdoestheintermediaryregularlyreviewandreflectonitssuccessesandfailures,changesininternal

organizationalcapacity,andexternalcontextual

conditionsinordertolearnandmakemidcourse

corrections?

• Staffisopenandhonestaboutsuccessesandfailures• Staffhavearticulatedgoalsfortheirownimprovement• Comprehensiveandintegratedperformancemanagement

systemisusedthatconsidersbothprocessandoutcomes• Benchmarking(internalorexternal)isintegratedandused

regularly• Qualitativeandquantitativedataalignedtobothleading

andlaggingindicatorsarereviewedatregularintervals• Demonstratedpracticeofmakingadjustmentsand

improvementsbasedonqualitativeandquantitativedata• Third-partyexpertsbroughtintoassistinreviewing

progress,assessingneed,anddefiningapathforward

Leadership

Towhatextentdoestheintermediary’s

leader/leadershipteamdemonstratecharacteristics

thatallowthemtoeffectivelyandefficientlymanagetheorganization?

• Leadershiparticulatesaclearvisionforwhatispossibleinthefutureandhowtheintermediary’sstrategywillachievethisvision

• Leadershiptakesstepstomotivate,inspire,andrallyothersbehindtheintermediary’svision

• Leadershippromotestheimportanceoflearningandimprovementthroughwordsandactions

• Leadershippromotescreativityandrisktakingthroughwordsandactions

• Staffarticulatefeelingsofsupportandempowermentbyleadership

• Staffarticulateconfidenceinleadershipknowledge,skills,anddispositions

FieldInfluence

TowhatextentistheorganizationconsideredaninfluentialleaderintheK-12

educationfield?

• Reports,tools,orotherresourcescreatedbytheintermediaryarewidelyusedacrossthefield

• Intermediaryhaspositiveandwidenamerecognitionwithinthelocalcontextitseekstoimpact

• IntermediaryhaspositiveandwidenamerecognitionwithintheK-12educationsector

• Intermediaryispresentatkeysector-specificconferencesorotherprofessionalevents

Phase II: Interim Progress InterimProgressassessesanintermediary’simplementationofitsstrategyandtheresultsthatstemfromit.Thephaseincludesthreedimensionsforevaluation:

• ImplementationofStrategy-Theextentandqualityofanintermediary’sactivitiesandthedegreetowhichthesemeettheintermediary’stargets.

• Short-termOutcomes-Theearlychangestopartner,network,andsystemcapacitiesandconditionsstemmingfromactivitiescompleted.


20

• ContinuousLearning-Anintermediary’sowninternallearningandimprovementprocessesandoutcomes.

TheInterimProgressdimensionsandcriteriashouldbeusedtoassessanintermediaryassoonasitbeginsrecruitingandprovidingresourcesandsupporttopartnersandatregularinternalsafterthattime.ThesixcriteriaintheShort-TermOutcomesdimensionbelowaredirectlyalignedtotheDiffusionLeversdiscussedintheprevioussectionofthisreport—CapacityBuilding,NetworkWeaving,andSystemCultivating.WhileitislikelythatmostintermediaryorganizationswillemployallthreeoftheDiffusionLevers,andasaresultshouldevaluateforallsixoutcomes,asmallnumberofintermediariesmayonlyemployselectLevers.Theywouldthenonlyexpecttoseeselectcriteria.

Eachdimensionisfurtherspecifiedincriteriaforevaluationaswellasguidingquestionsandsampleindicators.

2. 1 Implementation of Strategy

To what extent and with what level of quality is the intermediary implementing its strategy?


ExtentofImplementation

Howmanyresourcesand/orhowmuchsupportdidtheintermediaryprovideover

theperiod?

• Numberofreportswritten• Hoursoftrainingprovided• Numberofcoachingsessionscompleted• Numberofemailcommunicationssent• Numbertimespartnerswereconvened• Numberofcollaborativeprojectsengagedinbypartners

QualityofImplementation

Towhatextentweretheresourcesandsupportsof

highquality?

• Satisfactionratesgatheredfrompartnersorfieldactors• Statementsofsatisfactiongatheredfrompartnersorfield

actors• Participantreturnratesacrossmulti-dayworkshopsor

events• Clickratesforresourcesdisseminatedthroughemailor

website• Shareratesforresourcesdisseminatedthroughemailor

website• Alignmentbetweeninternalorexternalqualitycriteria

andactualresourceorsupportsprovided(e.g.,betweenqualitycriteriaforpolicybriefandactualpolicybriefsdrafted)

2.2 Short-term Outcomes

To what extent has the intermediary achieved desired short-term outcomes expected of a Diffusion Intermediary, such as increased partner capacity, a strong network, and improved system conditions?


PartnerUnderstanding

Towhatextentareindividualpartnersexpandingtheir

understandingofinnovation?

• Number/percentofpartnerswhoreportattainingnewknowledgeorskills

• Number/percentofpartnerswhoreporthavingexisting


21

knowledgeorskillsreinforcedorchallenged• Extenttowhichpartnerscancorrectlyrecallknowledgeor

demonstrateskills• Extenttowhichpartnerscanspecifysimilaritiesand

differencesbetweenpriorandnewknowledgeandskills

PartnerPerceptions

Towhatextentareindividualpartnersdevelopingpositiveviewstowardtheinnovation?

• Number/percentofpartnerswhoareconfidentintheirabilitytousenewknowledgeandskills

• Number/percentofpartnerswhoreportnewknowledgeandskillschangedorreinforcedtheirviews,opinions,andbeliefsabouteducation

• Extenttowhichpartnershaveformedpositiveperceptionsofthenewknowledgeandskills

• Number/percentofpartnerswhointendtouseinformationandknowledgegainedfromKMoutput

NetworkForm

Whatindividualsororganizationsmakeupthe

network?

Howcantheconnectionsbetweennetworkmembers

becharacterized?

• Numberofmembers• Geographicdistributionofmembers• Sectorsrepresentedacrossmembership• Typesofmembersrepresented(i.e.,individualsversus

organizations)• Numberofconnectionsforagivenmember(degree)• Distancebetweenonememberandanother(closeness)• Numberoftimesamemberoccupiestheshortestpath

betweentwootherelementsorclusters(betweenness)• Densityofconnectionsaroundcentralhubs(networkcore)• Densityofconnectionaroundtheoutsideofthenetwork

(periphery)• Extentofcentralizationaroundasinglehub• Amountofinformationflowingthroughconnections• Typeofinformationflowingthroughconnections

NetworkFunction

Towhatextentdoesthenetworkhavetheresources,structures,andconditionsinplaceneededtooperateeffectivelyandefficiently?

• Amountoffinancialandmaterialresourcesavailablefornetworkoperations

• Efficiencyofcommunicationstructures• Effectivenessofdecision-makingstructures• Presenceofsupportiveconditions(e.g.,trust,

accountability,alignment)• Qualityofproductscreatedbythenetwork(research,

tools,etc.)

SystemPoliciesandPractices

Towhatextentaretheformalandinformalpoliciesandpracticesthatimpact

partnersenablingordemandinginnovation?

• Legislationthatdemandsinnovation• Humanresourcessystemsandstructuresthatenable

innovation• Accountabilitysystemsandstructuresthatenable

innovation• Professionalstandardsthataligntoinnovation• Organization-widestandardoperatingproceduresthat


22

aligntoinnovation

SystemEngagementandPublic

Will

Towhatextentisthepublicdemandingandsupporting

change?

• Frequencyofpositivemediacoverage• Sizeandfrequencyofpublicgatheringstodemonstrate

supportforinnovation• Dollarsofpublicandprivatefundingdistributedtothe

regionforinnovation• Fundingandinfrastructureconditionsthatenable

innovation

2.3 Continuous Learning

To what extent is the intermediary gathering insights from its successes and failures and proactively responding to these insights?


UnderstandingofProgress

Towhatextenthastheintermediaryidentifiedareas

ofsuccessandfailure?

Whatlessonslearnedhastheintermediarypulledfromitssuccessesandfailuresthatcouldbenefititsownworkor

thelargerfield?

• Qualitativeand/orquantitativeprocessdatathatrevealtheextenttowhichtheintermediaryimplementeditsstrategyasintended

• Qualitativeand/orquantitativeoutcomesdatathatrevealtheextenttowhichthisintermediaryachieveddesiredoutcomes

• Documentedsuccessesandfailuresrevealedbyprocessandoutcomedata

• Department-orprogram-widefamiliaritywiththesesuccessesandfailures

• Documentedlessonslearnedthatlinkbacktosuccessesandfailures

• Department-orprogram-widefamiliaritywithlessonslearned

MidcourseCorrections

Towhatextentistheintermediaryadjustingitsstrategyororganizational

capacityasaresultoflessonslearned?

• Identifiablechangestoarticulatedstrategybasedonlessonslearned

• Observablechangestoenactedstrategybasedonlessonslearned

• Identifiablechangestointermediary’sinfrastructureoroperationsbasedonlessonslearned

• Department-orprogram-wideunderstandingofchangesandhowtheyconnecttolessonslearned

Phase III: Impacts on the Field ImpactsontheFieldassessesanintermediary’sultimategoals—thediffusionofinnovationacrossitsregionandimprovementstosocialoutcomesresultingfromdiffusion.AsthisframeworkisspecifictoanintermediaryintheK-12educationfield,thedimensions,criteria,andindicatorslistedcontainfieldspecificlanguage.However,theycanbeeasilymodifiedtoalignwithotherfields,suchashealthcareorjuvenilejustice.PhaseIIIcontainstwodimensions:


23

• DiffusionofInnovation-Thedegreetowhichchangestonewandinnovativepracticesarescaledacrosstheregion.

• ImprovedStudentOutcomes-Theincreasedknowledgeandskillsofstudentsresultingfromchangestopractice.

ThecriteriausedtoassesstheDiffusiondimensionwereinformedbyresearchpertainingtoboththediffusionofinnovationandthescalingofeducationreform.Morespecifically,thedimensionsofSpread,Depth,Ownership,andSustainabilitywereinformedbyProfessorCynthiaCoburn’smultifaceteddefinitionoftheconceptofscale(Coburn,2003).Thisdefinitionlooksbeyondquantitativemeasuresofgrowthtoricherqualitativemeasuresofthequalityandembeddednessofchange.

Finally,thedimensionsofImprovedStudentOutcomesdescribedherewereinformedbythecurrentbestthinkingregardingtheknowledgeandskillsstudentsneedtobesuccessfulincollege,career,andlife.Indoingso,thedimensionsseektoestablishaholisticstudentoutcomesdefinition.However,thelanguageusedisgenericandsowilllikelyneedtobecustomizedtospecificregionalandeducationalcontexts.Eachdimensionisfurtherspecifiedincriteriaforevaluationaswellasguidingquestionsandsampleindicators.

Diffusion of Innovation

To what extent has the intermediary been able to diffuse the innovative practices it has championed broadly throughout the region?


Depth

Towhatextentarechangestoinstructionalpracticedeep

andconsequential,thusreflectingnewvaluesand

beliefs?

• Extentofchangestoteachers’underlyingbeliefsabouthowstudentslearn

• Extentofchangestoteachers’underlyingbeliefsaboutthenatureofcontent

• Extentofchangestoteachers’underlyingbeliefsaboutthedefinitionofhigh-qualityinstruction

• Extentofchangestostudentandteacherrolesintheclassroom

• Extentofchangestopatternsofteacherandstudenttalks• Extentofchangestohowteachersandstudentstreatone

another

Spread

Towhatextentarechangestoinstructionalpractice

expandingoutwardstomoreandmoreclassrooms,

schools,and/ordistricts?

Towhatextentarechangestoinstructionalpracticeexpandinginwardsand

influencingclassroom,school,anddistrictpoliciesandoperatingprocedures?

• Number/percentofindividualsororganizationschangingpractices

• Degreetowhichnewpracticesareinfluencingdecision-makingattheindividualororganizationallevel

• Specificchangestooperatingpoliciesandproceduresnotinitiallyrequiredbycompletedtobettersupportinnovation

• Degreetowhichteachersdrawonnewpedagogicalknowledge,values,andbeliefsinspacesnotinitiallytargetedbyreform

• Degreetowhichpedagogicalprinciplesofthereformareembeddedindistrictpolicyandprocedures


24

Ownership

Towhatextentisauthorityforthereformbeingtakenonbythedistricts,schools,and

teachers?

• Presenceofstructuresandmechanismsforongoingteacherlearningaboutinnovation

• Presenceofestablishedstrategiesforcontinuingtofundinnovationactivities

• Degreetowhichleadershiphastakenresponsibilityforcontinuingtospreadinnovation

• Useofreform-centeredideasofstructuresinschoolordistrictdecision-making

Sustainability

Towhatextentdochangestopracticeremaininplaceafter

externalsupportsarenolongerpresent?

• Number/percentofteacherswhocontinueusinginnovativepracticesafterexternalsupportsareremoved

• Number/percentofteacherswhobeginusinginnovativepracticesafterexternalsupportsareremoved

Equity

Towhatextentarechangestopracticeoccurringequally

acrossdifferentsocioeconomicgroups?

• Gapindepthofdiffusionacrossdifferentgeographicareasandsocioeconomicgroups

• Gapinspreadofdiffusionacrossdifferentgeographicareasandsocioeconomicgroups

• Gapinownershipofdiffusionacrossdifferentgeographicareasandsocioeconomicgroups

• Gapinequityofdiffusionacrossdifferentgeographicareasandsocioeconomicgroups

Improved Student Outcomes

To what extent are social outcomes improving in parallel with the changes in individual and organizational practice?


Knowledge

Towhatextentarestudentsdevelopingincreased

understandingofimportantcontentknowledge?

• Rateofmasteryoncontentknowledgeassessments• Percentgrowthinmasteryoncontentknowledge

assessments• Extentofcontentknowledgeunderstanding

demonstratedthoughperformancetasks

CognitiveandMetacognitive

Skills

Towhatextentarestudentsdevelopingkeymental

processingskillsaswellastheabilitytomonitorandassess

theuseoftheseskills?

• Regularitywithwhichkeyskillsaredemonstrated• Number/percentofstudentsdemonstratingkeyskills• Extenttowhichkeyskillsaredemonstratedthrough

performancetasks,behaviors,ordiscussions

HabitsandDispositions

Towhatextentarestudentsdevelopingthequalitiesand

mindsetsneededtobesuccessfulincollege,career,

andlife?

• Regularitywithwhichkeyhabitsanddispositionsaredemonstrated

• Number/percentofstudentsdemonstratingkeyhabitsanddispositions

• Extenttowhichhabitsanddispositionsaredemonstratedthroughperformancetasks,behaviors,ordiscussions


25

CLOSING Thesuccessfuldiffusionofnewpracticesisapersistentchallengewithinournation’sschoolsandschoolsystems.Educationalunderachievementandinequityarewickedproblemsthataredeeplyembeddedinthewaysinstitutionsandsocietyoperatepreventingeasysolutions.However,practitioners,funders,researchers,andotherseagertoseemeaningfulchangeinstudentopportunitiesandoutcomesareturningtoneworganizationalstructuresandstrategiestomovetheneedle.OneofthesestrategiesistheuseofnetworkedintermediarystructuresliketheNGLCRegionalFundsforBreakthroughSchoolsinitiative.

Thesestructureshavethepotentialtoprovidecustomizedsupportacrossadiverserangeofuniquelocalcontexts,whilealsofacilitatingcross-contextsharingandlearning.Tocapitalizeonthispotential,theintermediaryorganizationsthatoperateascrucialhubswithinthenetworkmustoftenworkatmultiplelevelssimultaneously.Theysupportandbuildthecapacityofindividualandorganizationalpartners,facilitatecommunicationandcollaborationacrosstheirpartnernetworksasawhole,andfostertheeconomic,social,andpoliticalconditionsthenetworkneedstoinnovate.

CPRLbeganitsworkwitharecognitionoftheimportantroleintermediaryorganizationsplayintransformingeducationandwiththedesiretoclarifythisroleandbuildgreatercapacitytomonitorandevaluateit.Thisreport,andtheaccompanyingtools,reflectwhatwehavecometounderstandabouthowintermediariesimpactchangewiththehopethatitwillsupportcurrentandfutureintermediaries,funders,andpartnerorganizationinbuildingsuccessfulnetworkedintermediarystructures.


26

REFERENCES

Capacity Building

Connolly,P.&York,P.(2002).Evaluatingcapacity-buildingeffortsfornonprofitorganizations.ODPractitioner.34(4):33-39.

Dalkir,K.(2011).Introductiontoknowledgemanagement.Knowledgemanagementintheoryandpractice.MITPress:Cambridge,MA.

Kirkpatrick,D.&Kirkpatrick,J.(2005).Thefourlevel’biggestchallenge.Transferringlearningtobehavior:Usingthefourlevelstoimproveperformance.Berret-KoehlerPublishers:Oakland,CA.

Ohkubo,S.,Sullivan,T.M.,&Harlan,S.V.(2013).Guidetomonitoringandevaluatingknowledgemanagementinglobalhealthprograms.USAID.Retrievedfromhttps://www.k4health.org/sites/default/files/km-monitoring-and-eval-guide_0.pdf.

Diffusion and Scaling

Ashley,S.R.(2009).Innovationdiffusion:Implicationsforevaluation.InJ.M.Ottoson&P.Hawe(Eds.),Knowledgeutilization,diffusion,implementation,transfer,andtranslation:Implicationsforevaluation.NewDirectionsforEvaluation,124,35–45.

Coburn,C.(2003).Rethinkingscale:Movingbeyondnumberstodeepandlastingchange.EducationalResearcher,32(6):3-12.

Rogers,E.(1962).Diffusionofinnovation.TheFreePress:NewYork,NY.

Intermediaries

Anthony,E.K.&Austin,M.J.(2008)Theroleofanintermediaryorganizationinpromotingresearchinschoolsofsocialwork:ThecaseoftheBayAreaSocialServicesConsortium.SocialWorkResearch,32(4),287-293.

Blank,M.J.,Brand,B.,Deich,S.,Kazis,R.,Politz,B.,&Trippe,S.(2003).Localintermediaryorganizations:Connectingthedotsforchildren,youth,andfamilies.Retrievedfromhttp://www.jff.org/publications/local-intermediary-organizations-connecting-dots-children-youth-and-families.

Cooper,A.&Shewchuk,S.(2015).Knowledgebrokersineducation:Howintermediaryorganizationsarebridgingthegapbetweenresearch,policyandpracticeinternationally.EducationPolicyAnalysisArchives,23(118).

Fisched,D.J.(2005).Workforceintermediaries:Poweringregionaleconomiesinthenewcentury.Retrievedfromhttps://eric.ed.gov/?id=ED485943.

Foley,E.(2010).ApproachesofBill&MelindaGatesFoundation-fundedintermediaryorganizationstostructuringandsupportingsmallhighschoolsinNewYorkCity.Retrievedfromhttp://files.eric.ed.gov/fulltext/ED510236.pdf.

FSG.(2013).Backboneeffectiveness:27indicators.Retrievedfromhttp://www.collaborationforimpact.com/wp-content/uploads/2014/01/Backbone_Effectiveness_Indicators.pdf.

GatewayCenterforGiving.(2012).MeasuringtheworkofintermediariesintheSt.Louisregion.Retrievedfromhttp://www.centerforgiving.org/Portals/0/Documents/Measuring%20the%20Work%20of%20Intermediaries%20Paper%20FINAL.pdf.


27

GatewayCenterforGiving.(2011).UnderstandingtheintermediaryinfrastructureinSt.Louis:Whotheyare,whattheydo,andcommonmisconceptions.Retrievedfromhttp://www.centerforgiving.org/portals/0/intermediaries%20report_final_v3.pdf.

GrantmakersforEffectiveOrganizations.(2013).Smarterrelationships,betterresults:Makingthemostofgrantmakers’workwithintermediaries.Retrievedfromhttp://www.geofunders.org/resource-library/all/record/a066000000AjtYcAAJ.

GrantmakersforEffectiveOrganizations.(2014).Whydofundersworkwithintermediaries?Retrievedfromhttp://www.geofunders.org/resource-library/all/record/a066000000H2hN3AAJ.

Honig,M.I.(2004).Thenewmiddlemanagement:Intermediaryorganizationsineducationpolicyimplementation.EducationalEvaluationandPolicyAnalysis,26(1),65-87.

Johnson,E.,Rothstein,F.andGajdosik,J.(2004),Theintermediaryroleinyouthworkerprofessionaldevelopment:Successesandchallenges.NewDirectionsforYouthDevelopment,104:51–64.

Lopez,E.M.,Kreider,H.&Coffman,J.(2005).Intermediaryorganizationsascapacitybuildersinfamilyeducationalinvolvement.UrbanEducation,40(1),78-105.

Lubienski,C.,Scott,J.,&DeBray,E.(2011).Theriseofintermediaryorganizationsinknowledgeproduction,advocacy,andeducationalpolicy.TeachersCollegeRecord,22.

Piha,S.&Pittman,K.Theimportantroleofintermediariesincollectiveimpactwork.TheForumforYouthInvestment.Retrievedfromhttps://collectiveimpactforum.org/sites/default/files/The%20Important%20Role%20of%20Intermediaries%20in%20Collective%20Impact%20Work.pdf.

Turner,S.,Merchant,K.,Kania,J.&Martin,E.(2012)Understandingthevalueofbackboneorganizations.StanfordSocialInnovationReview.http://www.fsg.org/publications/understanding-value-backbone-organizations-collective-impact.

U.S.DepartmentofHealthandHumanServices.(2008).Breakthroughperformance:10emergentpracticesofleadingintermediaries.Retrievedfromhttp://www.daremightythings.com/pdf/media_center/breakthrough_guide.pdf.

Scott,G.(2007).Fundercollaboratives:Aphilanthropicstrategyforsupportingworkforceintermediaries.Retrievedfromhttp://www.jff.org/sites/default/files/publications/FunderCollab.pdf.

Szanton,P.L.(2003).Towardmoreeffectiveuseofintermediaries.NewYork:FoundationCenter.Retrievedfromfoundationcenter.org/gainknowledge/practicematters/.

Wynn,J.R.(2000).Theroleoflocalintermediariesintheyouthdevelopmentfield.Retrievedfromhttps://www.chapinhall.org/sites/default/files/Intermediary-disc-paper.pdf

Network Weaving

Barabasi,A.(2002).Linked:Thenewscienceofnetworks.Cambridge,MA:PerseusPublishing.

Baran,P.(1964).Ondistributedcommunications:Introductiontodistributedcommunicationsnetworks.TheRANDCorporation.Retrievedfromhttp://www.rand.org/content/dam/rand/pubs/research_memoranda/2006/RM3420.pdf.

Bonbright,D.&Khagram,S.(2010).Innovationsforscalingimpactandkeystoneaccountability:Nextgenerationnetworkevaluation.Retrievedfromhttp://www.sdc-learningandnetworkingblog.admin.ch/uploads/2010/07/ME-of-Networks-2010.pdf.

Bryk,A.S.,Gomez,L.,Grunow,A.,&LeMahieu,P.(2015)Learningtoimprove.Cambridge,MA:HarvardEducationPress.


28

CenterforAdvancementofCollaborativeStrategiesinHealth.(N.d.)Partnershipself-assessmenttool.Retrievedfromhttp://www.lmgforhealth.org/sites/default/files/Center_for_the_Advancement_of_Collaborative_Strategies_in_Health_%28CACSH%29_Parternship_Self-Assessments.pdf.

Creech,H.(2001).Measuringwhileyoumanage:Planning,monitoringandevaluatingknowledgenetworks.Winnipeg,Manitoba:InternationalInstituteforSustainableDevelopment.Retrievedfromhttps://www.iisd.org/pdf/2001/networks_evaluation.pdf.

Creech,H.(2001).Formfollowsfunction.Winnipeg,Manitoba:InternationalInstituteforSustainableDevelopment.Retrievedfromhttps://www.iisd.org/pdf/2001/networks_structure.pdf.

Creech,H.&Ramji,A.,(2004)Knowledgenetworks:Guidelinesforassessment.Winnipeg,Manitoba:InternationalInstituteforSustainableDevelopment.Retrievedfromhttps://www.iisd.org/pdf/2004/networks_guidelines_for_assessment.pdf

Fowler,C.H.&Christakis,N.A.(2009).Connected:Thesurprisingpowerofoursocialnetworksandhowtheyshapeourlives.NewYork,NY:Little,BrownandCompany.

Holley,J.(2012).Networkweaverhandbook.Athens,OH:NetworkWeaversPublishing.

Lowell,S.(2006).Buildingthefieldofdreams:Socialnetworksasasourceofsector-levelcapacityintheafter-schoolworld.Retrievedfromhttps://barrfdn-prod.s3.amazonaws.com/documents/14/attachments/Building_the_Field_of_Dreams.pdf?1421961228.

Malinsky,E.,&Lubelsky,Chad.(2011)Networkevaluation:Cultivatinghealthynetworksforsocialchange.Retrievedfrom:http://socialinnovation.ca/networkevaluation.

NetworkImpact&CenterforEvaluationInnovation.(2014).Thestateofnetworkevaluation:Aguide.Retrievedfromhttp://www.networkimpact.org/the-state-of-network-evaluation-a-guide/.

NetworkImpact&CenterforEvaluationInnovation.(2009).Networkhealthscorecard.Retrievedfromhttps://www.networkimpact.org/downloads/NH_Scorecard.pdf.

Plastrik,P.&Taylor,M.(2006).Netgains:Ahandbookfornetworkbuildersseekingsocialchange.Retrievedfromhttps://networkimpact.org/downloads/NetGainsHandbookVersion1.pdf.

Plastik,P.,Taylor,M.,&Cleveland,J.(2014).Connectingtochangetheworld:Harnessingthepowerofnetworksforsocialimpact.Washington,DC:IslandPress.

Scearce,D.(N.d.).Networkeffectivenessdiagnosticanddevelopmenttool.Retrievedfromhttp://www.workingwikily.net/network_diagnostic.pdf.

Scearce,D.(2011).Catalyzingnetworksforsocialchange:Afunder'sguide.Washington,D.C.:GrantmakersforEffectiveOrganizations.

Scearce,D.,Kasper,G.,McLeodGrant,H.(2010).Workingwikily.StanfordSocialInnovationReview.Retrievedfromhttps://ssir.org/articles/entry/working_wikily.

Taylor,M.,Whatley,A.,&Coffman,J.(2015).Networkevaluationinpractice:Approachesandapplications.TheFoundationReview,7(2).

Kania,J.&Kramer,M.(2011).Collectiveimpact.StanfordSocialInnovationReview.Retrievedfromhttp://www.fsg.org/publications/collective-impact.

Parkhurst,M.&Preskill,H.(2014).Learninginaction:Evaluatingcollectiveimpact.StanfordSocialInnovationReview.Retrievedfromhttps://ssir.org/articles/entry/learning_in_action_evaluating_collective_impact.

Preskill,H.,Parkhust,M.,&SplanskyJuster,J.(2014).Guidetoevaluatingcollectiveimpact.Retrievedfromhttp://www.fsg.org/publications/guide-evaluating-collective-impact.


29

Systems Thinking and Change

TheBridgespanGroup.(2009).Thestrongfieldframework:Aguideandtoolkitforfundersandnonprofitscommittedtolarge-scaleimpact.Retrievedfromhttps://irvine-dot-org.s3.amazonaws.com/documents/64/attachments/strongfieldframework.pdf?1412656138

Meadows,D.(2008).Thinkinginsystems:Aprimer.WhiteRiverJunction,VT:ChelseaGreenPublishing.

Preskill,H.,Gopal,S.,Mack,K.,&Cook,J.(2015).Evaluatingcomplexity:Propositionsforimprovingpractice.FSG.Retrievedfromhttp://www.issuelab.org/resource/evaluating_complexity_propositions_for_improving_practice

Stroh,D.(2015).Systemsthinkingforsocialchange:Apracticalguidetosolvingcomplexproblems,avoidingunintendedconsequences,andachievinglastingresults.WhiteRiverJunction,VT:ChelseaGreenPublishing.

Latham,N.(2014).Apracticalguidetoevaluatingsystemschangeinahumanservicessystemcontext.LearningforAction.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4Zb19GOEE0Y1Q3SGM/view?pref=2&pli=1

Hargreaves,M.(2010).Evaluatingsystemchange:Aplanningguide.MathematicaPolicyResearch,Inc.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZRVYtODJld0drX28/view?pref=2&pli=1.

Coffman,J.(2009).Auser’sguidetoadvocacyevaluationplanning.HarvardFamilyResearchProject.Retrievedfromhttp://www.hfrp.org/.

Coffman,J.&Reed,E.Uniquemethodsinadvocacyevaluation.Retrievedfromhttp://www.pointk.org/resources/files/Unique_Methods_Brief.pdf.

HarvardFamilyResearchProject(2007).Advocacyandpolicychange.TheEvaluationExchange.13(1):1-32.

Forti,M.(2012).Measuringadvocacy-Yeswecan!StanfordSocialInnovationReview.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZbU5zLWFVX0hzb0E/view.

Reisman,J.,Gienapp,A.,&Stachowiak,S.(2007).Aguidetomeasuringadvocacyandpolicy.TheAnnieE.CaseyFoundation.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZX09HTktUYld4MjQ/view.

ADDITIONALRESOURCES

Measurement Toolkit for Intermediary Organizations AppendixtoIntermediaryOrganizationsandEducationInnovation


C E N T E R F O R P U B L I C R E S E A R C H A N D L E A D E R S H I P

31

Diffusion Levers Evaluation Toolkit: Capacity Building

TheDiffusionIntermediaryEvaluationFrameworkincludestwocriteriaforassessingshort-termCapacityBuildingoutcomes:PartnerUnderstandingandPartnerPerceptions.Thesecriteriaarebasedonareviewoftheliteraturelistedattheendofthissection.Thisreviewidentifiedacommonsetofstepsindividualsmovethroughastheydecidewhetherornottoadoptanewpracticeincludingbuildingawarenessandunderstanding,formingpositiveperceptions,implementing,andfinallyinternalizingthenewpractices.Thefirsttwostepsinformedthecriterialistedhere.LatterstepsarepartofPhase3intheFramework.

Evaluative Criteria, Questions, and Sample Indicators



Towhatextentareindividualpartnersexpandingtheirunderstandingof

innovation?

• Number/percentofpartnerswhoreportattainingnewknowledgeorskills

• Number/percentofpartnerswhoreporthavingexistingknowledgeorskillsreinforcedorchallenged

• Extenttowhichpartnerscancorrectlyrecallknowledgeordemonstrateskills

• Extenttowhichpartnerscanspecifysimilaritiesanddifferencesbetweenpriorandnewknowledgeandskills

PartnerPerceptions

Towhatextentareindividualpartners

developingpositiveviewstowardtheinnovation?

• Number/percentofpartnerswhoareconfidentintheirabilitytousenewknowledgeandskills

• Number/percentofpartnerswhoreportnewknowledgeandskillschangedorreinforcedtheirviews,opinions,andbeliefsabouteducation

• Extenttowhichpartnershaveformedpositiveperceptionsofthenewknowledgeandskills

• Number/percentofpartnerswhointendtouseinformationandknowledgegainedfromKMoutput

Tools

SampleSurveyQuestionstoAssessPerceptionsofNewInstructionalPracticesCenterforPublicResearchandLeadershipThesesamplequestionscanbecustomizedtocreateasurveyinstrumentforassessingpartnerperceptionsbeforeorafteracapacity-buildingevent.Thequestionswereinformedbydiffusiontheory,specificallythePerceivedAttributesofInnovationframework(Rogers,1962).


ü PartnerPerception

SampleSurveyQuestionstoAssessUnderstandingandIntentiontoActCenterforPublicResearchandLeadership


32

Thesesamplequestionscaninformthecreationofasurveyinstrumenttoassesspartnerlearningandintentiontoactfollowingacapacity-buildingevent. ü Partner

Understanding

ü PartnerPerception

Resources Connolly,P.&York,P.(2002).Evaluatingcapacity-buildingeffortsfornonprofitorganizations.ODPractitioner.34(4):33-

39.

Dalkir,K.(2011).Introductiontoknowledgemanagement.Knowledgemanagementintheoryandpractice.MITPress:Cambridge,MA.

Kirkpatrick,D.&Kirkpatrick,J.(2005).Thefourlevel’biggestchallenge.Transferringlearningtobehavior:Usingthefourlevelstoimproveperformance.Berret-KoehlerPublishers:Oakland,CA.

McCombs,B.L.&Whisler,J.TheLearner-centeredclassroomandschool.(1997).Wiley&Sons,Inc.SanFrancisco:CA.

Ohkubo,S.,Sullivan,T.M.,&Harlan,S.V.(2013).Guidetomonitoringandevaluatingknowledgemanagementinglobalhealthprograms.USAID.Retrievedfromhttps://www.k4health.org/sites/default/files/km-monitoring-and-eval-guide_0.pdf.


33

PERCEPTIONSOFNEWINSTRUCTIONALPRACTICESInformedbyDiffusionofInnovation,EverettRoger’s(1962)

RelativeAdvantage StronglyAgree Agree Disagree Strongly

Disagree

RelativeAdvantage

1. [Thenewinstructionalpractices]willmakemyjobeasier.

2. IwillbemoreeffectiveasateacherifIimplement[thenewinstructionalpractices].

3. Myexistinginstructionalpracticesbuildstudentknowledgeandskillsjustaswellas[thenewinstructionalpractices]will.

4. Idon’tthink[thenewinstructionalpractices]willtransformmystudents’learningenoughtobeworthtrying.

Compatibility

5. [Thenewinstructionalpractices]alignwellwiththewaymyschoolsystemoperates.

6. [Thenewinstructionalpractices]donotfitwellwiththewayIteach.

7. Ibelievethat[thenewinstructionalpractices]aretherightwaytoteacherstudents.

8. Mypeerswillnotsupport[thenewinstructionalpractices].

Thesurveyitemsbelowcanbecustomizedandusedtoassesstheextenttowhichindividualsaredevelopingpositiveperceptionsofanewpracticeorsetofpractices.Theitemsareorganizedintofivecategories:

• RelativeAdvantage-Thedegreetowhichaninnovationisperceivedasbeingbetterthantheideaitsupersedes.

• Compatibility-Thedegreetowhichaninnovationisperceivedasconsistentwiththevalues,pastexperiences,andneedsofpotentialadopters.

• Complexity-Thedegreetowhichaninnovationisperceivedasrelativelydifficulttounderstandanduse.• Trialability-Thedegreetowhichaninnovationmaybeexperimentedwithonalimitedbasis.• Observability-Thedegreetowhichtheresultsofaninnovationarevisibletoothers.


34

ResponseKeyQuestions1,2,5,7,10,13,16,19Stronglyagree=0Agree=1Disagree=2StronglyDisagree=3

Questions3,4,6,8,9,11,12,14,15,17,18,20Stronglyagree=3Agree=2Disagree=1StronglyDisagree=0

RelativeAdvantage StronglyAgree

Agree Disagree StronglyDisagree

Complexity

9. IhavetroubleimaginghowIcouldpossiblyimplement[thenewinstructionalpractices].

10. Overall,Ibelieve[thenewinstructionalpractices]wouldbeeasyformycolleaguestoimplement.

11. Iwouldstruggletoexplain[thenewinstructionalpractices]tootherpeopleinmyschoolsystem.

12. Gettingtheresourcesandsupportsneededfor[thenewinstructionalpractices]wouldbecomplicatedandtimeconsuming.

Trialability

13. Icouldgobacktomyclassroomtomorrowandeasilytrysomeof[thenewinstructionalpractices].

14. Experimentingwith[thenewinstructionalpractices]insmallwayswouldn’treallybepossible.It’sallornothing.

15. Itwouldbechallengingtotestout[thenewinstructionalpractices]withoutgettingadditionalsupportfrommyschoolsystem.

16. Iseepiecesof[thenewinstructionalpractices]thatIcoulddemonstrateformypeerswhenIgetbacktowork.

Observability

17. Ihavetroublevisualizing[thenewinstructionalpractices].

18. WhenIgointoclassroomswhereteachersareimplementing[thenewinstructionalpractices]Ihavetroubleidentifyingwhatisdifferent.

19. Classroomsusing[thenewinstructionalpractices]lookverydifferentfrommostotherclassroomsI’veseen.

20. IwishIcouldsee[thenewinstructionalpractices]inactionbutit’shardtofindhigh-qualityexamples.


35

SAMPLEPARTNERCAPACITY-BUILDINGQUESTIONS

Learning

Towhatdegreedoyoufeelyouhaveimprovedknowledgeorskillsinthefollowingareas?

Pleasecheckoneboxtotherightofeacharea.

NoImprovement

MinimalImprovement

ModerateImprovement

LargeImprovement

1.[Knowledgeorskillcategory]





6.Howwouldyoucharacterizetheoverallimprovementinyourknowledge/skillsasaresultofthisevent?

� NoImprovement� MinimalImprovement� ModerateImprovement� LargeImprovement

7.Howlikelyisitthatyouwillbeabletoapplytheknowledge/skillslearnedinthiseventwithoutsupportwhenyoureturntoyourschool?

� VeryUnlikely� SomewhatUnlikely� SomewhatLikely� VeryLikely

8.Pleasedescribeanyfactorsthatmayinterferewithyourapplyingtheknowledgeandskillsfromtheeventwhenyoureturntoyourschool.

Thesurveyitemsbelowcanbecustomizedandusedtoassesstheextenttowhichpartnersdevelopednewunderstandingsandpositiveperceptionsbecauseofcapacity-buildingactivities.


36

PerceptionandIntention

Forthefollowingstatementsindicatepossibleactionsyoumighttakeafterthisevent.Indicatetheextenttowhicheachstatementistrueorfalseforyou.9.Ihaveidentified1-2specificstrategiesasaresultofthiseventthatIwillfocusonintegratingintomypracticewithinthenextmonth.

� DefinitelyFalse� ProbablyFalse� ProbablyTrue� DefinitelyTrue

10.Inthenextmonth,IwillshareatleastsomeofwhatIlearnedinthiseventwithacolleague.


11.Inthenextmonth,IwillseekoutmoreinformationrelatedtowhatIlearnedtoday.


12.Whataspectsoftheeventweremostusefultoyou?


37

Diffusion Levers Evaluation Toolkit: Network Weaving

TheDiffusionIntermediaryEvaluationFrameworkincludestwocriteriaforassessingshort-termNetworkWeaving

outcomes:NetworkFormandNetworkFunction.Thesecriteriaarebasedonascanofthenetworkevaluationliteraturelistedattheendofthissection.Thescanidentifiedthreeprimaryareasforevaluationspecifictonetworks.These

includethetwoareaslistedhere—form(i.e.structure,shape,connectivity)andfunction(i.e.health,vibrancy,

operations)—aswellasnetworkimpacts.ThethirdareawasincorporatedintoPhase3oftheFrameworkbecauseit

resultsfromacombinationofallthreeDiffusionLevers.



NetworkForm

Whatindividualsororganizationsmakeup

thenetwork?

Howcantheconnectionsbetweennetwork

membersbecharacterized?

• Totalnumberofmembers

• Geographicdistributionofmembers

• Sectorsrepresentedbymembers

• Typesofmembers(i.e.individualsversusorganizations)

• Numberofconnectionsforagivenmember(degree)

• Distancebetweenonememberandanother(closeness)

• Frequencywithwhichamemberoccupiesshortestpathbetweentwo

othermembers(betweenness)

• Densityofconnectionsaroundnetworkcore

• Densityofconnectionsaroundperiphery

• Extentofcentralizationaroundasinglehub

• Amountofinformationflowingthroughconnections

• Typeofinformationflowingthroughconnections

NetworkFunction

Towhatextentdoesthenetworkhaveresources,structures,andconditions

neededtooperateeffectivelyandefficiently?

• Financialresourceavailablefornetworkoperations

• Materialresourceavailablefornetworkoperations

• Efficiencyofcommunicationstructures

• Effectivenessofdecision-makingstructures

• Peeraccountabilityacrossthenetwork

• Trustbetweenmembers

• Balanceofparticipationacrossmembers

• Extentofalignmentbetweenmembers

• Workproducedasaresultofcollaboration


38

Tools

KumuKumuInc.Awed-basednetwork-andsystem-mappingplatformwithasocialnetwork

analysisoptionthatallowsfornetworkdatatobemanuallyenteredoruploaded

throughExcelorGoogleSheets.Kumuallowsausertocalculatevariousnetwork

metrics(degree,closeness,betweeness,etc.)aswellascodetheresultingmap

usingdifferentcolors,gradations,andlinetypes.

ü NetworkForm

NetworkFunction

MapYourNetworkbyHandNetworkWeaverHandbookBasicdirectionsformappingnetworksbyhand.Thisprocessissuitableforsmall

networksandcanbeengagedincollaborativelyasanetworkweavingactivity.

ü NetworkForm

NetworkFunction

Network Design and Assessment Scales TheCenterforPublicResearchandLeadershipAplanningandreflectiontoolthatidentifiestencharacteristicsnetworksmay

varyinrelationtoalongwithscalesandguidingquestionsforself-assessment.

ThistoolwasinspiredbytheMonitorInstitute’sresearchonnetworkdesign.

ü NetworkForm

NetworkFunction

Network Effectiveness Diagnostic and Development Tool MonitorInstituteAflexibletoolforassessingandstrengtheninganetwork’sfunction.Itidentifies

variouscharacteristicsandalignedattributesthataredesirableforanetworkto

have.Thetoolalsoprovidesdifferentcharacteristicsfornetworkswithclear

boundariesandmembers(“boundednetworks”)andnetworkswithambiguous

boundariesandmembers(“unboundednetworks”).

NetworkForm

ü NetworkFunction

Network Health Score Card NetworkImpactandCauseCommunicationsA22-questionsurveythatassessesnetworkhealth,orfunction,infourareas(1)

networkpurpose,(2)networkperformance,(3)networkoperations,and(4)

networkcapacity.Eachquestionaskstherespondenttoratethedegreetowhich

theyagreewithaquestionona5-pointscale.

NetworkForm

ü NetworkFunction

PartnerSelf-AssessmentToolCenterforAdvancementofCollaborativeStrategiesinHealth

Avalidatedsurveycontaining67questionspertainingtocollaborationspread

acrosseleventopics:synergy,leadership,efficiency,administrationand

management,non-financialresources,financialandothercapitalresources,

decisionmaking,benefitsofparticipation,drawbacksofparticipation,benefits

anddrawbacksofparticipatinginthepartnership,andsatisfactionwith

participation.

NetworkForm

ü NetworkFunction


39

Sample Network Form Questions TheCenterforPublicResearchandLeadershipSamplequestionstocustomizeanduseingatheringinformationontheformof

anetwork.Thedatacollectedcanbevisualizedthroughasocialnetworkanalysis

orbecompiledandreportedoutthroughmoretraditionaltablesandgraphs.

ü NetworkForm

NetworkFunction

Sample Interview Questions for Network Members TheCenterforPublicResearchandLeadershipQuestionsthatcanbecustomizedandusedininterviewsorfocusgroupswith

NetworkMembers.Thesequestionswereinformedbyvarioussourcesandcover

boththeformandthefunctionofthenetwork.

ü NetworkForm

ü NetworkFunction

Resources CenterforAdvancementofCollaborativeStrategiesinHealth.(N.d.)Partnershipself-assessmenttool.Retrievedfrom

http://www.lmgforhealth.org/sites/default/files/Center_for_the_Advancement_of_Collaborative_Strategies

_in_Health_%28CACSH%29_Parternship_Self-Assessments.pdf.

Creech,H.&Ramji,A.,(2004)Knowledgenetworks:Guidelinesforassessment.Winnipeg,Manitoba:International

InstituteforSustainableDevelopment.

Creech,H.(2001).Measuringwhileyoumanage:Planning,monitoringandevaluatingknowledgenetworks.Winnipeg,

Manitoba:InternationalInstituteforSustainableDevelopment.

Creech,H.(2001).Formfollowsfunction.Winnipeg,Manitoba:InternationalInstituteforSustainableDevelopment.

Holley,J.(2012).Networkweaverhandbook.Athens,OH:NetworkWeaversPublishing.

Malinsky,E.,&Lubelsky,Chad.(2011)Networkevaluation:Cultivatinghealthynetworksforsocialchange.Retrievedfrom:http://socialinnovation.ca/networkevaluation.

InnovationsforScalingImpactandKeystoneAccountability:NextGenerationNetworkEvaluation(DavidBonbright&

SanjeevKhagram,2010)

MonitorInstiture.Engage:HowFundersCanSupportandLeverageNetworkforSocialImpact.Retrievedfromhttp://engage.rockefellerfoundation.org/what-could-a-network-help-me-achieve/what-network-design-would-

be-the-most-useful/

NetworkImpact&CenterforEvaluationInnovation.(2014).Thestateofnetworkevaluation:Aguide.Retrievedfrom

http://www.networkimpact.org/the-state-of-network-evaluation-a-guide/.

NetworkImpact&CenterforEvaluationInnovation.(2009).Networkhealthscorecard.Retrievedfrom

https://www.networkimpact.org/downloads/NH_Scorecard.pdf.

Plastik,P.,Taylor,M.,&Cleveland,J.(2014).Connectingtochangetheworld:Harnessingthepowerofnetworksforsocialimpact.Washington,DC:IslandPress.

Plastrik,P.&Taylor,M.(2006).Netgains:Ahandbookfornetworkbuildersseekingsocialchange.Retrievedfrom

https://networkimpact.org/downloads/NetGainsHandbookVersion1.pdf.

Scearce,D.(N.d.).Networkeffectivenessdiagnosticanddevelopmenttool.Retrievedfromhttp://www.workingwikily.

net/network_diagnostic.pdf.

Taylor,M.,Whatley,A.,&Coffman,J.(2015).Networkevaluationinpractice:Approachesandapplications.TheFoundationReview,7(2).


40

MAPYOURNETWORKBYHANDSource:NetworkWeaverHandbook,JuneHolley,2012

Steps

1. Haveeveryoneinthegroupdrawacircleoraddamovablestickertorepresentanodeforoneselfandlabelit.

Thendrawnodesforothersinthenetworkyouareworkingwithclosely.

2. Drawlinesbetweenyouandtheothersusingthekeybellow.Thendrawlinesbetweenanyofthepeopleinthe

networkwhoknoweachother.ThisistheCore.

3. Nextdrawnodesforotherindividualsandorganizationsthatyouareworkingwithontheproject,butless

frequently,anddrawlinestotheindividualsinyourprojectnetworkwhohavearelationshipwiththat

individual.ThisisthePeriphery.

4. Addotherindividualsorgroupstheyareconnectedto(whomightaddvaluetoyournetwork)anddrawlines

connectingthem.ThisinyourPotentialNetwork.

5. Aroundtheoutsideedge,putnodesforindividualsorgroupsyouarenotworkingwithinthisproject,butwho

couldaddvaluetotheprojectiftheywereinvolved.Thesemightbepeoplewithexpertise,peoplefromother

communitieswhohavebeeninnovatinginwaysthatwouldbeofvaluetoyournetwork,orpeoplewhoare

oftenleftoutofprojects.ThisisalsoyouPotentialNetwork.

Insmallnetworks,itisfeasibletomapthenetworkbyhandonalargeposter,awall,oranonlinemind-mapping

platformlikeMindMeister.Todothissuccessfully,itishelpfultohaveallormostnetworkmemberspresent.

RelationshipKey

Havecollaboratedonaproject(color1)

Gotothemfororofferadvice,information,resources,andexpertise

(color2)

Knowthepersonbuthaven’tcollaboratedorexchangesresourcesyet

(color3)


41

NETWORKDESIGNSCALESANDQUESTIONS

InformedbyWhatNetworkDesignWouldbetheMostUseful?,TheMonitorInstituteand

NetworkEvaluation:CultivatingHealthyNetworksforSocialChange,EliMalinskyandChadLubelsky,2011

PARTA:NETWORKDESIGN

Completethescalesandquestionsbelowbasedonthevisionforthenetwork6monthsfromtoday.

Size

� � � � � � � �

Hundreds Handful

Howmanymembersdoweaspiretohave?

Whatisourreasonforthisdesignchoice?

SectorRepresentation

� � � � � � � �

SameSector MixedSector

Whatsectorsdowehopearerepresented?


Networkscantakevariousforms.UsethescalesandalignedquestionsinPartAofthissurveytoclarifyand

documentnetworkdesignchoices.UsethescalesandalignedquestionsinPartBtoassessprogress.


42

Geography

� � � � � � � �

City-Based Global

Whatgeographicareadoweintendtocover?


MemberType

� � � � � � � �

Systems Individuals

Whattypeofmembersdoweseektohave?


Orientation

� � � � � � � �

Action Learning

Whatorientationdoweplantotake?



43

Centralization

� � � � � � � �

Centralized Decentralized

Whatdegreeofcentralizationdoweplantoestablish?


GoalAlignment

� � � � � � � � �

Maximum Minimum

Whatlevelofgoalalignmentshouldweseekacrossmembers?


Entry

� � � � � � � �

EntirelyClosed EntirelyOpen

Whatlevelofopennesswillwehavetonewmembers?



44

Leadership

� � � � � � � �

Distributed Top-down

Whatwillourapproachtoleadershipbe?


Focus

� � � � � � � �

Established Evolving

Whatconstancyoffocusdoweaspiretohave?


ValueProposition

� � � � � � � �

Individual Collective

Whatdegreeofdifferenceinmembervaluepropositionsdowewant?



45

NETWORKPROGRESSSCALESANDQUESTIONS

PARTB:NETWORKPROGRESS

Completethescalesandquestionsbelowbasedonthenetworkscurrentstatus.

Size

� � � � � � � �

Hundreds Handful

Howmanymembersdowehave?

Howdoweknowthis?Whatevidencedowehave?

SectorRepresentation

� � � � � � � �

SameSector MixedSector

Whatsectorsarerepresented?


Networkscantakevariousforms.UsethescalesandalignedquestionsinPartAofthissurveytoclarifyand

documentnetworkdesignchoices.UsethescalesandalignedquestionsinPartBtoassessprogressagainstinitial

goals.


46

Geography

� � � � � � � �

City-Based Global

Whatgeographicareadowecover?


MemberType

� � � � � � � �

Systems Individuals

Whattypeofmembersdowehave?


Orientation

� � � � � � � �

Action Learning

Whatorientationdowetake?



47

Centralization

� � � � � � � �

Centralized Decentralized

Whatdegreeofcentralizationwasestablished?


GoalAlignment

� � � � � � � �

Maximum Minimum

Whatlevelofgoalalignmentexistsacrossmembers?


Entry

� � � � � � � �

EntirelyClosed EntirelyOpen

Whatlevelofopennessdowehavetonewmembers?



48

Leadership

� � � � � � � �

Distributed Top-down

Whathasourapproachtoleadershipbeen?


Focus

� � � � � � � �

Established Evolving

Whatconstancyoffocusdidwehave?


ValueProposition

� � � � � � � �

Individual Collective

Whatdegreeofdifferenceinmembervaluepropositionsdowehave?


OVERVIEW

This is a tool for assessing the health of your network, and exploringactions to take to develop or strengthen it. This tool is intended for useby individuals working within or through social change networks.

Instructions:

1. Begin by identifying your network as either:

1. BOUNDED: a network with clear boundaries.The participants are known.

1. UNBOUNDED: a network with fuzzy boundaries.The participants are not all known.

2. Rate your network (high, medium, low) against attributes within eight areas ofnetwork health

3. Step back and jot down notes on your network’s performance in each area ofhealth. Note whether or not this is a priority area for strengthening. Dependingon where your network is at in its lifecycle, different attributes may be atdifferent levels of priority

4. Elicit multiple perspectives on your network’s health. Ask leaders from acrossyour network to take the diagnostic. Compare and aggregate results

5. Next, link your priority areas with actions for strengthening networks. Theactions are by no means prescriptive and do not correlate directly to theattributes within each area of network health. They are meant to spur yourthinking about the range of specific steps you might take to strengthen yournetwork

Sources: This tool was created with inputs from multiple sources – most significantlyresearch done by Monitor Institute for Packard Foundation grantees in 2008-09, and thework of the following network experts: Beth Kanter, June Holley, Marty Kearns, Pete Plastrikand Madeleine Taylor, Clay Shirky, and Jane Wei-Skillern.

Created by the Monitor Institute, www.monitorinstitute.com

Please direct queries about this tool to [email protected]

This work is licensed under the Creative Commons Share Alike 3.0Unported License.

Characteristic Desired Attributes HIGH MED LOW

Notes on OverallPerformance Potential Actions to Strengthen the Network

Value

Clearly articulated give and get forparticipants

• Engage network participants in framing network purpose and goals

• Clearly articulate value the network aspires to deliver to participants

• Regularly test the network value with participants and refine / update asneeded

• Ensure that the network is accountable to the community it seeks to serveDelivers value/outcomes to participants

Network value propositions are aligned andevolve with participant demand

Participation

Participation includes the necessary diversityknowledge, skills and capacity to achievepurpose

• Map the network --to determine who is in the network and how connectedthey are, and identify new participants and strategies for engaging them

• Determine network boundaries – who is in and who is out. Determine howporous these boundaries should be

• Welcome and orient new participants, develop a standard process fordoing so

• Hire a network weaver to bring the right participants into the mix andincrease connectivity throughout the network

• Create workspaces that invite community building and participation—online and in-person

• Identify ‘open triangles’ and close them – identify two people who youknow and who would benefit from knowing each other and introducethem

• Encourage small collaborative projects among just two or three networkparticipants

• Codify a code of conduct, share it broadly, and live by it

New participants can quickly becomeproductive within the network

High voluntary engagement in the network

Participants have a formal or informal codeof conduct and high level of trust with oneanother

Participants regularly interact andcollaborate with one another without goingthrough a central hub

Form

Network has a concept of its structure, how itsuits its purpose, and how it might evolve(e.g. from hub and spoke to multi-hubstructure)

• Map the network in order to visualize structure, diagnose strengths andweaknesses, and identify strategies for growing the network

• Facilitate an open strategic conversation that encourages participationfrom across the network; solicit the ‘wisdom of the crowd’

• Grow the number of people on the periphery of the network and createopportunities for their fresh ideas to flow into the network

• Create an innovation fund – a dedicated resource for cutting edge workthat creates a mandate for risk-taking

Balance of top-down and bottom-upstrategies for doing the work of the network

Network spaces invite self-organized action

1



Leadership

Leaders inspire and help participantsrecognize and work towards common goals

• Identify individuals with strong group process skills and aptitude for IT-enabled collaboration to take on more responsibility

• Develop a system for diversifying and refreshing leadership

• Get out of the way – target opportunities for network participants toconnect and collaborate directly

• Bridge difference. Connect people and ideas that normally don’t gotogether

Leaders seek out opportunities to highlightand bridge difference in service of networkgoals

Leadership is shared. Responsibility andcontrol is spread throughout the network

Leadership is refreshed and renewed toreflect the network as it evolves

Governance

Governance is reflective of diverseconstituencies in the network andtransparent

• Formalize governance system with an eye to identifying opportunities toshare decision-making power

• Create mechanisms for voices from the periphery to influence decision-making

Governance is formalized in a group,committee or board (not a single person)

Governing body rotates its members overtime

Connection

Network is resilient. If some highly connectedparticipants leave, the network remainsstrong

• Align communication tools with what members are comfortable using orcan be trained to use. Don’t assume the network should adopt the latestadvances

• Follow the 1/10/100 rule :1% create content, 10% comment on it, and100% view content

• Allocate time and budget for designing, facilitating and maintaining onlinenetwork communications

• Look to young people to guide your use of social media

• Design shared spaces –online forums and in-person common spaces thatencourage interactions

• Seek out lower cost opportunities for connecting network participants in-person – e.g. host a reception at a commonly attended conference

Ample shared space, online and in-person,allowing participants to easily connect

Network use of social media tools andstrategies are appropriate given participantskills and habits

Network use of social media tools andstrategies are a good fit for types ofinteractions needed to meet the purpose

2



Capacity

Participants know where resources are in thenetwork—knowledge , skills, and capacity

• Create systems to help participants find and share relevant expertise

• Broadcast basic needs to the network and tap excess capacity to fill them

• Surface the talent in the network. Don’t assume external expertise needsto be brought in to address network needsNetwork can identify and prioritize filling

knowledge, skills and capacity gaps

Effective model for financial sustainability inplace

Learning &Adaptation

Network gathers feedback, and captureslearning as stories

• Create mechanisms for regular gathering of feedback from networkparticipants

• Invite key network participants to develop a shared set of metrics

• Create a network map that will serve as a baseline measurement. Map thenetwork again in 1-2 years, or once there has been enough time for changein the network. Compare the two maps to assess change in the system

• Create a mechanism – with dedicated resources – for ongoing capture oflearning and stories throughout the network

Network has an agreed upon desired impactand a common set of metrics to measure thatimpact

Network regularly measures, evaluates, andreflects on its impact to refine its goals andactivities

3



Value

Clearly articulated give and get forparticipants

• Engage network participants in framing network purpose and goals

• Clearly articulate value the network aspires to deliver to participants

• Regularly test the network value with participants and refine / update asneeded

• Ensure that the network is accountable to the community it seeks to serveDelivers value/outcomes to participants

Network value propositions are aligned andevolve with participant demand

Participation

Participation includes the necessary diversityknowledge, skills and capacity to achievepurpose

• Map the network --to determine who is in the network and how connectedthey are, and identify new members and strategies for engaging them

• Determine network boundaries – who is in and who is out. Determine howporous these boundaries should be

• Welcome and orient new participants; develop a standard process fordoing so

• Hire a network weaver to bring the right participants into the mix andincrease connectivity throughout the network

• Create workspaces that invite community building and participation—online and in-person

• Identify ‘open triangles’ and close them – identify two people who youknow and who would benefit from knowing each other and introducethem

• Encourage small collaborative projects among just two or three networkparticipants

• Codify a code of conduct, share it broadly, and live by it

High voluntary engagement in the networkby participants

High voluntary engagement in the network

Participants have a formal or informal codeof conduct and trust one another

Participants regularly interact andcollaborate with one another without goingthrough a central hub

Form

Network has a concept of its structure, how itsuits its purpose, and how it might evolve

• Map the network in order to visualize structure, diagnose strengths andweaknesses, and identify strategies for growing the network

• Facilitate an open strategic conversation that encourages participationfrom across the network; solicit the ‘wisdom of the crowd’

• Grow the number of people on the periphery of the network and createopportunities for their fresh ideas to flow into the network

• Create an innovation fund – a dedicated resource for cutting edge workthat creates a mandate for risk-taking

Balance of top-down and bottom-upstrategies for doing the work of the network

Network spaces invite self-organized action

1



Leadership

Leaders inspire and help participantsrecognize and work towards common goals

• Identify individuals with strong group process skills and aptitude for IT-enabled collaboration to take on more responsibility

• Develop a system for diversifying and refreshing leadership

• Get out of the way – target opportunities for network participants toconnect and collaborate directly

• Bridge difference. Connect people and ideas that normally don’t gotogether

Leaders seek out opportunities to highlightand bridge difference in service of networkgoals

Leadership is shared. Responsibility andcontrol are pushed out to the network

Leadership is refreshed and renewed toreflect the network as it evolves

Governance

Governance is reflective of diverseconstituencies within the network andtransparent

• Formalize governance system with an eye to identifying opportunities toshare decision-making power

• Create mechanisms for voices from the periphery to influence decision-making

Connection

Network is resilient. Connectivity is strongenough throughout that if some highlyconnected participants leave, the networkremains strong

• Align communication tools with what participants are comfortable using orcan be trained to use. Don’t assume the network should adopt the latestadvances

• Follow the 1/10/100 rule :1% create content, 10% comment on it, and100% view content

• Allocate time and budget for designing, facilitating and maintaining onlinenetwork communications

• Look to young people to guide your use of social media

• Design shared spaces –online forums and in-person common spaces thatencourage interactions

• Seek out lower cost opportunities for connecting network participants in-person – e.g. host a reception at a commonly attended conference

Ample well-designed space, online and/or in-person, allowing participants to easilyconnect

Network use of social media supportsobjectives in external communications plan

Network use of social media is embraced andunderstood by network leaders andparticipants

2



Capacity

Network can identify and prioritize fillingknowledge, skills and capacity gaps

• Create systems to help participants find and share relevant expertise

• Broadcast basic needs to the network and tap excess capacity to fill them

• Surface the talent in the network. Don’t assume external expertise needsto be brought in to address network needsEffective model for financial sustainability in

place

Learning &Adaptation

Network gathers feedback, and captureslearning as stories

• Create mechanisms for regular gathering of feedback from networkparticipants

• Invite key network participants to develop a shared set of metrics

• Create a network map that will serve as a baseline measurement. Map thenetwork again in 1-2 years, or once there has been enough time for changein the network. Compare the two maps to assess change in the system

• Create a mechanism – with dedicated resources – for ongoing capture oflearning and stories throughout the network

Network has a clearly articulated desiredimpact and a set of metrics to measure thatimpact

Network regularly measures, evaluates, andreflects on its impact to refine its goals andactivities

3

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

o o o o o

Questionnaire

Instructions This questionnaire asks questions about different aspects of your partnership. It will take about 15 minutes to complete. The questionnaire allows you to express your opinions and provide information about your experiences anonymously. DO NOT WRITE YOUR NAME ANYWHERE ON THE QUESTIONNAIRE and your name will not be attached in any way to the responses you give. By answering the questions, you will help your partnership learn about its strengths and weaknesses and about steps that your partnership can take in order to improve the collaboration process. The answers that people in your partnership give will be used to generate a report for your partnership. Only the people in your partnership will have access to this report. There are no right or wrong answers to the questions. Thoughtful and honest responses will give your partnership the most valuable information. Please answer every question, and please check only one answer per question. To complete the questionnaire:

• Please use a BLUE or BLACK ink pen.

• Be sure to read all the answer choices before marking your answer. • Answer each question by placing a legible check mark or “X” in the box to the left of your

answer, like this:

[ √ ] Extremely well OR [ X ] Extremely well

• Please return the completed questionnaire in a manner that protects your anonymity, as instructed by your coordinator.

CENTER FOR THE ADVANCEMENT OF COLLABORATIVE STRATEGIES IN HEALTH 2

Synergy Please think about the people and organizations that are participants in your partnership.

a. By working together, how well are these partners able to identify new and creative ways to solve problems?

[ ] Extremely well [ ] Very well [ ] Somewhat well [ ] Not so well [ ] Not well at all

b. By working together, how well are these partners able to include the views and priorities

of the people affected by the partnership’s work?


c. By working together, how well are these partners able to develop goals that are widely understood and supported among partners?


d. By working together, how well are these partners able to identify how different services and programs in the community relate to the problems the partnership is trying to address?


e. By working together, how well are these partners able to respond to the needs and

problems of the community?



f. By working together, how well are these partners able to implement strategies that are most likely to work in the community?


g. By working together, how well are these partners able to obtain support from individuals and organizations in the community that can either block the partnership’s plans or help move them forward?


h. By working together, how well are these partners able to carry out comprehensive

activities that connect multiple services, programs, or systems?


i. By working together, how well are these partners able to clearly communicate to people in the community how the partnership’s actions will address problems that are important to them?



Leadership Please think about all of the people who provide either formal or informal leadership in this partnership. Please rate the total effectiveness of your partnership’s leadership in each of the following areas:

a. Taking responsibility for the partnership

[ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know

b. Inspiring or motivating people involved in the partnership


c. Empowering people involved in the partnership


d. Communicating the vision of the partnership


e. Working to develop a common language within the partnership



Please rate the total effectiveness of your partnership’s leadership in:

f. Fostering respect, trust, inclusiveness, and openness in the partnership [ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know

g. Creating an environment where differences of opinion can be voiced


h. Resolving conflict among partners


i. Combining the perspectives, resources, and skills of partners


j. Helping the partnership be creative and look at things differently



Please rate the total effectiveness of your partnership’s leadership in: k. Recruiting diverse people and organizations into the partnership


Efficiency

1. Please choose the statement that best describes how well your partnership uses the partners’ financial resources.

[ ] The partnership makes excellent use of partners’ financial resources. [ ] The partnership makes very good use of partners’ financial resources. [ ] The partnership makes good use of partners’ financial resources. [ ] The partnership makes fair use of partners’ financial resources. [ ] The partnership makes poor use of partners’ financial resources.

2. Please choose the statement that best describes how well your partnership uses the partners’ in-kind resources (e.g., skills, expertise, information, data, connections, influence, space, equipment, goods).

[ ] The partnership makes excellent use of partners’ in-kind resources. [ ] The partnership makes very good use of partners’ in-kind resources. [ ] The partnership makes good use of partners’ in-kind resources. [ ] The partnership makes fair use of partners’ in-kind resources. [ ] The partnership makes poor use of partners’ in-kind resources.

3. Please choose the statement that best describes how well your partnership uses the

partners’ time.

[ ] The partnership makes excellent use of partners’ time. [ ] The partnership makes very good use of partners’ time. [ ] The partnership makes good use of partners’ time. [ ] The partnership makes fair use of partners’ time. [ ] The partnership makes poor use of partners’ time.


Administration and Management We would like you to think about the administrative and management activities in your partnership. Please rate the effectiveness of your partnership in carrying out each of the following activities:

a. Coordinating communication among partners


b. Coordinating communication with people and organizations outside the partnership [ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know

c. Organizing partnership activities, including meetings and projects [ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know

d. Applying for and managing grants and funds


e. Preparing materials that inform partners and help them make timely decisions [ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know


Please rate the effectiveness of your partnership in:

f. Performing secretarial duties [ ] Excellent [ ] Very good [ ] Good [ ] Fair [ ] Poor [ ] Don’t know

g. Providing orientation to new partners as they join the partnership


h. Evaluating the progress and impact of the partnership


i. Minimizing the barriers to participation in the partnership’s meetings and activities

(e.g., by holding them at convenient places and times, and by providing transportation and childcare)



Non-financial Resources A partnership needs non-financial resources in order to work effectively and achieve its goals. For each of the following types of resources, to what extent does your partnership have what it needs to work effectively?

a. Skills and expertise (e.g., leadership, administration, evaluation, law, public policy, cultural competency, training, community organizing) [ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know

b. Data and information (e.g., statistical data, information about community perceptions, values, resources, and politics) [ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know

c. Connections to target populations

[ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know

d. Connections to political decision-makers, government agencies, other

organizations/groups [ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know


For each of the following types of resources, to what extent does your partnership have what it needs to work effectively?

e. Legitimacy and credibility


f. Influence and ability to bring people together for meetings and activities


Financial and Other Capital Resources

A partnership also needs financial and other capital resources in order to work effectively and achieve its goals. For each of the following types of resources, to what extent does your partnership have what it needs to work effectively?

a. Money [ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know

b. Space



For the following type of resources, to what extent does your partnership have what it needs to work effectively?

c. Equipment and goods [ ] All of what it needs [ ] Most of what it needs [ ] Some of what it needs [ ] Almost none of what it needs [ ] None of what it needs [ ] Don’t know

Decision Making

a. How comfortable are you with the way decisions are made in the partnership? [ ] Extremely comfortable [ ] Very comfortable [ ] Somewhat comfortable [ ] A little comfortable [ ] Not at all comfortable

b. How often do you support the decisions made by the partnership?

[ ] All of the time [ ] Most of the time [ ] Some of the time [ ] Almost none of the time [ ] None of the time

c. How often do you feel that you have been left out of the decision making process?

[ ] All of the time [ ] Most of the time [ ] Some of the time [ ] Almost none of the time [ ] None of the time


Benefits of Participation For each of the following benefits, please indicate whether you have or have not received the benefit as a result of participating in the partnership.

a. Enhanced ability to address an important issue [ ] Yes [ ] No

b. Development of new skills [ ] Yes [ ] No

c. Heightened public profile

[ ] Yes [ ] No

d. Increased utilization of my expertise or services

[ ] Yes [ ] No

e. Acquisition of useful knowledge about services, programs, or people in the

community [ ] Yes [ ] No

f. Enhanced ability to affect public policy

[ ] Yes [ ] No

g. Development of valuable relationships

[ ] Yes [ ] No

h. Enhanced ability to meet the needs of my constituency or clients

[ ] Yes [ ] No

i. Ability to have a greater impact than I could have on my own

[ ] Yes [ ] No


As a result of your participation in the partnership, have you experienced the following benefits:

j. Ability to make a contribution to the community [ ] Yes [ ] No

k. Acquisition of additional financial support

[ ] Yes [ ] No

Drawbacks of Participation For each of the following drawbacks, please indicate whether or not you have or have not experienced the drawback as a result of participating in this partnership.

a. Diversion of time and resources away from other priorities or obligations [ ] Yes [ ] No

b. Insufficient influence in partnership activities

[ ] Yes [ ] No

c. Viewed negatively due to association with other partners or the partnership

[ ] Yes [ ] No

d. Frustration or aggravation

[ ] Yes [ ] No

e. Insufficient credit given to me for contributing to the accomplishments of the

partnership [ ] Yes [ ] No

f. Conflict between my job and the partnership’s work

[ ] Yes [ ] No


Comparing Benefits and Drawbacks So far, how have the benefits of participating in this partnership compared to the drawbacks? [ ] Benefits greatly exceed the drawbacks [ ] Benefits exceed the drawbacks [ ] Benefits and drawbacks are about equal [ ] Drawbacks exceed the benefits [ ] Drawbacks greatly exceed the benefits Satisfaction with Participation

a. How satisfied are you with the way the people and organizations in the partnership work together? [ ] Completely satisfied [ ] Mostly satisfied [ ] Somewhat satisfied [ ] A little satisfied [ ] Not at all satisfied

b. How satisfied are you with your influence in the partnership?

[ ] Completely satisfied [ ] Mostly satisfied [ ] Somewhat satisfied [ ] A little satisfied [ ] Not at all satisfied

c. How satisfied are you with your role in the partnership?


d. How satisfied are you with the partnership’s plans for achieving its goals?



e. How satisfied are you with the way the partnership is implementing its plans? [ ] Completely satisfied [ ] Mostly satisfied [ ] Somewhat satisfied [ ] A little satisfied [ ] Not at all satisfied


73

AdditionalResources:MeasurementToolkitforIntermediaryOrganizations. . . . . . .

SAMPLENETWORKFORMQUESTIONS

Who You Are Your Name ___________________________________________________ Your Organization or School (if applicable) _______________________________________________ Today’s Date (MM/DD/YY): _________________ When did you join the network (MM/YY)? __________________ Which of the following groups do you primarily represent? Please choose only one.

☐ Individual School � Regional Education Service Agency

☐ School District ��Technical Assistance Provider

☐ Charter Management Organization ��Funder

☐ State Level Agency� ��Researcher�

Who You Communicate With For each person, check the box that best describes how often you communicated in the last six months (e.g. in writing, over the phone, face-to-face, or in meetings). If you don’t know the person, check the box marked “Don’t Know Person.” If it is you check the box labeled “It’s me.”

Person Not in Last 6 Months

Once in Last 6 Months

Multiple Time in Last 6 Months

Monthly Multiple Times a Months

Weekly Multiple Times a Week

Don’t Know Person

It’s me

Person A ___ ___ ___ ___ ___ ___ ___ ___ ___

Person B ___ ___ ___ ___ ___ ___ ___ ___ ___

… ___ ___ ___ ___ ___ ___ ___ ___ ___

Who You Learn From For each person, check the box that best describes how often they have provided you with information you used to do your work in the last six months (e.g. new idea, a report, contact

Use the following questions as a starting point for designing a network survey to assess the membership and structure (i.e. form, structure, or connectivity) of your network.


74

information, etc.). If you don’t know the person, check the box marked “Don’t Know Person.” If it is you check the box labeled “It’s me.”

Person Not in Last 6 Month

Once in Last 6 Month


Monthly Multiple Times a Month


Don’t Know Person

It’s me

Person A ___ ___ ___ ___ ___ ___ ___ ___ ___

Person B ___ ___ ___ ___ ___ ___ ___ ___ ___

… ___ ___ ___ ___ ___ ___ ___ ___ ___

Who You Collaborate With For each person, check the box that best describes how often you have worked together to do work in the last six months (e.g. plan an event, write a report, conduct an observation). If you don’t know the person, check the box marked “Don’t Know Person.” If it is you check the box labeled “It’s me.”*

Person Not in Last 6 Month

Once in Last 6 Month


Monthly Multiple Times a Month


Don’t Know Person

It’s me

Person A ___ ___ ___ ___ ___ ___ ___ ___ ___

Person B ___ ___ ___ ___ ___ ___ ___ ___ ___

… ___ ___ ___ ___ ___ ___ ___ ___ ___


75

Who You Seek Out For each person, indicate the primary benefit you currently receive from their membership in the network. If you don’t know the person, check the box marked “Don’t Know Person.” If it is you check the box labeled “It’s me.”

Person

Information that helps me act and

capitalize on opportunities

Access to funding or material resources

Access to key decision makers

Problem-solving interactions that push me thinking

Personal support including the

ability to brag or vent

Motivation and a reminder of the

importance of out work

Person A ___ ___ ___ ___ ___ ___

Person B ___ ___ ___ ___ ___ ___

… ___ ___ ___ ___ ___ ___

Who is Missing In the cells below, list up to 12 people you feel are missing from the above lists.

1. 2. 3.

4. 5. 6.

7. 8. 9.

10. 11. 12.

THANK YOU!


76

SAMPLEQUESTIONSFORNETWORKMEMBERS

Membership

1. Approximately how many individuals or organizations are in the network?

2. Who makes up the network? What sectors or roles are represented? • Practitioners? • Policy makers? • Funders? • Researchers

3. In your view, is this membership size and diversity sufficient to accomplish network goals?

• Why or why not? Structure

4. Who do you interact with most within the network? • What types of interactions do you have?

o Information or resources sharing? o Problem-solving support or thought partnering? o Collaboration on projects?

5. Who do you interact with least within the network?

• Why are your interactions with these individuals or groups limited?

6. Do you interact directly with other network members or are interactions typically facilitated and overseen by network leadership? • How do you interact and communicate? • How do leaders facilitate interactions and communications?

7. Which, if any, network members stand out because of their abundance of connections with

other members? • Who are these members connected with? • What interactions do these members have with others?

8. What, if any, connections across the network could be improved? • Are there any members who seem disengaged? • Are there connections that are fading?

The following questions can be used in interviews and focus groups with network members to assess network outcomes. Questions are organized according to the evaluative criteria of membership, structure, operations, and benefits. Questions should be selected and modified to fit a network’s context, design, and purpose.


77

Operations 9. What is working well about the network’s operations?

• Communication systems? • Meetings and events? • Resource sharing? • Leadership? • Collaboration?

10. What is not work well about the network’s operations and should be improved or ended?

• Communication systems? • Meetings and events? • Resource sharing? • Leadership? • Collaboration?

11. Do you believe your participation in this network is positively impacting your day-to-day work? Why or why not? • Is there timely and sufficient alignment between your work and topics explored in the

network? • Have there been any times when your participation in the network has felt burdensome or

disconnected from your interests and needs?

12. Does the network seem to have sufficient resources? • To your knowledge, does the network seem to be financial stable? • Does the network have access to facilities? • Does the network have technology resources and sharing platforms to support its work? • Is the allocation of staff or network members sufficient enough to support network

activities?

13. How effectively is the network led? • Are leaders organized? • Do leaders communicate proactively and transparently? • Are leaders knowledgeable and credible?

Benefits

14. When you first joined, what did you hope to gain by participating in the network? � • Knowledge? • Resources? • Connections? • Support?

15. Is there anyone or anything you hoped to influence through participating in the network?

• Policies? • Organizational structures? • Public perceptions? • Mindsets? • Behaviors?


78

16. When you first joined, what did you want to contribute to the network? � • Knowledge? • Resources? • Thought-partnership?

17. In your view what has the network accomplished or produced that stands out?

• Has the network had any breakthrough moments? • Has the network developed new knowledge, resources, or tools? • Has the network effectively addressed barriers to its work or developed new supports for its

work?

18. What do you think has changed about your work as a result of your participation in the network? • Changes to your knowledge, mindsets, or behaviors?

19. To what extent have you been able to influence the people and things you hoped to influence?

• Policies? Structures? Public perceptions? Mindsets? Behaviors?

20. How have you been able to contribute to the network? • Knowledge? Resources? Thought-partnership?

21. Do your contributions meet your expectation?

• Why do you think this is?

22. Do the network’s results meet your expectations? a. Why do you think this is?


79

Diffusion Levers Evaluation Toolkit: System Cultivating

TheDiffusionIntermediaryEvaluationFrameworkincludestwocriteriaforassessingshort-termoutcomesstemmingfromSystemCultivatingactivities:SystemPolicies&PracticesandSystemEngagement&PublicWill.Thesecriteriaarebasedonascanofthesystemthinkingandsystemchangeliteraturelistedattheendofthissection.



SystemPolicies&Practices

Towhatextentarethe

formalandinformal

policiesandpracticesthat

impactpartnersenabling

ordemandinginnovation?

• Legislationthatdemandsinnovation• Humanresourcessystemsandstructuresthatenableinnovation• Accountabilitysystemsandstructuresthatenableinnovation• Professionalstandardsthataligntoinnovation• Organization-widestandardoperatingproceduresthatalignto

innovation

SystemEngagement&PublicWill

Towhatextentisthe

publicdemandingand

supportingchange?

• Frequencyofpositivemediacoverage• Sizeandfrequencyofpublicgatheringstodemonstratesupportfor

innovation• Dollarsofpublicandprivatefundingdistributedtotheregionfor

innovation• Fundingandinfrastructureconditionsthatenableinnovation

Tools

BellwetherSurveySparkPolicyInstitute

Abellwethersurveyisacommonmethodfortrackingpoliticalwill.Itentailstalkingwitha“bellwether”—aninfluentialandpoliticallyinformedindividual—todeterminehowlikelyitisthatapoliticalissuewillbeactedonbasedonhowkeydecisionmakersarethinkingandtalkingabouttheissueandwhereitsitsinthepoliticalagenda.Asimilarprocesscanbeusedtoassesswillwithinthepublic,funders,researchers,orotherkeysystemactors.

ü SystemPolicies&Practices

üSystemEngagement&PublicWill

Coalition Building Self-Assessment AGuidetoMeasuringAdvocacyandPolicy

Thisself-assessmentcanbecustomizedtoanintermediary’stargetedsystem-levelstakeholdergroups.Itallowsanintermediaryorganizationtodeterminetheextenttowhichitmetitsgoalsaroundbuildingchampions,orindividualswhovocallysupporttheintermediary’svisionandstrategyanddevotetimeandresourcestoengagingothersinit.

SystemPolicies&Practices


MediaTrackingandAnalysisOverviewCenterforPublicResearchandLeadership

Thisdocumentprovidesabriefoverviewofmediatracking.Mediatracking SystemPolicies&Practices


80

monitorshowmediacoveragerelatedtoakeytopicevolvesovertime.Theprocessinvolvesusinganews-trackingservicelikeLexisNexistosystematicallygathercontentandthenreviewingthiscontentforpatternsinframing,frequency,lengths,etc.


PolicymakerRatingTemplateSparkPolicyInstitute

Policymakerratingisasystematicprocessofgaugingpolicymakersupportforspecificissues.Itinvolvespolicyadvocatesrating(1)apolicymaker’slevelofsupportforanissue,(2)thepolicymakerlevelofinfluencesonthepolicy,and(3)advocateslevelofconfidenceintheaccuracyofthefirsttworating.

ü SystemPolicies&Practices

SystemEngagement&PublicWill

Resources BridgespanGroup.(2009).Thestrongfieldframework:Aguideandtoolkitforfundersandnonprofitscommittedto

large-scaleimpact.Retrievedfromhttps://irvine-dot-org.s3.amazonaws.com/documents/64/attachments/strongfieldframework.pdf?1412656138

Coffman,J.(2009).Auser’sguidetoadvocacyevaluationplanning.HarvardFamilyResearchProject.Retrievedfromhttp://www.hfrp.org/.

Coffman,J.&Reed,E.Uniquemethodsinadvocacyevaluation.Retrievedfromhttp://www.pointk.org/resources/files/Unique_Methods_Brief.pdf.

Forti,M.(2012).Measuringadvocacy-Yeswecan!StanfordSocialInnovationReview.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZbU5zLWFVX0hzb0E/view.

Hargreaves,M.(2010).Evaluatingsystemchange:Aplanningguide.MathematicaPolicyResearch,Inc.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZRVYtODJld0drX28/view?pref=2&pli=1.

HarvardFamilyResearchProject(2007).Advocacyandpolicychange.TheEvaluationExchange.13(1):1-32.

Latham,N.(2014).Apracticalguidetoevaluatingsystemschangeinahumanservicessystemcontext.LearningforAction.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4Zb19GOEE0Y1Q3SGM/view?pref=2&pli=1

Meadows,D.(2008).Thinkinginsystems:Aprimer.WhiteRiverJunction,VT:ChelseaGreenPublishing.

Preskill,H.,Gopal,S.,Mack,K.,&Cook,J.(2015).Evaluatingcomplexity:Propositionsforimprovingpractice.FSG.Retrievedfromhttp://www.issuelab.org/resource/evaluating_complexity_propositions_for_improving_practice

Reisman,J.,Gienapp,A.,&Stachowiak,S.(2007).Aguidetomeasuringadvocacyandpolicy.TheAnnieE.CaseyFoundation.Retrievedfromhttps://drive.google.com/file/d/0BxI97tx1dL4ZX09HTktUYld4MjQ/view.

Stroh,D.(2015).Systemsthinkingforsocialchange:Apracticalguidetosolvingcomplexproblems,avoidingunintended

consequences,andachievinglastingresults.WhiteRiverJunction,VT:ChelseaGreenPublishing.


81

BELLWETHERSURVEYQUESTIONSSources:SparkPolicyInstitute

IntroductionThankyoufortakingthetimetotalktome.LetmestartbygivingyouabriefoverviewofwhyIwantedtotalktoyoutoday.Iamgatheringinformationfromkeypeopleofinfluenceaboutthepolicylandscapeandprioritiesinourstate.TodayI’mgoingtoaskyouaboutourstate’spolicylandscapeingeneral,andthensomemoredetailedquestionsaboutspecificpolicies.Weareinterestedinyouropinionsandreactionstoourquestions;ofcoursetherearenorightorwronganswers.

Questions1. Currently,whatthreeissuesdoyouthinkareatthetopofthe[state/federal/local]policyagenda?�2. Howfamiliarareyouwith[thepolicyofinterest]?�3. Whatindividuals,constituencies,orgroupsdoyouseeasthemainadvocatesfor[thepolicy]?Whodoyouseeas

themainopponents?�4. Consideringthecurrenteducational,social,andpoliticalcontext,doyouthink[thepolicy]shouldbeadoptednowor

inthenearfuture?�5. Lookingahead,howlikelydoyouthinkitisthat[thepolicy]willbeadoptedinthenext5years?�6. Currently,whatindividuals,constituencies,orgroupsdoyouseeatthemainadvocatesfor(yourpolicyissue)?Who

doyouseeasthemainopponents?7. If[thepolicy]isadopted,whatissuesdoyouthinkthestateneedstobemostconcernedaboutrelatedtoits

implementation?�

ClosingThankyoufortakingtimetoanswermyquestionsonpolicyissuesandhealthcarepolicyspecifically.Youranswerswillbesummarizedwithresponsesfromotherleadersinourstate.Yourindividualresponseswillnotbesharedwithanyoneoutsideourevaluationteam.

Abellwethersurveyisacommonmethodfortrackingpoliticalwill.Itentailstalkingwitha“bellwether”—aninfluentialandpoliticallyinformedindividual—aboutthepoliticalenvironmentandthestandingofspecificpolicies.Thefollowingquestionscanbecustomizedandusedtocompletebellwethersurveys.


82

COALITIONBUILDINGSELF-ASSESSMENTModifiedfromAGuidetoMeasuringAdvocacyandPolicy,TheAnnieE.CaseyFoundation(2007)

Powerfulfriends

aboundandare

willingandfrequent

spokespersonsforour

missionusing

political,social,or

economiccapitalto

gainsupportfrom

others

Somenewchampions

havebeenidentified

andaretakingsome

publicstepsto

supportourmission,

butitisnotgenerally

atthetopoftheir

agendas

Limitedchampions

withothersnot

seeinganyrelevance

ofourmissiontotheir

workorviewingusas

apossiblecompeting

demandonresources

Substantialprogress

hasbeenmadein

thelast6months

(checkforyes)

[StakeholderGroup-E.g.schoolsuperintendents,policymakers,funders]

1 2 3 4 5 *

Evidence:


1 2 3 4 5 *

Evidence:


Thisself-assessmentcanbecustomizedtoanintermediary’stargetedsystem-levelstakeholdergroups.Foreachofthestakeholdergroups,markanumberfrom1-5toindicatetheextenttowhichpowerfulconnectionshavebeenestablishedwithindividualsorgroupsmakingupthecategoryofstakeholders.Ifdesired,alsocheckthoseareaswheretheintermediaryhasmadesubstantialprogressinthelastsixmonths.


83

1 2 3 4 5 *

Evidence:


1 2 3 4 5 *

Evidence:


84

MEDIATRACKINGANDANALYSISOVERVIEW

Tracking

Useanews-trackingservicelikeLexisNexusorGoogleNewsAlertssystematicallygathercontent.Carefullyselectthekeywordstouseinthesearchaswellasthetypesofpublicationsthesearchwillpullfrom.Theseshouldremainconsistentthroughouttheprocesses.

Title Author Source Date Length TopicIsthemessageone

wewantedtoconvey?

Istheinformationaccurate?

Whoismentioned

inthearticle?

Analysis

Atregularintervalsreviewthedatabaseusingthequestionsbelowassampleprompts.

• Whatchangesinthefrequencyofrelevantmediacoveredhaveweseenoverthelast6months?o Iscoveredincreasing?Decreasing?Stable?o Whereisfrequencychanging?Whereisitstable?

• Whatchangesinthecontentofrelevantmediacoveredhaveweseenoverthelast6months?o Towhatextentiscoveragealignedwithourorganizationalvisionandstrategy?o Towhatextentiscoveragegainingnuanceandcredibility?o Towhatextentiscoveragehighlightingkeyplayersinthefield?

• Whatrelationshipbetweenfrequencyandcontenthaveweseenoverthelast6months?• Whyareweseeingthesepatterns?• Whatstepscanwetaketodisruptoracceleratethesepatterns?

Thisdocumentprovidesabriefoverviewofmediatracking.Mediatrackingmonitorshowmediacoveragerelatedtoakeytopicevolvesovertime.Theprocessinvolvesusinganews-trackingservicelikeLexisNexistosystematicallygathercontentandthenreviewingthiscontentforpatternsinframing,frequency,lengths,etc.


85

POLICYMAKERRATINGTEMPLATESource:SparkPolicyInstitute

Issue (Onesentenceorshortphrasethatclearlyarticulatesyourissue)PolicymakerName RaterName

Scale Rating DefinitionSupport(Supportfortheissue) �Notatall

supportiveorinopposition.

Noevidencethispersonhastakenaction,spokenabout,orotherwisedirectlysupportedthispolicyissue.OR,evidencethispersonopposestheissue.

�Notverysupportive

Thispersonhasverballyexpressedsomesupport,butprimarilyinone-on-oneconversationsandsmallgroupmeetings.

�Supportive

Thispersondemonstratessupportthroughactionssuchas:voting,speakinginpublic,quotedinthemedia,encouragingotherstosupporttheissue,helpingnegotiation/supportbills.

�Extremelysupportive

Thispersonisknownasachampionfortheissue,playsaleadershiproleinadvancingtheissue,andconsistentlymakestheissueapriorityontheiragenda.

Influence(extenttowhichpolicymakermeetsthefollowingcriteria)

• Majoritypartymember• Relevantcontentexpertise• Seniority/experience• Reputation/respect• Keycommitteemember• Formalleadershipposition

�Notatallinfluential Meetsnocriteriaoronecriteria.

�Somewhatinfluential Meetsatleasttwocriteria.

�Influential Meetingsthreeorfourcriteria.

�Extremelyinfluential Meetingsfiveorsixcriteria.

Confidence(yourconfidencelevelinyourrating)

�Notveryconfident

Ratingsbasedonthirdhandinformation.Notverifiable.

�Somewhatconfident

Ratingsbasedonconsistentinformationfromoneormoresource,butnot100%verifiable.

�Confident Ratingsbasedondirectcontactwiththepolicymakerorinformationfromahighlytrusted,verifiablesource.

Thissampleversioncanbeeasilyadaptedformanytopics.Primarily,youwillwanttoupdateittomatchyoursettinginthefollowingways:

1. Defineyourissueandputitinthefirstrow.2. Makesurethatthedefinitionsofsupportareappropriatetothesettingsofyourpolicymakers(e.g.ifthey

arenotlegislators,youmaynotwanttokeeplegislationasevidenceoftheirsupport);3. Updatethecriteriaforinfluencetomatchyourpolicymakers’setting.

ADDITIONALRESOURCES

Phrased Rubrics for Assessing Intermediary Organizations AppendixtoIntermediaryOrganizationsandEducationInnovation



AlignedtotheDiffusionIntermediaryEvaluationFramework

87

Phase I: Potential for Influence

ReadinessofLocalContext

CRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE

SupportforInnovation

Towhatextentisthelocalcontextopentoandsupportiveofthechangesandworkproposedbytheintermediary?

• Theexistingpolicyenvironmentisprohibitiveoftheideasandpracticestheintermediaryseekstoachieve,andfew,ifany,explicitavenuestoalterthesepoliciesexist.

• Thelocalityhasnofinancial,

informational,orpeersupportavailableforexperimentationorentrepreneurialismwithintheeducationsector.

• Thepublicisrisk-averse,

hasdeepmisconceptionsorproteststotheneweducationalideasandpracticesbeingproposed,and/orhasahistoryofprotestingeducationalreformandinnovation.

• Theexistingpolicyenvironmentispermissiveoftheideasandpracticestheintermediaryseekstoachieveandsomeisolatedpoliticalavenuesexisttosupportinnovation.

• Thelocalityhassome

formalorinformalsupportforexperimentationorentrepreneurialismwithintheeducationsectoravailableintheformofsomefunding,peertopeercollaborationstructures,and/orinformationalresources.

• Thepublicislargely

disengagedfromdiscussionsofeducationinnovationanddemonstrateneitherprotestnorsupport.

• Theexistingpolicyenvironmentpromotestheideasandpracticestheintermediaryseekstoachieveandpoliticalavenuesexisttosupportinnovation.

• Thelocalityhasvarious

formalorinformalsupportsforexperimentationorentrepreneurialismavailablewithintheeducationsectorintheformoffunding,peertopeercollaborationstructures,and/orinformationalresources.

• Thepublicdemonstratessomeengagementwithandsupportforeducationinnovation.

• Theexistingpolicyenvironmentalreadyrequirestheideasandpracticestheintermediaryseekstoachieveinpartorintotal.

• Thelocalityhasextensive

formalorinformalsupportforexperimentationorentrepreneurialismwithintheeducationsectoravailableintheformoffunding,peertopeercollaborationstructures,and/orinformationalresources.

• Thepublicdemonstrates

deepengagementwithandademandforeducationinnovation.



88

FieldNeedTowhatextentisthelocal

educationsystemdemonstratinganeedfor

improvement?

Towhatextentdoesthelocaleducationsystem

needadditionalsupportofthekindanintermediary

canprovide?

• Theeducationsystemdemonstratesnocurrentneedforimprovement—studentachievementratesarewellabovethenationalaverageandequalacrossdifferentracial,ethnic,andsocioeconomicgroups.

• Theeducationsectorwithinthelocalcontexthasnoneedforfurthersupportofthekindtheintermediaryprovides.

• Theeducationsystemdemonstratesaslightneedforimprovement—studentachievementratesareaverageatbestand/orgapsexistacrossdifferentracial,ethnic,andsocioeconomicgroupssimilartothoseseennationally.

• Theeducationsectorwithinthelocalcontextalreadyhasextensivesupportssimilartothoseofferedbytheintermediary.

• Theeducationsystemdemonstratesamoderateneedforimprovement—studentachievementratesarebelownationalaveragesand/orgapsexistacrossdifferentracial,ethnic,andsocioeconomicgroupsthatarehigherthantothoseseennationally.

• Theeducationsectorwithinthelocalcontexthasfewsupportssimilartothoseofferedbytheintermediary.

• Theeducationsystemdemonstratesanurgentneedforimprovement—studentachievementratestrailfarbehindnationalaveragesand/orgapsexistacrossdifferentracial,ethnic,andsocioeconomicgroupsthatareamongthehighestinthenation.

• Theeducationsectorwithinthelocalcontexthasnosupportssimilartothoseofferedbytheintermediary.

Vision&StrategyCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE

TransformativeVisionTowhatextentdoestheintermediary’svisionfor

changebreakfromtraditionaleducationalpracticesandsystems?

• Theintermediary’sdesiredimpactisunstated,incoherent,and/ordoesnotincludemeaningfullydefininginstructionortheinteractionsthatoccurbetweenstudentandteacher.

• Theintermediary’sdesiredimpactisslightlyunclearand/oronlyminimallyredefinesinstructionortheinteractionsthatoccurbetweenstudentandteacher.

• Theintermediary’sdesiredimpactisforthemostpartclearandredefinesinstructionortheinteractionsthatoccurbetweenstudentandteacherinnotableways.

• Theintermediary’sdesiredimpactisclearandsubstantiallyredefinesinstructionortheinteractionsthatoccurbetweenstudentandteacher.

ViabilityofStudentImpact

Howrobustistheorganization’sbasis(in

evidenceand/orlogic)forpredictingthatitsvisionwillpositivelyimpact

students?

• Thereisnoevidenceoforlogicalexplanationforarelationshipbetweenthenewpracticesproposedbytheintermediaryandimprovedlevelsofstudentsuccess.

• Thereisearlyorscatteredevidenceofarelationshipbetweenthenewpracticesproposedbytheintermediaryandimprovedlevelsofstudentsuccessorthereisanearlyhypothesisforwhyimprovedoutcomescanbeexpected.

• Thereissubstantial,thoughnotconclusive,evidenceofarelationshipbetweenthenewpracticesproposedbytheintermediaryandimprovedlevelsofstudentsuccessorthereisacredibleresearch-basedhypothesisforwhyimprovedoutcomescanbeexpected.

• Thereisconclusiveevidenceofarelationshipbetweenthenewpracticesproposedbytheintermediaryandimprovedlevelsofstudentsuccessorthereisaresearch-backedhypothesisforwhyimprovedoutcomescanbeexpected.



89

InformedStrategyTowhatextentdoestheintermediary’sstrategyincorporatethekeyDiffusionLeversofCapacityBuilding,

NetworkWeaving,andSystemCultivating?

• ThestrategydoesnotincludeactivitiesthatemploythekeyDiffusionLevers(i.e.,CapacityBuilding,NetworkWeaving,andSystemCultivating).

• ThestrategyincludesfewactivitiesthatemploythekeyDiffusionLevers(i.e.,CapacityBuilding,NetworkWeaving,andSystemCultivating)oractivitiestodemonstratelittleplanningregardingwhichLeverstoemployandtowhatdegree.

• ThestrategyincludesseveralactivitiesthatemploythekeyDiffusionLevers(i.e.,CapacityBuilding,NetworkWeaving,andSystemCultivating)andreflectssomeconsiderationofthehowtheLeverswillintersect.

• ThestrategyincludesanarrayofactivitiesthatemployallthekeyDiffusionLevers(i.e.,CapacityBuilding,NetworkWeaving,andSystemCultivating)andreflectscarefulconsiderationofhowtheLeverswillintersect.

OperationalAlignmenttoStrategy

Towhatextentaretheintermediary’sday-to-dayactionsandprogramslinkedtoitsstrategy?

• Coreprogramsandserviceslackalignmenttotheintermediary’sarticulatedstrategyand/orstaffmember’sday-to-daytasksdonotsupportcoreprogramsandservices.

• Someprogramsandservicesmapbacktotheintermediary’sarticulatedstrategywhileothersdonot,and/orsomeday-to-daystafftasksaligntotheseprogramsandserviceswhileothersdonot.

• Mostprogramsandservicesmapbacktotheorganization’sarticulatedstrategy,andday-to-daystafftaskslargelyaligntotheseprogramsandservices,minimizingwastedtimeandresources.

• Thereistightalignmentbetweenprogramsandservices,andtheorganization’sarticulatedstrategyandday-to-daystafftasksaligntotheseprogramsandservices,eliminatingwastedtimeandresources.

OrganizationalCapacityCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE

InternalResourcesTowhatextentdoesthe

intermediaryhavesufficientfinancialandhumanresourcestosupportitsstrategy?

• Theintermediarycurrentlydoesnothavesufficientresourcestoimplementtheactivitiesthatmakeupitsarticulatedstrategyandhasnoplanforattainingtheseresources.

• Theintermediaryisapproachingsufficientresourcestoimplementtheactivitiesthatmakeupitsarticulatedstrategyorhasacoherentplanforattainingtheseresources.

• Theintermediarycurrentlyhassufficientresourcestoimplementtheactivitiesthatmakeupitsarticulatedstrategy.

• Theintermediarycurrentlyhasmorethansufficientresourcestoimplementtheactivitiesthatmakeupitsarticulatedstrategy.

ConnectednessTowhatextentdoesthe

intermediaryhaverelationshipswithfield

(e.g.,K-12education)andsystempartners(e.g.,policymakesr,funders,technicalassistance

providers)acrossthelocalcontext?

• Theintermediaryhasnotbuiltrelationshipswithkeyfieldorsystempartnersandisisolated.

• Theintermediaryhasbuiltrelationshipswithsomekeyfieldand/orsystempartnersandisbecomingpartofanetworkoflike-mindedorganizations.

• Theintermediaryhasbuiltrelationshipswithmanykeyfieldand/orsystempartnersandispartofanetworkoflike-mindedorganizations.

• Theintermediaryhasbuiltrelationshipswithanextensivenumberofkeyfieldand/orsystempartnersandisawellknowplayerinanetworkoflike-mindedorganizations.



90

LearningOrientationTowhatextentdoestheintermediaryregularlyreviewandreflectonitssuccessesandfailures,changesininternal

organizationalcapacity,andexternalcontextualconditionsinorderto

learnandmakemidcoursecorrections?

• Theintermediaryisnotpreparedtocollectandadapttoevidenceofitsownsuccessesorfailures,oradapttochangesinorganizationalcapacityorcontextualconditions.

• Theintermediaryissomewhatpreparedtocollectandadapttoevidenceofitsownsuccessesorfailures,oradapttochangesinorganizationalcapacityorcontextualconditions.

• Theintermediaryismostlypreparedtocollectandadapttoevidenceofitsownsuccessesorfailures,oradapttochangesinorganizationalcapacityorcontextualconditions.

• Theintermediaryisfullypreparedtocollectandadapttoevidenceofitsownsuccessesorfailures,oradapttochangesinorganizationalcapacityorcontextualconditions.

LeadershipTowhatextentdoestheintermediary’sleadershipexhibitcharacteristics(e.g.,vision,strategicplanning,inspiration,

creativity,supportofstaff,alearningstance)thatallowthemtoeffectivelyandefficientlymanage

theorganization?

• Thereisnoevidenceoftheintermediary’sleadershiphavingthecharacteristicsneededtoefficientlyandeffectivelymanagetheorganization.

• Thereissomeevidenceoftheintermediary’sleadershiphavingthecharacteristicsneededtoefficientlyandeffectivelymanagetheorganization.

• Thereissomeevidenceoftheintermediary’sleadershiphavingthecharacteristicsneededtoefficientlyandeffectivelymanagetheorganization.

• Thereisextensiveevidenceoftheintermediary’sleadershiphavingthecharacteristicsneededtoefficientlyandeffectivelymanagetheorganization.

FieldInfluenceTowhatextentisthe

organizationconsideredaninfluentialleaderinthe

K-12educationfield?

• TheintermediaryisnotyetwellregardingintheK-12educationfield,anditsworkdoesnotyetserveasamodelforthefield.

TheintermediaryissomewhatwellregardedintheK-12educationfield,anditsworkservesasamodelforsomeinthefield.

TheintermediaryiswellregardedintheK-12educationfield,anditsworkservesasamodelformanyinthefield.

TheintermediaryisextremelywellregardedintheK-12educationfield,anditsworkconsistentlyservesasamodelforthefield.



91

Phase II: Interim Progress

2.1ImplementationofStrategyCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE

ExtentofActionsTakenHowmanyresources

and/orhowmuchsupportdidtheintermediary

provideovertheperiod?

• Theintermediaryhasnotmetitstargetsfortheamountofresourcesandsupportsitintendedtoprovideovertheperiod.

• Theintermediaryisapproachingitstargetsfortheamountofresourcesandsupportsitintendedtoprovideovertheperiod.

• Theintermediaryachieveditstargetsfortheamountofresourcesandsupportsitintendedtoprovideovertheperiod.

• Theintermediaryexceededtargetsfortheamountofresourcesandsupportsitintendedtoprovideovertheperiod.

QualityofActionsTakenTowhatextentweretheresourcesandsupportsof

highquality?

• Theintermediary’sresourcesandsupportsdidnotmeetstandardsofquality.

• Theintermediary’sresourcesandsupportsareapproachingstandardsofquality.

• Theintermediary’sresourcesandsupportsmetstandardsofquality.

• Theintermediary’sresourcesandsupportsexceededstandardsofquality.

2.2Short-termOutcomesCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE

PartnerUnderstandingTowhatextentareindividualpartnersexpandingtheirunderstandingof

innovation?

• Theintermediary’sindividualpartnershavenotyetincreasedtheirunderstandingofkeyknowledgeandskillsrelatedtotheinnovation.

• Theintermediary’sindividualpartnershavebeguntoincreasetheirunderstandingofkeyknowledgeandskillsrelatedtotheinnovation.

• Theintermediary’sindividualpartnershavesubstantiallyincreasedtheirunderstandingofkeyknowledgeandskillsrelatedtotheinnovation.

• Theintermediary’sindividualpartnershavegainedextensiveexpertiseinkeyknowledgeandskillsrelatedtotheinnovation.

PartnerPerceptionsTowhatextentareindividualpartners

developingpositiveviewstowardtheinnovation?

• Theintermediary’sindividualpartnershavenotyetimprovedtheirviewstowardtheinnovation.

• Theintermediary’sindividualpartnershavebeguntoimprovetheirviewstowardtheinnovation.

• Theintermediary’sindividualpartnershavesubstantiallyimprovedtheirviewstowardtheinnovation.

• Theintermediary’sindividualpartnershavebecomeengagedandactiveproponentsoftheinnovation.



92

NetworkFormWhatindividualsor

organizationsmakeupthenetwork?

Howcantheconnections

betweennetworkmembersbecharacterized?

• Thenetworkhasnotreacheditstargetspertainingtomembershipsizeandcomposition.

• Therearefewifany

connectionsbetweennetworkmemberssofarandconnectionsthatdoexistarenotproducingaddedvalueformembersorforthefieldatlarge.

• Thenetworkhasnearlyreacheditstargetspertainingtomembershipsizeandcomposition.

• Therearesome

connectionsbetweennetworkmembersandtheseconnectionsarebeginningtoproduceaddedvalueformembersorforthefieldatlarge.

• Thenetworkhasreacheditstargetspertainingtomembershipsizeandcomposition.

• Therearesubstantial

connectionsbetweennetworkmembersandtheseconnectionsareproducingaddedvalueformanynetworkmembersorforthefieldatlarge.

• Thenetworkhassurpassedittargetspertainingtomembershipsizeandcomposition.

• Therearesubstantial

connectionsbetweennetworkmembersandtheseconnectionsareproducingaddedvalueforallornearlyallnetworkmembersaswellasforthefieldatlarge.

NetworkFunctionTowhatextentdoesthe

networkhavetheresources,structures,andconditionsinplaceneededtooperateeffectivelyand

efficiently?

• Thenetworkdoesnotyethavetheinfrastructure,operationalsystems,orconditionsneededtofunctioneffectivelyandefficiently.

• Thenetworkhassomeoftheinfrastructure,operationalsystems,andconditionsneededtofunctioneffectivelyandefficiently.

• Thenetworkhasmostoftheinfrastructure,operationalsystems,andconditionsneededtofunctioneffectivelyandefficiently.

• Thenetworkhastheinfrastructure,operationalsystems,andconditionsneededtofunctioneffectivelyandefficiently.

SystemPoliciesandPractices

Towhatextentaretheformalandinformal

policiesandpracticesthatimpactpartnersenablingordemandinginnovation?

• Formalandinformalpoliciesandpracticesthatimpactpartnersareprohibitiveofinnovation.

• Formalandinformalpoliciesandpracticesthatimpactpartnersallowforbutdonotpromoteinnovation.

• Formalandinformalpoliciesandpracticesthatimpactpartnerspromoteinnovation.

• Formalandinformalpoliciesandpracticesthatimpactpartnersrequireinnovation.

SystemEngagementandPublicWill

Towhatextentisthepublicdemandingandsupportingchange?

• Thepublicisrisk-averseandactivelyprotestsinnovation.

• Thepublicislargelydisengagedfrominnovation,butdoesnotposeanaddedbarriertothework.

• Thepublicisbeginningtocallforandsupport,eitherverballyorfinancially,innovation.

• Thepublicisdemandinginnovationandisactivelysupportingitverballyorfinancially.



93

2.3ContinuousLearningCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCE

UnderstandingofProgress

Towhatextenthastheintermediaryidentifiedareasofsuccessand

failure?

Whatlessonslearnedhastheintermediarypulledfromitssuccessesand

failuresthatcouldbenefititsownworkorthelarger

field?

• Theintermediaryisnotcollectingevidenceofitsownsuccessorfailure.

• Thegranteehasnotpulledhelpfullessonslearnedfromitsprogressthatitcanrespondtointernallyorthatactorsinthefieldcanlearnfrom.

• Theintermediaryiscollectinglimitedevidenceofitsownsuccessorfailure.

• Thegranteehaspulledsomehelpfullessonslearnedfromitsprogressthatitcanrespondtointernallyorthatactorsinthefieldcanlearnfrom.

• Theintermediaryiscollectingavarietyofevidenceofitsownsuccessorfailure.

• Thegranteehaspulledvarioushelpfullessonslearnedfromitsprogressthatitcanrespondtointernallyorthatactorsinthefieldcanlearnfrom.

• Theintermediaryiscollectingcomprehensiveevidenceofitsownsuccessorfailure.

• Thegranteehaspulledan

extensivenumberofhelpfullessonslearnedfromitsprogressthatitcanrespondtointernallyorthatactorsinthefieldcanlearnfrom.

MidcourseCorrectionsTowhatextentisthe

intermediaryadjustingitsstrategyororganizationalcapacityasaresultof

lessonslearned?

• Theintermediaryhasnotadapteditsvisionandstrategyororganizationalcapacityasaresultoflessonslearned.

• Theintermediaryhasmadesomeinitialchangestoitsvisionandstrategyororganizationalcapacityasaresultoflessonslearnedbuthasnotfullyrespondedtotheselessons.

• Theintermediaryhasmadesubstantialchangestoitsvisionandstrategyororganizationalcapacityasaresultoflessonslearnedbuthasnotfullyrespondedtotheselessons.

• Theintermediaryhasmadesubstantialchangestoitsvisionandstrategyororganizationalcapacityasaresultoflessonslearneddemonstratingafullandcompleteresponsetotheselessons.



94

Phase III: Impacts on the Field

3.1DiffusionofInnovationCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCEDepths

Towhatextentarechangestopracticedeepandconsequential,thusreflectingnewvaluesand

beliefs?

• Changestoinstructionreflectnoshiftsineducators’viewsandbeliefsabouthowstudentslearn,theroleofteacherandstudent,andsubjectcontent.

• Changestoinstructionarebeginningtoreflectsomefundamentalshiftsineducators’viewsandbeliefsabouthowstudentslearn,theroleofteacherandstudent,andsubjectcontentthatsomewhatalignstoinnovation.

• Changestoinstructionreflectsubstantialshiftsineducators’viewsandbeliefsabouthowstudentslearn,theroleofteacherandstudent,andsubjectcontentthatmostlyalignstoinnovation.

• Changestoinstructionreflectacompleteshiftineducators’viewsandbeliefsabouthowstudentslearn,theroleofteacherandstudent,andsubjectcontentthatalignstoinnovation.

SpreadTowhatextentare

changestoinstructionalpracticeexpanding

outwardstomoreandmoreclassrooms,schools,

and/ordistricts?

Towhatextentarechangestoinstructional

practiceexpandinginwardsandinfluencingclassroom,school,andpoliciesandoperating

procedures?

• Innovationhasnotyetspreadoutwardtoevenasmallnumberofclassrooms,schools,and/ordistrictsinthetargetedregion.

• Innovationhasnotyet

spreadinwardstochangeclassroom,school,ordistrictstandardoperatingproceduresorareasnotoriginallytargetedforchange.

• Innovationisbeginningtospreadoutwardtoanumberofclassrooms,schools,and/ordistrictsinthetargetedregion.

• Innovationisbeginningtospreadinwardstochangeclassroom,school,ordistrictstandardoperatingproceduresorareasnotoriginallytargetedforchange.

• Innovationhasspreadoutwardtoasubstantialnumberofclassrooms,schools,and/ordistrictsinthetargetedregion.

• Innovationhasspreadinwardstosubstantiallychangeclassroom,school,ordistrictstandardoperatingproceduresorareasnotoriginallytargetedforchange.

• Innovationhasspreadoutwardtothevastmajorityofclassrooms,schools,and/ordistrictsinthetargetedregion.

• Innovationhasspreadinwardsandcompletelychangedclassroom,school,ordistrictstandardoperatingproceduresorareasnotoriginallytargetedforchange.

OwnershipTowhatextentis

authorityforthereformbeingtakenonbythedistricts,schools,and

teachers?

• Teachers,schools,ordistrictshavenotyettakenonauthorityformanagingandexpandinginnovationandexternalpartiesremainthedrivingforce.

• Teachers,schools,ordistrictsarebeginningtotakeonauthorityformanagingandexpandinginnovationbutexternalpartiesremainadrivingforce.

• Teachers,schools,ordistrictshavetakenonsubstantialauthorityformanagingandexpandinginnovationandexternalpartiesarenolongerthedrivingforce.

• Teachers,schools,ordistrictshavetakenoncompleteauthorityformanagingandexpandinginnovationandexternalpartiesarenolongeradrivingforce.



95

SustainabilityTowhatextentdochangestopracticeremaininplaceafter

externalsupportsarenolongerpresent?

• Innovationhasnotremainedinplacesincedepartureofexternalsupports;instructionalmodelandstandardoperatingproceduresareshiftingbacktotheirpreviousstate.

• Innovationhasremainedsomewhatinplacesincedepartureofexternalsupports;instructionalmodelandstandardoperatingprocedureshavenotcompletelyshiftedbacktotheirpreviousstate.

• Innovationhasmostlyremainedinplacesincedepartureofexternalsupports;instructionalmodelandstandardoperatingprocedureshaveonlyslightlyshiftedbacktotheirpreviousstate.

• Innovationhasremainedfullyinplacesincedepartureofexternalsupports;instructionalmodelsandstandardoperatingprocedureshavenotshiftedbacktotheirpreviousstate.

EquityTowhatextentarechangestopractice

occurringequallyacrossdifferentsocioeconomic

groups?

• Innovationisisolatedtosmallpocketsoftheregion,resultinginunequalopportunitiesforstudentsalignedtosocioeconomicstatus.

• Innovationismostlyisolatedtosmallpocketsoftheregion,resultinginunequalopportunitiesforstudentsalignedtosocioeconomicstatus.

• Innovationismostlybalancedacrosstheregion,resultinginlargelyequalopportunitiesforstudentsalignedtosocioeconomicstatus.

• Innovationisbalancedacrosstheregion,resultinginequalopportunitiesforstudentsalignedtosocioeconomicstatus.

3.2ImprovedStudentOutcomesCRITERIA WEAKEVIDENCE DEVELOPINGEVIDENCE SUFFICIENTEVIDENCE EXTENSIVEEVIDENCEKnowledge

Towhatextentarestudentsdeveloping

increasedunderstandingofimportantcontent

knowledge?

• Fewifanystudentshaveincreasedtheirmasteryofcontentknowledge.

• Somestudentshaveincreasedtheirmasteryofcontentknowledge.

• Moststudentshaveincreasedtheirmasteryofcontentknowledge.

• All,ornearlyall,studentshaveincreasedtheirmasteryofcontentknowledge.

CognitiveandMetacognitiveSkillsTowhatextentare

studentsdevelopingkeymentalprocessingskillsas

wellastheabilitytomonitorandassesstheuseoftheseskills?

• Fewifanystudentshaveexhibitingincreasedcognitiveandmetacognitiveskills.

• Somestudentshaveexhibitingincreasedcognitiveandmetacognitiveskills.

• Moststudentshaveexhibitingincreasedcognitiveandmetacognitiveskills.

• All,ornearlyall,studentshaveexhibitingincreasedcognitiveandmetacognitiveskills.

MindsetsandDispositions

Towhatextentarestudentsdevelopingthequalitiesandmindsets

neededtobesuccessfulincollege,career,andlife?

• Fewifanystudentshaveexhibitingimprovedmindsetsanddispositionsneededtobesuccessfulincollege,career,andlife.

• Somestudentshaveexhibitingimprovedmindsetsanddispositionsneededtobesuccessfulincollege,career,andlife.

• Moststudentshaveexhibitingimprovedmindsetsanddispositionsneededtobesuccessfulincollege,career,andlife.

• All,ornearlyall,studentshaveexhibitingimprovedmindsetsanddispositionsneededtobesuccessfulincollege,career,andlife.

Intermediary Organizations and Education Innovation · Center for Public Research and Leadership 6...

Documents

Transcript of Intermediary Organizations and Education Innovation · Center for Public Research and Leadership 6...