Maximizing Evaluation Impact by Maximizing Methods:Social Network Analysis Combined with Traditional Methods for Measuring Collaboration

Carl Hanssen, PhD & MaryAnn Durland, PhDAmerican Evaluation AssociationBaltimore, MDNovember 7, 2007

11/07/2007 2

Agenda

Social Network Analysis: The Method

SNA Results and Interpretation

Next Steps

11/07/2007 3

SNA Methodology

Network Analysis is the study of the relationships formed by the interaction or links between components in a “set”.

MMP sets are schools The components are individuals:

Faculty, both math and non math MTL (School level Math Teacher Leaders) MTS (District level Math Teacher Specialists)

Relationship is communication about math

11/07/2007 4

Measures

Indegree – popularity Density – how “thick”, how much,

out of potential

11/07/2007 5

Role in Evaluation

How much does the communication structure actually fit the theory and the design of the project

Can the structure be correlated with other measures of implementation and impact? Activities Proximal measures

11/07/2007 6

MMP Evaluation Logic ModelStudent

Achievement

Teacher Content& Pedagogical

Knowledge

Math FacultyInvolvement

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

NewCourses

DistrictBuy-in

MPA Ownership

MATCBuy-In

UWMBuy-In

ClassroomPractice

MMPActivities

ProximalOutcomes

DistalOutcomes

11/07/2007 7

MMP Report Card Indicators

19 indicators in 7 domains derived from in-school data collection, online surveys, and MPS data

1. MTS Assessment2. Collaboration3. Learning Teams4. Classroom Practice5. Professional Development6. Teacher MKT7. Student Achievement

11/07/2007 8

SNA In Context: Evaluation Results

StudentAchievement

Teacher Content& Pedagogical

Knowledge

Learning TeamEffort

SchoolBuy-in

TeacherInvolvement

ClassroomPractice

WKCEMean % Proficient = 44%

Overall rating = 3.5Gap MTL v. other teacher = .2Teacher Engagement = 3.2

Overall IRT = -0.34Algebra IRT = -0.18

Team Functioning = 3.5MMP Principles = 3.6LT Quality = 3.1

PD Hrs. = 17.8Facilitation Hrs. = 1.0PD Quality = 3.1

Network density = 6.7% / School density = 17.6%MTL Role = 13.8 / MTS Role = 5.3

SR MTL Engagement = 4.4 / MTS Quality = 3.0

MTS Assessment = 38.3 of 55

11/07/2007 9

Data Collection

Math stakeholders in each school were asked to name individuals with whom the communicated about mathematics

Statistical analysis focused on1. Network and in-school density2. Importance of MTL and MTS

11/07/2007 10

MMP Impact Continuum

Low High

Loose WebMTL Not CentralFew Links to MTLMTS OutsideFew Links to MTS

Tight WebMTL Central

Many Links to MTLMTS Inside

Many Links to MTS

11/07/2007 11

LowSchool n Total Named

Network density

Density in school

MTL Role--In Degree

MTS Role--In Degree

G 17 40 6.2% 9.0% 17.31 3.85

Average 21.1 54.0 6.7% 17.6% 13.81 5.31 SD 6.8 17.6 2.6% 9.6% 7.2 4.9 Median 19 48 6.2% 15.4% 13.07 3.75

11/07/2007 12

MediumSchool n Total Named

Network density

Density in school

MTL Role--In Degree

MTS Role--In Degree

I 28.0 75.0 4.0% 12.2% 23.31 0.33

Average 21.9 57.1 6.3% 12.2% 18.84 2.69 SD 8.0 16.7 2.6% 5.0% 6.90 3.70 Median 22.0 51.0 5.7% 11.4% 17.56 0.92

11/07/2007 13

HighSchool n Total Named

Network density

Density in school

MTL Role--In Degree

MTS Role--In Degree

B 23 55 11.4% 31.1% 28.24 18.52

Average 21.1 54.0 6.7% 17.6% 13.81 5.31 SD 6.8 17.6 2.6% 9.6% 7.2 4.9 Median 19 48 6.2% 15.4% 13.07 3.75

11/07/2007 14

Student Achievement & In-School Network Density

11/07/2007 15

Student Achievement & MTL In Degree

11/07/2007 16

Conclusions

Distributed leadership—a key program goal is manifested by a tightly webbed network

School-level adoption of program principals is manifested by positioning of key individuals within the network

There may be a natural evolution of school networks that is indicative of program impact in that school

11/07/2007 17

Next Steps Continue school-level analysis to

strengthen our hypothesis about the relationship between social networks and other proximal and distal outcomes

Develop cross-school (or district-wide) networks

11/07/2007 18

Contact Information

Carl Hanssen, PhDHanssen Consulting, LLCcarlh@hanssenconsulting.com616-808-2867

MaryAnn Durland, PhDDurland Consultingmdurland@durlandconsulting.com630-650-9944