COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL

A GUIDE TO PROGRAM EVALUATION

The Saguaro SeminarJohn F. Kennedy School of Government

Harvard University

Thomas Sander, Executive DirectorStephen Minicucci, Ph.D., Principal Investigator

Working Draft, Not for Citation or Quotation without Permission

COMMENTS WELCOMESend remarks to Tom_Sander@harvard.edu

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

May 24, 2023TABLE OF CONTENTS

WHO SHOULDN’T USE THIS GUIDE................................................................................................................3A NEED TO MEASURE SOCIAL CAPITAL IMPACT..........................................................................................4HOW TO USE THIS GUIDE.................................................................................................................................4

PHASE ONE: PLANNING..........................................................................................................................................6

STEP 1: MOBILIZING RESOURCES......................................................................................................................6STEP 2: UNDERSTANDING SOCIAL CAPITAL AS IT RELATES TO ORGANIZATIONAL MISSION........7

A. CONSIDERING WHAT TYPES OF SOCIAL CAPITAL TIES YOU ARE BUILDING.....................................8B. ANALYZING ORGANIZATIONAL GOALS.......................................................................................................8C. HOW DOES ORGANIZATIONAL MISSION MAP ONTO SOCIAL CAPITAL?..............................................9D. WHAT COMMUNITY ARE YOU SERVING?...................................................................................................9

STEP 3: IDENTIFYING PROGRAM LINKS TO SOCIAL CAPITAL.................................................................10A. PROGRAM’S RELATION TO COMMUNITIES SERVED.............................................................................11B. SOCIAL CAPITAL FORMATION STRATEGIES............................................................................................12

1) DIRECT SOCIAL CAPITAL BUILDING..............................................................................................122) INDIRECT SOCIAL CAPITAL BUILDING STRATEGIES.................................................................13

C. WHICH TIES IS ORGANIZATION TRYING TO BUILD?.............................................................................14D. IS SOCIAL CAPITAL AN ESSENTIAL PART OF PROGRAM OPERATIONS?............................................15

PHASE TWO: EVALUATION.................................................................................................................................16

STEP 4: DESIGNING THE EVALUATION..........................................................................................................17A. WHO IS THE STUDY GROUP?.....................................................................................................................17B. HOW CAN WE BEST IMPLEMENT A BEFORE- AND AFTER-SURVEY FRAMEWORK..........................18C. SHOULD WE USE A COMPARISON GROUP?............................................................................................20D. WHAT SHOULD WE MEASURE?.................................................................................................................23(1) THE RESPONDENTS’ GENERAL LEVEL OF SOCIAL CAPITAL..............................................................24(2) PROGRAM-SPECIFIC TIES AND SOCIAL CAPITAL.................................................................................24(3) RESPONDENTS’ EXPOSURE TO AND USE OF THE PROGRAM............................................................24(4) DEMOGRAPHIC AND OTHER RELEVANT RESPONDENT CHARACTERISTICS...................................25

STEP 5: CONDUCTING AN EVALUATION.......................................................................................................25A. INTERVIEW FORMATS..................................................................................................................................25B. HOW MANY CASES?......................................................................................................................................26C. MAKING SURE SAMPLES ARE RANDOM (AND NON-RESPONSE ERROR)............................................27D. WHO SHOULD CONDUCT THE EVALUATION?.......................................................................................27E. HOW TO CONSTRUCT THE QUESTIONNAIRE..........................................................................................27

PHASE THREE: ACTION........................................................................................................................................28

STEP 6: INTERPRETING THE RESULTS............................................................................................................28STEP 7: REVISING PROGRAMS..........................................................................................................................30

DRAFT2

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

THE PURPOSE OF THIS GUIDEIncreased amounts of social capital benefit individuals, organizations and communities in ways that academics are increasingly observing1 and which many non-profits have sensed intuitively for a while.2 More recently, writers have showed how businesses with high levels of social capital can gain a competitive advantage.3

This guide is not intended to convince you that social capital is critically important to your programs, although if you want to read more about why social capital is important, click here. Similarly, while many organizations reading this guide already believe that they are creating social capital, you might want to look at this non-exhaustive list of kinds of organizations that might want to use this guide or read the “Social Capital Formation Strategies” section to get a sense of how some organizations build social capital.

The guide is intended for organizations that have already decided that social capital is an important aspect of their work, and want to know how to measure how well they are building social capital, whether it be in the community, among their staff, with their board, etc.

Organizations that may be interested in this guide include schools, hospitals, community service groups, businesses, or community foundations, neighborhood associations, to name but a few. [For a list of organizations that might want to use this Guide, click here.] We intend the guidebook to be used by program developers, program evaluators, and leaders who are already aiming to build social capital through their organizations.

It is our intention that this guide will enable you to measure the effect that you have on the social capital of your community, improve the effectiveness of services delivered, and enable you to claim credit for the community benefits that your organization is creating.

WHO SHOULDN’T USE THIS GUIDEWe try to be as clear as possible on the evaluation process in this guide, but we strongly recommend that organizations that have never evaluated any aspects of their program not cut their “evaluation teeth” using this guide. If this is the first time you are thinking about conducting an evaluation, and you have no staff with prior evaluation experience, we recommend that you find an outside evaluator to work with you. [Perhaps the evaluator could also teach your organization about the evaluation process as she proceeds.] Organizations that want more background on evaluation in general might also wish to read the W.K.Kellogg’s Evaluation Handbook; in their typology, the type of evaluation we discuss in this guide is a “project-level outcome evaluation”.

1 A review of the international social science literature found 20 articles on “social capital” prior to 1981, 109 between 1991 and 1995, and 1003 between 1996 and March 1999. [Winter, Ian, “Major themes and debates in the social capital literature: the Australian connection,” in Social Capital and Public Policy in Australia, Ian Winter, ed. (Melbourne: Australian Institute of Family Studies, 2000), 17.]2 In the past decade or so, there has been a dramatic shift toward treating and strengthening communities instead of simply treating individuals, and there has been a new focus on the overall health of our communities. Groups like the Healthy Cities movement do not confine themselves to traditional medical problems or solutions, but rather embrace the entire community as their “patient.” Treating violence as a public health problem has become common, and there has been a renewed interest in community policing strategies that bring neighborhood watch groups and other associations together with police to prevent crime. And parental involvement in their children’s schools is now widely seen as the pathway to more effective education and schools. 3 See Don Cohen, Laurence Prusak, In Good Company: How Social Capital Makes Organizations Work.

DRAFT3

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

A NEED TO MEASURE SOCIAL CAPITAL IMPACTIn an era of increased focus on the bottom-line and quantification of impact, we underinvest in abstract concepts that are difficult to measure. Social capital is an excellent example of this phenomenon.

For example, in the service delivery world (social work, medical care, and the like), programs have to focus on program funding and reimbursement for services. This requires documenting and measuring elements of service delivery, such as the number of hours of daycare provided, number of vaccinations given, number of job placements made, number of people housed in shelters. Such measurements ignore measuring the social capital health of the communities, despite the intuitive awareness that strong communities are essential to these programs being effective.

This guide is intended to provide a means for measuring social capital. Once measured, we hope that service and funding organizations will publicize the impact that their programs have on building social capital and use this fact to justify greater investments in building it.

HOW TO USE THIS GUIDEThe process outlined in this guide falls into seven steps:

Phase I: Planning Step 1: Mobilizing Resources Step 2: Understanding Social Capital as it relates to Organizational Mission Step 3: Identifying Program Links to Social Capital

Phase II: Evaluation Step 4: Designing the Evaluation Step 5: Conducting the Evaluation

Phase III: Action Step 6: Interpreting Results Step 7: Revising the Program based on the evaluation

Note: After reviewing the entire guide, each organization must decide how much time to spend on each step. For example, large organizations with research shops and on-going evaluations may want to skip “Mobilizing Resources” (step 1). Similarly, some organizations may be more familiar than others with social capital and how they would like to build it and thus will spend much less time in steps 2 and 3 tracing the intersection between the organization and program on the one hand, and social capital on the other. For every organization, steps 4, 5, and 6 will require the most time and resources. We also highly recommend that you read pages 28-44 of the W.K. Kellogg Foundation’s Evaluation Handbook which has a useful overview of “outcome evaluation” that will be a helpful complement to this Guide.

Throughout this guide, we’ll have a hypothetical non-profit called “Jumpahead” going through the same Evaluation process. You may want to find out at each stage how Jumpahead dealt with these decisions or conducted this step of the process.

DRAFT4

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

For a thumbnail description of Jumpahead and why they decided to measure social capital, click here.

PHASE ONE: PLANNING

Before implementing an evaluation tool designed to tell them how much social capital they are building, community-based organizations must take preliminary planning steps.

Determine the resources needed and mobilize them. Develop a shared and adequate understanding about what social capital is and how the basic mission

of the organization relates to it. Identify how you expect that specific program services, activities, and/or events impact social capital.

An effective evaluation design can then build upon this foundation.

We envision that this planning phase will engage a wide subset of the organization, involving not just the leaders and executives but a broad representation of service delivery staff and volunteers. [We imagine that the members of the organization come together in a social capital retreat and a meeting to discuss and plan this evaluation.] The second phase, the evaluation itself (described below), will require smaller working groups and may require outside research help as well.

STEP 1: MOBILIZING RESOURCESResource constraints make it difficult for community organizations to conduct high quality evaluations and, if they do collect evaluation data, to use this information effectively to improve their programs.

Task: Identify adequate resources to ensure that the evaluation can be planned and executed, and the results analyzed and acted upon.

There is no simple formula we can give you to determine the out-of-pocket costs of the evaluation and the existing staff time that will need to be devoted to this project. So much of the out-of-pocket costs will depend on the scale of the evaluation (how many people are being surveyed over what time period –discussed in “Whose Social Capital” and “How Many Cases” below) and what outside expertise you need (to conduct the survey, analyze the results, etc.). The level of staff time needed for this project will also depend on the familiarity of staff with evaluation, and with how much thinking the staff has previously done about how the organization’s programs build social capital. If you want a list of some of the activities that will absorb internal staff time and out-of-pocket costs, click here. The W.K. Kellogg Foundation generally recommends that evaluation consume roughly 7-10% of a project budget, although obviously it is ideal if you find cheaper ways of conducting a high-quality evaluation.

Task: Building Support for Evaluation

There may be real opposition to using scarce organizational resources for evaluation either because: a) previous evaluation data has not been used effectively; and/or b) because evaluation shifts resources away from service delivery, at least in the short run.

DRAFT5

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

Ineffective program evaluation: This is a clear risk; evaluation is only valuable if it produces data that strengthens or helps in program design and only if there is adequate follow-through to ensure effective use of the information gathered. To the extent possible, we try to incorporate a discussion of these issues in the “Designing the Evaluation” and “Action” phases (steps 4, 6 and 7) discussion below.

Taking short-term resources away: Although this undertaking may pull needed resources away from direct programmatic efforts in the short run, in the long term (i.e., after the post-survey that measures gains from the baseline pre-survey), the evaluation should increase available resources in two ways. First, by quantifying the program’s strengths, it will make the program more attractive to current and new funders. Second, identifying weaknesses can lead to programmatic improvements, which will enable the organization to use whatever level of resources it has more effectively.

Of course, even given the logic of these points, some organizations may simply lack the political will to free up any staff time and other resources for evaluation work, since this may require heightened workloads or cutting back on something. If costs are a key concern, we recommend a number of possible cost-reducing strategies.

Finding Partners. A logical way to overcome resource constraints, and one strongly consistent with the idea of social capital, is to form partnerships with other organizations wishing to conduct similar evaluations. A community funder or some other umbrella organization might convene a working group of such partners and perhaps also provide some technical or financial support for the project. The consortium can meet to discuss the social capital concept and how it relates to each organization's program offerings, to share research plans and survey questionnaires, and to discuss results.

Using and Sharing Volunteers. Staffing the interviews and collecting the evaluation data is likely to be the largest single cash cost of the evaluation. One way to reduce this cost is to use volunteer interviewers. In order to ensure high quality, unbiased results, the interviewers will require some training and should not be involved in day-to-day program operations. These considerations are discussed further in “Interview Formats” in step 5. [One possibility is for partners with whom you are already cooperating to provide volunteer interviewers for each others’ programs.]

Collaborating with local colleges and universities: Developing this evaluation as a research project for graduate students in social work or some other social science could also provide a useful field research project for the students and provide rigorous researchers for the organization doing the evaluation at low or zero cost. Finally, it might also provide a later pool of volunteers on which to draw.

Ongoing example: how did Jumpahead mobilize resources?

STEP 2: UNDERSTANDING SOCIAL CAPITAL AS IT RELATES TO ORGANIZATIONAL MISSIONWe recommend that you convene key employees in a social capital retreat to discuss their theory of how they expect that your program leads to increased social capital, beginning with a discussion of social

DRAFT6

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

capital. To that end, everyone at the table should have, at a minimum, read this section. Here are also other recommended readings on social capital if you want to dig deeper.

Goal of the retreat: to develop a shared understanding of social capital and how the organization would like to build it or believes it is already building it.

FIGURE TWO ABOUT HERE

Questions to consider at such a retreat (all discussed below) A) What kind of social ties are you building? Strong or weak ties? Individual level ties or

broader community ties? Are you more trying to increase bonding or bridging social capital? B) Analyzing Organizational Goals. After clarifying the organization’s mission, if necessary,

consider:o How central is social capital to what you do? o Do you believe in building more social capital because it is an essential strategy of

how to provide your services or is it an ancillary benefit of your program, but not a key goal?

o Do you build social capital in the hopes of alleviating some of the problems you are tackling?

C) Discuss how the mission of the organization maps onto the concept of social capital, i.e., which of the four goals of building social capital are you attempting to increase: i) Individual Bonding and Bridging; ii) Community Bridging; iii) Community Bonding for Collective Action; iv) Informal Community Bridging.

D) What community are you serving?

A. CONSIDERING WHAT TYPES OF SOCIAL CAPITAL TIES YOU ARE BUILDINGAfter reading the materials on social capital, you should discuss this in the context of your own organization. Are you trying to build informal or formal ties? Are you trying to build bonding or bridging social capital? How does social capital express itself within your organization?

Ongoing example: hear how Jumpahead’s categorized the types of social capital

B. ANALYZING ORGANIZATIONAL GOALS

Before analyzing a specific program or programs, it is worthwhile to briefly consider how the organization as a whole and its mission relates to social capital. We define three closely related tasks below, which will frame the evaluation design.

MISSIONTask: clarify the organization’s mission if necessary.

Throughout this process, keep the organizational goals clearly in focus. To begin with, the meeting leaders should rearticulate the organization’s central mission. This mission might range from “poverty

DRAFT7

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

alleviation” to “increasing public access to the fine arts.” For most organizations, this is not problematic. However, if the mission is not clear, currently in flux, or in dispute, we recommend that the meeting spend a considerable amount of time on this. We hope that a consideration of social capital goals will also help the group clarify the organization’s central mission. [For example, an organization striving to alleviate poverty might realize that its micro-lending approach depends on social capital to be effective: namely, individuals trusting and relying on one another to repay small loans.]

Ongoing example: hear how Jumpahead analyzed their organizational goals

C. HOW DOES ORGANIZATIONAL MISSION MAP ONTO SOCIAL CAPITAL?

Task: How does social capital relate to the organization’s mission?

Determining how social capital relates to the organization’s basic mission may be difficult in some cases. The following questions may be helpful:

Across all of the services, activities, and events organized or provided by your organization, is one of the four social capital goals predominant?

If more than one of these goals is evident, do they reinforce or conflict with one another?

This task is important because it helps the organization determine what type of social capital should be the focus of its evaluation. Programs that aim, among other things, to strengthen ties between individual participants and their communities, however defined, would expect to be able to measure a strengthened community. If the main social capital aim of the program is to link participants to resources and opportunities in the larger society, however, an evaluation focused solely on community-bonding ties may be inappropriate. On the other hand, if an organization is trying to unify and mobilize a community to meet some challenge, it may not be sensible for it to evaluate the bridges built between community members and the whole society. In making these choices, an organization is not labeling the other kinds of ties as unimportant, but only as less central to its mission. Our measurement strategy will explore those relationships that we have identified as critical in greater depth, but include some questions on all aspects of social capital.

Ongoing example: how does social capital relate to Jumpahead mission?

D. WHAT COMMUNITY ARE YOU SERVING?

Tasks: Define the communities that are served by the organization. Decide how the organization expects to affect these communities.

DRAFT8

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

The central issue in the whole evaluation process is to clearly define the organization’s relationship to the community. This is complicated by Americans’ loose use of the term community. To evaluate the social capital built, we need to first define the population served. This should be easy for most organizations to do, but the response to this query has implications for the whole evaluation planning and design process, so we want the clearest possible answer. Second, the organization must answer the harder question of how it expects to affect the community or communities served.

Defining the Community ServedRoughly, organizations define communities served in three overlapping ways.

First, they serve a community of place imprecisely defined by geographic boundaries. This may be a list of places from which customers or participants come, or may have been fixed historically. It may also vary quite dramatically in scale, ranging from a well-defined single neighborhood or town, to a number of neighborhoods, to whole cities or regions.

Second, an organization may have a central commitment to serving a community of identity, such as a racial, ethnic, or religious group. This commitment does not require or imply exclusion. The organization may serve a community of identity with shared cultural elements. By doing so, it can provide individual community member support or strengthen the whole community by either enabling it to act collectively or to preserve the community’s distinctiveness.

Finally, an organization may be tied by its programs to distinct population groups defined by age or some other demographic category, service-need, vocation or avocation, and so on. For lack of a better term, we’ll call this a program community. Members of such a group may or may not see themselves as members of a common community. For example, visually impaired or gay residents may see their primary community as being visually impaired or gay, or conversely may see this as a far less important community than their ethnic identity or their neighborhood. Law can also define a program community: for example, residence requirements that determine a school community.

These three types of communities (place, identity, and program) can combine in virtually every conceivable way, e.g., an organization may serve African-American teens from a certain neighborhood, and another organization may serve all adults across a whole city who lack job skills. Because of these overlapping communities, it is important for the organization to evaluate each type.

Ongoing example: what community does Jumpahead serve?

Note: If your organization operates only one program, the thinking just done in Step 2 may significantly overlap with this next step (Step 3).

STEP 3: IDENTIFYING PROGRAM LINKS TO SOCIAL CAPITALOnce the organization, in its social capital retreat has restated its mission in social capital terms, it can move on to an examination of the specific program or programs to be evaluated. Staff and volunteers who work directly with participants and the public must be very involved in this step. We recommend that you bring a diverse group of stakeholders together for your social capital retreat. While this diversity of thought may make it more difficult to conduct such retreats and manage the conversation, it increases the chance that you will not miss out on an important observation. The ultimate decision as to what you want to evaluate is yours, and this may be important to identify up-front, but you will benefit from diverse perspectives.

DRAFT9

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

We expect that program specifics will be discussed in a second meeting that may be held a week or more after the first meeting or meetings that dealt with the issue from an organization-wide perspective. If the members of the organization cannot answer some of the questions that follow, it may be necessary to undertake a small internal “study” of program operations.

A. PROGRAM’S RELATION TO COMMUNITIES SERVED

Task: Define How the Program Expects to Affect The Relevant Communities

Whether the program, for example, expects to impact any random citizen in a certain geographic area, or only a program participant matters a great deal in considering how to evaluate the social capital impact of a program. Measurement and evaluation choices hinge on the visibility of a program and the scale of its impact. Community health organizations, general education and youth services programs, community advocates and organizers, and antipoverty agencies, among others, typically operate highly visible local efforts serving relatively small geographic communities. Community health centers and youth development agencies (like the YMCA or Boys and Girls Clubs) aim to be a part of the everyday life of their communities. Other programs, like community development corporations, value visibility but don’t directly touch many residents’ lives as often. Because of their institutional position in community life, it is reasonable for these organizations (like the YMCA or the Boys and Girls Clubs) to be concerned with creating social capital in the geographical community served, and not simply among program participants.

Programs involving a small percentage of city residents. Other programs operate on a broad geographic scale but involve a small fraction of residents. There are at least two such program types.

i) Programs Serving Disparate Population: First, a program may serve a very disparate general population. Many arts and cultural institutions fit this model, as do programs that organize, coordinate, or support a number of different local organizations. These programs have a much more diffuse impact on community social capital (defined geographically) and therefore the social capital impact is harder to measure. This does not mean that they cannot contribute to the creation of social capital. The events and activities they sponsor may have a memorable impact on the participants’ lives even if they don’t affect a random community member in that town.

City- and region-wide programs can create bridging social capital in ways that neighborhood and town programs cannot. Obviously, it would be asking a great deal to expect these activities to measurably increase social capital in the whole community they serve (of perhaps one million or more!). Instead, the evaluation should likely focus on how these programs change how individual participants relate to the particular communities of which they are part.

ii) Programs providing special services: Second, programs can also provide special services tailored to relatively narrow population groups. They might provide this service in a centralized location (a large residential facility or a special school, for example) or operate small-scale programs in varied communities.

While programs may have a number of social capital goals, their most likely aim is to serve their program participants heavily – whether that service is participation in an extra-curricular arts program, a homeless shelter, or a choral group. Because the population served may comprise only a small part of a whole neighborhood or town, the effect that these programs is likely to be seen on the program participants, not on the social capital of the community as a whole.

DRAFT10

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

B. SOCIAL CAPITAL FORMATION STRATEGIES

Task: Next, identify the ways in which program4 activities build social capital.

The first question is how the program’s services, activities, and events are intended to meet the organization’s overarching social capital creation goals. We identify a number of basic approaches, broken roughly into direct and indirect social capital building:

1) DIRECT SOCIAL CAPITAL BUILDINGA community-based organization can create social capital directly through its programs. While by no means an exhaustive list, here are some basic ways in which programs might do this:

Fostering Social Ties. If a program creates new social ties or strengthens existing ones it may directly create social capital among those who come into contact with it. (As you’ll remember, we are particularly interested in new or stronger ties that expand the scope of information the individual is exposed to, provide stronger emotional support and/or facilitate increased trust and reciprocity.)

To consider whether a program does this, put yourself in successive turns in the shoes of a typical member of various groups (staff member, program participant, project partner, parent of participant, etc.) and ask yourself whether he/she is likely to come out of this experience with significantly stronger ties. Look for repeat encounters and opportunities to build trust. Before conducting this step, you may want to even conduct some informal focus groups of one or more of these audiences, learning more about their experiences with the program.

The planning group needs to inventory all the social interactions embedded in the program and assess them in terms of their potential social capital effects. Because this may be a new way of seeing the program, we hope that the organization's team takes the time to really “walk through” the program, looking for every opportunity for meaningful interactions. Once you’ve identified the broader inventory of what links are created you can decide where to focus in the “Which Ties?” step. It is also our hope that a side benefit of organizations undertaking this review will be to identify new opportunities to shift program structures or resources in order to diversify and extend the interactions among participants, their families, staff, and the community.

Community Coalition-Building. An important variation on the above theme involves efforts to act directly on the community as a whole. As with the “Fostering Social Ties” analysis above, it may be helpful to inventory all the coalition-building initiatives and analyze their potential social capital effects. Examples of this type of coalition-building include programs that directly organize or mobilize residents, that organize community-wide events, or that work with voluntary community associations to increase their effectiveness.

Individual Awareness of Social Capital. Some programs directly try to make participants aware of social capital and try to develop more social capital through broader networks and civic participation. Sometimes this is a sub-part of a larger program goal (a job search, for example), and sometimes this awareness is itself a program goal (a voter registration drive, for example). In

4 Throughout, we will use the term program in a generic way to refer to any integrated set of one or more services, activities, and events through which a community organization achieves its mission. Even given the dizzying array of such programs [hypertext link to “3b Box 1”] we will try to make some generalizations about social capital.

DRAFT11

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

these cases, program evaluation may include an assessment of how well the message was retained as well as how participants actual behavior and social networks changed as a result.

2) INDIRECT SOCIAL CAPITAL BUILDING STRATEGIESOrganizations may conversely try to increase community social capital indirectly by dismantling barriers to social capital formation or creating new community resources that will facilitate neighborhood organization and sociability. Here are two ways in which programs might indirectly create social capital:

Removing Barriers. Because social capital is rooted in simple social interactions, ranging from ordinary socializing to more formal group activities, anything that keeps people apart, thus limiting their interactions, also limits new social capital formation. If an organization successfully identifies and mitigates one or more barriers to social capital formation, it may help build more social capital. The most common barriers to social capital are safety and discrimination.

o Physical safety is a basic human need, and the fear of violence can easily keep people off streets and out of parks and make them wary of intervening when children misbehave or minor neighborhoods disputes arise. If cooperative efforts, additional or changed police protection, new lighting, private security, or other changes make people feel safer, this may well lead to more time spent outdoors together and more social capital created. The process of enhancing safety (such as through a neighborhood crime watch) can also enhance community collective action.

o Exclusion of groups, discrimination, and segregation represent the real manifestations of community divisions that community-building efforts seek to overcome. While a direct approach to building bridging social capital might bridge people from different groups together for a common purpose (e.g., wealthy and poorer residents working together on a park, or black and white kids on the same basketball teams), an indirect approach might identify and publicize patterns of exclusion, or seek changes that will make public spaces more inviting for groups who now feel uncomfortable there.

Enhancing Community Resources. Another strategy to create social capital involves enhancing community resources underpinning social interactions. These resources build opportunities for more social capital just as road improvements support new business expansion. Here are a few ways in which programs can build the foundation for activities that create social capital:

o Creating and Reviving Public Spaces. The creation or revival of public spaces where people can meet and visit regularly is a prime example of building social capital infrastructure.5 A new playground or community center may become a gathering place for parents and children. Public spaces like parks, libraries, schools, museums, public meeting rooms, gyms, flea markets, farmer’s markets, the post office, and even the town dump can become places for regular social interactions and places where people learn about their community and what's going on in it.

o Creating and Reviving Civic Events. Closely related to public spaces are public events, like parades and celebrations. Shared experiences, like shared spaces, help create a sense of community that may help foster deeper forms of civic engagement.

5 A new public safety initiative may reclaim these spaces, so our categories of “removing barriers” and “creating public space” are closely related.

DRAFT12

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

o Improving Neighborhood Information Sources. The lack of information about one's community may exclude civic participation, so entities like neighborhood newspapers and community cable television programs may be forms of social capital infrastructure.

o Flextime and Release-time for Volunteering. Freeing up time for workers to participate in work-related social activities or to volunteer in the community similarly represents a resource that can be tapped to create new social capital.

This non-exhaustive list of examples shows how a new resource can improve the possibility of social capital creation. Since those creating these resources expect that increased social capital will be an outcome (whether they label it “social capital”), it is certainly reasonable for them to measure social capital. It is also possible that other basic changes in a community — a shift in the housing mix, a business boom, for example — may ultimately enhance social capital. However, unless community organizations have special reasons to look for these second order effects, we do not recommend including them in program evaluation design.

[Note: Direct and indirect interventions could also be combined in the same program; for example, revitalizing a community center using considerable neighborhood participation might both build social capital in the process and create social capital infrastructure (the community center) which indirectly increased social capital.]

C. WHICH TIES IS ORGANIZATION TRYING TO BUILD?Social capital grows out of the concrete interactions of people. The appropriate place to begin to explore the social capital implications of programmatic interventions, then, is to examine the types of social interactions embedded in the program.

Social capital is very diverse and complex. For an organization just beginning to evaluate its own social capital, it may make sense to focus on just one aspect of social capital, leaving other categories of social capital for later. By doing this, an organization can ensure that the evaluative venture doesn’t take over the entire organization. There are at least six classes of ties, which each comprise a category of social capital:

1) Linking program staff and volunteers to participants and their families. These ties are intended to improve program operation by improving information flow and building trust and cooperation between staff/volunteers and participants and their families.

2) Binding participants to participants. These ties get participants to work together, to provide support, information and collectively address common problems. The evaluation should be sensitive to the possibility that these ties extend beyond the boundaries defined by the program (for example, lasting friendships, or cooperation on different types of problems).

Note: Achieving permanence is not the only way these ties can have a social capital effect. Through participant-to-participant interaction, individuals may acquire key skills that will facilitate creating new ties outside the program, or they may become interested and involved in the community in a other ways. This is why we need to measure how their general level of social connectedness and civic engagement has changed as well as their ties to other participants.

3) Strengthening community bonds. A special case of the “participant-to-participant” category arises when the program is open to or acts upon the whole community. In short, it treats the entire

DRAFT13

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

community as a participant. Although the particular goals of these programs will vary widely, embracing both bridging and bonding social capital, most seek to facilitate social capital creation by acting on each participant's attachment to the community. It is likely that the program expects to create few permanent participant-to-participant bonds (because other participants may be little more than faces in a crowd). [An example of this might be a campaign to invoke civic pride and identity with your town, such as a public awareness campaign showing different types of folks with the caption “We’re Pittsburgh.”]

4) Integrating participants into family and community. These ties integrate individual participants into family and community, either by re-attaching them to key personal support networks, replacing or augmenting those networks, or by enriching the range of ties each have with other members of the community. These ties to community are likely to be more instrumental: e.g. building character, providing community job referral networks or recreational programs.

5) Linking participants to the larger society. An extension of the family to community ties are social ties that link participants, or the whole community, to resources in the larger society. An example might be teaching kids about the larger metropolitan area in which they live, or adults about job opportunities in the whole regional economy. This category includes efforts to improve the whole community's ability to represent itself externally (to city or state government, for example).

6) Strengthening community organizations. Programs may facilitate the formation and operation of other community organizations that both build and represent a form of social capital. This intervention may be through training, material support (such as space, money, or printing), or mediation and meeting facilitation (leadership), or help with organization and mobilization.

Click here for a graphic depiction of these six types of links. The two types of links not covered in the guide (among community organizations and community-based governance) are depicted with dashed instead of solid lines.

D. IS SOCIAL CAPITAL AN ESSENTIAL PART OF PROGRAM OPERATIONS?Whether the social capital identified in the above inventory of program-related ties is an essential element or by-product of a program will affect the evaluation design, so this should be discussed.

1) What Social Capital is Essential for Program Effectiveness?The success or failure of some programs hinges directly upon social capital. There are two ways in which this occurs.

Key program goals may be expressible in social capital terms, so that program success is virtually synonymous with social capital creation. For example, programs expressly seeking to “build community” embrace social capital building as a primary goal, regardless of how they seek to effect this goal. It is certainly appropriate to evaluate these programs in terms of their success at building those features of community that they have explicitly targeted.

Social capital within the program may be essential to operate effectively. For example, a school-based program may believe that cooperative and trusting relationships between teachers and students are essential for the program. Note that members of the organization should identify and agree upon these ties in theory because they reflect how the program ought to work, although

DRAFT14

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

further reflection may reveal that there is little or nothing about the program that is actually likely in practice to build such ties.

2) Is Social Capital a By-Product of Social Operations?If increasing social capital seems like a frill rather than a key determinant of program success, it is a by-product. This determination could be the source of some discussion and disagreement. For example, some might view establishing a coffee hour after meetings as a key way to deepen relationships among program staff and participants. Others might view this as a wasteful use of space, cleanup time, time away from pressing work, and resources. Similarly, one daycare provider might see opportunities to create interactions among parents and staff as tangential to the program effort, another may see this as the best way that the daycare center can serve these families.

NOTE: DON’T GLOSS OVER STEP 3; PROGRAM LINKS TO SOCIAL CAPITAL ARE CRITICALBecause members of the organization and the participants in such social capital retreats know their programs and organization well, it is tempting to give this step short shrift in the evaluation process and move on to concrete design questions. But evaluation design depends on specifying the pathways of social capital creation at this step, so we advise you to move slowly and be very thorough. We are interested at this point in all the interactions that might be important, not just the principal one(s).

To return to our educational program example, the program might be viewed as only involving classroom interactions, when non-classroom interactions might be far more important setting for social capital creation. These social ties might range from student-teacher interactions outside traditional instructional settings, student-student interactions, and parent-student or parent-teacher interactions. Based on their programmatic expertise, members of the organization must decide which ties are central to program effectiveness. We hope that this guide and the prior discussion of social capital will help you decide this.

Some program evaluation occurs even at this theoretical step. Fundamentally, interactions which programs are striving for on paper must include a plausible pathway by which the program forms these. If parent-teacher interactions are vital in theory, does the program’s design create reasonable opportunities for these interactions? How are program participants supposed to interact with each other and with the staff and volunteers who run the program? While it will be difficult pragmatically to separate out the social capital effects of each type of interaction, an analysis of program goals and structures should at least suggest which of these relationships ought to be important. It may be helpful to draw a theory model that traces out how you expect to have the impact. For an example of this, click here.

Ongoing example: what were the inventory of social ties for Jumpahead, and how did they expect them to increase social

capital?

PHASE TWO: EVALUATION

Once an organization has identified the social capital elements in its programs and specified pathways through which its activities contribute to social capital formation, it can formulate plans to evaluate its social capital impact. Somewhat artificially, we break down this process into steps 4 and 5: Step 4 involves the basic design questions, such as the study’s timeframe and the choice of study populations;

DRAFT15

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

Step 5 concerns the nuts and bolts of fielding a survey-based evaluation. You can also access a list of social capital survey questions intended to help organizations develop their own survey instruments.

STEP 4: DESIGNING THE EVALUATION

Note: We presume that the type of evaluation being done is a before-and-after evaluation to determine whether social capital was built among a specific target audience. We envision a quantitative evaluation, although there may be cases where organizations are more interested in gathering qualitative data.

OUTLINE OF STEP 4

The design of the evaluation hinges on answering a sequence of questions discussed below. The answers to these questions flow directly from the earlier discussion of what the program is trying to do. The first big question is:

A. Who is the “study group”—the focus of the evaluation effort? That is, in whose social capital are we interested?

Once we know whose social capital we are measuring, we still have to figure out how to do so. This gives rise to a sequence of specific design choices:

B. How can we best implement a basic “before and after” study framework?

C. Should we use a “comparison group,” and if so, who?

And the big practical question of

D. What should we measure?

A number of more technical questions, such as how many cases are required, who should do the work, and how much it will cost are left to Step 5, below.

A. WHO IS THE STUDY GROUP?Which population will be the focus of the evaluation effort? Since the members of the organization may feel that they have a positive impact on the social capital of a wide variety of different groups, choosing one or more group for special examination might be difficult. [You can choose multiple “populations”, such as parents of program participants and program participants themselves, but each additional population chosen will increase the cost and complexity of the evaluation so you may want to stick with one group and you probably don’t want to choose more than two or three maximum.]

In the evaluation-planning phase, we strove for completeness and worked mainly in the realm of ideas and words. Now, as we shift to the practical problem of fielding an actual evaluation, concerns about cost and practicality emerge. An organization may find that it cannot afford or manage the ideal evaluation of its program. This does not mean the whole effort is wasted; we can gain tremendous insight from a “second-best” evaluation implemented with full knowledge of its limitations. A less intensive and less conclusive evaluation may also form the basis of a proposal to fund a more sophisticated evaluation.

Task: Specify a study group.From your Step 3 inventory, look at groups of individuals whose direct and indirect ties you thought were critical to impact. Depending on the program, these might be, for example: program participants and/or

DRAFT16

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

their families, staff, volunteers, residents of a certain neighborhood, etc. If only one group appears on this list, this decision is straightforward. If two or more critical groups appear, the situation becomes more complicated and you may well have to make choices. Because of cost or other considerations, an organization may decide not to ask questions of both volunteers and program participants, for example. In this case, the evaluators may choose to focus on one group at a time. Later, we consider how the same evaluation can include different sets of respondents. The group whose social capital we decide to measure is called the study group. For most purposes, program participants and/or their immediate families are the appropriate group to study, but there are certainly cases in which the focus should be staff or volunteers, or specific outside groups with which the organization interacts, such as members of a local crime watch group. Sometimes organizations may also aim to increase social capital among a much wider audience. For example, groups that organize large community events or serve a large percentage of the neighborhood (the community-health center, or a major youth center such as the YMCA or Boys Club). In these cases, a general sample of the neighborhood or community (however defined) may make more sense to study. We’ll make a general distinction here between the latter, a community sample, and all of the former cases, which we'll simply call program samples.

We’ll worry later about the right size of this sample.

B. HOW CAN WE BEST IMPLEMENT A BASIC “BEFORE AND AFTER” STUDY FRAMEWORK?

METHODOLOGY OF BEFORE- AND AFTER-SURVEY Once the decision about whom to study is made, the next basic design question concerns the type of comparison upon which the organization will base its evaluation.

In all cases, we expect organizations to conduct before and after surveys of the social ties and social capital. Basically, you interview the study group (for example, program participants) and ask them questions before exposure to the program (for example, early on the first day). This “before” survey is also called the baseline. Then, you question them “after” a program cycle6, which, depending on how long the program lasts, might be a semester later, a year later, etc.7 The increase or decrease of specific questions is a reasonable estimate of how much of that type of social capital was created or lost over the program cycle. However, a number of circumstances complicate this simple design.

The model is difficult to apply in cases where participation is spontaneous and the program cycle very short – for example, one-time community events and other programs open to the general public. In these cases, a short baseline survey can be administered during a community event, or immediately afterwards. It is not practical to gather baseline measures of attitudes (such as trust in others or attachment to the community) since one typically cannot talk with participants before they are exposed to the one-time event (e.g., a festival). However, answers to actual behavioral questions (such as the level of recent social

6 One should conduct the survey at the end of the same program cycle for all participants regardless of whether they intend to do another program cycle.7 For programs with no fixed program cycle, e.g., a homeless shelter with no limit on how long residents can stay there, find a common denominator of how long people stay with the program, or if there is not such commonality set the follow-up survey artificially at a year later.

DRAFT17

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

activities or voluntarism) may form more valid baseline measurements.8 You might try to obtain e-mails or phone numbers and follow up with these interviewees in a month or a year to see if there seems to be any persistence to the social capital responses of the participants.

The model is also difficult to apply in programs that mix new and continuing participants. We assume that the study group will be a sample of new participants. Note that a simple comparison of the social capital of new and longtime participants, while possibly very useful, will not represent a good test of social capital creation because participants choosing to return to the program may differ systematically from the truly new participants. For example, participants might get the largest social capital benefit from their first year of program involvement. Including many returning participants would thus dampen the program impact demonstrated because many of those sampled would show smaller social capital gains in their past year than first year participants. Likewise, if you are interested in the social capital among returning participants, including the responses of new clients may alter the true results for this group.

A less serious problem with mixing new and returning participants is that it forces comparison between groups of participants while the before-and-after design employs comparisons of individual participants. While both comparisons are valid, comparisons of individuals (rather than groups) are more likely to produce statistically significant results.

Task: Determine when to conduct the baseline or “before” measurement.

There is not a lot to be said about this task. You should ideally conduct the baseline measurement among the study group at or just before the start of the program, if this is possible. Elsewhere we discuss how many people need to be surveyed, survey question design, and making sure that you chose a representative population (if you are not surveying everyone). If you employ a comparison group (discussed in the next section), you should conduct the “before” measurement of this comparison group at the same time as the study group (if possible).

Task: Determine when to conduct the “after” measurement.

You must also consider when to conduct the post-survey of this pre- and post-measurement. The answer will depend on the types of impacts that are desired and on practical issues of conducting interviews.

For many programs, the best option is an “exit interview” in which the interviewer questions the participant on site near the end of his/her program exposure (or at the end of a cycle of a continuing program). “Entrance” and “exit” interviews then form the basis of a comparison. The strength of exit interviews is that respondents are generally willing to be interviewed and answering questions is easier since the program details are still fresh in their minds. If participants are likely to continue their relationship with the program (or with the organization), exit interviews are particularly appropriate. There are two cautions however:

(1) Difficulty in discerning longer-term impacts: Some anticipated program effects, such as becoming involved in other community organizations, or in local politics, may take much longer to emerge and therefore not be evident in such an early “after” snapshot. If the evaluation team chooses the exit interview approach, they should de-emphasize asking about social capital effects not expected to emerge in the short-run.

8 While in principle program exposure should not affect questions of a respondent’s past behaviors, the problem is that many social capital questions have socially desirable responses (e.g., people would like to say that they recently volunteered), which elements of your program elements may augment.

DRAFT18

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

(2) Social Capital Euphoria: The exit interview may exaggerate other social capital aspects, such as participant-to-participant ties: many respondents may report making long-term friends through the program that they will not in fact maintain. Also, depending on the interviewer, participants may feel pressure to give politically correct answers; for example, claiming that they do more volunteering if the organization with which they have been involved has an ethos of volunteering.

An alternative to an “exit” interview is conducting “follow-up” interviews some time after program exposure. The strength of this approach is that it reduces these problems—long-term social capital building may be more evident and short-term “euphoria” may have abated. The key weakness with the follow-up approach is that respondents are harder to locate and schedule for interviews and they are more likely to refuse interviews. Also, the further into the future that the follow-up survey is done, the harder it will be for the organization to show that they were responsible for the result rather than other things that participants had done since. As a practical matter, the follow-up interviews are more likely to be conducted over the telephone (see discussion in Step 5 below). When should such follow-up interviews take place? Two to six weeks will likely ensure that the short-term program euphoria has passed, but six months or a year are likely necessary for real long-term effects to become evident. However, you should note that response rates will continue to decline the longer into the future you wait.

Persistence: A Third Survey? For those cases in which long-term effects are the central concern, evaluators should consider a three-step evaluation in which clients are interviewed on entry, exit, and then some time later (perhaps a year), assessing different aspects of social capital in each snapshot.

Ongoing example: hear what group Jumpahead decided to evaluate and what social ties they decided to measure ?

C. SHOULD WE USE A COMPARISON GROUP? AND, IF SO, WHO?

Task: Determine what comparison group, if any, will be used.

Whenever possible, organizations should employ a comparison group against which the study group’s progress can be compared. [Click here for an example of why comparison groups are important.] Simply looking at pre- and post-tests of a study group can be misleading for at least three reasons described below: selection effects, placebo effects, and climate effects.

For cost or other reasons, an evaluation using a simple pre- and post-survey is often the best organizations can do, but we wanted to alert you that such a test (without a comparison group) provides only a rough sense of the causal role of the program in social capital creation. Here are some other factors, besides the program itself that might explain the observed shift in social capital:

Selection Effects . First, the study group may be significantly predisposed to increase social capital or, alternatively, be an especially resistant population. These predispositions are critical because participants (and volunteers, and staff) are typically “self-selected”; that is, they themselves choose whether to get involved with a program. This selection bias means the intervention does not fit a standard experimental design model, in which individuals exposed to some treatment are randomly selected. These selection effects can bias the results positively or negatively.

In a positive selection bias example, individuals participate in a program because they are looking to change their lives (“getting their act together”) and may be making other lifestyle shifts that

DRAFT19

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

also affect their social capital. In these cases, some or possibly most of the observed change in their social capital is because of participants’ earlier decision to change their lives, not the result of the program.

In a negative selection bias example, people attracted to a program have below-average social capital; for example, community newcomers, people with certain disabilities, mental illness, or a substance abuse problem. In these cases, a program might have a very large impact because it is building off a very low base or a very small impact because of the difficulty of getting such a group to make connections with others. A proper assessment of the program would take these selection effects into account.

Placebo Effects. A second factor that skews the program’s assessment of its causal role in social capital creation is the evaluation itself. This may sound strange, but by asking people about their social connections, memberships, and level of social trust, we stimulate them to think about these issues. When we ask them again a few months, even a year later, they may have given the questions more thought and have more accurate and less biased answers. The questions themselves may have prompted them to act to build their own social capital. This situation is somewhat analogous to the placebo effect that medical researchers sometimes find when patients assert improvement even though they were only given a sugar pill as part of a control group.

The Civic Climate. Finally, community changes occurring between the pre- and post-evaluation may distort our sense of program’s impact on increasing social capital. We’ll call this the “civic climate effect.” Some of these shifts, such as a breaking political scandal, an economic downturn, or a local crisis, might temporarily depress the level of social trust in the general population. Some other changes, like a local sports team championship, might raise spirits and appear to raise social capital as well. If the pre- and post-tests are taken in different seasons, the climate effect may be just that, a result of the weather! 9

Because of factors like “selection,” “placebo,” and “climate” effects, the evaluation results are much more convincing if the change in social capital of the study group is contrasted with the corresponding shift in an appropriate comparison or control group. Because the addition of this group is an attempt to re-create a classic experimental design that includes “treatment” and “control” groups, this evaluation approach is often called quasi-experimental.

If a comparison group can be found that is similar to the study group in terms of their baseline level of social capital or, especially, their ability to acquire new social capital, then a comparison of the changes in social capital between the two groups will yield a much better estimate of the actual program impact. This can be a tall order. Some organizations will not have access to a good comparison group. For example, an organization that implements a school-wide initiative would ideally compare their success with a similar school where this initiative was not tried, but that requires access to another school. The evaluation of many workplace innovations has this problem. Comparisons are probably also not practical for those assessing community-wide impacts.

Finally, for those who deal with very specific populations, at-risk youth or single mothers for example, it is not always possible to identify an appropriate comparison group. By answering the question, “Who is comparable to this group?” we identify what we think are the relevant specific group characteristics for

9 We rush to add that we do not understand all of these connections well, but this is precisely why they can “contaminate” our program-based measurements.

DRAFT20

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

social capital. This is unavoidable theorizing, as with the program review in Step 3, but at least we can make these assumptions explicit.

Who are good comparison groups? The simplest case involves comparisons of two program sites operated by the same organization, but where the program change is implemented at only one site. This is not a perfect comparison since one site might offer different programs, serve a different population, or have different staff. Still, the two sets of participants are often quite comparable.

Even better, for example, a school program with limited enrollment may admit qualified students by lottery, allowing one to compare admitted students with a random selection of those not selected in the lottery. These situations are very close to a true experiment and should produce good results. Sometimes this set-up is called a “natural experiment”.

Many organizations don’t have multiple sites, or may be unwilling to withhold a worthwhile program or service from a group of clients in the name of research. (This is a basic ethical problem that frequently arises in medical studies of interventions that are expected to help patients.) If an ideal control group is unavailable, what are some second best alternatives? Three possible sources for comparison groups should be considered:

(1) People affiliated with your organization but not in the program being evaluated. Some organizations may have access to such potential participants, for example kids who use a youth center for a basketball league, but not for the after-school mentoring program being evaluated. If all these kids come from the same neighborhood, the basketball league youth may be similar enough to use as a comparison group. It may be a good idea to draw a relatively large sample of the comparison group in case a subset of them emerges as particularly good (e.g., kids with single parents). Another example might be a community health center, evaluating a specific health program, recruiting other inpatients as a comparison group.

(2) Using a community organization or program as a comparison site, perhaps one interested in duplicating the program under review. For programs that serve children, partnerships with local schools might serve this purpose. For adult populations, it is a little harder to imagine where a comparison sample might be recruited. This option is obviously more plausible if community organizations and institutions have good working relationships.

(3) For programs serving the entire community, using the general population as a comparison group, i.e., contrasting the social capital of program participants with a sample of the entire community. The comparison group here can be drawn randomly from the listings in the appropriate telephone exchanges. If you wanted to further refine the sample you might filter or screen out candidates to make sure respondents are of the right age group, neighborhood, gender, homeowner status, or family structure (parents, single parents). For such a sample to be practical the desired characteristics must not be too rare (if they are, many more calls must be made and the cost or time to complete these surveys rises quickly).10

10 For example, if you are only trying to reach African American households with children in a 10 block area and these represent only 5% of the phone numbers in a given exchange (the first three digits of a seven digit phone number), this is what the math will look like. First, you will place calls to random numbers and try to find people willing to do the survey (which often only has a success rate of only 1 in 3). Then in the 1/3 of cases that you get a willing respondent, you’ll have to exclude 19 of every 20 calls to ensure that all the respondents are African American households with children in this 10 block area. Since the result is that it will take roughly 600 calls to get 10 suitable respondents, or 6000 calls to get 100 respondents, this becomes very time consuming and expensive.

DRAFT21

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

Ongoing example: hear who Jumpahead used as comparison group ?

Summary of Design Choices: Timing and PopulationSo far, we have introduced four basic evaluation design questions:

Whom to study When to conduct the baseline or “before” interview When to conduct the “after” interview, Whether to use a comparison group, and if so, who

There are often no right answers to these questions, and resource and other constraints may preclude us from pursuing the best choice. To maximize the study’s credibility and impact, it is essential that members of the organization can justify the choices made at each step and have a good understanding of the trade-offs involved.

D. WHAT SHOULD WE MEASURE?

Task: Identify Measurable Aspects of Social Capital

Now that we know whom we are asking questions of, what questions are we asking them? This will vary substantially from program to program, but the survey instruments that every organization develops will probably include four categories of questions assessing:

(1) The respondents’ general level of social capital; (2) Program-specific ties and social capital; (3) The respondents’ exposure to and use of the program; and (4) Demographics and other relevant respondent characteristics.

Note for programs using control groups: control or comparison group respondents will likely be asked only the questions in categories (1) and (4) and possibly one or two questions from category (3). 11 The program participants would get all four categories.

(1) THE RESPONDENTS’ GENERAL LEVEL OF SOCIAL CAPITALThe general assessment of social capital lies at the core of the evaluation. Our Social Capital Community Benchmark Survey of over 30,000 Americans identified 11 dimensions of social capital. In the realm of intelligence, not all people strong on one dimension, say spatial intelligence, are strong on another, say verbal skills. But as with types of intelligence, most dimensions of social capital roughly correlate with each other: an individual exhibiting an above average level of informal socializing is probably likely to be above average in formal group involvement or social trust, even though there are clearly many exceptions. Even though your program may be interested in only one or two social capital dimensions specifically, we recommend that you ask questions from all the dimensions so you can get a fuller sense of the types of social capital being built or not being built.

11For example, we need to ask whether the comparison subject has been exposed to the program under evaluation or one like it.

DRAFT22

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

We recommend that all programs ask as many of the core questions on social capital as possible. This should leave additional time in the survey for demographics and exploring in further the types of social ties that the program is interested in exploring more in-depth.

(2) PROGRAM-SPECIFIC TIES AND SOCIAL CAPITALThese questions can be separated into: a) those measuring social ties directly associated with the program and b) those that measure the level of trust and respect in the program or surrounding community. The specific form of these questions depends so much upon program and organizational characteristics that we cannot recommend exact questions. However, the generic social capital measures, provide a model for these more particular questions, in addition to providing useful questions from different domains (workplace, family, neighborhood, school-based social capital, etc.) that may well give you good ideas for questions in the domains in which you are most interested.

(3) RESPONDENTS’ EXPOSURE TO AND USE OF THE PROGRAMThe analysis in Identifying Program Links to Social Capital (Step 3 above) should identify the various opportunities created by the program for participants, volunteers, staff, and the community-at-large to interact with one another. These opportunities hopefully initiate the ties that the program-specific measures of social capital will measure. These questions ask how the study group actually interacted with these program features.

We want to measure how much time they spent, for example: Using common spaces Socializing before or after classes, meetings, or events Attending lectures, classes, and workshops Whether they followed referrals or other recommendations How convenient they found participation in various aspects of the program How comfortable they felt socializing with others in program settings Impressions on how important different aspects of the program were Which program components they enjoyed the most

Note that the factual questions related to program use can sometimes be complemented with non-survey sources, such as staff observation or administrative records (time cards, sign-in sheets, attendance records, etc.). These sources provide a good cross-check against survey-based information sources. To this end, organizations should examine their existing evaluation procedures to explore the possibility of expanding them to include considerations of social capital. These questions will help give program ideas about whether participants are building social capital through the channels that the organization expects. For example, a daycare provider might speculate that a weekly coffee half-hour was essential for daycare providers and parents to meet one another and form ties. However, if an evaluation showed that the kind of social ties that parents had with the daycare providers had no correlation with whether parents went to this coffee half-hour (or how often they went), it should tell the organization that the coffee half-hour is not the key ingredient in these social relationships forming, although some other factor might be key.

(4) DEMOGRAPHIC AND OTHER RELEVANT RESPONDENT CHARACTERISTICSClick here for a short list of demographic concepts that research has shown to be related to social capital formation. These include education, age, income level, race, gender, religion, marital and family status, homeownership status, and the number of years the respondent has lived in the same community or location. Organizations surveying staff or volunteers may also want to know the respondent’s years of experience and their type of employment.

DRAFT23

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

Ongoing example: what are some questions Jumpahead asked ?

STEP 5: CONDUCTING AN EVALUATIONThere are a number of concrete issues that an organization fielding an evaluation still needs to resolve. This section considers five key practical questions:

A. What is the right interview format? B. How many interviews do I need (“How Many Cases”)? C. How to make sure that the samples are random?D. Who should conduct the evaluation? E. How to construct the questionnaire?

Together with the earlier “Whom to Study?” discussion, the answers to these questions largely determine the overall evaluation cost. Of course, if the resulting cost is too high, an organization may have to scale back, compromising on these four issues until the project is affordable. At that point, the organization will have to ensure that the scaled-down evaluation will still be worthwhile.

A. INTERVIEW FORMATSThere are at least six basic formats, although combinations are also possible:

In-person interviews conducted on-site In-person interviews conducted off-site (in the person’s home, at school, etc.) Self-administered questionnaires answered on-site Telephone interviews Mail or e-mail surveys Interactive or Internet evaluations (using a computer program).

Each of these models has advantages and disadvantages. [If you are undecided about what approach you will take and want to read about the relative strengths and weaknesses of these approaches, click here.]

Ongoing example: hear about Jumpahead’s interview format

B. HOW MANY CASES? Deciding how many interviews to conduct depends both on a number of technical factors and overall budget constraints. Obviously, no other design question has as great a bearing on the overall cost of the evaluation as the number of people surveyed, so organizations have a strong interest in minimizing the number of people surveyed. On the other hand, they do not want the number of respondents to be insufficient to reach a conclusion.

If you are interviewing everyone in the study group or control group, called a census sample, you don’t have to worry that your data doesn’t match the group as a whole. But if you are interviewing less than 100% of the study group or control group, such as a sample from the community, you must worry about sample size. Basically the size of the sample relative to the size of the study group (or control group) will provide you with a confidence interval (a range in which you are relatively certain that the entire group ‘s

DRAFT24

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

responses would lie if all the population had been surveyed). [This confidence interval, although mathematically easy to predict, is only one of four possible sources of error. Click here for a description of bias from patterns of non-response and nonrandom samples; validity problems; and reliability problems.

For those not interviewing everyone in your study group, click here for more specific guidance on sample size. However, we encourage you to consult a statistician (someone who is well versed in statistics) to advise you on these questions, as they can be complex.

Sample Size Recommendations for In-Person (not paper-based) and Telephone Interviews

Table 3If Populationis:

We recommend a Sample of: Approximate

Sample Fraction Confidence Interval30 30 1 060 45 ¾ 7.4%100 65 2/3 7.2200 98 2 7.1500 163 1/3 6.31000 213 1/5 6.01500 313 1/6 5.52500 363 1/7 4.710,000+ see Table 2: but 500

individuals probably sufficient

Note: if after the evaluation you are planning to compare one part of your sample against another part (for example men versus women), please read the section on stratification.

Recommendations for In-Person Paper-Based SurveysBecause the cost of additional cases can be quite small when administering in-person surveys on paper, we suggest that organizations stay with universal samples (in other words trying to interview all of the individuals in the group you are interested in) much longer, perhaps even up to 100 or 200 cases. But, see the discussion on “A Word on Census Samples”.

C. MAKING SURE SAMPLES ARE RANDOM (AND NON-RESPONSE ERROR)For programs not conducting a census of the study group, you will need to figure out how to draw the selected number of cases. Regardless of the survey’s format, all samples are judged by the same, central principle: that every person in the group you are surveying has an equal chance of being selected.

For a detailed discussion of how best to find and use a list from which to draw participants, and how to reduce the error of participants not responding, click here.

DRAFT25

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

D. WHO SHOULD CONDUCT THE EVALUATION?Larger organizations with an internal evaluation or research department should probably use such a department for the study. Smaller organizations typically need external help for Steps 4-6 (the “Evaluation” phase and “Interpreting the Results”), but all organizations should be actively involved in Steps 1-3 (the “Planning” phase), even if using an outside evaluator. If you want guidance on how best to select an external evaluator, see the W.K. Kellogg Evaluation Handbook chapter 5, step 4 (pp. 57-68).

Interviewing represents the most time-intensive element of the evaluation process. Most interviews will take 15-30 minutes, permitting only 2 surveys to be conducted per hour on-site or by phone, and fewer off-site. The time necessary to field 100 or 200 interviews quickly mounts. (Also bear in mind that you often need to call, for example, 300-600 numbers to get 100-200 willing respondents.) We outline below a number of options for staffing the interviews. There are only two restrictions: (1) interviewers should not have had significant prior contact with the respondents (or it could upset the objectivity of responses); and (2) they should be trained in the survey instrument (i.e., asking the survey questions).

TrainingAll interviewers should go through a short training in which they are introduced to the basic goals of the survey (without biasing interviewers by telling them your preconceived expectations), go over common points of difficulty with the instrument, and interview one another for practice. Note: this is the first occasion that the evaluation team will have to pre-test the instrument: i.e., see if respondents find question wording vague or unclear, see if many respondents want to answer with an option that is not listed, etc. If you have to replace interviewers or substitute in new ones in the midst of your survey, make sure all new interviewers are trained in this same way. The interviewers should also each conduct one or two pre-test interviews on qualified respondents before interviewing the actual population being surveyed. The survey is conducted as if it was real, but the results are discarded and not entered.

Interviewers also need to understand what confidentiality procedures exist for respondents.

Ongoing example: hear about Jumpahead’s sample size, sampling approach, and who conducted the evaluation

E. HOW TO CONSTRUCT THE QUESTIONNAIREIn “Recommended Survey Questions” we supply recommended core questions on social capital measurement and a longer list of pre-tested questions. [In general, you should use pre-tested survey questions, since new questions that sound good may not actually provide useful data.] We invite you to use as many of these questions as you want. Since, in your construction of a survey you want to nonetheless draft your own questions, especially on issues relating to the specifics of your program, we have a separate section on how-to’s for survey construction. Note that writing questions is easiest when the survey writer has a good sense both of what she wants to measure and how to communicate with the study and comparison populations. You may want to read the section on “Determining Data Weighting” in advance of gathering the data.

DRAFT26

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

ConfidentialityBecause the evaluation design involves both before- and after-interviews, you will need to keep contact information to recontact survey respondents and both surveys will need respondent identifiers. Special efforts therefore must be made to protect respondent confidentiality. There are a number of ways in which this can be done during and after the study period and in the creation of computer data files.

A simple approach is to assign unique identifier numbers to each respondent that allow the organization to share the data with others or to join the data from different survey waves, without referencing respondent names. A separate file can be kept (with paper backup) that contains the mapping of names to respondent identifiers. We recommend that the survey instruments be numbered on all pages in advance and that the cover sheets only contain respondent's name, address, or other contact information. This page can then be easily separated from the rest of the questionnaire before being given to the organization.

Confidentiality is especially important to underscore, especially if the surveys are being conducted face-to-face or by phone with anyone from the organization. Where possible, we suggest that mail surveys be sent to some group other than the program being evaluated so the responses can be entered confidentially.

Congratulations! You’re ready to go conduct the survey. Come back to step 6 when you have gathered the results of the survey and are ready to input them.

PHASE THREE: ACTION

STEP 6: INTERPRETING THE RESULTS

Data entry: The first step of interpreting the results after you have conducted the survey is to enter the results of the survey in a database or a software package. Excel is fine if you simply want to know what the averages were on the various questions or other simple statistical functions (like standard deviation, etc.). If you want to do more sophisticated analysis, like for example correlate the level of volunteering with the intensity of program involvement, and control for the participant’s education level, you will probably need a statistical package (like SAS, SPSS, Stata, etc.).

In general, each row of your database will correspond to a different survey respondent and will probably have a field indicating the respondent ID number (to preserve confidentiality), and each column will correspond to a different question asked (like the gender of the respondent, or how many times they volunteered in the last 12 months, or how many social events of the program they attended, etc.). In statistical packages each cell would have a coded number for that answer and you could then see, for example, that a “2” under the column “RELIMP” meant that religion was somewhat important to the respondent and a “1” meant it was very important.

Anyone (a volunteer, a program participant, etc.) can enter the data as long as he/she didn’t know the identity of the respondent. But it is critical that the responses be entered accurately as it will produce erroneous analysis later if the answers are misentered.

DRAFT27

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

Data analysis: For each question, determine the percentage change among the study group between the pre- and post-survey, and compare it to analogous percentage change among the control group. To see how to do this, click here.

You may want to think about combining some of the social capital questions into indices along some of the 10 dimensions (like social trust, inter-racial trust, etc.). You may want to determine some overall social capital indices based on the general social capital questions or based on the program-specific questions. [See “Determining Data Weighting” below.]

If you are well versed in data analysis, or have an evaluator or local college or university helping you, you may want to conduct more sophisticated analyses to try to understand what is driving some of these changes. For example, are the changes between pre- and post-survey responses not true across-the board and only for respondents of a particular gender, age range, or marital status? Are the factors driving a decrease in respondents’ social trust different from the factors leading to an increase in respondents’ social networks? Etc.

Determining data weighting: Prior to your analysis and reporting on your results, you should figure out whether you intend to summarize the data into certain indices showing program impact. This is something that you may want to consider back at Step 5 (“Constructing the questionnaire”); your results will have more integrity if you form this weighting prior to looking at the data, rather than first entering the data and then seeing which weightings of the data and variables show a significant improvement or prove your hypotheses. For example, you might decide that you want to combine 10 questions and have them each count equally. Conversely, maybe 20 questions are important, but 3 of them (for example on bridging social capital where the program put its emphasis) should count much more than others.

Report on your analysis: If you are sharing the results of your evaluation with an external audience (funders, policy makers, the surrounding community, etc.) and/or quite possibly if you are sharing them with program staff, you will need to write up your findings in a report. [For a very brief checklist of some things to consider in writing a report, click here.] If you are sharing this with external audiences, you probably want to write a draft version of this report, circulate it to program staff and get their reaction, before revising it and sharing it externally. You’ll have to decide how much of the findings you are willing to share with the external audience, although generally it is less ethical to only report on those items showing social capital improvement and not to report items showing no change or even negative social capital impact.

Assuming you decide to do a report, it should probably contain some or all of these sections: Executive summary / Intro: A quick summary of why you undertook the evaluation, how you

measured the impact, and your headline findings. Program description: a thumbnail description of what your program does, the mission of your

organization and the population it serves. Goals and objectives of study: why you undertook the study, and what specifically you were

attempting to measure (for example, whether children in a school who were randomly assigned to participate in a mandatory service learning program showed higher trust and stronger friendship networks than ones who didn’t).

Research design: how you asked the questions, who you surveyed, etc.o Data collection procedures/Methodology: how you went about gathering the data (for

example were the surveys done by phone, in person, mailed, or self-administered), where did you survey individuals; when; who collected the data (an outside firm, trained volunteers), how many people were surveyed, whether and how confidentiality was

DRAFT28

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

maintained; who entered the survey; any incentives given to respondents to participate; how many follow-up calls were made for hard-to-reach individuals; etc.

o Instrument: what questions you asked of your study and control groupso Sampling plan: how you selected individuals for your study and control groups

(assuming that you didn’t survey everyone in that group) Data analysis/results: what were the more interesting findings (good and bad) from the data, and

in the report (or an appendix) the change you observed on all of the questions asked (showing means, and possibly standard deviations for all questions pre- and post-survey, and with the corresponding pre- and post- means for the control group if you had one)

Implications/ recommendations for programmatic change: (see immediately following discussion)

The final stage in program evaluation is making recommendations, which can be politically charged, and is one advantage of using external researchers. We recommend that a lot of dialogue take place between the research team, agency staff, and, if possible, the program participants, in drafting the recommendations. You should consider whether a potential new program element ought to be expanded (if it is showing the impact you expected), retrofitted (if it is not showing the impact you hoped for in places or at all), or possibly discontinued.

If the data show findings different from what you expected (for example, you observed expected large increase in volunteering, but unexpectedly found no increase in bridging social ties), you might think about whether there are things that your program could be doing to make it easier to forge such relationships. Some sample questions in such a case that you might consider are:

Is this an issue about the composition of your program participants (or staff) or the types of partners with whom you work?

Would having smaller diverse groups as part of your program facilitate this? Are there social activities that you could foster that would make it easier for these ties to form?

Depending on your survey, this process of thinking about program recommendations may involve multiple steps. Looking at and discussing the data may raise additional questions that you may be able to go back to the survey data to test. If, for example, you are not observing a big increase in political participation among participants, this may raise additional questions. Does it depend on how many program sessions the participant attended? Does it depend on whether the participant had a mentor in the program? Did women and men or rich or poor participants show different trajectories? All of these questions can help tease out what is going on in a program and suggest why things might or might not be working as you thought.

[Do we need to discuss other statistics concepts like: Reliability change index, Proportion of variance, Myths about statistical significance, Understanding trends, Using statistics]

Ongoing example: learn how Jumpahead analyzed the results?

STEP 7: REVISING PROGRAMS

The ideal evaluation process is a circle. Programs get formally or informally evaluated, and an organization gets valuable feedback on what parts of the program are or are not having the intended effect. On the basis of this data feedback, the programs are revised (new program elements are offered, a

DRAFT29

COMMUNITY ORGANIZATIONS AND SOCIAL CAPITAL A GUIDE TO PROGRAM EVALUATION

part of the program is discontinued, staff are trained differently, participants or partners are recruited differently, parents are engaged differently, etc.).

With these changes to the program comes a need to test whether these revised programs then meet the objectives through additional evaluation, and this data in turn suggests new revisions. And the cycle continues.

We think it is highly useful, in advance of doing the evaluation, to discuss internally or with your evaluator, how you envision the results might affect your actions and programs. While you can’t anticipate what the data will tell you, this up-front discussion will increase the likelihood that your program evaluation will lead to useful changes for your organization.

If your program learns that a program is having exactly the impact that it hoped for or expected, you can trumpet this to funders, potential staff members, and policy makers. You can also think about how to build upon this success. Would it work to expand the number of program participants or to expand to another location? Are there additional program enhancements that you want to try as well? Was there another program that you had hoped to evaluate but were unable to that you want to evaluate in the next cycle?

Ongoing example: learn how Jumpahead used the results?

FEEDBACKWe hope that the Social Capital Program Evaluation Guide has been helpful. Software developers label first versions “1.0”, and this is definitely a “1.0” Social Capital Program Evaluation Guide. We hope that you will provide us with your feedback: what parts of this guide worked well, ideas for improvements, good case studies or examples to add, additional glossary items, etc. With your help, we can improve on this in a 2.0 version incorporating your recommendations.

DRAFT30