Inclusion Outcomes and Indicators of Success

Holly MatulewiczInstitute for Community Inclusion, ICI(617) 287-7640Holly.Matulewicz@umb.edu

Lucy BayardNational Service Inclusion Project, ICI(617) 287-4355Lucy.Bayard@umb.edu

www.serviceandinclusion.org

Toll-free hotline: 888-491-0326 (voice/TTY)

Training and Technical Assistance

Values that Guide & Current Best Practices on Inclusion Person-first Language & Inclusion EtiquetteWho are People with Disabilities? Who is a "Qualified" Person with a Disability? To Disclose or Not to Disclose – What is the Impact? Creating an Environment that Encourages DisclosureHow to Conduct Effective Outreach & Recruitment to People with DisabilitiesInterviewing: Welcoming & Appropriate Questions & StatementsUnderstanding the ADA & Section 504Beyond the Laws: Creating a Welcoming, Inclusive EnvironmentAccommodation Strategies & Adaptive Products: No Tech, Low Tech & High TechDemonstrations & "hands-on" activities of how to use Accommodation ProductsHow to develop Inclusive Service Descriptions How to Ensure Access for Everyone: architectural, programmatic, communication, &

alternative format access Member & Volunteer Management - supervising & performance concerns Recruitment & Networking Resources: Disability Organizations and AgenciesPreparing for Accessibility Site VisitsReasonable Accommodations: definition & responsibilities Disability Benefits: Supplemental Security Income & Social

Security Disability IncomeSupporting Inclusion through Community Asset Mapping Emergency Preparedness for People with Disabilities

Objectives:

To develop an understanding of:

• Indicators of successful inclusion• Addressing program issues with data- driven management• Measuring constructs• Introduction to quantitative measure• Introduction to qualitative measures

What is inclusion?

Inclusion means that all people, regardless of their abilities, disabilities, or health care needs,

have the right to:

• Be respected and appreciated as valuable members of their communities

• Serve as a member or volunteer in Senior Corps, AmeriCorps or Learn and Serve America programs

• Work at jobs in the community that pay a competitive wage and have careers that use their capacities to the fullest

• Participate in service learning opportunities with peers from elementary school through college and continuing education

What are indicators of successful inclusion?

• Individuals with disabilities are actively recruited and welcomed at the organization / service site• Availability of accommodations is openly publicized and reasonable accommodations are provided upon request• Individuals with disabilities assist in reviewing materials & policies

What are indicators of successful inclusion?

• Buildings and programs are accessible• Interviews, meetings, events and social

gatherings are held in accessible locations• Individuals are asked about their

experience and satisfaction• Individuals evaluate the effectiveness of

products and strategies• Materials available in alternative formats such as large print, electronic, Braille etc.

What does successful inclusion look like?

Members in leadership roles:

• In addition to her service, Anna, an AmeriCorps member, serves in a leadership role on Nebraska’s InterCorps Council, which connects all AmeriCorps programs in the state. She conducts peer-grant reviews, and plans events and statewide service days.

• Justen has been an AmeriCorps*VISTA for two years and, in his third year, has become a VISTA Leader in his Corps. He provides support to 32 members and fulfills the goals of the organization.

Presentation Overview

• Addressing program issues with data-driven management

• Measuring constructs

• Introduction to quantitative measures– administrative data

– surveys

• Introduction to qualitative measures– 1:1 Interviews

– focus Groups

– field Observation

Part I. Addressing program issues

with data-driven management

What is “data-driven” management Integration of data into

your management practices

• Setting measurable goals & measure progress towards them

• Setting benchmarks or standards

• Using data in presentations to staff, funders

Why is it useful?

• Make decision based on evidence not instinct, assumptions, or perceptions

• Have more information to use for analyzing issues / developing solutions

• Helps managers and funders see big picture, accountability for outcomes

• Helps identify trends over time

• Provides benchmarks for staff

Using data-driven management

• How am I already using this today?

– Do your goals have measurable outcomes?

– What data do you use to measure progress?

– How often do you review data for:• Member/ volunteer level outcomes• Agency-level outcomes• State-level outcomes

– How often do you share these data with• Members / volunteers, staff, agency upper

management, funders

• How can I do more in the future?

How many programs can answer these questions?• Using the data you collect now – can you report on:

– # of members / volunteers in each program

– # of partnerships made between National Service and Disability organizations

– # of applications for National Service of persons with disability

– # / Type of recruitment efforts to disability community

– Disability Organizations – type / volume of info shared about National Service opportunities

– Change in these numbers over time

Strategies for becoming “outcomes driven”

• Focus on data that matter to you

• Nurture the “inquisitive mind”

• Help others see the benefits of using data

• Build systems / procedures for enhancing data quality

Part II. Measuring constructs

Building Blocks of Data-driven Management

• Data is not as scary as people may think!

• To gather data efficiently and effectively – follow some basic steps:

1. Identify the “construct” to measure.2. Conceptualize the construct. 3. Operationalize the definition. 4. Develop method for: collecting, entering,

analyzing, and reporting these data.

Step 1. Identify the construct to measure• Looking at your member / volunteer, staff, and

agency outcomes:– What questions do you want to answer?– What information is needed to answer the question?

• Classifying units of analysis (say individuals) into categories (satisfied with course not satisfied with service experience).

• Often times we want to “measure” things that in and of themselves are intangible in the social world (“satisfaction” with course, “quality of life,” etc).

Step 2. Conceptualize the construct

• Process of specifying what we mean when we use particular terms. Begin by clarifying what we “mean” by a concept.

– Example: “Quality of experience” among members / volunteers.

• What does that “mean” to those interested in measuring it?

• What does that look like? What are examples of it?

• Produces an agreed upon meaning for a concept for the purposes of research.

Step 3. Operationlize your definition• Things like “satisfaction with life” or “fear

of crime” are hard to measure directly, so we have to make inferences.

• Process of defining specific ways to infer the occurrence of specific phenomena.

• Indicators are observations we think reflect the presence or absence of the phenomena to which the concept refers.– How do we “know it when we see it” or “when

someone experiences it”

As you develop your measures

• Good research / evaluation strives for both reliability and validity. Applies to qualitative and quantitative measures.

– Reliable: Using your method – could others replicate your research and get similar results?

– Valid: When operationalized – will your measure “truly

show” the concept you want to study?

Earl Babbie, Basics of Social Research, 2002

Step 4. Select a data collection strategy• Once you have gone through these steps:

1. Are you already gathering these data?2. If not - select a data collection strategy

• Today we will briefly cover 5 strategies to collect data:

– Quantitative: case record abstraction and surveys– Qualitative: 1:1 interviews, focus groups, field

observation

• Meant as introduction to key tasks – not exhaustive summary!

Two kinds of data: Qualitative and Quantitative• Data come in many shapes, sizes, and formats.

Distinction is between numerical (quantitative) and non-numerical (qualitative) data.

• Different data will be needed to:– Answer different types of questions– Measure different kinds of outcomes

• These two types of data– Require different data collection and analysis techniques. – Both are useful and valuable tools. – Each have advantages and disadvantages.

Qualitative Vs. Quantitiave Data

• Most all data start out as initially a qualitative measure - must be quantified so that the researcher can perform statistical analysis.

• Much controversy about whether using qualitative or quantitative methods and data will “prove the point” better.

Reporting on your data

• Determine your audience and craft report accordingly. – Who will hear or read this report - how much background knowledge do

they have?– What questions will they bring to the table?– What topics are of most interest to them?– What do you want them to “take away” from the findings

• Present your question or point of evaluation clearly for your audience. – Use jargon / lingo appropriate to the audience: Same data may be

packaged differently.

– Identify themes, recurring ideas, or common experiences

– Ensure the report is true to the data – not just highlighting those supporting your ideas

– Map out key findings visually (charts / graphs)

• Clearly state your data collection processes.– Methods transparent: audience should be able to use your process to

replicate results. – Point out any qualifications or conditions (shortcomings).

Part III.Introduction to quantitative measures

Quantitative Data

• In numerical format naturally – monetary values, counts, dates

• Coded / assigned numerical values for analysis – placement= 1, non-placement=2– coding into categories with numerical representations (e.g. 1 = yes, 2

= no).

• Used for: – Describing information in aggregate, identifying trends overtime,

quantifying outcomes, conducting statistical analysis

• Able to be analyzed using statistics

• Examples include:– # clients served at agency, # staff, % of staff with BA degrees, % of

clients placed in jobs, agency cost per client served

Advantages & disadvantages of using quantitative data

Advantages• Aggregate large volume of

data

• Numbers can be “persuasive”

• Track trends over time

• Measure relationship between different variables

Disadvantages• Training / expertise

required to collect, enter, and analyze these data

• May not shed light on the “whole story” or the “why” of a situation

• Participants may feel limited to preset response categories

Administrative Data or Case Record Abstractions

• Collection of existing data from case records or files:– “Abstracting” key variables from the data for

reporting / analytical purposes– Because the data are not collected for research

purposes – files may have missing data or vary in how items were recorded

• Unobtrusive research– study social behavior without affecting it

Value of Record Abstractions / Administrative Data• Minimal costs, effort to gather these data – no burden on

participants

• Provides a snapshot of key indicators for your state or agency

• Gives you ability to aggregate these data for snapshot on progress towards goals / outcomes:

– Recruitment: # or types of agencies tapped for recruitment partnerships, # or type of sites where applications were distributed

– WEBBERS on members / volunteers: gender, age, race, education

– Agencies: # and type of service opportunities members / volunteers placed in

What is a survey?

• Way to collect standardized information from large group of individuals.

• Collection of data from a scientifically selected group of people. Results can be representative of a larger population.

• Data collected are used to address specific issues.

• A standard set of procedures are followed.

Advantages and Disadvantages of Surveys

Advantages• Collecting original data on

population too large to observe directly

• Results can be generalizable to whole population (when using scientific sampling methods)

• Paper / web give respondents the flexibility to return data at their convenience

Disadvantages• Can be extremely costly to

conduct

• Item non-response and unit non-response

• Accounting for sampling bias (based on mode), can leave out some members of the population (reading level, non telephone household, non-English speakers, persons with disability)

4 Modes of Survey Administration

1. Self-administered: Mail

2. Self-administered: Web

3. Interviewer Administered: Telephone

4. Interviewer Administered: In Person

1. Mail Surveys

• Most common mode of survey data collection

• Low in cost

• Response Rates generally low – need multiple waves of follow-up

• Used a great deal in business surveys when directed at specific groups (such as members of professional organizations)

• Who does it exclude?

2. Web Surveys

• New technology, seen most prevalently in convenience samples

• High costs associated with programming, yet once programmed:

– data available immediately– structure quex. In such a way to eliminate item non-response

(benefits / drawbacks of doing this)– Respondents can answer at any time, like paper instrument– Response options can be personalized based on previous

responses

• Currently still LARGE bias in general population using web mode alone, therefore not recommended (alone) for general population study

3. Telephone Surveys

• Most large-scale surveys in the US are conducted by telephone using CATI - improves the quality of the data collection

– Must be tested for correct routing / branches of Qs– Avoids an important error - omissions!

• Can increase cooperation rates.

• Faster, less expensive than in-person interviewing.

• Who doesn’t it reach? – When might this be a problem?

4. In-Person Surveys(Interviewer Administered)

• Presence of interviewer may have effects. – Increase in cooperation– Possible to get immediate clarification on issues in the

instrument– Possible for bias because of interviewer presence

• High quality of data - training the interviewers in a classroom like environment– Good interviewing techniques stressed

• Professionalism• Avoiding bias

• Why might this mode not get used as often?

Regardless of mode: Use Advance / Cover letters• Key Components:

– Explain the purpose of the survey

– Organization sponsoring the survey & any relevant endorsements or supporters

– Lets person know you will be contacting them (or they may contact you) with any relevant details about items needed for survey

– Provide details on deadlines or submission requests

• Write from reader’s perspective: “Why should I participate?

Schedule of survey data collection

• Set up your calendar with mail dates

• Identify total field period (start to finish)– Allow sufficient time to – Prep mailings– Recruit, hire, train interviewers– Design / test web survey– Process survey returns / enter data

• Goal: work backwards from end goal or deadline

Preparing to field your survey

• If interviewer-administered (on phone or in person) you must hire interviewers and supervisors.

– Train them on your survey– Ensure they have basic interviewer training– Specify DC schedule, QC rates, production rates, and

response rate expectations.

• For mail surveys, training staff on schedule, receipt and follow-up procedures

• For all methods: developing QC process check on completion of work, including collecting and editing documents.

Once the quantitative data are collected …• Once the data have been: quality checked, edited, and entered

you can begin your analysis! Your analysis should focus on answering the questions you posed when designing your data collection forms (abstractions or surveys).

• Spreadsheets may be useful to you for– Simple entry procedures– Few case records– Ease of reporting– Disadvantages of spreadsheets?

• Databases may be useful to you for– Simple entry, once entry form is designed, Minimizing entry error– Ability to: link several datasets, program reports into database / create

on-line for field access– Disadvantages of databases?

Part IV. Introduction to qualitative

measures

From survey to ethnography...

• This model of data collection allows for more freedom … not making the respondents feel “boxed in” to prescribed answer categories.

• Using a standardized format implicitly assumes that all respondents will understand and interpret the questions in the same way

• Structure can limit researchers ability to gain in depth knowledge about an issue

Qualitative data

• Qualitative data are non-numerical

• Used for:

– Examining social world through stories, images, and experience

– Probing more deeply into constructs, examining the “how” or “why” types of questions

• Examples include:

– Transcripts from 1:1 or group interviews– Observations made in the field– Pictures, texts

Why aren’t qualitative data used more?

• Capturing and analyzing qualitative data sets has been a tough business.

• Extremely costly process, quite time consuming, often necessitating small sample sizes.

• “Numbers” can be perceived as more persuasive.

1:1 Qualitative interviews

• Interaction between participant and interviewer where interviewer has “general plan” of inquiry – but not set questions

• No specific order of questions

• Interviewer must be well trained, very knowledgeable in subject matter (for probing)

• Essentially a “conversation” but participant does 95% of the talking

Advantages and disadvantages of 1:1 qualitative interviewing

Advantages

• Participants share info in 1:1 format may not share in group

• Allow participant to explore concepts more freely / fully

• Researchers not limited to script or preset response categories

• Great for exploratory work where you may have limited info on topic

• Focus on verbal and non-verbal cues.

Disadvantages

• Relies heavily on skill and knowledge of interviewer

• Costly to implement – per interview costs may limit sample size

• Due to small number, limits to generalizability

• Large volume of data to transcribe / analyze

Focus groups

• Group interviews - as they are like in-depth interviews– Guided discussion on topics of interest

• Purpose is to explore rather than describe or explain in a definitive sense– Group of 7-12 people too atypical to generalize to whole population

• Very flexible form of DC, allow participants to frame answers and construct meaning as they wish

• Examples: – member / volunteer service experience: successes, challenges– application experience: how heard of opp, why appealed, app

process– retention issues: why left service, what can be changed– agency partnerships: quality of service provided by members /

volunteers

Advantages & Disadvantages of Focus Groups

Advantages

• Socially oriented research method

• Flexible – group may raise topics researcher didn’t foresee or anticipate

• Speedy results

• Low in cost

Disadvantages

• Less control than individual interviews. Tendency to produce “group think” where people may not readily express ideas that deviate from group’s.

• Data can be difficult to analyze.– Difference between groups can

be troublesome.

• Moderators must be skilled and discussion must be conducted in a conducive environment.

• Groups are difficult to assemble.

Field Observation

• Methods of collecting data on people, likely in their natural settings.

• People from somewhere going somewhere else & sharing what they find

• “Informant” who gives you your data (like a narrator)

– Participant observation: performed by those who take part in the activities they observe. Gain “verstehen” by immersing themselves in the daily lives of those they study.

– Non-participant observation: made by an observer who remains as aloof as possible from those being observed.

Using Field Observation

• Decide on a topic where field observations are appropriate

• Identify your research questions and constructs to measure

• Can include narrative and quantitative measures

• Examples can include:– observation of worksite or agency – observation of member / volunteers with those they serve– Attending recruiting events – observing candidates and

recruiters

Advantages and disadvantages of field observation

Advantages

• Direct observation, rather than descriptions or interpretations (via interviews) from participants’ bias / perspectives

• Data can richly supplement other sources of info

Disadvantages

• Disruption of natural setting

• Time consuming / labor intensive

• Relies heavily on skills of observer

• Can rely on honesty – level of disclosure of informants

• Can have ethical dilemmas (participant vs. non-participant)

• Act of study can change behavior of those observed

How can I assess if my program is inclusive and accessible?

An accessibility checklist provides guidelines for assessing your program(s) to help ensure :

• compliance with the law (Section 504 of the Rehabilitation Act and Title II of the Americans with Disabilities Act)

• how to create an environment that makes people with disabilities feel welcome

• how to design programs and services so that people with disabilities can fully participate

Tips for conducting accessibility checklist at your organization /

program• Involve Program Directors, service site supervisors, and

all relevant staff at site / organization

• Provide an opportunity for members, including members with disabilities, to provide feedback and share their experiences regarding accessibility and inclusion

• Be willing to collaborate with disability organizations in your community to access resources and assistive technology

• Assessing your program and improving areas in need may take time, so it is important to keep it as a priority and be patient!

Thank you!

Holly MatulewiczInstitute for Community Inclusion, ICI(617) 287-7640Holly.Matulewicz@umb.edu

Lucy BayardNational Service Inclusion Project, ICI(617) 287-4355Lucy.Bayard@umb.eduwww.serviceandinclusion.org