No Place Like HomeLarimer County Evaluation Training

General Staff Survey

• Please open your browser to: https://www.surveymonkey.com/s/COSTAFF

• Read the informational page, and if you agree to take the survey, please click “Next” to begin

• All of your responses are CONFIDENTIAL• It should take approximately 15 minutes to

complete

NPLH Summary

• Evaluation of three child welfare agencies’ use of family meetings with families receiving in-home services– Process

• What does practice look like?• Tools: focus groups, observations, surveys

– Outcomes• Child welfare placements (number and type), re-

reports, recurrence of maltreatment, etc.• Family functioning and protective factors• Satisfaction and service provision• Tools: administrative data, surveys

Your role…

• Is critical!• To provide data and feedback about your

experience working in child welfare and with family meetings– As part of a federally-funded study that will be

disseminated nationally• To recruit families to participate, so that they

can provide data and feedback about their experience

• To participate in the feedback loop – ongoing communication

Who What is the value?

…the field •Research is mixed and the evidence-base of FGDM is moderate—more research is needed•Additional research can support the national spread of family meetings or suggest the need for alternative practices

…families •Families receive interventions with known effectiveness•Opportunity for families to provide input about their experiences with CPS

What’s in it for…?

Who What is the value?

…CPS Staff and Agencies (You!)

•Accountability. •Opportunity to get input from staff who know the most about the families (You!)•Knowledge of effectiveness of family meetings can increase investment in the practice•Visibility and respect for advancing the field and systematically examining internal practices•Increased knowledge about the use and potential of evaluation•Results can impact system and practice change

What’s in it for…?

Human Subjects

• Overview–History of human subjects research–Historical laws and documents

protecting human subjects–Your role in protecting the rights of

participants*

Human Subjects History

Before there were human subjects protections:

• Nuremberg (1930’s – 1940’s)– Concentration camp prisoners, twins

• Willowbrook Hepatitis Study (1956)– Mentally retarded, institutionalized children

• Brooklyn Jewish Chronic Disease Cancer Study (1963)– Poor, eldery

• San Antonio Contraception Study (1971)– Mostly poor, women

• Tuskegee Syphilis Study (1932-1971)– Poor, black males

Human Subjects History

Attempts to protect human subjects:• The Nuremberg Code (1947)

– Informed consent. Benefits outweigh risks. Voluntary participation.

• Declaration of Helsinki (1964…2000)– “Concern for the interests of the subject must always prevail

over the interests of science & society”

• National Research Act (1974)– Tuskegee Institutional Review Board (IRB) approval required if

getting U.S. Dept. of Health and Human Services funds

• The Belmont Report (1979)– “Justice, beneficence, respect for persons”

Our Role

Our Role in Protecting Participants Rights

• Give participants all the information that they may need to make an informed decision.

• Ensure that participants understand the information• Give participants an opportunity to consider

involvement in the project on an initial and ongoing basis

News You Can Use

• What’s important to know:– Risks to you personally are minimal– There are potential benefits to children, families, and

to staff in the agency – Your participation is voluntary

• Privacy and Confidentiality– All survey information is confidential and used only

for the study– Survey data is maintained by Kempe Center NPLH

evaluation team– Individually identifiable information will NOT be

shared outside of the Kempe Center evaluation team

Evaluation Terms

• Evidence-Based Practice• Confidentiality vs. Anonymity• Bias• Model Fidelity• Validity• Process vs. Outcome evaluation

– How vs. What• Qualitative vs. Quantitative methods

– Depth vs. Breadth

Evaluation Design - CO

General Staff

Survey

Families meet eligibility criteria and are notified

about the study; Caregiver Survey

Group Assignment – Propensity Score Match

Meeting participants

notified about the study; Fidelity

Survey

Follow Up Survey Follow Up Survey

Case-Specific Questionnaire

Right now

Intake or FAR worker makes 1st FSRT or FUM referral

FUM/FGC

1-2 months post-FUM/FGC

Ongoing case closure/transfer to out-of-home services

NPLH Surveys

1. General Staff Survey

2. Caregiver Survey

3. Meeting Fidelity Surveya) Participant

b) Facilitator

4. Case-Specific Questionnaire

5. Follow-Up Surveya) Caregiver – Intervention

b) Caregiver – Control

c) Meeting Participant

Surveys that are distributed TO you

• General Staff Survey–All staff who have a role in the

evaluation (Ongoing, Intake, FAR workers, facilitators, supervisors)

• Facilitator Meeting Fidelity Survey–Facilitators only

• Case-Specific Questionnaire–Ongoing workers only

General Staff Survey

• Purpose:–To understand more about:

• Who utilizes family meetings in CPS• How much experience staff have with

family meetings• How staff perceive family meetings• The context in which meetings are

happening.–We can use this information to inform

our process and outcome evaluations

General Staff Survey

• Who? You!– Case workers (Intake, FAR, Ongoing)

providing in-home services– Family meeting facilitators– Supervisors of both

• When? Now

• How? Via Survey Monkey

Facilitator Fidelity Survey

• Purpose:– To understand more about:

• How, why, and when family meetings are used• Who is invited to and who attends family

meetings• Other characteristics of family meetings

–Fidelity–Logistics

–Meeting fidelity is part of our process evaluation

Facilitator Fidelity Survey

• Who? Meeting facilitator• When? After the family meeting

(FUM or FGC)– So as not to over-survey you, only the first 2

FUMs held during an in-home case will be surveyed (and all FGCs)

– Once back at your desk/computer (same day or day after)

• How? Via Survey Monkey

Case-Specific Questionnaire

• Purpose:–To understand more about:

• What services in-home families receive• Are services provided impacted by family

meetings?–Services provided is one of our outcome

evaluation variables

Case-Specific Questionnaire

• Who? Ongoing workers• When? At case closure OR transfer from

in-home services to out-of-home services (foster care, kinship care, etc.)– Transfer does NOT mean voluntary placements– It does mean a formal or legal change in

custody, as indicated in TRAILS• How? Via Survey Monkey

– Deb Bowen will send you a ‘tickler’ or reminder

Surveys that are distributed BY you

• Caregiver Survey– By Intake or FAR workers– At face-to-face meetings with family– At point of referral for the first family meeting

(FSRT or FUM)• Participant Fidelity Survey

– By facilitators– At in-home FUMs and FGCs

What you ARE responsible for

To distribute:1. Information Sheet

• Explains project to participants.• Formal/signed consent is not required but participation is voluntary.• What is voluntary? Completion of surveys NOT participation in child

welfare services.

2. Surveys• Paper and pencil surveys with

– Caregiver Survey – self-addressed and stamped envelope with a flyer with a phone number to call for assistance

– Meeting Participant – large envelope for all surveys in the center of the room/table

To pre-fill:

-Assessment or Case ID fields before distributing Caregiver Surveys or Participant Fidelity Surveys to families/meeting participants.

What you are NOT responsible for

• Explaining the content of surveys to families– People with questions can call the number provided

on the Information Sheet (have extra copies available for them to keep)

• Helping families/meeting participants fill out surveys– Other household members or family supports, etc.

can help. • In order to maintain confidentiality, it is

important that you merely handout surveys and provide no more information than what is included on the information sheets.

Caregiver Survey

• Purpose:– To understand more about:

• The characteristics of families receiving in-home child welfare services (e.g. demographics)

• The protective capacities of parents • The parent-child relationship and child behavior

– This information can be linked to outcomes (through TRAILS Case IDs)

Caregiver Survey

• Who? Parents or legal guardians• When?

– Face-to-face meeting– At time of referral for first family meeting (FSRT or

FUM)• How?

– Paper and pencil– Self-addressed and stamped envelope– Distributed by Intake or FAR worker– Parents can call the number on the information

sheet to take the survey via phone

Participant Fidelity Survey

• Purpose:– To understand more about:

• How, why, and when family meetings are used• Who is invited to and who attends family meetings• Other characteristics of family meetings

–Fidelity–Logistics

– Meeting fidelity is part of our process evaluation

– We want to hear from families and other participants as well as staff.

Participant Fidelity Survey

• Who? Any meeting participant who agrees to fill it out

• When? AT the family meeting (1st or 2nd FUM or FGC)– Section 1: Before the meeting starts– Section 2: After the meeting ends

• How? Paper and Pencil– Meeting participants will fill out paper-pencil surveys

and place them in a large envelope, to be sealed at the end of the meeting

– The facilitator will collect them in a safe spot to be mailed to the evaluation team

Surveys administered by evaluation team

• How? Paper and pencil; via mail• 3 follow-up survey versions with the

following components:– Meeting Follow-Up

• Who? All meeting participants who filled out the initial Meeting Participant Fidelity Survey

– Caregiver Follow-Up• Who? All (intervention and control group) parents

– Family Satisfaction• Who? All (intervention and control group) parents

AND kin meeting participants

NPLH Survey Summary

• Case workers:– Complete:

• General Staff Survey (1 time)• Case Specific-Questionnaire (Ongoing - for every study case)

– Distribute:• Caregiver Survey (FAR/Investigations - to every study case)

• Facilitators:– Complete:

• General Staff Survey (1 time)• Facilitator Fidelity Survey (for every study family meeting)

– Distribute and Collect:• Meeting Participant Fidelity Survey (for every study family

meeting)

Where can I find survey materials?

Receptionist

2573 Midpoint • You can find extra survey materials here

– Caregiver surveys with self-addressed, stamped envelopes

– Extra information sheets– Meeting Participant fidelity surveys and envelopes

• Facilitators can drop-off completed participant fidelity survey packets here after meetings for bulk mailing at a later date

• When you are running low on materials, we will deliver more

When can I find them? When do we start?

October 22nd, 2012!

THANK YOU!

If, throughout the life of this

project, you have any questions

please contact:

Heather Allan, NPLH Project Coordinator

Email: heather.allan@childrenscolorado.org

Phone: 303.864.5134