Findings from the National Evaluation Of Child Welfare Training Grants:

Lessons and Implications

Mary Elizabeth Collins, MSW, Ph.D.Maryann Amodeo, MSW, Ph.D.Cassandra Clay, MSW, EdM.

Boston University School of Social Work

May 2007

Grant #9OCT0124 funded by U.S. DHHS, Children’s Bureau

National Evaluation of Child Welfare Training Grants

Funded by the Children’s Bureau (Oct 2003- Sep 2006)

Large project: multiple data collection tasks and several research questions

Overall goal: Greater understanding of federally-funded CWT and how to use information from the evaluation to improve our training strategies.

National Evaluation of Child Welfare Training: Four Main Components

Case studies of 9 previously federally funded training projects – Independent Living

Comprehensive review of child welfare training literature

Survey of faculty of (mostly) schools of social work

Survey of state (and some county) child welfare training administrators

Update on Four Components

Multiple case study of IL projects – DONE, report available

Review of child welfare training literature – DONE, report available

Phone interview of state child welfare training directors – 90% done, response rate=48/51, great data!

Mail survey of faculty – 60% done, N=97; response rate=54%

Conceptual Model CONTEXTUAL FACTORS TRAINING PROJECT

ACTIVITIESINDIVIDUAL PROJECTOUTCOMES

CLUSTER OUTCOMES LONG TERMOUTCOMES

State Agency Nature of collaboration Level of personnel

involved Organizational

constraints Training policies Project match with

agency need Children and Family

Service Reviews

Children’s Bureau RFP Grantees’ meeting

Grantees’ Organization Level of support Experience with CWT Seniority of project staff

State-Level Issues Politics Funding Regulations and laws County, Tribe or

community issues

Other (media, role oftechnology)

Develop Materials Curricular

development skills

Deliver Training Trainer skill Number of trainees Cost Effectiveness

Involve Youth Planning training Delivering Training Other

Evaluation of Project Process Outcome

Disseminate Findingsand Materials Presentations Articles/reports Curricula/tapes

Collaboration Development

Other Project Activities

Worker Skills Knowledge Attitude Behavior

Supervisor Skills Knowledge Attitude Behavior

Training of Trainers Knowledge Attitude Behavior

Agency Culture Youth focused

Impact on Youth Skill development Empowerment

KnowledgeDevelopment(evaluation)

Knowledge Sharing(dissemination) Institutionalization of

curricula

Collaboration Institutionalization of

collaboration

Other Outcomes

Literature Review Survey of Schools of

Social Work Survey of Child Welfare

Administrators

Improvement in AgencyPractice

Improvement in School ofSocial Work Youth-focused curricula

Improvements in the ChildWelfare Field (privateagencies, ancillaryagencies, child advocacyorganizations)

Improvement in thePositive YouthDevelopment Field

Other Outcomes

Improvement in Child,Youth, and Family Well-being

Recommendations from Case Study Report: Children’s Bureau, Future Grantees, Public Agencies

Youth/Consumer Involvemente.g., Public agencies have been making progress in emphasizing more partnership with services users; training initiatives and strategies are another mechanism by which agencies can engage consumers in their work.

Evaluatione.g., Public child welfare agencies need to cooperate with the evaluation task. This will include allowing evaluators to conduct follow-up with trainees from the state agency. Examination of the transfer of learning to the agency setting and mastering skills in the work setting is sorely needed. This will nearly always require follow-up in the practice setting which will include data collection involving interviews, observations, case record reviews, and other methods. Additionally, access to comparison groups may be needed.

Collaboratione.g., Grantees should aim to establish collaborations with organizations rather than individuals to provide more stability to the collaborations. Collaborators should be chosen in part to facilitate long term institutionalization. Thus, at the start of projects, active advocacy should occur to secure organizational collaborators who will share the work and responsibility for outcomes

Recommendations from Case Study Report: Children’s Bureau, Future Grantees, Public Agencies

Institutionalizatione.g. The Children’s Bureau should aim to fund the type of training projects that are of critical need to public child welfare agencies. If the funding priorities of the Children’s Bureau are not aligned with the needs of the field, public child welfare agencies have no reason to engage in long-term institutionalization of training programs. After projects are completed, the Children’s Bureau is the only entity with appropriate infrastructure to keep the products that have been developed at the forefront of child welfare practice.

Knowledge Developmente.g., Grantees should recognize that their projects are opportunities to develop learning about the field of child welfare training that can and should be shared with wider constituencies. Although “lessons learned” are typically requested as a part of project reporting, the content is generally thin and lessons are not shared. Grantees should think conceptually about the core lesson of their project and disseminate the contributions via conference presentations and journal articles. These should be less focused on promoting projects and more focused on linking project innovations to the wider field of child welfare training.

Findings from the Literature Review Adult Learning Theory

e.g., Organizational realities that are barriers to training based on these theories need to be confronted head-on. These barriers include time and space for sophisticated training; professional orientation toward workers that recognizes their expertise; and development of a learning culture in which organizational learning and development is the norm. Without addressing these barriers, training programs will have limited impact.

Training Implementatione.g., The field should move toward the development of training systems rather than training courses. Training systems infuse training with an organizational context and have greater potential for viewing training holistically, with connection to achievement of organizational outcomes. This will add to better conceptualization of the purposes of training and the anticipated linkage of training content and training activities with expected outcomes

Findings from the Literature Review Agency/University Partnerships

e.g., As with other forms of training, better evaluation is needed to assess the results of partnership efforts. Most of the literature emphasizes the benefits of partnerships, but there are limitations. Sometimes these are discussed as logistical problems or the melding of academic and practice cultures. Partnerships may limit the voice of schools of social work to critique child welfare practice and policy, and may cause schools to shift toward vocational education rather than professional education.

Evaluation of Traininge.g., The methodology for evaluating outcomes related to knowledge and attitude change are fairly well developed and easily implemented. The measurement of skill development is more complicated and requires more methodological sophistication. Measures such as case plans and action plans should be more fully developed for use in training evaluation. Also observational methods and measures should receive attention so that the field can begin to measure the interaction of worker and client.

Survey of State and County Child Welfare Training Administrators -- Content

Main challenges agency faces in providing effective CWT Experiences with federally-funded CWT

Description Strengths/Weaknesses Impact

Involvement with IV-E partnerships Description Strengths/Weaknesses Impact

State/county funded training programs

Survey of State and County Child Welfare Training Administrators -- Content

Administrative supports for training Impact of CFSRs on trainingExisting mechanisms to institutionalize trainingRelevant historical factors that have influenced CWTFuture initiatives in CWTTraining evaluationWhat could federal government do to help states and

counties deliver effective CWT?What should be federal priorities (topics) of CWT in the

next few years?

Survey of State and County Child Welfare Training Administrators: Training Evaluation

Many (n=18) refer to “levels” of evaluationLevel 1/satisfaction, n= 8; “Would like to do more”Level 2/pre-post change, n=13 with some moving

toward more complex evaluation for some training.Additional evaluation steps e.g., longitudinal follow-up,

n=20Most developed training systems, n=4

Survey of State and County Child Welfare Training Administrators: Training Evaluation

“The success of training is first measured when trainees take a test. … followed by a field based assessment piece, which is left up to various supervisors. The supervisor’s evaluation includes an evaluation form, document review and observation of workers in various case management activities. Also an online tracking system, which tells them who passes competencies and other information.”

Survey of State and County Child Welfare Training Administrators: Training Evaluation

“Evaluation method varies based on the kind of program. Some Level 1 (happy sheets) that are standardized. Available electronically—when a course is offered, evaluation information is put in immediately, can access it. Use pre/post knowledge test for basic training and all specialty programs. Occasional attempts at level 3 & 4 for very special situations; it’s expensive and hard. Also have a system to do field evaluations – conduct an on-site field review, either scheduled randomly or a project manager requests it.”

Survey of State and County Child Welfare Training Administrators: Training Evaluation

Utilization:

Generic statement about reviewing results (n=9)

Primarily to review courses and trainers (n=23)

With information system to tract participation, link with retention, overall quality assurance (n=12)

Survey of State and County Child Welfare Training Administrators: Training Evaluation

The information is compiled and examined quarterly. It is then utilized for the following: 1. to look at trainers; 2. are workers getting what they want? (i.e., if they need to change curriculum); 3. how is the environment? (are environmental changes needed). Future plans include a practice reform initiative with [the University] which will have its own formal evaluation and they are working towards developing a model that will include behavioral anchors for new workers.

Survey of Colleges/Universities: Relationship with Child Welfare Agency

N=97

Your Relationship Your School’s Relationship

Quality 3.46 3.16Amount 3.22 2.97Mutuality/Reciprocity 3.09 2.83Productivity 3.13 2.82Desire for Future Collaboration 3.73 3.44

Scale: 1 (low) – 4 (high)

Survey of Colleges/Universities: Perceived Impact of Federally-Funded Projects (N=30)

Moderate/Strong

n percentRecruitment of new CW staff 15 50%Development of current CW staff 26 87%CW worker knowledge 30 100%CW worker attitude 23 85%CW worker skills 27 90%CW supervisor knowledge 22 76%CW supervisor attitude 16 59%CW supervisor skill 18 69%Macro level CW policy/practice 11 47%

Triangulation: Common Findings

Reluctance of state agency to fully support training and its evaluation

Potential and pitfalls of partnerships between agencies and universities

“Training” knowledge dominated by curriculum development and training delivery; expansion needed in evaluation, organizations, and systems

Training knowledge dispersed in variety of fields and ranges from highly micro (e.g., instrumentation) to macro (e.g., policy implementation)

Discussion Topics

Relative roles of faculty, state/county training agencies, and the federal government in advancing the state of training.

Enhancing the knowledge base of child welfare training.Addressing conceptual and methodological challenges

to conducting training.Assessing what is known and unknown in child welfare

training.Challenges of linking training outcomes to performance

outcomes.

For more information, contact:

Mary Elizabeth Collins, Ph.D.Boston University School of Social Work264 Bay State RoadBoston, MA 02215617-353-4612mcollins@bu.edu