February 2012

Camp Fire USA Program Quality Intervention

2011 Report

David P. Weikart Center for Youth Program Quality

2

Contents Findings and Recommendations ......................................................................................................... 3

Findings Summary ............................................................................................................................. 3

Recommendations .............................................................................................................................. 3

Introduction and Background ............................................................................................................. 5

The Camp Fire Program Quality Intervention (CFPQI) ........................................................................ 5

Fidelity and Participation.................................................................................................................... 7

Instructional Quality Improvement ..................................................................................................... 8

Camp Fire PQA Form A - Low Scoring Items ..................................................................................... 9

Camp Fire PQA Form A - High Scoring Items ..................................................................................... 9

Management Practices ...................................................................................................................... 10

Camp Fire PQA Form B ................................................................................................................... 10

Low Scoring Organizational Items – Camp Fire PQA Form B ............................................................ 10

Improvement Planning ..................................................................................................................... 11

Satisfaction with the YPQI Process ................................................................................................... 13

Survey Responses ............................................................................................................................. 13

Summary of Camp Fire Council Executive Interviews ........................................................................ 14

References ....................................................................................................................................... 16

Appendix A*: Summary of Youth Work Methods Course Training of Trainers Implementation ........... 17

Appendix B: Data Table Camp Fire PQA - Form A .......................................................................... 18

Appendix C: Data Table Camp Fire PQA - Form B ........................................................................... 22

Appendix D: Open-ended Survey Responses ...................................................................................... 25

Appendix E*: Camp PQA Summary and Scores ................................................................................ 29

3

Findings and Recommendations The findings below are described in detail throughout the remainder of the report.

Findings Summary

Voluntary participation is growing and implementation is deepening. Approximately 80% of Camp Fire USA

councils voluntarily participated in some element of the CFPQI process. Forty-one percent of councils who

originally ordered a box set completed all of the required data collection and submitted a Program Improvement

Plan. The intervention has been expanded to emphasize improvement efforts. Over 579 direct delivery staff

participated in the live Youth Work Methods Courses. A strong group of advocates for the work is growing and

they will be instrumental in moving forward.

Levels of quality were consistent with national comparison data. Councils appear to offer safe and supportive environments to young people in their programs. Interaction and Engagement domains offer opportunities for improvement. Improvement plans were effectively targeted at the lowest scoring areas on Forms A and B – suggesting that these areas will improve in sites that implemented improvement plans.

Most council staff found the process helpful. Councils reported the process helped to improve motivation and

contributed to a higher level of engagement among participating youth. There is a high approval rate from 80% of

executives from 30 councils who responded to the survey. Many cited the benefits of having a “common language”

of quality with which to discuss specific program improvements.

Some councils struggled with staffing and time. There were implementation challenges in councils with lower

capacity. Some councils reported they felt pressed for time and struggled with staffing limitations when completing

the observations, assessments, and improvement planning. Even larger councils found the timeline caused them to

rush through parts of the process they felt needed more careful consideration.

The Camp Fire USA system is making good progress. While participation in CFPQI is not yet universal, Phase III

has shown good progress in the depth of engagement of many councils and their optimism for the future benefits of

this work. Understanding of the different levels of engagement offers insight in how to target councils to support all

in reaching higher levels of engagement.

Recommendations

To increase participation in the quality improvement system, consider councils’ stated barriers to participation.

Examine barriers to participation and implementation and determine what capacity challenges exist that get in the

way of participation. Through examination of councils’ previous engagement, this will offer the chance to identify

the needs of councils who are struggling and need extra support. It will also offer the opportunity to target efforts at

councils that are on the verge of high engagement to move them into the group that can serve as advocates and

supports for other councils. Consider whether the same approach and model is appropriate for all councils based on

capacity, council structure, and programming. Also, additional emphasis on a year round calendar which includes

other reporting requirements and supports even during non-assessment periods can help to embed CFPQI practice

into Camp Fire USA culture.

Maximize Camp Fire investment in CFPQI through Learning Communities among councils.

Expand and transform available supports and technical assistance from Weikart Center and Camp Fire USA to

create a supportive learning community among councils. Regular calls that provide a space to learn from others will

provide the needed improved communication and opportunities for increased mutual accountability among

councils. Strong councils can be the best advocates for the work and can support struggling councils through these

4

learning communities. Lessons can be learned from TACSS Initiative Technical Assistance and Coaching model in

Michigan and Oklahoma or other mentor models to help in pairing councils to support one other.

Integrate with other data systems and requirements. Work to increase relevancy of CFPQI by aligning efforts at

quality improvement with other evaluation and outcome measures used within Camp Fire USA. Make explicit

connections between PQA data and other data that councils collect, which could be done through leading indicators

framework which includes surveys of youth and families or through including other data sources in CFPQI

improvement planning. This will support and deepen youth and staff experience, especially in supporting councils

to see how the various reporting requirements can be meaningful and useful.

Make local connections to increase networking and training opportunities with existing quality systems. There

are over 65 networks nationwide that are using the YPQI process. Seek ways to create partnerships between

councils and these local systems familiar with the YPQI and the continuous quality improvement. Opportunities to

attend live trainings will provide increased understanding of the CFPQI process and promote staff experience in

seeing that they are part of a quality movement that extends even beyond Camp Fire USA. This will also support

councils in becoming leaders in their local communities.

To increase youth voice and engagement, emphasize efforts on promoting youth participation in program

decision making and governance structures. Focus efforts at improvement in the point-of-service at engagement,

where scores are among the lowest. Attention should be placed to support councils in offering more opportunities

for youth voice and governance. The Weikart Center’s Youth Driven Spaces model offers one approach to

increased youth governance practices. Additionally, councils could increase adult-youth partnership, planning, and

reflection practices to improve engagement scores.

Engage key stakeholders in administrative decision-making. Families and other key stakeholders should be

engaged in meaningful ways to support administrative decision-making. While Camp Fire programs appear strong

administratively, communication with families is shallow and program leaders would benefit from exploring ways

to include these and other important stakeholders in more significant roles, such as through evaluation processes.

5

Introduction and Background

The Camp Fire Program Quality Intervention (CFPQI)

The Camp Fire Program Quality Intervention (CFPQI) is based on the Youth Program Quality Intervention (YPQI)

model. The Youth Program Quality Intervention model defines instructional quality as a set of practices summarized in

Figure 1 and assessed by the Youth Program Quality Assessment (Youth PQA), a standardized measure of

instructional practices. These practices are drawn from developmental science (Durlak & Weissberg, 2007; Eccles &

Gootman, 2002; Gambone, Klem, & Connel, 2002) and the ongoing research around the Youth Program Quality

Assessment (Youth PQA; High/Scope Educational Research Foundation, 2005; Smith & Hohmann, 2005). The

original Youth PQA (Form A) is composed of four domains, 18 scales (summarized in Figure 1) and 60 observable

items. In addition to these 60 original items, 12 additional observable items specific to Camp Fire National Program

Standards were included as were administrator interview items (Youth PQA Form B) aimed at assessing the degree

to which organizational structure supports staff in their efforts to facilitate high-quality programming. Higher scores

on Form A are associated with higher levels of youth engagement defined as belonging, interest and challenge,

while very low scores are associated with youth disinterest. In addition, programs with high quality instruction

provide youth with opportunities to practice emerging social and emotional skills (e.g., efficacy, communication,

empathy, problem solving) that support success in adolescence and early adulthood. Higher scores on Form B are

thought to be associated with higher levels of staff empowerment to carry out the behaviors identified in Form A as

important staff practices.

The Youth Program Quality Intervention follows the Assess-Plan-Improve sequence depicted in Figure 2 to help

program staff improve the quality of instruction that they provide for youth. While the sequence is designed to produce changes in both policies and organizational settings, the ultimate goal is to improve quality at the point of service – the place where instruction and youths’ program experiences occur. This approach to quality improvement and workforce development is currently being implemented in several thousand agency, school, and community-

based settings in over 38 states.

Figure 1 – Pyramid of Instructional Quality

Figure 2 – YPQI Model

6

This report presents findings from Phase III of the Camp Fire USA Program Quality Intervention (CFPQI) work

that was implemented in Camp Fire USA councils and programs nation-wide, over the course of 2011. In order to

support Camp Fire’s mission to build caring, confident youth and future leaders,1 Camp Fire USA originally

commissioned the David P. Weikart Center for Youth Program Quality (Weikart Center) in 2009 to help build and

implement a quality improvement system for Camp Fire councils using the research based Youth Program Quality

Intervention as a model. Phase I included development of program standards and aligned assessment tools. Phase II,

in 2010, was the pilot rollout of the Camp Fire Program Quality Intervention. Now in its second full cycle of

implementation, Camp Fire USA and its councils have an opportunity to capitalize on their significant investment

in quality improvement. The goals of Phase II were to continue to support participation in quality assessment and

improvement processes and to deepen the experience. Phase III was divided into two assessment and improvement

periods, a spring and summer group. Councils chose which group to participate in, and had access to live and

online trainings, webinar supports, etc. Additionally, opportunities for deepening the experience included options

for conducting external assessment and a larger focus on the improvement phase by rolling out the Youth Work

Methods training series by training a cadre of council based trainers.

1 For more information, see Camp Fire website: http://campfire.org/Mission_and_Core_Values.aspx

7

Fidelity and Participation

In a recent randomized field trial, funded by the William T. Grant Foundation, the Youth Program Quality

Intervention model produced positive and sustained effects on both managers’ continuous improvement practices and

the quality of instruction delivered by individual staff. Notably, these effects were strongest in sites that implemented all elements of the model (Smith et al., in 2012). For field demonstrations like the Camp Fire Program

Quality Intervention, to the extent that both T&TA supports and intervention components are implemented well,

inferences about changes in quality are strengthened. Table 1 describes primary components of the YPQI in comparison to the Camp Fire Program Quality Intervention. The right-hand column provides details regarding fidelity to both the T&TA model and implementation of YPQI practice components. In the Camp Fire Intervention, councils varied in their level of implementation, with some completing as many as five out of the seven primary components of the YPQI and others failing to implement any of the components. Notably, Phase III implementation expanded to include the limited use of external assessment, in addition to the creation and submission of Program Improvement Plans and the utilization of Youth Work Methods trainings (see Appendix A for detailed description of Youth Work Methods utilization) – three important

components of the YPQI process that had not been emphasized in Phases I or II.

Table 1 - Alignment between Youth Program Quality Intervention & Camp Fire Program Quality Intervention

Element YPQI Camp

Fire Notes on Training and Action

External assessment at baseline

(PQA Form A)

Form A external assessments were completed for 9

programs in 4 councils (7%).

Program self assessment at

baseline (PQA Form A)

Form A self-assessments were completed in 123

programs in 38 councils (64%) during the spring,

summer, and fall of 2011.

Program self assessment at

baseline (PQA Form B)

Form B assessments were completed in 33 councils

(56%).

Improvement Planning using self

and/or external assessment ratings

Planning with Data webinars in June and September

2010 used program self assessment data. Program

improvement plans were required to be submitted this

year and 24 councils (41%) were able to complete

them.

Youth Work Methods trainings

(High/Scope Active Participatory

Approach aligned to PQA)

20 people, from 15 councils, were certified as Youth

Work Methods (YWM) Course Instructors. 579

council staff, from 23 councils (39%), have been

trained through live YWM courses. Approximately

1,438 courses total have been completed. Online

courses were also available.

TA Coaching for site managers

(focused on continuous

improvement practices, managers

receive support in the YPQI

process)

Phone and email technical assistance coaching from

Weikart Center staff was available for each Council.

Quality Coaching for staff

(focused on instruction, managers

receive coaching workshop and

support front-line staff through

strengths-based feedback using the

PQA)

8

Instructional Quality Improvement

The aggregate data for the Camp Fire PQA (both self assessment and external assessment) shows a pattern that is typical across domains. Sites tended to score highest in the Safe Environment domain and then progressively lower in the Supportive Environment, Interaction, and Engagement domains (see Figures 4 and 5). This suggests that councils in the Camp Fire Program Quality Intervention are successfully laying the foundation for higher level staff practices and higher levels of student engagement. Furthermore, external assessment scores were comparable to national averages in most domains, suggesting Camp Fire is on par with normative standards, with the exception of the Engagement domain, on which Camp Fire scored significantly lower. However, only four sites were included in the external assessment process, limiting the degree to which such results are being accurately inferred.

Figure 4 Camp Fire PQA Form A-- Overall Self Assessment Observational Quality Scores

Figure 5 Camp Fire PQA Form A-- Overall External Assessment Observational Quality Scores*

* The National Sample scores are derived from use of the original Youth PQA which does not contain the 11 supplement items in the Camp Fire PQA.

4.56 4.37

3.76

3.24

1

2

3

4

5

I. Safe Environment II. SupportiveEnvironment

III. Interaction IV. Engagement

Sco

re

Program Self Assessment N=63 Sites

4.19

3.83

2.99

2.13

4.41

3.84

3.01

2.56

1

2

3

4

5

I. Safe Environment II. SupportiveEnvironment

III. Interaction IV. Engagement

Sco

re

External Assessment N=4 Sites National Sample* N = 913

9

Camp Fire PQA Form A - Low Scoring Items

During the program self assessment process, ten practices were identified that scored a “1” in a substantial

proportion of the observed offerings (see Table 4). Scoring a “1” on the PQA means that particular practice was not

present during the offering. As such, a high percentage in the far right column of Table 4 reflects a clear target for

improvement in the respective instructional practice. Items in scales IV.P (Planning) and IV.R (Reflection)

consistently scored low, suggesting that these areas should be the focus of quality improvement efforts.

Table 4 Low Scoring Items in Observational Assessment

Item/Scale % Scoring

“1” n=123

IV.P.1 Youth have multiple opportunities to make plans 38%

I.A.3 Intentional opportunities for youth to discuss culture 37%

IV.R.1 Youth engaged in intentional process of reflection 37%

IV.R.3 Structured opportunities for youth to present to group 36%

IV.P.2 Planning strategies 36%

III.N.3 Youth have opportunities to lead a group/act as mentors 36%

III .P Ambient Item: Staff guides youth in deliberate planning 35%

III.N Ambient Item: Staff provides youth with opportunities to lead other youth 34%

IV.Q.2 Youth have opportunity to make at least one open-ended process choice 33%

IV.R.2 Youth given opportunity to reflect on activity in two or more ways 32%

Camp Fire PQA Form A - High Scoring Items

Table 5 presents high scoring items from the Form A program self assessment. These are Camp Fire PQA items

where at least 80% of offerings scored a “5,” indicating that the practice is present at a high level. All of the high

scoring items are concentrated in the Safe and Supportive Environment domains. This indicates that youth program

staff members are able to consistently provide emotional and material support to youth and lay the foundation for

cognitive and social development.

Table 5 High Scoring Items in Observational Assessment

Item/Scale % Scoring

“5” n=123

II.F.3 During activities, staff generally smile, use friendly gestures 95%

I.A.2 Mutual respect and inclusion 91%

II.G.3 Enough materials and supplies for all youth 89%

II.K.1 Staff actively involved with youth 89%

II.F.2 During activities, staff use a warm tone and respectful language 85%

I.A Ambient Item: Staff provide a welcoming experience for youth 85%

II.G Ambient Item: Staff are prepared for the activities 81%

10

Management Practices

Camp Fire PQA Form B

Figure 6 summarizes overall quality scores measured by the Camp Fire PQA Form B, which is structured as an

organizational interview and designed to assess the quality of organizational policies and practices in place to

support youth programs. Councils scored lowest in the Youth Centered Policies and Practices domain, which

measures the involvement of youth and families in creating program structures. They scored the highest in the

Access domain, which measures the program’s accessibility to all youth. See Appendix C for the full details of the

Form B data.

Figure 6 Camp Fire PQA Form B-- Overall Organizational Quality Scores

Low Scoring Organizational Items – Camp Fire PQA Form B

Overall, Form B ratings were quite positive. During the program self assessment process, five areas of program

structure scored a “1” on the Camp Fire PQA Form B in 30% or more of the programs assessed (see Table 6).

Scoring a “1” on the PQA means that particular program structure is not in place. The lowest scoring items related

to feedback, evaluation and decision-making structures within the organization.

Table 6 Low Scoring Items - Camp Fire PQA Form B

Item % Scoring

“1” n=62

V.D.5 Youth and families are involved in staff training and evaluation 74%

V.C.1 Youth and staff share decisions about physical environment 50%

VI.K.2 Intentional strategy to increase diverse participation 43%

VI.N.6 Short-term staff participate in self-evaluation 43%

V.D.8 Youth, their families and staff share program governance 30%

3.77 3.98

4.22

1

2

3

4

5

V. Youth Centered Policies andPractices

VI. High Expectations for Youthand Staff

VII. Access

Sco

re

CFPQA Form B N=62 Sites

11

High Scoring Organizational Items – Camp Fire PQA Form B

Table 7 presents organizational items in the Camp Fire PQA in which at least 80% of assessed Council programs

scored a “5,” meaning that the program organizational practice is in place at a high level. The highest scoring items

are distributed among the three domains: (V) Youth Centered Policies and Practices (VI) High Expectation for

Youth and Staff, and (VII) Access, suggesting high overall organizational quality. Last year’s high scoring self

assessment items were concentrated in only two scales: (1) Program Policies Enhance Health and Safety and (2)

Staff Qualification support a Positive Youth Development Focus.

Table 7 High Scoring Items - Camp Fire PQA Form B

Improvement Planning

After self assessment, sites were asked to complete and submit a Program Improvement Plan as part of the Planning with Data training. Improvement planning is often where the process becomes more real and meaningful for sites, as they are able to tailor their goals to the direct needs of their programs. This year 41% (24/59) of participating

councils (i.e. those who ordered box sets) were able to complete their Program Improvement Plans. Sites were asked to identify up to three areas on the PQA (either Form A or Form B) they would like to work on in the upcoming program year (see Tables 8 and 9). Those areas that were popular targets of improvement – such as III.P “Youth have opportunities to set goals and make plans” (Table 6) or V.D “youth and family members have an influence on program” (Table 9) – were consistent with those items that frequently scored low during program self assessment (refer to Tables 4 and 6). Previous studies of the YPQI model have found that, when implemented well, sites tend to improve over time in the areas they select in their Program Improvement Plans (Smith et al., 2008; Smith et al., 2012).

Item % Scoring

“5” n=62

V.A.1 Program administrator has youth development experience and training 91%

V.A.3 Program director has 3 or more years of relevant experience 91%

VI.N.3 Short-term staff participate in self eval/twice each program cycle 85%

VI.O.2 Program admin creates welcoming atmosphere for staff 83%

VII.Q.2 More than 33% of staff have returned 83%

VII.Q.1 More than 75% of staff were in program for entire cycle 82%

VI.M.1 Ratio of adults to youth meet needs of activity and national guidelines 80%

12

Table 8 – Domain and Scale level Form A PQA Items Selected for Improvement by 4C Distinction

Form A PQA Item/Domain Cares Camps Contribute Connects Total

IA. Psychological and emotional safety is supported. 1 1

IB. The physical environment is safe. 1 1 IC. Appropriate emergency procedures are present. 1 1 2 IIF. Staff provides a welcoming atmosphere. 2 2 IIG. Session flow is planned, presented and paced for youth.

1 2 2 5

IIH. Activities support active engagement. 2 2 IIi. Staff supports youth in building new skills. 2 1 3 IIK. Staff uses youth centered approaches to reframe conflict.

2 2 1 5

IIIL. Youth have opportunities to develop a sense of belonging.

1 3 1 5

IIIM. Youth have opportunities to participate in small groups.

2 2 1 5

IIIN. Youth have opportunities to act as group facilitators. 8 2 10

IIIO. Youth have opportunities to partner with adults. 2 1 3 IVP. Youth have opportunities to set goals and make plans. 6 3 1 10 IVQ. Youth have opportunities to make choices based on their interests.

6

1

7

IVR. Youth have opportunities to reflect. 2 6 1 9 IV. Engagement 1 1

Table 9 – Domain and Scale level Form B PQA Items Selected for Improvement by 4C Distinction Form B PQA Item/Domain Cares Camps Contribute Connects Total

V.B. Staff support youth in building new skills. 2 2 V.C. Youth have an influence on the setting/activities. 1 2 3 V.D. Youth and family members have an influence on

program.

1 2 3

VI.F. Organization promotes staff development. 1 1 2

VI.G. Organization promotes supportive social norms. 1 1

VI.i. Organization is committed to ongoing improvement. 2 1 3 VI.J. Program is an integral part of the councils’ services plan.

2 1 3

VI.K. Intentional strategies that support an overall pattern of growth.

1 1 1 3

VI.N. Program administrators assess job performance and satisfaction of staff.

1 3 1 5

VI.P. Programs are evaluated for consistency with goals and business plan.

1 1

VII.S. Organization communicates with families, other organizations and schools.

1 1

13

Satisfaction with the YPQI Process

Survey Responses

In January 2011, 47 executives representing 30 councils and 26 staff representing 16 councils participated in an

online survey about their participation in the Camp Fire USA Program Quality Intervention. The majority of

executives and staff were satisfied with the YPQI process (see Figures 7 and 8), however staff were less likely to

report satisfaction than executives.

In addition, participants were asked to respond to several open-ended questions, including what changes they have

seen in staff, and what they would change about the YPQI process (see Appendix D for responses). With respect to

changes in staff and youth, both direct service staff and executive staff noted that for most of the sites any

meaningful changes were not likely to appear for another program cycle. Those who did note changes mentioned

increased motivation and improved morale for both direct service staff and youth. Executive staff also specifically

noted management’s increased ability to target specific areas of needed improvement.

One area in need of change cited by both executives and direct service staff was the need to make the CFPQA more

flexible with respect to the variety of setting features available across Camp Fire councils. For example, one

respondent discussed how it was difficult to answer planning questions because their site offered a set “calendar of

events” for which campers signed up. As a result of this structure, engagement was high, but did not reflect the same

type of student choice that might be present in other settings. Several respondents cited concerns about time,

including: “…we don’t have enough resources in terms of hours available,” and “make it less time consuming.”

They also noted that competing reporting requirements (e.g. SACERS) made the process feel repetitive.

Additionally, the fixed deadlines frustrated some who wanted to spend more time on their evaluations, having

derived most benefit from the discussions inspired by the quality improvement process.

Finally, both direct service staff and executive staff were asked what additional supports would assist them in

completing the quality improvement process. Both groups asked for additional training with respect to identifying

specific behaviors on the CFPQA. Additional in-person trainings and video examples were suggested. Both groups

cited that they felt they had good support from both Weikart Center and Camp Fire USA.

Figure 7 Council Executives’ Views of the Overall Usefulness of the CFPQI.

84% 80% 76% 73%

16% 20% 21%

20%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

It has helped our staffunderstand quality

It has helped improvethe quality of youth

programs

It has led to changes orimprovements in staff

training

It has been worth ourtime and effort

Agree Neutral Disagree

14

Figure 8 Direct Service Staff’s Views of the Overall Usefulness of the CFPQI

Summary of Camp Fire Council Executive Interviews

Telephone interviews were conducted with 15 council executives within Camp Fire USA. Interviewees were

specifically chosen based on the diversity of their councils, as well as the variation in their levels of engagement in

the CFPQI. Of the council executives chosen, four were determined to be fully engaged in the process, five had a

hard time getting started, three were new and still struggling but invested in learning and, finally, three were not able

to participate in the process at all, despite having shown initial interest as evidenced by the purchase of box sets.

One of the most often repeated comments about the CFPQI process was its ability to provide participants a

“common language” with which to communicate notions of improvement and standards of quality. Participants

referred to the “clearly articulated standards” and the powerful benefits of the “experiential education” process, as

realized through the multiple training opportunities (e.g. webinars, Youth Work Methods trainings), as well as the

ongoing coaching and support councils received from both Camp Fire USA and the Weikart Center.

Youth Work Methods trainings were cited for their direct usefulness to frontline staff as both a personal

development tool and a practice development tool. Youth Work Methods “Planning and Reflection” was

mentioned twice as a particularly useful experience. Several of the council executives interviewed mentioned the

importance of participating in quality improvement work as a way of strengthening councils’ legitimacy among

participants and their families; staff; and funders, as it demonstrates a commitment to youth in the form of

continuous improvement set against a backdrop of established standards that make sense to practitioners.

Other council executives mentioned the usefulness of developing the Program Improvement Plan. It provided these

councils a “roadmap,” developed organically through discussions which often brought in participants that would

not normally be involved in the day-to-day workings of the programs. Council executives spoke of engaging board

members and long-time volunteers in highly illuminating and productive discussions, creating a “university of

minds” driven by a common desire for program improvement and guided by the “easy to translate” expectations of

the CFPQA. Council executives described the process as helping “staff feel more in control of their programs.”

Although most of the council executives expressed high praise for the process, some difficulties were also revealed,

particularly among the less engaged participants. For some council executives, especially those with very small

59% 59% 50%

62%

31.00%

27% 36% 19%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

It has helped our staffunderstand quality

expectations

It has helped us improvethe quality of our youth

programs

It has led to changes orimprovements in staff

training

It has been worth outtime and effort

Agree Neutral Disagree

15

staffs, initial exposure to the process left some with the sense that the reporting requirements would be very difficult

to accomplish. One interviewee stated: “the process scared us, it seemed to be too much….” For two of the

councils interviewed, this initial exposure (one webinar and one live training) left them certain they would be

logistically unable, due to lack of available staff, to complete the requirements and so they did not pursue the

process any further. For instance, another interviewee stated,

“I sat through the webinar, an hour or two at the computer and…with the idea that we were

supposed to have five or six people to observe, that was just not practical for our situation…I have

one teacher who does all the self-reliance programs… and an after school childcare program… two

teachers are in charge of that. Those are basically the two programs we have.”

Lack of clarity about the process also contributed to unwillingness of some to participate, as evidenced by

statements such as, “I guess I’m not real sure exactly what the outcome is supposed to be or how it would make our

situation any better.”, “...we did not see that we had anything to gain from doing it…” and, “We don’t really get

anything out of it that is specific to what we do or specific to how we could do it better.”

Two council executives mentioned difficulties with adapting the CFPQA Form A and Form B to their setting. For

example, in camp settings, CFPQA items relating to the physical environment were not directly applicable. Also,

clubs headed by volunteers (who could not receive training due to cost prohibitions) and programs offering diverse

services struggled in answering some items on both Form A and Form B. Attempting to explain differences in these

various programs extended the reporting time for completing the self assessment.

All council executives interviewed mentioned the extensive reporting requirements and the time involved as a

drawback to the CFPQI process. For instance, one stated, “I think it needs to be less labor intensive…” The

monetary costs associated with the materials and training were noted as prohibitive for some councils, as well, with

one interviewee noting that, “…it would probably be quite expensive for us in terms of staff time.” and another

stating, “… if it takes five people, that would be quite expensive for us.”

Overall, the most frequently cited barrier was the time involved in seeing the process through all the stages. Both

high and low implementing sites identified problems with arranging staff time and problems with meeting fixed

deadlines (e.g., “the time it took to go through assessments,” “difficulty in arranging meeting times,” “timing not

helpful”). For low implementing sites, lack of available staff and issues of relevancy were the most frequently

identified problems.

Despite reports of barriers such as time limitations and lack of clarity around purpose for some councils, the overall

process is consistently described as useful when implemented by councils with the capacity to properly carry out the

work. This suggests that, while some councils may need more time and support in getting their continuous quality

improvement process off the ground, the investment will pay off.

16

References

Akiva, T., Pearson, L., Sugar, S. A., Peck, S. C., Smith, C., & Denault, A. (2010). Linking after-school instructional

practices to youth engagement: A pattern-centered approach. Paper presented at the Society for Research on

Adolescence Biennial Meeting.

Durlak, J. A., & Weissberg, R. P. (2007). The impact of after-school programs that promote personal and social skills.

Chicago, IL: Collaborative for Academic, Social, and Emotional Learning (CASEL).

Eccles, J. and Gootman, J.A. (Eds.) (2002). Community Programs to Promote Youth Development. Denver, DC:

National Academies Press.

Gambone, M. A., Klem, A. M., & Connell, J. P. (2002). Finding Out What Matters for Youth: Testing Key Links

in a Community Action Framework for Youth Development. Philadelphia: Youth Development Strategies,

Inc., and Institute for Research and Reform in Education.

High/Scope Educational Research Foundation. (2005). Youth PQA program quality assessment: Administration

manual. Ypsilanti, MI: High/Scope Press.

Smith, C., & Hohmann, C. (2005). Full findings from the Youth PQA validation study. Ypsilanti, MI: High/Scope

Educational Research Foundation.

Smith, C., Lo, Y.-J., Sugar, S. A., Akiva, T., Frank, K. A., Devaney, T., et al. (2012). Continuous quality

improvement in afterschool settings: Impact findings from the Youth Program Quality Intervention study.

Ypsilanti, MI: David P. Weikart Center for Youth Program Quality.

17

Appendix A*: Summary of Youth Work Methods Course Training of Trainers Implementation

Regularly cited as one of the most useful aspects of the YPQI process for direct care staff, Youth Work Methods

trainings have seen a substantial improvement in system-wide impact this year. Since May 2011:

20 people, representing 15 councils were certified as Youth Work Methods (YWM) Course instructors.

579 council staff was trained through YWM courses. These represented trainings for afterschool, club,

camp, and other council staff from 23 councils.

Most staff participated in multiple sessions and participated in several YWM courses with approximately

1438 total courses completed.

Three joint-council staff development days were held using the Youth Work Methods Courses.

o Snohomish County and Central Puget Sound councils hosted 34 participants from five Washington

Camp Fire councils.

o First Texas and Lone Star councils hosted 33 participants from five Texas Camp Fire councils.

o Columbia Council hosted 65 participants from two Oregon Camp Fire councils.

Methods were used by council YWM trainers to increase knowledge and deepen the impact of CFPQI.

Trainers conducted YWM sessions for all the council administrative and program staff.

Trainers included CFPQI content at monthly council staff meetings.

National and councils added program quality tips, resources and information to National and council

newsletters.

Counselor-In-Training (CIT) youth were trained in CFPQI and had these youth contributed to the self

assessment process.

Councils included testimonials from club leaders in how they have used program quality techniques in

monthly meetings.

Trainers conducted YWM sessions for AmeriCorps members.

Councils conducted in-service days for all youth development professionals (full time, part-time and

volunteer) three times a year using YWM courses as part of the curriculum.

In interviews conducted with council-based administrative personnel, YWM trainings and online course materials

were regularly cited as supportive of a “common language” among staff and several interviewees noted participation

with Youth Work methods courses as the “most useful” tool in understanding the CFPQI, largely contributing to a

growth in the depth of participation of the 23 councils that have utilized this resource. A summary of those

interviews follows this section.

*This summary contributed by DD Gass, Director of Program Planning & Effectiveness for Camp Fire USA

18

Appendix B: Data Table Camp Fire PQA - Form A

Data Table for Camp Fire USA PQA

All Camp

Fire N= 123

Care n= 25

Camp n= 57

Contri-butes n=

11

Connect n= 30

I.SAFE 4.50 4.44 4.49 4.69 4.86

IA: Psychological and emotional safety is promoted 3.99 4.18 3.89 4.01 4.03

I.A 1 -The emotional climate of the session is predominantly positive 4.49 4.42 4.43 4.64 4.60

I.A 2-There is no evidence of bias 4.80 4.83 4.86 4.45 4.80

I.A 3 -Intentional opportunities for youth to discuss culture 2.89 3.34 2.72 2.45 3.00

I.A 4 - Youth culture affirmed by staff 3.52 4.04 3.19 4.60 3.30

IB: The physical environment is safe and free of health hazards. 4.72 4.56 4.69 4.95 4.81

I.B 1 -The program space is free of health and safety hazards 4.58 4.28 4.61 4.82 4.70

I.B 2 - The program space is clean and sanitary 4.71 4.48 4.59 5.00 5.00

I.B 3 -Ventilation and lighting are adequate in the program space 4.93 5 4.88 5.00 4.92

I.B 4 -The temperature is comfortable for all activities in the program space 4.73 4.67 4.65 5.00 4.80

IC-Appropriate emergency procedures and supplies are present. 4.40 4.48 4.38 4.85 4.23

I.C 1 -Written emergency procedures are posted in plain view 4.02 4.42 3.98 4.71 3.57

I.C 2 -At least one charged fire extinguisher is accessible and visible from the

program space 4.54 4.65 4.32 5.00 4.71

I.C 3 -At least one complete first-aid kit is accessible and visible from the

program space 4.36 4.48 4.39 4.43 4.20

I.C 4 - Other appropriate safety and emergency equipment is to the program

offering as needed, can be located by staff and is maintained in full-service

condition

4.77 4.75 4.76 5.00 4.67

I.C 5 - All entrances to the indoor program space are supervised for security

during program hours 4.11 4.09 4.09 4.43 4.08

I.C 6 -Access to outdoor program space is supervised during program hours 4.42 4.52 4.45 5.00 4.09

I.C.7- Youth supervised 4.73 4.66 4.74 5.00 4.66

ID: Program space and furniture accommodate the activities offered. 4.81 4.81 4.83 4.95 4.72

I.D 1 - Program space allows youth and adults to move freely while carrying

out activities 4.66 4.58 4.67 5.00 4.60

I.D 2 -Program space is suitable for all activities offered 4.85 4.92 4.85 5.00 4.73

I.D 3 - Furniture is comfortable and of sufficient quantity for all youth

participating in the program offering 4.83 4.73 4.79 5.00 4.92

I.D 4 -Physical environment can be modified to meet the needs of program

offering 4.91 5 5.00 4.82 4.70

IE: Healthy food and drinks are provided. 4.67 4.73 4.67 4.57 4.62

I.E 1 -Drinking water is available and easily accessible to all youth 4.72 4.5 4.74 4.71 4.87

19

I.E 2 -Food and drinks are plentiful and available at appropriate times for all

youth during the session 4.90 4.88 4.94 4.60 4.92

I.E 3 - Available food and drink is healthy 4.15 4.78 4.00 4.60 3.78

I.E 4 - safe food management 4.80 5.00 4.62 5.00 4.89

II.SUPPORT 4.30 4.06 4.29 4.62 4.42

IIF: Staff provides a welcoming atmosphere. 4.72 4.49 4.69 4.84 4.91

II.F 1 -All youth are greeted by staff within the first 15 minute of the session 4.64 4.42 4.50 4.82 5.00

II.F 2- During activities, staff mainly us a warm tone of voice and respectful

language 4.71 4.58 4.66 5.00 4.80

II.F 3 -During activities, staff generally smile, use friendly gestures, and make

eye contact 4.90 4.84 4.89 5.00 4.93

II.F 4- Staff greet youths family 4.56 4.15 4.69 3.00 4.90

IIG: Session flow is planned, presented, and paced for youth. 4.49 4.36 4.50 4.69 4.49

II.G 1 -Staff start and end session within 10 minutes of scheduled time 4.69 4.92 4.65 4.64 4.59

II.G 2 -Staff have all materials and supplies ready to begin all activities 4.64 4.75 4.62 4.82 4.52

II.G 3 -There are enough materials and supplies prepared for all youth to

begin activities 4.77 4.83 4.69 4.82 4.86

II.G 4 - Staff explain all activities clearly 4.56 4.5 4.41 4.82 4.80

II.G 5 -There is an appropriate amount of time for all of the activities 4.26 3.83 4.33 4.82 4.27

II.G 6 - Variety of activities 4.04 3.26 4.33 4.27 4.06

IIH: Activities support active engagement. 4.23 4.2 4.14 4.54 4.32

II.H 1 -The bulk of the activities involve youth in engaging with material or

ideas or improving a skill through guided practice 4.51 4.5 4.63 4.64 4.27

II.H 2 -The program activities lead to tangible products or performances that

reflect ideas or designs of youth 4.16 4.17 3.96 4.45 4.40

II.H 3 -The activities provide all youth one or more opportunities to talk about

what they are doing and what they are thinking about to others 4.17 4.25 3.98 4.45 4.33

II.H 4 -The activities balance concrete experiences involving materials, people

and abstract concepts 4.10 3.92 3.94 4.64 4.31

IIi: Staff support youth in building new skills. 4.33 4.08 4.43 4.45 4.31

II.I 1 -All youth are encouraged to try out new skills or attempt higher levels

of performance 4.35 4.00 4.38 4.82 4.40

II.I 2 All youth who try out new skills receive support form staff despite

imperfect results, errors, or failure; staff allow youth to learn from and correct

their own mistakes and encourage youth to keep trying to improve their skills

4.31 4.33 4.42 4.09 4.17

II.I 3 - Appropriately challenging activities 4.37 4.04 4.49 4.45 4.40

IIJ: Staff support youth with encouragement. 4.06 3.86 3.95 4.51 4.28

II.J 1 -During activities, staff are almost always actively involved with youth 4.65 4.67 4.49 5.00 4.79

II.J 2 -Staff support at least some contributions or accomplishments of youth

by acknowledging what they’ve said or done with specific, non-evaluative

language

3.76 3.58 3.76 3.91 3.83

II.J 3 -Staff make frequent use of open-ended questions 3.80 3.33 3.59 4.64 4.24

IIK: Staff use youth-centered approaches to reframe conflict. 3.99 3.53 4.02 4.54 4.20

20

II.K 1 -Staff predominantly approach conflicts and negative behavior in a non-

threatening manner 4.66 4.39 4.62 5.00 4.91

II.K 2 -Staff seek input from youth in order to determine both the cause and

solution of conflicts and negative behavior 3.90 3.35 4.02 4.43 4.05

II.K 3 -To help youth understand and resolve conflicts and negative behavior,

staff encourage youth to examine the relationship between their actions and

consequences

3.66 3.26 3.69 4.71 3.67

II.K 4 -Staff acknowledge conflicts and negative behavior and follow up with

those involved afterward 3.83 3.29 3.94 4.14 4.00

II.K 5 - In conflict, staff acknowledge feelings of youth 3.75 3.31 3.72 4.42 4.05

III.INTERACTION 3.61 3.35 3.64 3.65 3.76

IIIL: Youth have opportunities to develop a sense of belonging. 4.07 4.02 3.93 4.18 4.33

III.L 1 -Youth have structured opportunities to get to know each other 3.79 3.92 3.59 3.80 4.07

III.L 2 -Youth exhibit predominately inclusive relationships with all in the

program offering, including newcomers 4.13 4.17 3.98 3.91 4.45

III.L 3 -Youth strongly identify with the program offering 4.36 4.08 4.28 4.45 4.71

III.L 4 -The activities include structured opportunities to publicly

acknowledge the achievements, work or contributions of at least some youth 4.02 3.92 3.92 4.45 4.10

IIIM: Youth have opportunities to participate in small groups. 3.48 3.2 3.72 3.10 3.40

III.M 1 -Session consists of activities carried out in at least 3 groupings- full,

small or individual 3.45 2.65 3.85 2.80 3.62

III.M 2 -Staff use 2 or more ways to form small groups 3.16 3.09 3.39 3.00 2.85

III.M 3 -Each small group has a purpose and all group members cooperate in

accomplishing it 3.83 3.58 4.11 3.40 3.67

III.M 4 - Interdependence of tasks 3.44 3.43 3.49 3.20 3.44

IIIN: Youth have opportunities to act as group facilitators and mentors. 3.08 2.8 3.03 3.28 3.38

III.N 1 -All youth have multiple opportunities to practice group-process skills 3.71 3.33 3.56 4.43 4.19

III.N 2 -During activities, all youth have one or more opportunities to mentor

an individual 2.68 2.58 2.74 2.71 2.63

III.N 3 - During activities, all youth have one or more opportunities to lead a 2.80 2.50 2.69 2.71 3.29

IIIO: Youth have opportunities to partner with adults. 3.78 3.37 3.87 4.00 3.89

III.O 1 -Staff share control of most activities with youth, providing guidance,

and facilitation while retaining overall responsibility 3.32 2.75 3.37 3.20 3.76

III.O 2 - Staff always provide an explanation for expectations, guidelines, or

directions given to youth 4.25 4.00 4.33 4.80 4.11

IV.ENGAGEMENT: Engagement 3.05 2.6 2.92 4.10 3.25

IVP: Youth have opportunities to set goals and make plans. 2.84 2.05 2.96 3.52 3.09

IV.P 1 - Youth have multiple opportunities to make plans for projects and

activities 2.78 2.04 2.96 3.00 3.00

IV.P 2 - In the course of planning the projects or activities, 2 or more planning

strategies are used 2.76 1.87 2.96 3.86 2.86

IV.P 3 - Guidance for youth planning 3.07 2.73 2.95 3.00 3.60

21

IVQ: Youth have opportunities to make choices based on their interests. 3.13 2.9 2.98 4.21 3.18

IV.Q 1 -All youth have the opportunity to make at least one open-ended

content choice within the content framework of the activities 3.00 2.83 2.96 3.29 3.14

IV.Q 2 -All youth have the opportunity to make at least one open-ended

process choice 2.93 2.75 2.74 3.86 3.21

IV.Q 3 - Youth given a chance to control direction of activity 3.41 3.19 3.30 4.45 3.35

IVR: Youth have opportunities to reflect. 3.10 2.85 2.81 4.09 3.46

IV.R 1 -All youth are engaged in an intentional process of reflecting on what

they are doing or have done 3.05 2.33 2.81 4.27 3.62

IV.R 2 -All youth are given the opportunity to reflect on their activities in 2 or

more ways 2.81 2.42 2.56 4.27 3.07

IV.R 3 -In the course of the program offering, all youth have structured

opportunities to make presentations to the whole group 2.91 3.17 2.56 3.55 3.07

IV.R 4- Staff initiate feedback from youth 3.50 3.33 3.23 4.09 3.90

IV.R 5 - Staff support for reflection 3.26 3.08 2.92 4.27 3.64

TOTAL PQA Score: SAFE, SUPPORT, INTERACTION,

ENGAGEMENT 3.87 3.70 3.94 4.00 4.11

TOTAL PQA and Ambient Item Score 3.94 3.66 3.98 4.19 4.01

22

Appendix C: Data Table Camp Fire PQA - Form B

Data Table for Camp Fire USA PQA Form B All Camp Fire

Councils

N=62

V. Youth Centered Policies and Practices 3.77

A. Staff qualifications support a positive youth development focus. 4.48

1. Administrator: youth development experience 4.83

2. Program Director: youth development training/ education 4.12

3. Program Director: youth development job experience 4.80

4. Staff: relevant education/training 4.16

5. Staff: job experience 4.32

6. Staff: reflect demographics of program participants 4.29

B. Program offerings tap youth interests and build multiple skills. 4.01

1. Influence format or content of program offerings 3.35

2. Program offerings: programmatic focus on skill building 4.35

3. Program has a major and specific focus on developmental domains 4.32

C. Youth have an influence on the setting and activities in the organization. 2.74

1. Shared decisions about physical environment 2.34

2. Shared decisions about program offerings 2.83

3. Youth lead sessions 3.19

D. Youth and family members have an influence on the structure and policy of the organization. 3.07

1. Staff respond to family questions 4.18

2. Staff create opportunities for families to participate 3.50

3. Youth and family members participate in program quality reviews 2.96

4. Training or policy for when and how to inform parents of conflict or misbehavior 4.04

5. Youth and families involved in training and evaluating staff 1.59

6. Youth and families share responsibilities for recruitment 2.89

7. Youth and families involved in community outreach 3.03

8. Youth and families share responsibilities on program governing boards 2.62

E. Program policies and procedures exist to enhance the health and safety of all participants 4.91

1. Document checklist 4.91

VI. High Expectations for Youth and Staff 3.98

F. Organization promotes staff development. 4.04

1. New staff pre-service orientation 4.40

2. Professional development within organization 4.47

3. Staff skills and training 4.06

4. Adult education principles used when training staff 3.83

5. Support for staff to attend courses, conferences, workshops 3.77

6. Professional development outside the organization 3.35

7. Program administrators meet during program cycle 4.23

8. On-site directors and direct delivery staff meet during program cycle 4.21

G. Organization promotes supportive social norms. 4.18

1. Young people identify with organization 4.41

2. Expectations for personal and social interaction 4.29

23

Data Table for Camp Fire USA PQA Form B All Camp Fire

Councils

N=62

3. Encouraging youth to share personal concerns 4.22

4. Staff pursue knowledge on issues and cultures of the youth 3.80

H. Organization promotes high expectations for young people. 4.25

1. High expectations for young people 4.22

2. Acknowledge achievements 4.29

I. Organization is committed to ongoing program improvement. 4.21

1. Assesses youth outcomes 4.40

2. Staff evaluations 3.81

3. Assesses program quality 4.44

4. Program improvement based on assessment 4.23

J. This program is an integral part of the council's program service's plan. 3.51

1. Feedback from young people in program 4.00

2. Feedback from parents/guardians 4.77

3. Feedback from past participants 3.28

4. Feedback from stakeholders and/or the general public 3.26

5. Demographic information 3.48

6. Programs offered based on data 3.38

7. Primary audience for each program based on data 3.38

K. There are intentional strategies that support an overall pattern of growth in services to youth 3.01

1. Intentional strategy that supports growth in services 3.54

2. Intentional strategy to increase participants and growth is evident from data 2.49

L. For each program, there is a program business plan which includes … 4.09

1. Camp Fire USA mission statement, and core values 4.40

2. Program activities relate to desired outcomes 3.91

3. Written outcomes for youth 3.91

4. Plan supports the needs of the target population. 4.16

M. There is a supervision plan in force that includes a ratio of youth to adults to maximize safety 4.54

1. Ratio of adults to youth 4.65

2. Staff circulate 4.66

3. Minimum number of adult staff 4.30

N. Program administrators assess job performance and satisfaction among staff 3.63

1. Program staff are supported in their work. 4.60

2. Team work 4.21

3. Staff demonstrate a sense of purpose 4.70

4. Staff receive continuous feed back 3.29

5. Year-round staff participate in self-evaluation 2.83

6. Short-term staff participate in self-evaluation 2.23

7. Supervisors receive training 3.44

O. The program administration provides for sound leadership and management 4.17

1. Strategies to support sustainability 3.98

2. Welcoming atmosphere for staff 4.70

3. Key stakeholders are involved in decision making 3.62

4. Compliance with standards and licensing. 4.37

24

Data Table for Camp Fire USA PQA Form B All Camp Fire

Councils

N=62

P. Each program is evaluated for consistency with the goals and business plan. 4.20

1. Goals for sound financial management 4.31

2. Population and participation projections 4.21

3. Results of youth development outcomes 4.08

VII. Access 4.22

Q. Staff availability and longevity with the organization supports youth-staff relationships 4.68

1. Staff with the program the entire program cycle 4.67

2. Staff have returned from previous program cycle 4.70

R Barriers to participation are addressed 4.16

1. Availability of eligible youth 4.11

2. Distance, transportation, neighborhood safety barriers 3.91

3. Cost barriers 4.41

S. Organization communicates with families, other organizations, and schools. 4.15

1. Mechanisms for communication 4.35

2. Communication with other organizations and schools 4.00

T. Parents are informed of the program's purpose, schedules… 4.01

1. Structured opportunities for families to be involved 4.12

2. Formal communication with families 4.01

3. Family members involved in decision-making 3.14

4. Schedules are completed and announced in advance 4.70

TOTAL 4.17

25

Appendix D: Open-ended Survey Responses

Prompt: Overall, what would you change, if anything, about the quality improvement process that you participated in?

Because of our Camp Fire Kids Unplugged Program, I feel that the YPQA could you a little tweak or two. This program is designed in a way that does not fit

the measurement tool exactly. For instance, we create a calendar of events for families to choose from. They do not, however, have very much of a say as to

what the activities will be. That being said, our Unplugged program does have the most participant engagement and interaction.

I would love it if more people from the council could participate in the external assessor training. I am hopeful more and more folks will become involved with

the process and we continue to implement our improvement plan.

Many of these programs have come and gone, costing councils money and using the most valuable resource any council has - the time of their staff , leaders,

and volunteers. All of these programs have great ideas and wonderful feel-good offerings. But at the end of the day, if the goal is to truly impact youth in a

positive, life-long and meaningful way, we must engage them - we must *listen* to them, and we must stop pretending that programs such as these have an

impact on anything other than what has become to true bottom line for youth work - funding. The cart has come before the horse for some time, and Camp

Fire USA as a national organization is dying. Given how very much we have to offer, it's a terrible shame.

It seems very limiting to measure only one time of year formally with National. Our clubs and a lot of our Self Reliance programs are active in the fall and it

would be nice to have those pieces of information on a regular basis. We are doing some observations on these, but the impetus is not as high because it is not

"required".

That we need to be more intentional in how we think about, implement, and report on quality programming.

Internally I wish we had more time and energy to devote to the process and definitely to focus attention to the improvement plans in a more timely manner.

We occasionally needed better clarification during training or in written materials to help determine what we were looking for during the assessment process. It

would have possibly streamlined the scoring sessions for Form A and the interview sessions for Form B.

Make it less time consuming

More training at the local level. NOT ONLINE

We are located in Massachusetts and we need to participate in a Quality Rating Improvement System process. This process requires a formal assessment to

rate where we are according to standards. The tool we use is the School Age Childcare Environmental Rating Scale. (SACERS) Although I really think

assessment is important, having to do two assessments at the same time and filling out different paperwork is very time consuming. I would love it if I could

use one for the other. I feel focusing on one rather than doing two would get a better result.

We have already tweaked it a bit and really know that it is important to have others from outside our program come in and score the forms and provide

feedback. We have a better lens that way and we really need to train our seasonal staff in more depth so they "get it" and not view it as a negative.

As we've gone through this process, we question a little how this is going to be an annual process and what the expectations will be for our council. We see

value in assessment and improvement. Unfortunately, the best people to complete these assessments and improvement plans are our frontline staff and site

directors and we simply don't have enough resources in terms for hours available for them to complete this process as in depth as would be necessary due to the

many other things we have put on their plates. Our front line staff are all part time individuals and we simply don't have enough time in a day to have them

fully integrate this process. We have competing priorities and we have been focused on improvement to quality for some time and some of this seems a little

behind where we are at.

I get it now but it was not until I went through the external assessment and youth work methods trainer course that things really clicked. I think the in person

trainings are far more effective than webinars. I think that more methods courses will help people understand what we are trying to do. The PQA basics course

trains us to understand what YPQA is and helps us understand domains, indicators, etc. Methods helped me understand HOW the PQA connected with day

to day programs. I think that it what is missing from the Basics course.

I would like more instruction. I felt blind at times, and that’s not helpful to anyone. Best practices for the various common programs (ie Club, Camp, Out of

School) would be fabulous

The time required to participate in these trainings has cost us more money. We are a small council and do not have a lot of time or money for these things.

Some of the questions are too black and white. It makes it difficult, when observing for a short period of time, to score everything accurately - while we may be

doing what the question is asking we may not be doing it during that time period. I realize there are processes for this but I am not sure it give a true

representation of the program that lasts all day (like our resident camp).

More resources for external assessment and training

No, I believe that this process is a great tool for our council and we are planning to train all our staff using this process.

The assessment is too time consuming and I had help from National!! Also- I've listed only the STAFF that participated in this survey because that was all that

was asked. We've also had volunteers participating. I'm not sure where that fits into your questionnaire.

If there were any ideas on how to shorten the time in scoring but still receive the same results.

Easier to use digital tools that include room for anecdotes. More training materials to use with staff. More models for how to engage diverse stakeholders in

the process. Funding for the additional staff time required to do this process well.

Getting advice from others that have completed the process was valuable.

More staff input. This process has come down to 1-2 staff making it happen and not a system wide process where different managers do observations and bring

things to the table. Even going over the observation data, only the observers participated in the meeting. As many were part time staff or volunteers, priority

was not to "pay" for their time to come in and sit down and talk about what was observed. Also, from National, more impetus on the upper management to

create an atmosphere of YPQA in the councils versus 1-2 staff who try to implement it. This could have been better to make board members and CEO's

responsible for this.

Just a person to contact if there are questions.

The resources such as the webinar and manual were sufficient.

How best can our council integrate data captured from previous years into national's concentration on the YPQA?

26

I was given the support that I needed during this process.

We have made a commitment to the improvement process. For us that means we are able to change programs, training, and anything else related to process

that is identified in our improvement plan to get moving toward the quality target that we have set.

A more detailed webinar about scoring the YPQA.

You provided good support when we asked.

More time

We appreciated our support for the Weikert Center--very helpful. Just having gone through the process will inform our future assessment work.

Training opportunities closer to local councils.

We felt very supported in this. The Weikart Center was always very responsive when we had questions.

Probably more staff and of course revenue and time.

I think that the online training was great. It was a really useful tool in training not only myself and my staff but our partners.

The assessment is too time consuming and I had help from National!! Also- I've listed only the STAFF that participated in this survey because that was all that

was asked. We've also had volunteers participating. I'm not sure where that fits into your questionnaire.

If there were any ideas on how to shorten the time in scoring but still receive the same results.

More suggestions from the program improvement organizers.

Easier to use digital tools that include room for anecdotes. More training materials to use with staff. More models for how to engage diverse stakeholders in

the process. Funding for the additional staff time required to do this process well.

Getting advice from others that have completed the process was valuable.

See above. Also, more staff input. This process has come down to 1-2 staff making it happen and not a system wide process where different managers do

observations and bring things to the table. Even going over the observation data, only the observers participated in the meeting. As many were part time staff or

volunteers, priority was not to "pay" for their time to come in and sit down and talk about what was observed. Also, from National, more impetus on the upper

management to create an atmosphere of YPQA in the councils versus 1-2 staff who try to implement it. This could have been better to make board members

and CEO's responsible for this.

Just a person to contact if there are questions.

The resources such as the webinar and manual were sufficient.

PROMPT: As a result of participating in this program quality improvement process, what changes, if any have you seen for your staff (i.e. in turnover, morale, quality of instruction, etc.)

The YPQA has helped management and administrative staff gain a better understanding of how to improve the support we give our point of service instructors.

Now that we have a streamlined outline of what program quality is, we can look for our shortcomings and strengths in greater detail. Given that insight, the

YPQA has helped our coucil focus on our areas of need and reach our goals quicker.

Make most staff actually motivated to work here

I feel like our staff are beginning to connect with the kids more and hear what the kids want their their idea's more. Reflection has also become more in depth

although we still have some work to do in that area.

Higher level of engagement with youth and a greater contribution from each staff member.

being that we are only a 1 full time & 1 part time staff council- it has taken time away from other activities & so has all of these survies that we are being asked

to fill out.

As a leader I see a continued push towards the type of metric that looks great on paper and does little for youth.

I have never heard of this program, I know nothing about any training or webinar, I don't even know what the initials stand for. This survey is totally wasted

on me.

Morale

I have not been here long enough to see great changes. I have only attended one workshop, VOICE and CHOICE, and I do not know if it was our teacher or

the curriculum, but I did not learn much.

Instruction was improved.

I honestly don't know what this is, unless I know it perhaps by another name? Some of the Youth Methods Workshops sound similar to what we did at

training in August/September, but I am not entirely sure.

I was not working for Camp Fire when these activities were held

undertstanding of the difference in programs offered

It has helped to increase our staff awareness of how we measure quality. It has also helped us define and understand areas that we need extra help/training to

address. Out of the YPQA process, we are implementing a Youth Methods Course training for our volunteers in February to increase Voice and Choice for our

children and youth.

None yet. While we completed the assessement portion before the end of August, we are just beginning to address the improvement plan phase. To date the

experience has impacted myself and those serving on a Effective Programming Committee. As we move forward, I anticipate various reactions from our staff.

I've seen some growth in morale and quality of instruction.

Too early to tell. Because of timing and staff availability, the observation and assessment portion of the project was conducted by myself (Executive Director),

27

board members, and volunteers. Due to the delays in implementing the second part of the process (which will take place primarily after the first of the year),

it's too early to really gauge the reaction and attitude of our program staff. I anticipate some will be very engaged and with others it will take time to really gain

buy-in.

We conducted a summer survey and we will hopefully see results next year.

Too soon to know yet they are all planning better. All staff has been on board less than 3 months

To help them understand that they are not just babysitter but that they are professional educators with a hard job of teaching the youth they work with in a

comprehensive manner. To be quality it has to be intentional not merely reactive.

Staff are excited about the tool but not excited when they see results. It's been a struggle to get them to open up to the idea that maybe they are not as good at

what we do as they think they are. Will take time to adjust and learn that every program has room to improve and youth development is always changing and

growing as new ideas and information is learned/formed.

We are still in the process of training our frontline staff on these courses, therefore we cannot assess whether or not there has been any changes in staff at this

point.

Deepened their knowledge of youth development, helped direct service staff in working with youth. helped staff feel more connected as professionals, formed a

link between assessments, creating improvement plans and carrying them out which then provides better programming for youth, generating opportunities for

teamwork

The process has helped our staff understand what quality programs look like. The indicators provide specific behaviors for us to train towards. Using YPQI

as a guideline has made our programs more valid....all of us...board, staff and volunteers have a greater understanding of what makes "youth development"

different from childcare. We knew it was different but now we have a common language to help us articulate why.

Haven't had any training classes yet for our volunteers

I have seen some staff take the opportunity and run with it, growing as youth development professionals, but overall the change has been quantifiably

negligible. I cannot tell you the effect of our staff training, but I like to think that it was successful in boosting camper experience.

Haven't had any training classes yet for our volunteers

More work for very small staff.

Morale, quality of instruction, more training and involvement of our children in programming

Staff who attended the YMW trainings have new skills for their bag of tricks. They trainings were very helpful and enlightening.

The biggest changes will take place in next season's summer staff for us. We did our observations toward the end of our summer program this year so there

was not much time to implement changes.

Common language, targeted improvements and accountability

Morale

Quality of instruction, intention during staff trainings.

PROMPT: As a result of participating in this program quality improvement process, what changes, if any, have you seen for the youth with whom you work? (outcomes, motivation, attachment to program, etc.)

Youth have benefited from the YPQA in terms of inclusiveness. It has also helped our service learning curriculum a great deal. Allowing the students voice and

choice has empowered them as individuals. It has also incited interest in other areas of our program. By ensuring that the kids are safe and supported, they are

free to engage and interact more naturally with their instructors and each other.

Not been working long enough to see what could be improved with youth

I've found myself giving the youth more control, instead of planning everything myself I ask what they want and really try to implement their ideas.

A greater degree of input from youth into program activities and culture.

no changes to the youth or the way we run our programs- we just have to turn in paperwork now on what we do...

Again, I see little positive improving fort youth. I see a focus on numbers vs impact, paperwork vs youth, and 'training' vs the type of skills that lead to actually

engaging youth voice during the critical Jr High years - which is when we lose most of our youth in CFUSA.

My kids are great without the program.

Motivation has improved

Youth have less conflicts.

ownership of participation

I think it is still early to see these changes because only a few staff have really been involved with this process. We are in the process of rolling these quality

improvement steps out with our volunteers (see previous question) so that they understand what we expect when we say program quality.

None yet.

PROMPT: What kind of supports would you like to have provided the next time you go through a program improvement process?

External assessors would really help us. As long as they are trained!!! Our council is so small, that asking managers, directors and all other hands on deck to

help assess is a great strain on all of us. Given that the tool is so time consuming, having an extra pair of hands that is trained in YPQA would be a gigantic

time saver. It would also almost guarantee a more objective assessment.

To go to it first.

I found the hard-copy materials sent to be very helpful. A good video with an overview of the process and some short live shots of program time with

commentary on scoring (maybe one example for each section or something) would be helpful.

The support of the council for the diversity of what youth want, vs what the council wants youth to want. Programs that begin by engaging youth instead of

trying to force multiple standards straight out of a typical school onto a program that youth participate in for pleasure. An understanding of the fact that in

28

order to lead youth you must first reach them, and to reach them you must treat them like people with unique needs and goals of their own.

NONE

It was great.

All the support was great.

More Youth Methods courses for the different areas available. We are fortunate that our CEO has gone through the training and I can utilize her as a resource

to help train volunteers and staff in identified areas of growth. It would also be great to see what other councils have as improvement plan goals and how they

are addressing those goals.

I feel good about the supports that we've been provided.

Possibly more examples of actual assessments. In meeting with one of our staff who took training to do external assessments, she shared how much time was

spent on equitable scoring. While that probably isn't realistic for internal assessment training for staff and volunteers, I'd like to see a move in that direction.

Our volunteers struggled with what "they" were really looking for on some of the questions - they were happy to answer honestly but weren't always sure they

knew what they were answering.

External support

Need tools on how to help younger seasonal staff that are really tied to the way it has been done in the past to move through that and open up to change.

Camp Fire HQ and the Weikart Center have been extremely supportive to us whenever we have needed it. Response time is usually quick and answers are

complete.

I think I have it down now. I have always received the support I needed and asked for. DD and Amanda have been great!

More volunteers - that's all I've used

TRAINING! and Best Practices

More volunteers - that's all I've used

More planning and help to do the improvement part of the process.

Our Council's challenges in completing the self assessments and improvement process were related to staff turnover. We look forward to participating in 2012

with a new team. Thanks to DD and Amanda for your understanding and patience with us!

More resources for external assessment and training of site staff

I feel that the support has been great!

PROMPT: Do you have any other comments about your experience?

I think this is a great tool and we are finally starting to see the value of it in our programs this 2nd year.

I love YPQA and was excited to learn new things; this was such a breath of fresh air after all of my educational training.

Simply going through the process of evaluation and review with the team offered a lot of learning opportunities. I am SO glad Camp Fire has partnered with

Weikert.

PLEASE MAKE IT STOP!!! JUST KIDDING! I UNDERSTAND THE PROCESS OF TRYING TO EVALUATE & COMPARE WHAT EVERYONE

IS DOING FOR THE SAKE OF IMPROVING BUT IT'S NOT FUN!!!

If youth want to do a project, and we can help them with that project , why not ? We can meet our goals if we only help them to meet theirs. We have a

growing number of people who are very good at paperwork and filling out forms, but when I see these people handling a large group of youth on a camp-out

and they cannot get the youth to cooperate on the simplest task, or to be interested in or engaged in an activity that many regard as fun - I have to wonder why

these are the new shining stars of Camp Fire.

Enjoyed it.

I started in September of 2011.

The staff person who attended this training is no longer with us. I am unsure how to answer any of these questions. We will need to start all over again for this

year.

A comment more regarding the this survey. It really hard to accurately remember what webinars or GROW sessions I participated in during the last two

years. I tried, but may not have been a 100% accurate. It took me a minute to remember where GROW actually was the last two years :) Also the question

that asked about staff time - only allowed a maximum of 15 hours. I estimate I have spent close to 45 hours since late April.

We have a small program with only two full time staff members, some of the questions did not apply to us . It is also hard to find time to work on the plan and

to take the surveys when you are running a council with only two people.

There are only two staff in our council and finding time for the surveys, webinars is very hard.

Please make the recording on the system simpler. It took me forever to figure it out and then to get it into the system. Amanda helped by explaining some. It

would be great if there could be some type of upload from Excel or a document that would simplify the process time-wise.

Some of our team understand and can at least partially utilize this tool, but it;s not universal and that is the first thing that needs to be fixed. If the leadership

doesn't get it, how can you expect it to trickle down.

Hopefully we will be able to participate more fully this year.

I love it

29

Appendix E*: Camp PQA Summary and Scores

The American Camp Association (ACA) collaborated with the Weikart Center to develop a Camp PQA using some

of the research validated assessment questions which are in the YPQA, and also add some nature and outdoor

program specific items. The Camp PQA is shorter than the YPQA and does not include any safe environment

indicators, as the ACA is recommending this tool be used with accredited camps, which already meet safety

standards. ACA is still trying to determine how they will make the tool available to camps in future years.

This past summer, Camp Fire USA worked with the ACA to have five Camp Fire councils field test the Camp

PQA. ACA had field tested the tools two summers ago, with four camps, and were interested in having Camp Fire

council staff, who were trained in the YPQA process also field test the assessment tools. The councils chosen had

completed CFPQA assessments in camp programs previously. The five councils who completed the process were:

Green Country Council, Snohomish County Council, Columbia Council, Georgia Council, and Heartland

Programs. The councils primarily conducted Camp PQA assessments with resident camp, but some day camp

assessments were also done.

Camp Fire USA will continue to work with ACA to determine how the Camp PQA might work with the CFPQA

process. Camp Fire USA will continue to ask councils to complete the CFPQA in all programs, however strategies

are being developed to determine if the Camp PQA might be used in alternative years with day and resident camp

programs.

Data Table Camp PQA: Form A

Mean

Scores

N=13

I. Activity Climate 4.25

IA: Staff Friendliness 4.56

1. Staff mainly use a warm tone of voice and respectful language. 4.67

2. Staff generally smile, use friendly gestures, and make eye contact. 4.75

3. When campers approach them, staff never exhibit non-attentive behaviors. 4.25

4. Staff appear to like the campers they’re working with. 4.67

IB: Staff Circulation. 4.33

1. Staff circulate (and spread out if multiple staff) to interact with every camper (in groups or individually) at some

point during the activity. 4.58

2. Staff interact one-on-one at least once with every (or almost every) camper during the activity. 3.92

3. Staff are actively involved with campers (e.g., they provide di-rections, answer questions, work as partners or

team members, check in with indi-viduals or small groups). 4.75

4. Each camper experiences personal attention from staff. 4.08

IC: Emotional Safety. 4.31

1. There are no incidents in which a camper or campers are made fun of or every such incident is addressed by staff. 4.33

2. Staff shows respect for all campers and insists that campers show respect for each other (e.g., use kind words,

take turns, help each other). 4.25

3. Campers appear to feel free to be themselves. 4.20

ID: Feelings and Conflicts. 4.00

1. Every time there is a conflict or an incident involving strong feelings, staff ask about and/or acknowledge the

feelings of the campers involved. 5.00

2. In a conflict situation, adults ask the campers what hap-pened. 5.00

3. When strong feelings are involved, staff consistently help campers respond appropriately (e.g., staff encourage

campers to brainstorm possible solutions, take time to “cool off,” find an appropriate physical outlet, etc.) 5.00

4. Staff handle campers’ feelings and conflicts supportively. 4.00

IE: Support for Belonging. 3.63

II. Learning Environment 3.69

IIF: Aim High. 4.28

30

1. All campers are encouraged to try out new skills or attempt higher levels of performance. 4.33

2. Staff provide intentional opportunities for development of specific skills (as opposed to activities with just a

recreation or ‘having fun’ focus) for all campers in the session. 4.33

3. The staff has high expectations for campers. 4.17

IIG: Appropriate Challenge. 3.83

1. There is sufficient time for all of the activities (e.g., campers do not appear rushed, frustrated, bored, or

distracted; most campers finish activities). 4.00

2. Activities are appropriately challenging (not too easy, not too hard) for all or nearly all of the campers; there is

little or no evidence of boredom or frustration on the part of campers. 3.75

3. Campers seem challenged (in a good way) by the activities. 3.75

IIH: Asking Questions. 3.43

1. Staff ask 3 or more open-ended questions (i.e., questions without predetermined, correct answers). 3.83

2. Staff ask 3 or more challenging questions (i.e., questions that make campers think, require more than a quick

answer, require campers to analyze, evaluate, make connections, etc.) 3.08

3. Staff uses questions effectively with campers. 3.38

IIi: Collaborative Opportunities. 2.92

1. The activity includes opportunities for all campers to work towards shared goals (e.g., small groups build things

or create performances). 2.33

2. The activity includes opportunities for all campers to work cooperatively together. 3.33

3. The activity involves interdependent tasks for all campers (i.e., campers have different tasks or roles, that come

together for the task or project). 3.00

4. Campers have the experience of collaborating with others. 3.00

IIJ: Support for Active Learning. 4.00

III. Engagement 2.40

IIIK: Planning. 2.13

1. Campers have multiple opportunities to make individual or group plans for projects and activities (e.g., Written

or sketched plan for a building project, verbal plans about an art project, staff asks, “what’s your plan?”). 2.08

2. There is a specific time or times for planning during the session routine. 2.17

3. Campers engage in deliberate planning. 2.13

IIIL: Voice. 2.70

1. All campers have the opportunity to make at least one open-ended choice within the content framework of the

activities (e.g., campers decide roles, tools or materials, topics within a given subject area, subtopics, or aspects of a

given topic).

3.17

2. Staff actively encourage campers to take an activity in a new or unplanned direction (e.g., staff says, “Can you

guys think of a better way to do this or how we might change this activity to make it more interesting or

challenging?”; Staff supports camper

2.75

3. Staff share control of most activities with campers, providing guidance and facilitation while retaining overall

responsibility (e.g., staff use youth leaders, semiautonomous small groups, or individually guided activities). 2.33

4. Campers have a say in how they spend their time in the activity. 2.54

IIIM: Reflection. 2.38

1. All campers are engaged in an intentional process of reflecting on what they are doing or have done (e.g., writing

in journals; reviewing minutes; sharing progress, accomplishments, or feelings about the experience). 2.58

2. All campers are given the opportunity to reflect on their activities in 2 or more ways (e.g., writing, role playing,

using media or technology, drawing). 2.08

3. Activities involve 1 or more structured times in which staff ask campers debrief questions (e.g., questions that ask

campers about the experiences they had in the activity). 2.67

4. Campers have opportunities to look back on things they are doing and make learning connections. 2.33

TOTAL 3.57

Data Table Camp PQA: Form A2

Mean

Scores

N=8

I. Activity Climate 4.10

IA: Staff Friendliness 4.59

1. Staff mainly use a warm tone of voice and respectful language. 4.66

31

2. Staff generally smile, use friendly gestures, and make eye contact. 4.66

3. When campers approach them, staff never exhibit non-attentive behaviors. 4.41

4. Staff appear to like the campers they’re working with. 4.62

IB: Staff Circulation. 3.50

1. Staff circulate (and spread out if multiple staff) to interact with every camper (in groups or individually) at some

point during the activity. 4.16

2. Staff interact one-on-one at least once with every (or almost every) camper during the activity. 2.83

3. Staff are actively involved with campers (e.g., they provide di-rections, answer questions, work as partners or

team members, check in with indi-viduals or small groups). 3.58

4. Each camper experiences personal attention from staff. 3.41

IC: Emotional Safety. 4.16

1. There are no incidents in which a camper or campers are made fun of or every such incident is addressed by staff. 4.00

2. Staff shows respect for all campers and insists that campers show respect for each other (e.g., use kind words,

take turns, help each other). 4.33

3. Campers appear to feel free to be themselves. 4.16

ID: Feelings and Conflicts. 4.25

1. Every time there is a conflict or an incident involving strong feelings, staff ask about and/or acknowledge the

feelings of the campers involved. 5.00

2. In a conflict situation, adults ask the campers what hap-pened. 5.00

3. When strong feelings are involved, staff consistently help campers respond appropriately (e.g., staff encourage

campers to brainstorm possible solutions, take time to “cool off,” find an appropriate physical outlet, etc.) 5.00

4. Staff handle campers’ feelings and conflicts supportively. 4.25

IE: Support for Belonging. 3.63

II. Nature Supplement 4.61

NA. Physical Setting 4.83

1. Availability of natural outdoor areas 5.00

2. Programs utilize natural outdoor areas 4.66

NB. Program Implementation 4.39

1. Campers having fun 5.00

2. Staff is enthusiastic 4.83

3. Staff help campers explore nature 3.75

4. Campers interact with nature 4.00

TOTAL 3.57

Data Table Camp PQA: Form B

Mean

Scores

N=4

I. Program Support 3.72

IA: Support for climate 4.11

1. Young people identify with camp 5.00

2. Administration encourages one-on-one relationships 4.33

3. Mechanisms support close relationships 3.00

IB: Support for learning 3.66

1. Camp intentionally promotes learning 4.33

2. Administration promotes challenging activities 3.00

IC: Support for youth engagement 3.44

1. Policies promote camper planning 3.66

2. Campers have opportunities to choose activities 3.00

3. Policies promote reflection 3.66

ID: Continuous improvement 3.66

1. Camp assesses youth outcomes 4.33

32

2. Camp assesses program quality 3.66

3. Camp engages in program improvement 3.00

II. Nature Supplement 3.61

ND. Camp policies and practices 3.00

1. Staff required to have knowledge of outdoors 3.00

2. Specific examples of sustainable practices 3.00

NE. Nature focus 2.83

1. Nature themes and focuses 3.66

2. Camp uses nature curriculum 2.33

3. Nature curriculum addresses local issues 2.33

4. Campers reflect on nature topics 3.00

NF. Outdoor experiences 5.00

Outdoor programming in natural areas 5.00

Camp programs use outdoors most of the time 5.00

Informal activities occur in outdoor areas 5.00

TOTAL 3.57

*This summary contributed by DD Gass, Director of Program Planning & Effectiveness for Camp Fire USA