Koontz 713 Program Evaluation Full Report

7/28/2019 Koontz 713 Program Evaluation Full Report

1/85

Running Head: CHECK-IN CHECK-OUT

Check-in Check-out Program Evaluation

Full Program Evaluation Report

Joseph Koontz

Submitted in partial fulfillment of the requirements for:

EDLP 713 Evidence-informed Perspectives on Practice II

April 17, 2013


2/85

CHECK-IN CHECK-OUT 2

In order to protect the anonymity of the participants in this evaluation, the school being

evaluated will be referred to as Suburban Elementary. Suburban Elementary is a pre-

kindergarten through grade five school with approximately 500 students located in Central

Virginia. The majority of the students at Suburban Elementary receive free or reduced lunch,

and Suburban Elementary receives Title I funding. While it has not performed well in the past, it

has recently made great gains in student performance. A portion of that recent success may be

attributed to the school-wide positive behavior support program. This program evaluation will

focus on a component of that school-wide system called Check-in/ Check-out (CICO).

History and Background

From 2008 - 2010, Suburban Elementary failed to earn Virginia Department of Education

(VDOE) academic accreditation. Schools that failed to earn state accreditation were required to

implement a specific school improvement plan model. There were many academic improvement

plans, as well as plans to reduce discipline. Disruptive behaviors lower overall student

achievement because they impede the learning of all students and monopolize a significant

portion of teachers and administrators time. (Anderson, 2009)

In 2009, Suburban Elementary implemented a school-wide discipline program. This

program is based on a system developed by VDOEs Training and Technical Assistance Center

(TTAC). This school-wide program is designed to create a universal set of expectations and

consequences for students behavior. In September 2010, Suburban Elementary implemented

phase II of TTACs program. This second phase consisted of a positive behavior support system

for students who needed individualized help beyond the school-wide discipline plan.

Positive behavior support is a proactive way of dealing with student behavior.

Traditional methods of discipline, such as suspension, time outs, and loss of privileges are


3/85


reactive. Positive behavior support is intended to reduce the exclusionary practices of traditional

discipline in favor of proactive behavior management. (Luiselli, Putnam, Handler, & Feinberg,

2005)

Suburban Elementary realized that student behavior was taking up a large portion of

teachers and administrators time. By reducing discipline, Suburban Elementary hopes to

increase the time students are on task and learning. Their school-wide discipline program and

the CICO portion of that program are part of the larger school improvement plan aimed at raising

overall student achievement. Suburban Elementary is starting the third year of the CICO

program and is ready to evaluate the program to determine if the time and resources invested are

helping toward their goal of increased student achievement.

Program Goals

The ultimate goal of CICO is to prevent disruptive behavior in students who need support

beyond the effective school-wide discipline program. The program is designed for students to

check-in with an adult in the morning, carry an individualized behavior plan throughout the day,

and check-out with the same adult at the end of the day. The check-in is intended to be a

positive start to the day. The adult conferences with the student and together they talk about the

expectations for the day and set behavior goals. The student then takes a behavior chart with him

or her that teachers use to record their behavior at different intervals throughout the day. The

intervals are designed to be short time periods with specific behavior expectations. These

behavior expectations are written in simple terms with a rating scale that has only three

indicators for the teacher to circle or mark. The individualized behavior charts are designed to

be easy for the student to understand and the teacher to document. At the end of the day, the

students again conference with the same adult and they chart their individual progress toward


4/85


goals they set with their adult mentor. Luiselli et al. (2005) summarizes the goals of positive

behavior support systems:

1. Setting behavior expectations.

2. Teaching interpersonal skills.

3. Provide reinforcement for reaching.

4. Monitors behavior through data analysis.

5. Involves all stakeholders including the students, parents, teachers, and administrators.

The effective school-wide discipline program is beneficial for the majority of students

who attend Suburban Elementary. For those students who need more personalized attention, the

CICO system helps minimize their negative behavior.

Logic Model

Figure 1 shows the logic model for the CICO program at Suburban Elementary. This

logic model was developed with input from the client and it is a helpful guide when analyzing

the program.


5/85


Resources/

InputsActivities Outputs Outcomes Impact

Check-in/

Check-out

positivebehavior support

systemframework

documentation

School

Administration

Personnel

Student

incentives

Individualizedstudent behavior

plans

Funds for

student

incentives.

Identification of

students who are

unsuccessfulusing the Tier 1

School-wideDiscipline Plan

Pairing upstudents in

CICO with staff

mentor/advisor

Progress

monitoring astudent in the

CICO program

Daily check-in

and check-outbetween

students and

administrator

Counseling/ goal

setting withstudents in theCICO program

Rewardingstudents for

reaching set

behavior goals

Teachers fill out

students daily

behavior chart

Student

behavior plan

completed withteacher

documentation

Student referrals

Qualitative

information

from daily

studentconferences

Discipline Data

Students build

relationships

with an adult inthe school

building

Students take

ownership oftheir own

behavior

Decrease instudent

disciplinereferrals

Less teacher

and

administrativetime is spent on

student

discipline

Decrease in

negative

studentbehavior

Increased

student

achievement

Improved

school climate

Figure 1. Check-in / Check-out Logic Model for Suburban Elementary


6/85


Context

Political context. Suburban Elementary was on a school improvement plan that was

supervised by the state of Virginia. The county of Henrico and the state of Virginia invested a

myriad of resources into Suburban Elementary. Academic and behavioral data was scrutinized.

This type of scrutiny brings political pressure because public funding is involved. Part of the

reason Suburban Elementary implemented this program was because it was approved by the

VDOE through TTAC.

In recent months, the political pressure on school discipline has increases because of a

series of articles run in the Richmond Times-Dispatch. Virginia State Sen. A. Donald McEachin,

D-Henrico, expressed concerns in an October 2, 2013 letter co-written by Henrico County

District Supervisor Tyrone E. Nelson about the suspension rate of African American males.

(Martz, 2012) The principal made an appearance before the school board to discuss her

discipline practices. This CICO program is a critical part of the process for reducing their

suspensions.

Cultural context. Student discipline has the potential to have cultural implications.

Cultural and ethnic groups view discipline consequences differently. Furthermore, some parents

may see this type of individualized plan as punitive, rather than preventative. Parents may feel

their child is being singled out or they are somehow being stereotyped by this plan. The other

students in the school could also identify the students in the CICO program. There is the

potential for teasing and bullying based on participation in this program because the students

have a behavior chart they must carry to all classes. This differentiates these students from the

other students who do not have a plan. This program evaluation will help determine if the

positive outcomes outweigh the potential negative consequences of CICO.


7/85


Evaluation Purpose

The administrative team at Suburban Elementary has already confirmed that this program

is successful at reducing the negative behavior of students. The data is clear and for that reason

this will not be a quantitative study looking at suspension rates or discipline referrals only. This

will be a responsive evaluation using mostly qualitative data to look at the efficiency of the

program. Suburban Elementary invests substantial resources in the CICO program, which is part

of the School-wide Positive Behavior Support (SPBS) program. Two of the three members of the

administration meet with students every morning and afternoon. The schools principal would

like to know if this investment is as effective and efficient as possible. The purpose of this

evaluation was to find areas in need of improvement in the CICO process. The evaluation was

guided by the following questions.

1. Are there aspects of the Check-in Check-out program that can be improved or run moreefficiently?

2. What are the perceptions of the Check-in Check-out program from the differentstakeholders?

This was a responsive evaluation where the first goal was to understand the CICO

program from the perspective of different stakeholders to be able to make recommendations for

improvement.


8/85


Literature Review

Thousands of schools across the country have implemented school-wide positive

behavior support (SPBS) programs as a way to decrease problem behavior and improve school

climate. (Cohen, Kincaid, & Childs, 2007) These programs are designed to take a strategic,

proactive approach to student discipline and school climate. Traditionally, school discipline

policies tendto be mostly reactive and consequence driven. SPBS is a proactive way of dealing

with student behavior. It focuses on positive interactions, clear expectations, and consistent

consequences. SPBS programs also recognize that student behavior and school climate are

directly related to student achievement. SPBS programs are often part of schools

comprehensive improvement plans, which tie directly to state and local accreditation

requirements. This program evaluation will focus on a component of the SPBS called check-in/

check out.

School-wide positive behavior support programs are part of a larger initiative called

Response to Intervention (RTI). RTI can have academic or behavioral based programs that have

multiple implementation tiers. Tier-one implementation refers to school-wide expectations and

outcomes. Tier-one is the foundation for a successful SPBS program, but there will be a portion

of the school population that will need additional support beyond the tier-one program. Those

students would be served through tier-two intervention, also know as targeted intervention. This

literature review will focus on the research-supporting tier-one SPBS as well as tier-two, targeted

intervention.

Education Research Complete Database and Google Scholar were used to find peer-

reviewed articles dealing with school-wide positive behavior support programs. The following

key word search phrases were used to locate articles:


9/85


School-wide Behavior Program Positive Behavior Support Programs School-wide Positive Behavior Support Targeted Behavior Intervention Tier Two Behavior Intervention School-wide Behavior Support Evaluation

Articles were selected for review based on their relevance to the topic and based on the

type of study that was conducted. Articles that simply described SPBS and did not conduct or

review any study were rejected.

Tier-one: School-Wide Positive Behavior Support

A 2011 research article by Caldarella, Shatzer, Gray, Young, and Young used a quasi-

experimental design to prove the effectiveness of a SPBS on middle school climate and student

outcomes. The researchers used a non-equivalent two-group model with a pretest and posttest to

measure outcomes. This study included more than 300 teacher responses and 10,000 student

responses. Two schools were studied with one school serving as the control over a four-year

period.

In the school that implemented a SPBS program, there were statically significant

improvements in how teachers rated the school as well as a decrease in office referrals. The

study also found there was a statically significant correlation between the SPBS program and

student tardies and unexcused absences. If students are present on a regular basis, there is less of

a chance they fall behind due to missed work. In addition to the positive behavior outcomes, it

seems student participation increases. This is not a surprising correlation, but it does highlight

how behavior and school climate affect morethan just the number of office referrals.


10/85


The SPBS program in this study had similar key features to many programs like it. The

program promotes school-wide teaching of social skills, praise notes from teachers to students,

posting of school rules, proactive screening for students at risk for emotional and behavioral

disorders, and referrals of at-risk students for targeted intervention. (Paul, et al., 2011) One of

the ways SPBS is different than traditional discipline programs is the emphasis on positive adult

interactions. Furthermore, there are clear expectations for the students and the teachers. This

study also talks about targeted intervention, which will be covered in subsequent articles. SPBS

programs recognize that not all students are receptive to the school-wide model. This study

indicates 15% of students may require a more intensive and individualized plan. These

students would require tier-two intervention.

A 2005 study by Luiselli, Putnam, Handler, and Feinberg highlights the link between

behavior and academic performance. This study sites research that suggests that behavior

problems in young children predict maladjustment later in life. Luiselli et al. (2005) contend that

behavior has a direct link to academic performance and that early intervention is critical to

success. The authors of this study use a before and after design model using a suburban

elementary school of approximately 600 students. The school volunteered for the study because

the administration had a desire for improved academic achievement and reduced discipline. The

school would implement a positive school-wide behavior support program and the discipline and

academic data would be tracked for a three-year period to show if the SPBS program was

effective. This study also used a questionnaire to gauge the social validity of the SPBS program.

Teachers were given a questionnaire in years two and three of the study to assess perceptions of

the program effectiveness and the impact on student learning.


11/85


The results of this study affirmed the effectiveness of a school-wide behavior program.

Luiselli et al. stated, by virtue of reducing discipline problems, teachers can devote more time to

instruction and other learning opportunities that maximize educational progress. (2005, p. 195)

Over a three-year period, the study noted a decrease in suspensions and office referrals. The

study also noted an increase in student performance measured by the Metropolitan Achievement

TestSeventh Edition. The teachers also saw a positive effect on the school climate. The

number of teachers that indicated, the school discipline is effective (Luiselli et al., 2005,

p.194) went from less than 50% in year 1 to 100% in year 2. This study is limited because it

only looks at one school, but it does illustrate a successful school-wide implementation.

According to these studies and many others, SPBS programs have a clear benefit for

students, teachers, and administrators and have a positive impact on a schools climate and

achievement.

The previous studies have proved the value of positive school-wide behavior support

programs. Once a SPBS program is adopted, implementation quality and fidelity are critical

factors to a successful SPBS program. An article by Cohen, Kincaid, and Childs (2007) uses a

qualitative pilot study to look at the most effective ways to measure school-wide positive

behavior support implementation. According to their findings, more than 4,000 schools have

implemented some type of SPBS program. Cohen et al. credit these programs for decreasing

student discipline referrals and increasing positive climate. The authors contend the degree to

which a school implements a program with fidelity has an effect on the quality of the program.

The School-wide Benchmark of Quality (SBQ) is a tool to evaluate a SPBS program.

The SBQ is a 53 question rating scale that measures the degree of fidelity with which a

school is implementing a SPBS program. (Cohen et al., 2007, p. 204) 47 schools in two states


12/85


piloted the use of the SBQ. Their study finds this tool to be reliable and valid tool for evaluating

SBPS implementation. Their data suggests there is a direct correlation between a high score

(>70) and a decrease in office discipline referrals. For school administrators, having some way

to quantify the fidelity of implementation of programs is important.

Miramontes, Marchant, Heath, and Fishcer (2011) contend that social validity is an

important factor to the success of a school-wide discipline program. According to Miramontes et

al., social validity describes the value that society places on something. Miramontes et al. (2011)

contend that programs should be evaluated on more than goals, procedures, and outcomes and

should go beyond clinical judgment. (p. 446) The higher the degree of social validity, the more

individuals will buy into and be satisfied with a program. Their study attempts to measure the

social validity of positive behavior intervention and support programs using a questionnaire to

measure the social validity of stakeholders who were involved with a SPBS program.

One of the most significant findings from the study is the relationship between fidelity of

implementation and social validity. Miramontes et al. (2011) state, small yet statistically

significant correlations were found between faithful program implementation and increased

social validity, indication that the more accurately a program was implemented, the more

valuable the consumers perceived the procedures. (p.463) Implementation fidelity is a

consistent theme with several studies. This article highlights the need for stakeholders to find the

program relevant and practical in their everyday practice.

Tier-two: Targeted Intervention

Once a school has established an effective SPBS program, the next step is implementing

a tier two intervention program, also referred to as targeted intervention. Students who are not

responsive to SPBS will require targeted intervention. This targeted intervention for a small


13/85


percentage of students will look different for each student. Most targeted intervention involves

some type of check-in / check-out system. Students check-in with an adult in the morning, they

follow an individualized behavior plan during the day, and check-out with that same adult in the

afternoon.

According to a 2010 study by Cheney, Lynass, Flower, Waugh, Iwaszuk, Mielenz, and

Hawken, children with emotional and behavioral disabilities are at riskand often interfere

with teachers efforts to educate other children in the classroom. (p.152) Targeting these

students will not only benefit the students with significant behavior challenges, it will also help

the other students in the class by limiting disruptions. Cheney et al. (2010) also note that these

students are more likely to fail courses, drop out of high school, and are more likely to have

difficulties with relationships and employment when they are older.

This meta-analysis by Cheney et al. uses data from 18 urban elementary schools in the

Seattle metropolitan area. This article notes that their research shows 70% of students in targeted

intervention respond favorably, but 1015% of the students may need even more services such

as coaching on problem-solving skills and conflict resolution. There will be a small percentage

of students who do not respond to the increased support and they may have significant

disabilities that require special education services.

A study by Hawken and Horner (2003) prove the effectiveness of targeted behavior

support as a component of a SPBS program. This study shows a relationship between

intervention implementation and a reduction in problem behavior. This study confirms the

proportion of students who typically do not respond to the tier-one program.

This study looked at a group of students within a middle school in the pacific northwest

who had five or more office referrals and who were currently not receiving targeted intervention.


14/85


The students were added to the CICO program where they check-in with an adult for a quick

conference upon arrival, they carry with them a data collection tool for their teachers to fill out

about their behavior, and they check-out at the end of the day with the same adult. In this study,

there was also a daily home-school report for reinforcement by the parents.

This study found while overall levels of problem behavior were reduced following the

intervention, the larger effect was that students became more consistent in participating in class

without problem behavior. (Hawken & Horner, 2010, p. 237) The study also had some

important suggestions for schools implementing a targeted intervention program. They found

that the match up between the students and the adult had to be a good fit. Hawkin and Horner

(2010) found that if students do not find adult attention reinforcing the BEP [targeted

intervention] will be less effective. (p.237) This study acknowledges the importance of the

quality of the support provided by the adult in the CICO system. Further study would be needed

to identify the degree to which to adult verses the behavior chart had more effect on the students

behavior.

A study by Medley, Little, and Akin-Little (2008) illustrates the importance of having a

strong tier-one system before implementing a tier-two system. Their study finds that schools

with a SPBS as a foundation write more effective targeted assistance plans. This study used an

instrument called the Behavior Support Plan-Quality Evaluation (BSP-QE) to rate targeted

assistance plans. They found using the BSP-QE that schools the foundation that is created by the

school-wide program creates an environment that is more conducive to targeted assistance.

Unfortunately, just because a school has a SPBS program in place, does not ensure a quality

targeted intervention program. The study found that all schools could improve their intervention


15/85


plans. The implications for administrators would be that they should use a tool like the BSP-QE

to evaluate their own intervention plans.

A multi-tiered behavior intervention system can have potential negative outcomes if not

properly implemented. A 2009 study by McIntosch, Campbell, Carter, and Dickey studied 34

students in a school district that were identified for tier-two behavior support. The findings

clearly showed a positive correlation between a tier-two intervention program and a reduction in

negative behavior, the study also highlighted some areas of caution for school personnel. The

authors of this study highlight the potential for over-identifying students for special education

services. Students who are not successful with a tier-two intervention such as CICO are often

referred for special education.

The authors contend there is not enough research regarding the move between tier-two

and tier-three or special education services. McIntosch et al. (2009) caution schools to avoid the

one-trick pony phenomenon where schools only have one form of tier-two intervention. (p.89)

Just as the school-wide plan does not fit all students, a check-in/check-out program might not fit

all students participating in a tier-two program. Furthermore, just because a student is

unsuccessful in a specific tier-two program like CICO, does not necessarily mean they should be

referred for special education. McIntosch et al. (2009) states a lack of student response to a tier

two intervention may be misconstrued as evidence of a disability when an equally likely

explanation may be that there was a poor match between the intervention and the function of the

students behavior. (p.89)

Conclusion: Impact for School Leaders

Current research is in agreement that School-wide Positive Behavior Support Programs

are an effective way to decrease negative student behavior and increase positive school climate.


16/85


The research also indicates for some students, this school-wide program is not enough. For those

students, tier-two or targeted intervention is needed. The check-in/check out model is proven to

be effective for many of those students who need extra support.

For school administrators, there are several key themes that emerged from the current

literature. There is a need to evaluate school programs. There are tools available that can

quantify the fidelity of implementation and effectiveness of the programs. Administrators need

to go beyond the number of discipline referrals and results from climate surveys. There is also a

clear theme of implementation fidelity. Having all staff implement a SPBS program correctly

and consistently is critical to the success of the program. Finally, school leaders cannot forget

about the need to create buy-in from their stakeholders. The research shows the importance of

social validly to the success of a program.

Method

When this study was first conceptualized, it was originally a quantitative study looking at

the effectiveness of the CICO program in terms of the discipline data such as office referrals and

suspensions. After an initial conversation with the client, it was clear that this evaluation would

be much more qualitative in nature. The client has a clear understanding of the quantitativedata.

She knows that this program is effective in reducing the number of office referrals and

suspensions. She was more interested in the perceptions of the program and the efficiency of the

daily process. For this reason, the evaluation became a responsive evaluation using a mixed

method approach.

Participant Characteristics


17/85


There are many stakeholders involved in the CICO program. The administrators at

Suburban Elementary are responsible for checking-in and checking-out with the students in this

program each day. The students in the program are the most significant stakeholders. They are

the reason for the program and they are the students with the most significant behavior

challenges. They also have a substantial impact on the overall instruction and climate of the

school. Their behavior effects instruction and the teachers ability to effectively run his or her

classroom. During this program evaluation, all of these stakeholders may need to be involved

because each play a different role in the program.

The principal is ultimately responsible for all programs at Suburban Elementary. That

being said, the principal is intentionally removed from the CICO process for several reasons.

The adults involved in the CICO system need to maintain a positive relationship with the

students participating. The principal needs to maintain a certain detachment in order to be the

disciplinary figure if students in the CICO program need immediate consequences. Furthermore,

the responsibilities of a principal make it difficult to be available every morning and every

afternoon. The adults involved with the CICO system need to have consistent availability.

The school counselor and the resource teacher are the two primary individuals

responsible for this program. They each have a caseload of students who check-in and check-out

with them each day. They are also responsible for determining which students participate in the

program and for how long. They are also responsible for progress monitoring these students to

ensure the interventions are working. Any stakeholder has the opportunity to refer a student to

the CICO program, but the associate principal and resource teacher make the final determination.

The teachers are responsible for completing the individualized behavior charts throughout

the day for the children in this program. It is important for the teachers to implement the


18/85


requirements of the program with fidelity. They need to follow the daily behavior chart and

make accurate reports in order for the student to be successful. Cohen, Kincaid, and Childs

(2007) found the higher the degree of implementation fidelity, the more successful a school-wide

discipline is.

Parents are also significant stakeholders, although in conducting the study, it is clear that

parents are not a significant part of the CICO program in its current form.

The evaluation used enterprise documents, interviews, focus groups, and surveys to

collect information to help evaluate the CICO program at Suburban Elementary.

Evaluation Design

An initial interview with the schools principal and a review of the schools internal

documents helped shape the initial understanding of the program. With the help of the client, an

initial logic model was created and a tentative plan for conducting an evaluation was outlined.

The literature review was clear that any behavior intervention program is only as good as

the people implementing it. Therefore, a responsive evaluation method was chosen. Robert

Stake states, it [responsive evaluation] is not particularly responsive to program theory or stated

goals; it is responsive to stakeholder concerns. (Stake, 204, p.89) In the initial evaluation

design, stakeholder identification and interaction was emphasized. A meta-evaluation was also

important. A consultation with the principal before each data collection activity was built into

the design. The principal approved all questions and processes for collecting data from

stakeholders.

Interviews. Interviewing key stakeholders was part of the initial understanding process.

After an initial interview with the principal, an interview was conducted with the resource

teacher and the school counselor. Both of these individuals were chosen for interviews because


19/85


they are the stakeholders who run the CICO program on a daily basis. The students in the

program check-in and check-out with these two individuals. Interviewing these two people

would help get a better understanding of the process. Furthermore, the opinions of these two key

stakeholders would be valuable. For that reason, they were interviewed individually, not in a

focus group. The transcription of these interviews can be found in Appendices A and B.

Focus Groups. Focus groups are an efficient method for gathering data from a group of

stakeholders. Focus groups also allow for interplay between the stakeholders that can also

provide valuable information and context. Two focus groups were conducted with teachers from

Suburban Elementary. A complete transcription of the two focus groups can be found in

Appendices C and D.

Surveys. In designing this evaluation, it was important to have a way of gathering some

anonymous information from the stakeholders. In focus groups and interviews, stakeholders

might hold back or not be truthful out of fear of repercussions or because they are simply not

comfortable. Surveys allow individuals a way to be as truthful as they want. A survey was

designed for the teachers and the students with an area on the teacher survey for open ended

responses. A copy of the teacher and student surveys can be found in Appendices E and F

respectively.

Initially, a parent survey was developed, but after consulting with the client, it was

abandoned. For various reasons, the client did not want to involve the parents in this program

evaluation.

Steps taken to enhance credibility. There were several aspects of the evaluation design

that enhanced the credibility of the qualitative data collection.


20/85


Multiple data sources. Having multiple data sources from multiple stakeholders was a

critical aspect of the program design that enhanced the credibility. Having interviews, focus

groups, observations, and surveys helped to enhance the credibility of each individually.

Member checks. Before and after each data collection process, a member check was

conducted with the client. This helped shape the next data collection tool and it allowed for

clarification of perceptions and interpretations. These member checks are referenced in the data

collection flow chart in Figure 2.

Progressive subjectivity. The research journal and reflections of the evaluator serve as a

way to increase credibility. Novice evaluators will learn as they go and the reflective thought

throughout the data collection process helps to drive the credibility of the evaluation.

Steps taken to enhance dependability. There were several features built into the design

of the study that enhanced the dependability of the data collected. First, the data was collected

sequentially. This allowed for the data collection questions and methods to evolve with new

understanding. If the surveys, interviews, and focus groups had been conducted concurrently,

there would not have been the opportunity to incorporate new understanding. When there was

more than one interview or focus group that were meant to be identical, planning was done to

make sure they were conducted as close together as possible. Both administrator interviews

were conducted on the same day and the teacher focus groups were conducted only one day a

apart.

Also, the evaluator made sure to transcribe the data in a timely manner, but waited to

code the transcriptions until the end of the evaluation. By waiting to code at the end, it ensures

that the same codes and thought process is used throughout.


21/85


Data Collection

A flow chart of the data collection process can be seen in Figure 2 below. After each

data collection process, there was a member check with the schools principal. This allowed for

the data collection process to evolve with increased understanding.

Data Collection Process Flow Chart

Figure 2. Flow chart depicting the process for data collection, check for understanding, and plan

modification throughout the evaluation.

The initial interviews were extremely helpful in shaping the focus group and survey

process. One of the first discussions centered on how to survey the students. The principal did

not want to have to get permission from parents for the evaluator to conduct a focus group. The

evaluator and the client decided that someone from within the school would administer a brief


22/85


survey to the students. This would be someone the students were confortable with. A simple,

three-question survey was developed and administered to the students in the program. While this

was not ideal, it was a compromise that the client was comfortable with.

The teacher survey was sent out using Google Docs. The survey provided data that

helped shape the focus group questions. After discussing the results with the client, it was

decided to add additional questions about the Tier One, school-wide behavior support program.

The research indicated that having a strong Tier One program was important for the success of a

Tier Two program. The evaluator wanted to make sure there was some data collected about the

Tier One program.

There was also a compromise made on the teacher focus groups. Initially, the plan called

for separate focus groups for teachers who had students in the CICO program and teachers who

did not have students in the program. This was initially set up, but weather cancelations forced

the groups to be rescheduled. With the short notice, the focus groups had to be a mix of teachers.

While this was not the original plan, it ended up enriching the discussion. The evaluator relied

on the client to set up the focus groups. If these were conducted again, it would have been

helpful for the evaluator to have a list of the participants and to be able to touch base with them

ahead of time. It should also be noted that during one of the focus groups, the power went out.

There was a window in the room, so the blinds were opened and the focus group continued. The

participants indicated that they did not mind and this event did not seem to interfere with the data

collection.

Given more time, a follow up survey and focus group would have been helpful. Time

and logistics did not allow for this, but further questions about the students/teacher interaction

during the day would have been helpful.


23/85


Data Analysis and Interpretation

Several overriding themes emerged from the data analysis. The overriding themes of

Time and Communication were evident throughout the multiple data sources. Within the theme

of Time, subthemes of equity, fidelity, and consistency all surfaced. Within the theme of

Communication, oral, written, and parent communication were prominent subthemes. Each of

these themes will be addressed within the different data collection methods below where they

apply.

Observations

Time. Time was the major theme that surfaced through the observations. The students

and CICO administrators seemed rushed through the process. Both CICO administrators later

confirmed this observation. These observations of the process were conducted early on in the

evaluation. If time had permitted, observations would have been beneficial after the interviews

and focus groups because the evaluator had a better understanding of the process. Also,

observing the teacher in the classroom filling out the behavior chart would have been helpful.

Time and scheduling did not permit such observations.

Interviews

The interviews were an invaluable part of the evaluation. Through the interviews, the

evaluator was able to gain an in-depth understanding of the CICO process. The themes of time,

communication, and parent involvement were all evident.

Time. The initial interviews with the program administrators confirmed some of my

observations. The interviewers agreed that time was a major factor. Both indicated that more

time with the students would improve the CICO program, especially at the end of the day. They


24/85


also indicated that they wanted more time to counsel the students. One example came up where

the interviewee talked about a report that he got from the teacher. He indicated that if he had the

time, he would have made sure the student understood the ramifications of her actions. When

each CICO administrator was asked what was the most effective aspect of the CICO program,

they both indicated the relationships they formed with the students. When asked what could

strengthen the program, they both indicated a desire for more individual time with the students.

Relations and time go hand in hand and there seem to be an agreement by both administrators

that time is an issue.

Communication. There are several subthemes that surfaced in the interviews

surrounding communication. The administrators wished they could communicate more with the

students but they both felt like that had the relationships formed with the students for them to

communicate effectively. Communication between the teacher and the CICO administrator is

done primarily through the students daily behavior chart. Significant emphasis is placed on the

score the students receive on this behavior chart. See Appendix I for a sample student behavior

chart. Students are striving for a 54 each day, but anything over a 40 is considered a good day.

Teachers can also leave anecdotal notes about the students day. These notes can help provide

some context for the score, but not all teachers leave a note.

Communication consistency also emerged as a theme. One interviewee mentioned that

he had modified his students form so that it worked for him. This could create some

inconsistencies for the teacher who is filling out the forms. The interviewees did not indicate if

there was a consistent communication between the teacher and the CICO administrator.

The topic of parent involvement and communication came up with both interviewees.

There were inconsistencies between the two interviews. One interviewee sent home the


25/85


students behavior chart each week, and one did not. They both talked about calling parents

when they needed to for follow up at home.

The same inconsistency occurred with the communication about expectations for the

kids. Each CICO administrator had different number expectations for the behavior chart. One

administrator wanted the students to strive for a perfect score; while the other set the goals lower

and had the students work up to a larger goal.

When each CICO administrator was asked about training and what training they received

to administer this program, they had different answers. One administrator said she went to a

conference five or six years ago. The other administrator said he had not received any formal

training. Both indicated that they did not communicate regularly with each other about the CICO

program.

Focus Groups

Two focus groups were conducted. The first focus group had five participants and the

second had eight. There was a mix of teachers from all grade levels and subjects. The focus

shifted slightly from the interview and survey questions. The focus group discussion started by

asking the participants about the Tier One behavior program at suburban elementary. This was

done strategically based on the literature review. During the literature review phase, it became

clear that a school needed to have a strong Tier One program in order for their Tier Two program

to be successful. When the evaluator was not charged with reviewing the Tier One program,

gathering some information about that program could be helpful in evaluating the Tier Two

program. The focus group data indicated that the Tier One program at Suburban Elementary was

strong and they all felt like it was effective at dealing with the behaviors of the majority of the

students. A common theme that emerged as a strength of the Tier One program was consistent


26/85


communication and expectations. Several times the participants mentioned the common

language and consistent message that the Tier One program provided. The client may be able

to take some of the strengths of the Tier One program and incorporate them into the Tier Two

program.

Time. Similar to the interview responses, time is a factor for the teachers in several

ways. One of the questions asked the teachers if they thought the student chart was manageable

from the teacher perspective. The teachers who had one student in CICO said yes, it was

manageable and there was enough time. Three teachers had more than one student in the

program and they all indicated that it was much more difficult to manage when they had multiple

students in the program. This will be something the school administration will want to consider

when placing students in the program.

The teachers also talked about the lost instructional time at the end of the day. The

students leave their classrooms early at the end of the day in order to check-out with their

assigned administrator. The teachers questioned wheather the students were being as efficient as

they could with this time. Several teachers also had concerns that some of their biggest behavior

problems had unsupervised time as they transitioned from their classroom to the check-out

location.

A theme that emerged across both focus groups was the concept of equity versus

equality. The teachers seemed to understand that these students needed a higher level of support,

but they also had some cognitive dissonance about rewarding students for mediocre behavior.

They also vocalized a desire to recognize and reward the students who consistently demonstrate

good behavior. The teachers worried that they were not setting the students up for the real

world where individuals are not rewarded for every correct behavior.


27/85


Time for more classroom guidance emerged as a theme during the focus groups. The

teachers talked about a need for specific guidance with all students on how to interact with one

and other. This would include small group and classroom guidance.

Communication. There seemed to be a misunderstanding about the student chart and

what goes home to parents. Some teachers were under the impression that the behavior chart

went home to parents each day. Some teachers had been writing notes each day thinking they

were making it home to parents. There did not seem to be a clear consensus among the teachers

about what should be communicated on the chart each day. The teachers also indicated that they

had little interaction with the CICO administrator. School-wide, teachers use the students

agenda books to communicate with parents about student behavior. Some teachers still use this

means of communication with their CICO students and some did not. This will be something the

school administration should clarify with the teachers.

All the teachers who have students in the CICO use the student behavior sheet. This is

the most consistent way they communicate their students behavior on a dailybasis. This

quantifiable data point is subjective. This seemed to be another area where the teachers did not

agree. The behavior scale used goes from one to three, where a one is poor behavior and a three

represents good behavior. How one teacher grades a three could be very different than another

teacher. The teachers expressed a desire to have more clarification about how to use the chart so

they are consistent between classes. The teachers talked about the proscribed nature of the Tier 1

program. They expressed a desire to have the same consistency using the CICO program.

The teachers were also unsure about how to formally refer a student to the CICO

program. Some teachers voiced a concern that a teacher who writes up a student many times

gets preference. The teachers do not want their students to be overlooked because they have not


28/85


written them up enough. They expressed a desire for more transparency in the CICO selection

process.

The teachers had many suggestions for how the CICO could be improved. These ideas

will be fully discussed in the Discussion section of this report.

Quantitative Measures

Two surveys were conducted. One with the students in the CICO program and one was

given to the whole staff. An employee at Suburban Elementary gave the survey to 13 students.

The staff survey was conducted using a Google Doc form. It was sent to 38 staff members with

29 responses. The data from the teacher and student responses can be found in Appendices G

and H respectively.

Student survey. The student survey only had three questions. The first question asked,

What do you like about the Check-in/Check-out program? Six of the 13 students indicated

they like the candy or prizes. Three students indicated they like that the program helped them

control their behavior and the final four students responded with various other answers. 12 of

the 13 students were able to articulate that they behaved better because they were in the CICO

program.

Teacher survey. The teacher survey categorized the teachers into three groups, teachers

who currently have students in the CICO program, teachers who have previously had students in

the program, and teachers who have never had students in the program. The intent was to see if

there was a difference in the perception of the program among these groups. On nine out of the

10 questions on the survey, there was a difference between the teachers who have or have had

students in the CICO program and the teachers who have never had students in the program. The

largest difference among the groups was on question three which asked, The students behavior


29/85


chart is easy for the classroom teacher to complete. The teachers responded using a five-point

scale where a 1 represented disagree and 5 represented complete agreement. The teachers

who have students currently in the program rated this question almost a full point higher than

teachers who have never had a student in the program. That indicates there is a misperception

among the teachers who have never been a part of the program.

One question that all three groups rated lower than the other question had to do with

parent involvement. The question asked the teachers, I feel the level of parent communication

in the Check-in/Check-out program is sufficient. The teachers with students in the CICO

program gave this question an average rating of 2.93, the lowest of any category. One of the

survey respondents left the following message that may be of interest to the school

administration. He or she wrote, I do not know about the parent communication so am

interested in learning more. Also maybe clearer guidelines about when a student leaves the

program. Another respondent wrote, In the past when I have had students participate it

seemed that there was not clear communication between the teacher and the mentor about what

the expectations were. Both of these statements speak to a lack of communication and

understanding that could be addressed by the school administration.

When asked about the behavior chart itself, the teachers who had students in the CICO

program responded favorably to the question that asked if the behavior chart was easy for the

students to understand. The teachers agreed with this statement more than all other statements

with an average response of 4.14 on the five-point scale.

Figure 1 below displays the average response to each of the ten questions in the CICO

survey. The teacher responses are grouped by the type of teacher. The open-ended responses

can be found in Appendix G.


30/85


Table 1

Average Response by Teacher Groups from CICO Survey

Teacher

Group

Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10

Currently

Using CICO

4.00 3.43 4.07 4.29 4.14 2.93 3.43 4.07 3.36 3.93

Used CICO

in the Past

4.25 3.38 3.63 3.75 3.75 3.13 3.38 3.50 3.25 3.50

No CICOExperience

3.57 3.71 3.14 3.57 3.43 3.14 3.14 3.86 3.14 3.29

Discussion

Time and communication were overarching themes that surfaced among all stakeholders

in this evaluation. The CICO program at Suburban Elementary is constrained by the hours in the

school day and the available resources. The program is also influenced by external factors such

as home environment and parental involvement that are beyond the control of the school. There

are areas where time and resources could be used more efficiently and there are channels of

communication that could be more productive.

Stakeholder Perceptions of the Check-in / Check-out Program

There are mixed perceptions of the CICO program at Suburban Elementary. The staff

perceptions of the program definitely differ with their level of involvement. When asked if the

CICO program is a worthwhile investment of time by the school administration, the teachers

who currently have students in the program more strongly agreed with that statement. The


31/85


qualitative feedback was positive overall. There are a few teachers who do not see the value in

the program. This seems to be because of a bad experience with the program or because they

have multiple students in the program and the added work load makes it less effective.

The students perception of the program was almost all positive. All but one student who

was surveyed indicated that they behaved better because they were in the CICO program.

During observations of the check-out process, it was clear the relationship the CICO

administrator and the students had. It is hard to quantify the value of a relationship, but it was

evident. Furthermore, when the CICO administrators were asked what the most valuable part of

the CICO program was, the first response from both administrators was they relationship they

have with the students.

This program evaluation did not specifically focus on the Tier One school wide behavior

program but, the perception of that program from the focus groups was positive. Both groups

talked about the common language that it creates from grade level to grade level. Phrases such

as, we dont do that here were mentioned several times. Many teachers commented how the

Bones allowed them to reward students throughout the school. The current research indicates

that it is essential to have a solid Tier One program in order for the Tier Two program to be

effective.

Improving perception. Suburban elementary may want to consider additional training

and information sessions about the CICO program. As new teachers come on, they will need to

be trained, even if they do not have students in the program. You may also want to consider

having teachers sit in on the Effective School-wide Discipline (ESD) meetings on a rotating

basis. Through the interviews and focus groups, there seems to be a perception that the ESD

group had substantial input regarding the CICO program. Having teachers participate in those


32/85


meetings would let them feel like they are a part of the process and it may also help them

understand the limited resources the school administration is dealing with.

Parent involvement and communication was also a recurring theme. Having the parents

more involved in the CICO program may also increase the positive perception by the teachers.

The teachers understand that the students have difficult home lives and they want the parents to

be aware of their challenging behavior and be part of the solution, if possible.

Overall communication transparency may also help with the perception of the program.

Letting teachers know more about the selection process and why certain students get in the

program and why certain students leave the program are important. The teachers had varied

understandings of the program and misinformation can lead to misunderstanding. That was clear

from the survey results and the discrepancy between the teachers who had students in the

program and those that did not.

Program Improvement and Efficiency

The Check-in Check-out program at Suburban Elementary seems to be running smoothly

and does not need major changes. There were areas that could be modified, based on the

feedback from the stakeholders in this evaluation. Any suggestions for improvement will want

to be carefully considered and studied prior to implementation.

Time considerations. The teachers and CICO administrators all indicated that there was

not enough time to counsel the students as much as they would like. While it is not realistic to

have the students leave class earlier than they already do, perhaps there could be an extended

check-out period once per week. The additional time could be for the whole group or on a

rotating basis with individual students. That extended block does not necessarily have to be at

the end of the day. It could be during lunch or some other consistent time. The purpose of this


33/85


extended blockwould not be to go over that days behavior chart, but instead go over a weeks

worth of data or spend extra time talking about the students behavior goals for the future. It will

take some planning and organization by the school administration, but an extended time period

would be a helpful modification to the CICO program.

The school administration will want to look closely at the time this program takes for

teachers who have more that one student in the program. In both focus groups, the teachers who

had more than one student in CICO indicated that keeping up with multiple student charts was

difficult. One student and one chart seemed to be manageable. The school administration will

want to further investigate alternative reporting methods for teachers with multiple students in

the program.

During the check-out process, there is a limited amount of time. The students have to

catch the bus or day care van and the CICO administrator may have an afternoon duty. The

more students the CICO administrator has, the less individual attention the administrator can

provide. The school administration may want to consider identifying more staff to serve as

CICO administrators. This could be any staff member that is not a classroom teacher. By

reducing the number of students in each group, it creates more time to work with each student at

the end of the day. This would take more coordination and oversight, but it could have a positive

effect and help alleviate some of the time constraints at the end of the day.

Communication. There are several areas where communication methods and processes

can be improved throughout the CICO program. The two major areas for potential improvement

are parent communication and staff communication.

There seems to be a misunderstanding and some inconsistencies regarding parent

communication. The CICO behavior sheet has a place for a parent signature, shown in Appendix


34/85


I. One CICO administrator does not send home the behavior sheets and the other CICO

administrator sends them home once per week. The teachers in the focus group definitely had a

misunderstanding about how to communicate with parents through this program. One teacher

stated in a focus group, see Ive been writing notes all year thinking parents were seeing the

notes at least on Fridays but I guess theyre not. The other participants also seemed unsure

about wheather or not their parents saw the CICO behavior sheets. This could be easily fixed

with a clear policy about the level of parent communication.

The teachers also indicated a desire for the parents to more involved in the process.

There were several suggestions such as having the parents come in for an orientation to the

program. When a child enters this program, they have been a consistent behavior problem and

most likely have been suspended. The parents should be motivated to come in and learn about

the program if it will keep their child from being suspended. This could also build some good

will between the school and the parent because they will see the time and personal attention that

the school is committing to their son or daughter.

There also could be more consistent communication between the teacher and the CICO

administrator. The CICO administrator plays a significant role in the students education. The

teachers indicated a desire for more consistent communication to provide some connection to the

behavior sheets. The numbers on the behavior sheets often do not tell the whole story about the

students behavior. Also, the teacher or the CICO administrator may find things out about the

students personal life that they would not put on the chart, but would want the other person to

know. Along those same lines, there needs to be consistency in the teachers reporting on the

sheet. The numbers on the sheet are the main way the teacher communicates the behavior.

Having the teachers using the CICO program participate in a norming exercise may be helpful.


35/85


The teachers could be presented with a scenario and they could discuss how they would score

that on the behavior sheets. This would help them be more consistent. It may also be helpful to

create a forum for the teacher in the CICO program. These teachers all have students who have

been identified as needing Tier Two behavior intervention. Creating a forum for these teachers

to share ideas and have a support network may also be helpful. This could be in the form of a

quarterly meeting or some other digital communication forum.

Program and school-wide considerations. The school administration may want to

consider several modifications to the CICO program. One of the ideas that surfaced through the

focus groups was having a different model for the younger students and the older students. The

teachers seemed to think that the program and the behavior chart was better suited for the older

students. Further study will need to be conducted to see if there is an alternative behavior sheet

or check-out model for the students in the younger grades.

The behavior chart itself can be a trigger for students. The whole point of the CICO

program is to reduce negative behavior, but the chart can be a trigger at the end of the day for

students who did not get the score they wanted. For certain students, the CICO administrators

may want to consider an alternative communication method to prevent the chart from becoming

a trigger for negative behavior.

An area where the teachers had a great deal of cognitive dissonance revolved around

focusing so much energy on the most disruptive students and rewarding them for normal

behavior. One of the focus group participants stated:

I think were setting some of them up for a really harsh reality check for when they get

out to real world kind of things and theres not a cop with a fanny pack full of lollipops,


36/85


because theyre expecting to get some type of reward for doing every little thing.

(Appendix D, 2013)

The teachers also wanted to recognize the students who consistently exibit positive behavior.

The teachers stated a desire to recognize these students more than every nine weeks. The school

administration may want to involve the teachers in coming up with ways to recognize the most

well behaved students as well.

A final recommendation that bridges the gap between Tier One and Tier Two behavior

support is the need for more classroom guidance. One teacher stated, We need more [guidance

lessons]. With as many issues and when I look at my class, I have twenty-six students, more than

half of my students have anger management issues. The teachers agreed that some type of

weekly guidance lesson that was consistent would be helpful. This would take a significant time

and financial commitment by the school and may be something that is provided by a grant or

some other outside funding source.

Conclusion

The school administration at suburban elementary will have many decisions to make on

how to implement the recommendations from this evaluation or how to further evaluate different

aspects of the Check-in Check-out program. The limits of this evaluation should be considered

when considering the findings.

Limitations of this evaluation. Time and resources were a factor in this evaluation.

There was only one evaluator conducting this research and therefore the suggestions and findings

are biased by a single point of view. Furthermore, because of the confidential nature of this

evaluation, access to certain stakeholders was limited. The evaluator worked directly with the

school principal and did not go through the school system central administration. This was


37/85


mutual decision between the client and the evaluator, but it meant that there were limitations to

how the students could be surveyed and interviewed. Also, not including the parents in this

evaluation was a strategic decision made by the school administration. This evaluation mostly

focused on the perceptions of teachers and administration at suburban elementary and therefore

should not be considered a complete representation of all stakeholders.

The evaluator only conducted one round of interviews, focus groups, and surveys. This

was done out of respect forthe school staffs time at a busy time of the year for school personnel.

Ideally, there would be follow up with multiple groups over time.

Lastly, to truly evaluate the CICO program, it would need to be evaluated within an

evaluation of the entire school-wide behavior support program. The evaluator only included a

small portion of the questioning on the Tier One program.

Recommendations for future evaluations. The school administration at Suburban

Elementary may wish to further the investigation of the behavior programs. Based on the results

of the current program evaluation, the following students could be considerations for follow up

studies.

Implementation fidelity. The research indicates that a Check-in Check-out program is

only as good as the degree to which the school staff implements the program with fidelity.

Future studies should focus on observing the daily behavior sheet implementation. This current

study did not look at the teacher-student interaction, only the CICO administrator-student

interaction. A future study designed to track the fidelity of the implementation by the teachers

would be a logical follow-up.

Multiple students per class. One of the concerns that surfaced during the focus groups

was the effectiveness of the program when there were multiple students in the program within a


38/85


single class. A follow up study should be conducted to attempt to identify if there is a drop off in

effectiveness when more than one student in a class enters the CICO program.

Parent and student perceptions. A follow up study should be conducted with the proper

approval from the central office administration that involves more students and the parents of the

students in the CICO program. The data from the student survey was limited and no parents

were involved in the current evaluation. These are two important stakeholder groups that were

marginalized and should be included in future evaluations.

Longitudinal study. The administration at Suburban Elementary should consider

tracking the long-term outcomes of the CICO program. This could include students staying in

the program for multiple years as well as students who graduate from the program. This type of

study would investigate the outcomes and impact referenced in the log model in Figure 1.

Conclusion

The findings from this study will be helpful to the school administrators at Suburban

Elementary or to a school administration looking to implement a Tier Two behavior support

program such as Check-in Check-out. This type of program relies on relationships that are

formed between adults and students. Relationships take time and are difficult, if not impossible

to quantify. The current research and the findings from this study both confirm the need for

teachers and staff that are dedicated to students and willing to follow a program. Even if the

program is not implemented with complete consistency and fidelity, the relationship are what

make the difference in childrens lives.


39/85


References

Anderson, E. (2009). The effectiveness of a proactive school-wide discipline plan on office

discipline referrals at the elementary school level. (Unpublished doctoral

dissertation). Liberty University, Virginia.

Caldarella, P., Shatzer, R. H., Gray, K. M., Young, K., & Young, E. L. (2011). The Effects

of School-wide Positive Behavior Support on Middle School Climate and Student

Outcomes.Research In Middle Level Education Online, 35(4), 1-14.

Cheney, D., Lynass, L., Flower, A., Waugh, M., Iwaszuk, W., Mielenz, C., & Hawken, L.

(2010). The Check, Connect, and Expect Program: A Targeted, Tier 2 Intervention

in the Schoolwide Positive Behavior Support Model.Preventing School Failure,

54(3), 152-158.

Cohen, R., Kincaid, D., & Childs, K. (2007). Measuring School-wide Positive Behavior

Support Implementation: Development and Validation of the Benchmarks of

Quality.Journal Of Positive Behavior Interventions, 9(4), 203-213.

Hawken, L. S., & Horner, R. H. (2003). Evaluation of a Targeted Intervention Within a

Schoolwide System of Behavior Support.Journal Of Behavioral Education, 12(3),

225-240.

Horner, R., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A., & Esperanza, J. (2009).

A randomized, wait-listed controlled effectiveness trial assessing school-wide

positive behavior support in elementary schools.Journal of positive behavior

interventions, 11(3), 133-144.

Hoyle, C. G., Marshall, K. J., & Yell, M. L. (2011). Positive Behavior Supports: Tier 2

Interventions in Middle Schools.Preventing School Failure, 55(3), 164-170.


40/85


Luiselli, J., Putnam, R., Handler, M., & Feinberg, A. (2005). Whole-school positive

behavior supports: Effects on student discipline problems and academic

performance.Educational Psychology, 25(April-June), 183-198.

Martz, M. (2012, November 9). Henrico points to habitual offenders for high student

suspension rate: School suspensions examined.Richmond Times-Disbatch. Retrieved

from http://www.timesdispatch.com/news/henrico-points-to-habitual-offenders-for-high-

student-suspension-rate/article_2657cb19-24c4-535a-9b29-ee9bc83c0ec1.html

Medley, N. S., Little, S. G., & Akin-Little, A. (2008). Comparing Individual Behavior

Plans from Schools With and Without Schoolwide Positive Behavior Support: A

Preliminary Study.Journal Of Behavioral Education, 17(1), 93-110.

Miramontes, N. Y. & Marchant, M. & Heath, M. A. & Fischer, L.(2011). Social Validity

of a Positive Behavior Interventions and Support Model.Education and Treatment

of Children 34(4), 445-468. West Virginia University Press.

McIntosh, K., Campbell, A., Carter, D., & Dickey, C. (2009). Differential effects of a tier two

behavior intervention based on function of problem of behavior. Journal of positive

behavior interventions, 11(2), 82-93.

Stake, R. E. (2004). Standards-based and responsive evaluation. Thousand Oaks, CA: Sage.

Todd, A., Campbell, A., Meyer, G., & Horner, R. (2008). The effects of targeted intervention

to reduce problem behaviors: Elementary school implementation of check in - check

out. Journal of positive behavior interventions, 10(1), 46-55.


41/85


Appendix A

Interviewee: Resource TeacherInterviewer: Joe Koontz, Program Evaluator

Location: Highland Springs Elementary

Date: February 4, 2013

Koontz: Tell me about the check in check out process from beginning to end.

DV: It really is for our heavy hitters. It tends to be our kids from St. Josephs Villa, but itcan be othersour heavy hitters. They come it in the morning when they get off the bus.

They get their breakfast and then they come to check-in. The check in process is fairly

simple. They come and they fill out their charts, we talk about the goals for the day and

then they leave. Im pretty structured so its almost independent. Because Im out on thebus loop, the kids can come in the fill out their thing, the date is already up there they fill

in their name the date and generally they tend to pick the same thing. These are the target

behaviors right here (refers to the CICO sheet)

Koontz: What do you mean they pick?

DV: They pick their goals for the day. A lot of them tend to pick self control becausethats their biggest issue. Ive started to

Koontz: Self control would be keep your hands to yourself?

DV: Yea but it could also be as simple as self control and they write it here. With me

what they do name date and we do their goal. Now if you were to fill this out and you

wrote that your goal was self control, I would ask you before you go out the doorwhatdoes that mean? What does that mean to you? How does that look? Thats where they

would say to me, it looks like I would keep my hand to myself, Im not talking back, if

somebody hits me Im not going to fight back. Then I say, alight, youre ready. Have agood day. Then they give this thing (CICO Sheet) to their teacher. Their teacher then, all

different parts of the day can rate them along these three categories, but also keeping in

mind how their goal goes into that. Three is the highest score.and then.someteachers dont total it up. Im ok with doing it. The highest you can get is a 54. When

they come back to me in the afternoon, we look at it. We kind of talk about what where

their challenges and difficulties and things like that. The teacher sometimes writes

comments.

DV: Let me show you a completed one. The teacher wrote.

Koontz: Is there a target that they are trying to reach?

DV: 54, 54 is the highest you can get. We always say we want a 54 day. Is what I say to

them and when they come in in the afternoon they go to that thing behind (pointing to thecharts) you and they record their scores.


42/85


Koontz: For the month?

DV: We just go week to week. Today is Monday, so well start a brand new line. That girl

(name deleted) you can tell that her scores have fall all over the place.

Koontz: So is a 53 still a pretty good day?

DV: A 53 is still a pretty good day. I dont necessarily group them like thatyou might

have gotten a 45, but your teacher said, you worked really hard. You had a roughmorning, but you got it together. I think thats a pretty good day. For my reward system,

the person with the highest score gets to go to my treat box or gumball machine. Its a

little friendly competition just between them. If multiple people get a 54, whoever that is

gets to go to the treat basket. So, if you get a 54 every day, you get to go to my treat boxevery single day, if not, we just go by who has the highest score. Then they lean them

(the sheets) in their folders. They dont take them home because if they took them home

Ill never see them again. Then their right here for the next day.

Koontz: When a kid come in with a 40, how does that go with keeping track of the points?

Does it makes it better or worse? The competition of it?

DV: Before they walk in and before I see their folder, I can tell by the way they walk in

the door what kind of day they had. If theyre really excited then theyll come in and tell

you, I got a 54! Or, they come inthis one particular kids just kind of come in and pushis folder down and watches me and then I kind of knownot too goo ha? Sometimes

Ill look at it and say, thats not too bad. (Student name) for instance I had to call his

mom because I kept telling him, if you dont get out of the 30s, Im going to have to call

your mom. We were working on.his last 39, I said I need today to be a 40 day. I justwant you to aim for a 40. A number with a 4 in front. When he came in he was like, I

got it I got a 4 he knew hes in second grade he knew he had because he could tell and

he was excited. Of course you can see the next day he went down to a 29.

Koontz: How are students selected for the program?

DV: Based on the most number of behavior referrals. I hate to say reputation, but

reputation. And what I mean by that is (name) for instance. He is currently suspended.

He hasnt been suspended in over a year, but he got into a fight on Friday. (Name) does

not have any suspensions and he didnt have any referrals, so from his past history fromlast year and the year before that, thats why we keep him on a tight leash (in the

program), plus him and I have a great relationship, we fear that if we take him off he

might start to regress. A lot of it is prevention, but some of it is you need to be in here.

Koontz: Is there a referral process for teachers?

DV: These are kids who need a little extra. How does a teacher refer. There is a teacher


43/85


in first grade she wanted me to take on a new kid because a kid had graduated from the

program who kept getting 54s. The teacher was like, Ill take him back if you take a

different one. We told her she couldnt just do that. We told her she neededdocumentation that says he needs to be in there because this is tier 3. He who youre

referring is not really ready for this yet, but I can check in with him when he needs me or

call for me and Ill come check in with him, he needs to get in a little more trouble beforeentering this program.Koontz: Child study team ever refer a student as a strategy out of child study?

DV: It could come up. I dont know how many of them this year. We just look at thenumber of time in the office.

Koontz: Is there a formal process for that like when they hit a certain number of referrals?

DV: It big picture, its anyone who goes to New Bridge (suspension intervention

program) because we want to keep tabs on them. There is a kid right now (name) who is

a 4

th

grader he is not in CICO, but when he comes back he will be because he has hadmultiple issues this school year. Hes been there about a month now.

Koontz: What do you think is the most effective part of the CICO program?

DV: The relationship building.

Koontz: Tell me a little more about that.

DV: What I mean is Im kind of one of those teachers that is kind of cool. I dont think that

any of my child that I have in CICO think that it is a bad thing or because their bad. They

are excited to come to me, they want to show me their folders. They take responsibilityfor it. I instill an expectation that you have to take care of this folder. Ive bade it pretty

much easy for them. So, relationship building and it gives them a sense of responsibility.

It helps them to monitor their own behaviors because they know they have to report abouttheir day.they may think about it a little more. I see kids throughout the day that

sayI got all 3s so far! They want to tell me.

Koontz: It sounds like it gives you a vocabulary to quantify a qualitative action?

DV: It gives them something to strive for something to keep in the front of their mind. I need

to make sure my day is going good because they know they need to answer me at the endof the day. It also helps when A kid hasnt been doing so well, when the teacher picks up

the folder to mark it sometimes their behavior can go down.

Koontz: What are the biggest challenges with the program?

DV: I would say consistency. Because if a teacher hasnt bought into it , if the teacher is not

consistent enough to keep this folder at a certain part and fill it out at the end of the day.I dont want to wait till the end of the day and their just circling things because you just


44/85


want to circle something. I would like it to be meaningful where they can actually talk

about or either give a kid a chance so their having a little off task and you are repeating

and talking to them to many times. You say now if I speak to you one more time Imgoing to mark you a 1 on your chart. At least thats what I would do so were having that

dialog where Ive spoken to you and you have an understanding of my understanding. So

consistency and teacher buy-in and teacher application for them to use it effectively in theclassroom.

Koontz: So the teacher has more to do with this than you do. Do you have any expectations

for the teachers in terms of a teacher that is getting a kid for the first time? Anytraining?

DV: Yes, (name)s teacher is new. He is coming off a weeks suspension. I went to the

teacher and said, he is having a meltdown in the morning when it comes to filling thisout. I told her, dont send him to me, Im going to bring it to you and you and him can

work on it whenever you get a chance. If you dont fill it out all the way, thats ok

because hes in kindergarten so it frustrating for him to write and he, when his medicinewas off he was flipping tables over. So, I go on a case by case basis. There is no initial

training because theyve all seen it.

Koontz: Is it more effective with the older students?

DV: Yes, because they definitely take more ownership. The younger student couldnt get that

is was a daily thing.

Koontz: Can kids ever exit the program?

DV: Yes. We meet once a quarter to look at their average scores. For a kid who averages a36, hes not really ready. A student who averages a 52 like (name) who was only a

behavior problem when he was hungry. We dont want to hold that against him. When

he was on, he was on. Talking to his teacher he could um, not be in CICO. We hersupport, we could take him off and so he graduated out. All the other kids would say

where is (name)? I would say, he graduated from the program. They kids would then

ask, when do I graduate?

Koontz: When kids graduate, do the teachers keep them on some type of behavior plan?

DV: I dont think so. I dont think they do.(Interruption here)

DV: No I dont think Ms. Hunt has him on any specific behavior thing. She knows that

(name) and I have a good relationship. Sh

Koontz 713 Program Evaluation Full Report

Documents

Transcript of Koontz 713 Program Evaluation Full Report