Behavioral Interventions, Vol. 10, No. 4, 225-236 (1995)

A SYSTEM OF PROGRAM EVALUATION AND THE USEOF FEEDBACK AS A MEANS OF IMPROVING SERVICE DELIVERY

Dawn M. Smith Bancroft, Inc., Hopkins Lane, Haddonfield, NJ, 08033, USA

Agencies providing services to individuals with developmental disabilities must demonstrate quality programing as evidenced through a high level of staff compliance to organizational, licensing, and accreditation standards. In addition, economic and practical considerations impose the need for cost-effective procedures for evaluating and improving staff performance. This study utilized a combination of checklists, as an evaluative tool to measure staff performance, and private feedback to improve staff performance. A multiple baseline design was used to determine the effects of providing staff with (a) orientation training, (b) task-analyzed checklists of job skills, (c) private feedback from the agency trainer, and (d) private feedback from their immediate supervisor. Results indicate that training alone does not ensure competency; checklists alone do not ensure competency; private feedback from trainers improves competency; and private feedback from direct supervisors improves competency even further.

Since the passage of Public Law 94-142 (1975), public school districts have been mandated to provide free, appropriate, individualized education plans for children with developmental disabilities. As a direct result, many schools have opted to fund individuals to receive that education at private agencies. In addition, many adults having developmental disabilities are also provided services by private agencies.

With the continual move toward deinstitutionalization and mainstreaming, along with the high cost of services at private agencies, funding sources are searching for programs which offer quality services at a reasonable cost. Therefore, it has become incumbent upon the agencies to bear the burden of proof that their programs are of high quality as evidenced through program evaluation and accreditation.

Preparation of this manuscript was completed to fulfil the requirements for a Master’s Degree from Temple University.

The author would like to thank Todd Jarger for his assistance in data collection, and Drs Kathy Dyer and Saul Axelrod for their comments on earlier drafts.

Address correspondence to Dawn M. Smith, Bancroft, Inc., Hopkins Lane, Haddonfield, NJ, 08033, USA.

CCC 0084-558 1/95/04O22SlZ 01995 by John Wiley & Sons, Ltd.

226 D . M . Smith

One means of ensuring quality programing is to provide staff with adequate training. Unfortunately, many studies show that providing workshops and in- service training is not sufficient to maintain a high level of services (Hollander & Plutchik, 1972; Iwata, Bailey, Brown, Foshee, & Alpern, 1976; Panyon, Boozer, & Morris, 1970; Quilitch, 1975; Realon, Wheeler, Spring, & Springer, 1986). Knowing what is expected does not ensure compliance with those expectations. Therefore, agencies must develop methods of evaluating programs via staff compliance, as well as means of improving that compliance. A system of evaluating current programs and staff performance is needed to identify where quality programing exists and which areas require improvement.

Many studies in the literature have measured client gains as a means of program evaluation (Green, Willis, Levy, & Bailey, 1978; Pomerleau, Bobrove, & Smith, 1973), thereby implying that client gains result from improved staff performance. However, many variables of staff performance cannot be adequately measured through client gains. To be more inclusive, an evaluation of quality programing should include direct observation and measurement of actual staff performance in the work environment.

Repp and Deitz (1979) studied staff compliance with certain administrative duties (i.e., the timely submission of accurate monthly summary reports) and found that feedback for “on-time’’ and “approved” reports resulted in a high level of staff compliance. They suggest that studying various other areas of staff performance would produce similar results.

Lattimore, Stephens, Favell, and Risley (1 984) also studied staff performance and compliance by using prescriptive checklists which were effective training and management tools. They found that when the results of the checklists were shared with staff, their performance greatly improved.

The use of feedback for staff performance has been widely studied in the literature (Ivancic, Reid, Iwata, Faw, & Page, 1981; Panyon et al., 1970; Quilitch, 1975; Welsh, Ludwig, Radiker, & Krapfl, 1973). In most cases, however, feedback has been correlated with other contingent reinforcers. For example, Realon et al. (1986) found that providing feedback and praise had mixed results in maintaining staff performance, but when feedback and praise were paired with contingent monetary reinforcement, staff performance immediately improved and was maintained at a high level. Iwata et al. (1976) used feedback with a performance-based lottery. Staff were provided feedback and an opportunity to win days off from work as a reinforcer for improved performance. After a review of the literature, Iwata et al. (1976) found many studies that indicate that feedback alone is effective. They suggest that feedback may actually be combined with other consequences of a reinforcing or

Program evaluation and feedback 227

punishing nature, thereby altering its inherent effectiveness, even though the role of those correlated events is not acknowledged.

Unfortunately, the use of tangible reinforcers is generally impractical and costly for agencies providing services. Therefore, providing feedback alone may have to be an alternative method of improving staff performance. The question is whether or not feedback alone will be effective.

Many studies have addressed the issue of feedback alone and have provided ambiguous results regarding its effectiveness. One dimension which has been studied is the effectiveness of private vs. public presentation of feedback.

The research has shown that public posting of performance is generally effective in improving staff performance. For example, Ivancic et al. (1981) found that providing feedback by public posting of performance improved staff-client interactions during daily care routines. Green et al. (1978) evaluated client gains to measure staff performance and found that feedback and public posting improved performance significantly. Several other studies support this as well (Quilitch, 1975; Repp & Deitz, 1979; Welsh et al., 1973).

Less emphasis seems to have been placed on the use of private feedback; however, some studies have shown it to be effective (Dyer, Schwartz, & Luce, 1984; Panyon et al., 1970). Dyer et al. (1984) found that providing staff with immediate private feedback following observations improved the staffs performance of engaging clients in age-appropriate, functional activities. Panyon et al. (1970) suggest that initiating private feedback as soon as possible following training is an effective means of maintaining high performance levels.

Private feedback may be more pragmatic in community-based service programs. It is easier to deliver and more immediate than public posting. In addition, many staff in community-based programs are isolated from one another, so public posting would seem impractical.

Who delivers the feedback is another dimension involved in its effectiveness. Many studies suggest that feedback is significantly more effective when provided by supervisory personnel (Ivancic et al., 1981; Page, Iwata, & Reid, 1982; Panyon et al., 1970; Quilitch, 1975; Repp & Deitz, 1979). Yet, Burgio, Whitman, and Reid (1983) surveyed staff involved in an intervention program and found that supervisory feedback and praise were only moderately effective.

This author has attempted to study further both the effects of private feedback and whether feedback delivered by supervisors is more effective than that provided by agency trainers.

As mentioned above, programs must identify means of evaluating and improving staff performance that are practical and cost effective. Therefore, it can be hypothesized that having supervisors provide private feedback to staff fits those criteria.

228 D. M. Smith

This study utilized a combination of checklists as an evaluative tool and medium for feedback to staff. The purpose of the study was:

1. to replicate findings that show in-service training alone is not sufficient to ensure competency of job skills;

2. to determine whether providing staff with written task-analyzed checklists of job skills improves competency;

3. to determine whether direct private feedback from the agency trainer increases competency of job skills, and;

4. to determine whether private feedback from direct supervisors has more effect on improving competency of job skills than feedback from the trainer.

METHOD

Subjects and Setting

The subjects of this study were direct-care staff at a private agency providing services for children and adults with developmental disabilities. These four staff persons work in a community-based apartment program serving adults with challenging behaviors. One subject was female, the other three were male with a mean age of 23. All had some college level coursework and average length of employment was 7 months.

Instrument

A number of checklists was developed for use in this study by the author and two other staff education specialists employed by the organization. Each checklist identifies specific job skills in which the staff have been trained, as well as other items deemed necessary by the organization, state licensing agencies, and accreditation standards for quality programing to exist. An example of one of the checklists can be seen in Figure 1.

Not all checklists applied to all situations and observations; therefore, the researcher and checklist developers surveyed supervisory personnel to determine the content validity of the checklists (per regulatory standards) as well as which checklists were relevant and valid for each of their particular units.

Procedure

As mentioned above, the researcher surveyed program supervisors to determine the content validity and relevance of the checklists for each of the

Program evaluation and feedback 229

Evaluator:

Program Site: Date :

Instructions: Administer checklist by direct questioning, observation, and/or permanent product.

A. Unlocks box of medications

B. Finds correct medication to give from the medication administration record

C. Finds corresponding pharmacy bottle

D. Compares label, IIAR, & prescription.

E. If liquid, dispenses correct dosage into

P. If pill or capsule, dispenses correct

measuring unit

dosage

i. into cap

ii. into disposable cup

G. Recaps pharmacy bottle

H. Offers water and/or juice

I. Witnesses swallowing of medication

J. Completes medication administration record

K. Returns items to locked area.

- 1.

2.

3.

4.

5 .

6.

7.

8 .

9.

10.

11.

12.

Feedback received (initial h date):

Figure 1. Example checklist.

units. The researcher then made up packets of appropriate checklists for each unit. In addition, staff were surveyed to determine which particular job skills cause difficulty for them and in which areas they would like more training.

The first phase of the study was to establish baseline measures. Baseline measures were calculated as the percentage of compliance to the items on the checklists (the number of “yes” responses over the number of total possible “yes” responses multiplied by 100). Two of the agency trainers went to the respective units and observed staff in their program areas. Each observation

230 D . M . Smith

took no longer than 30min. During observations, the trainers checked the appropriate column on the appropriate checklist. Some checklists could not be completed by direct observation, in which case the trainers questioned staff and recorded answers. The trainers visited each program area at least four times to establish stable baselines and to calculate interobserver reliability.

Next, staff were given the task-analyzed checklists and told that those job skills were the ones being observed by the trainers (Checklists Alone). In doing so, staff knew exactly on which job skills competency was expected. The author continued to visit program areas and complete checklists with periodic reliability checks.

Third, the trainer continued to make observations and complete checklists, but now provided private verbal feedback and shared the completed checklists with the staff following each observation session (Checklists with Trainer Feedback). Feedback consisted of pointing out areas in which the staff did well, along with constructive suggestions on how to improve areas which may have been lacking.

Finally, the staffs direct supervisors conducted observations and provided the feedback without the trainer (Checklists with Supervisor Feedback). Again, interobserver reliability checks were conducted.

Design

A four tier multiple-baseline design (Baer, Wolf, & Risley, 1968) was used across subjects. As indicated above, there were four phases in the study:

A - Baseline B - Checklists Only/No Feedback C - Checklists With Trainer Feedback D - Checklists With Supervisor Feedback

Results can be seen graphically in Figure 2.

Reliability

Interobserver reliability was calculated throughout the study. Initially, two agency trainers visited program areas and conducted observations simultaneously. After each of the first four observation sessions, the percent of agreement was calculated by dividing the number of agreements by the

Program evaluation and feedback 23 1

TABLE 1. Interobserver reliability

Checklist Low High Mean

Recfleisure Mealtime Pen. hygiene Info. on persons Diffusion Incident reports Med. documentation Med. administration Operations manual Family communication

71% 55%

100% 69 % 66 % 78 % 85%

100% 60%

100%

100% 94 %

100% 100% 100% 100% 92 %

100% 100% 100%

89% 81%

100% 90% 91% 89% 91%

100% 82%

100%

number of agreements plus disagreements and multiplied by 100. Reliability was calculated for both occurrence and nonoccurrence of staff behavior.

Interobserver reliability was also calculated when supervisors began observations. The same formula as above was used to determine the percent of agreement between the supervisor and the trainer (who did the reliability checks). Interobserver reliability was calculated for each of the utilized checklists for 23% of the observations.

Reliability scores ranged from 55% agreement to 100% agreement with varying means for each checklist. The mean reliability was over 80% on all checklists. Overall means for reliability on individual checklists can be found in Table I .

RESULTS

Following each observation session, the overall percentage of compliance and adequate staff performance (as indicated by the number of “yes” responses on the checklists) was calculated and graphed. The data for all subjects are presented in Figure 2.

For Subject 1, the mean percent correct on the checklists during baseline was 73 (range 72-74), during the checklists only phase was 75 (range 73-76), during the checklists with trainer feedback phase was 90 (range 77-98), and during the checklists with supervisor feedback phase was 100 (range 100-100).

For Subject 2, the means for each of the phases were 71 (64-77), 77 (73-80), and 90 (78-96). This subject did not have scores for the final phase because employment was terminated prior to completing the study.

For Subject 3, the means were 66 (59-71), 74 (7&78), 85 (82-88), and 99 (97-100).

232 D. M. Smith

Figure 2. Percent correct on task-analyzed job skill checklists.

Program evaluation and feedback 233

And for Subject 4, the means were 69 (60-74), 74 (71-77), 78 (65-85), and 99 (97-100).

DISCUSSION

The results of this study show that staff competency and the performance of required job skills can be improved through the utilization of a system of checklists, observations, and private feedback.

It appears that feedback was reinforcing, especially when delivered by the direct supervisors; however, it could be argued that staff compliance and performance were avoidance responses. Although performance improved when feedback was provided by the trainer, it never reached the level attained when the supervisors provided the feedback. It is possible that the high rate of staff compliance was performed in an effort to avoid punishment from the supervisors for noncompliance. This was not an issue when the trainer provided feedback because the trainer had no authority over the staff and/or their jobs. A follow-up questionnaire may have shed some light on this issue; unfortunately, this particular program area has a high staff turnover rate, and each of the subjects left employment and were unavailable following the study.

One issue of concern regarding the results might be the possibility of improvement due to reactivity. This researcher, however, does not believe this to be the case. Many observations were conducted by the trainer during the baseline condition and again after the checklists had been distributed. Figure 2 shows there was virtually no change in the data until feedback was implemented, thereby indicating that improvement was due to feedback, not reactivity.

Another area of concern is the lack of data regarding maintenance and generalization. Long-term maintenance was impossible to measure due to the high staff turnover previously mentioned. Each of the subjects left employment shortly after the completion of the study. Interestingly, this researcher often had difficulty receiving the completed checklists from the supervisors in a timely manner. Therefore, it could be argued that the results do show maintenance since the supervisor’s observations and feedback were conducted in such an intermittent fashion. It can be reported anecdotally that the supervisors in this study did not receive feedback from their supervisors. It would be interesting to determine if the treatment effect would continue up the supervisory line and on to administrative personnel.

234 D . M . Smith

In addition to difficulties measuring maintenance, there were no measures of generalization. Each subject reported to and received feedback only from their immediate supervisor. Future research should focus on this potential problem.

Although the discussion thus far has indicated several areas of concern, it should be noted that the results of this study generally answer the experimental questions. Indeed, baseline measures support previous research by Realon, Wheeler, Spring, and Springer (1986) that indicate that in-service training is not sufficient to maintain competence and performance of job skills. Each of the subjects had attended a 40hr week of in-service training and all had demonstrated competency prior to the completion of training. However, as can be seen, this competency was not sufficiently demonstrated on the job in program areas.

When checklists similar to those used by Lattimore, Jones, and Calvert (1979) were introduced and distributed to staff, there was virtually no improvement in performance. These data support the conclusion by Luce and Dyer (in press) that staff performance may not reach optimal levels without systematic feedback.

Another aspect of this study was to determine whether private feedback from the agency trainer would improve competency. The data presented here indicate that indeed it does; however, it is obvious that the feedback from the trainer was not as effective as that provided by the supervisors. This has been shown to be true in several other studies (Dyer, 1989; and Dyer, Murray, Hoffman, & Luce, 1990) which indicate that staff will not display competency when feedback is provided by nonsupervisory personnel.

Of major importance is the simplicity with which this study was conducted. This system of using checklists, observations, and feedback requires very little response effort on the part of the supervisors as opposed to the system utilized by Reid, Parsons, and Green (1989). Observations may be conducted on an intermittent basis and are not terribly time consuming for the supervisors. In addition, there is high efficiency and low monetary cost to the organization. Furthermore, a recent study by Schepis and Reid (1994) suggests that delayed feedback may achieve the same results as immediate feedback, thereby lessening the burden on the supervisors even more. This avenue of inquiry should be studied using the system discussed herein.

Future research efforts should focus on the above mentioned areas of maintenance and generalization, immediate vs. delayed feedback, staff acceptance of this system, and the possibility of self-management if supervisors do not provide feedback.

In conclusion, the decentralization of services due to mainstreaming and deinstitutionalization means that staff are less frequently monitored and

Program evaluation and feedback 235

supervised. As Luce and Dyer (in press) have stated, these staff are required to gain knowledge of many job skills and perform them competently. Ferguson and Cullari (1983) suggest that procedures for improvement in staff compliance in these job skills should focus on techniques that can be practically implemented without a large time requirement. The results of this study indicate that this system of checklists, observations, and feedback can do just that.

REFERENCES

Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, I , 91-97.

Burgio, L. D., Whitman, T. L., & Reid, D. H. (1983). A participative management approach for improving direct-care staff performance in an institutional setting. Journal of Applied Behavior Analysis, 16, 3 7-53.

Dyer, K. (1989, May). Preference and choice: Classroom applications. Paper presented at the Annual Convention of the Association for Behavior Analysis, Milwaukee.

Dyer, K., Murray, M., Hoffman, K., & Luce, S. C. (1990, May). Zntegrating choice andpreference into classrooms for students with severe handicaps. Paper presented at the Annual Convention of the Association for Behavior Analysis, Nashville.

Dyer, K., Schwartz, J. S., & Luce, S. C. (1984). A supervision program for increasing functional activities for severely handicapped students in a residential setting. Journal of Applied Behavior Analysis, 17, 249-259.

Education of the Handicapped Act (EHA) (P.L. 94142) (1975). Ferguson, J. D., & Cullari, S. C. (1983). Behavior modification in facilities for mental retardation.

In S. B. Breuning, J. L. Matson, & R. P. Barrett (Eds.), Advances in mental retardation and developmental disabilities: A research annual (pp. 145-192). Greenwich, CT: JAI Press.

Green, B. F., Willis, B. F., Levy, R., & Bailey, J. S. (1978). Measuring client gains from staff- implemented programs. Journal of Applied Behavior Analysis, 11, 39S412.

Hollander, M. A., & Plutchik, R. (1972). A reinforcement program for psychiatric attendants. Journal of Behavior Therapy and Experimental Psychology, 3, 297-300.

Ivaneic, M. T., Reid, D. H., Iwata, B. A., Faw, G. D., & Page, T. 3. (1981). Evaluating a supervision program for developing and maintaining therapeutic staff-resident interactions during institutional care routines. Journal of Applied Behavior Analysis, 14, 95-107.

Iwata, B. A., Bailey, J. S., Brown, K. M., Foshee, T. J., & Alpem, M. (1976). A performance- based lottery to improve residential care and training by institutional staff. Journal of Applied Behavior Analysis, 9, 417-431.

Lattimore, J., Jones, M. L., & Calvert, T. L. (1978). Roadrunner project: Caregiver manual. Morgantown: Western Carolina Center.

Lattimore, J., Stephens, T. E., Favell, J. E., & Risley, T. R. (1984). Increasing direct-care staff compliance to individualized physical therapy body positioning prescriptions: prescriptive checklists. Mental Retardation, 22(2), 79-84.

Luce, S. C., & Dyer, K. (in press). Behavioral organizational systems useful in the treatment of severe behavior disorders. The Behavior Analyst.

Page, T. J., Iwata, B. A., & Reid, D. H. (1982). Pyramidal training: A large scale application with institutional staff. Journal of Applied Behavior Analysis, 15, 335351.

Panyon, M., Boozer, H., & Moms, N. (1970). Feedback to attendants as a reinforcer for applying operant techniques. Journal of Applied Behavior Analysis, 3, 1-4.

236 D. M. Smith

Pomerleau, 0. F., Bobrove, P. A., & Smith, R. H. (1973). Rewarding psychiatric aides for the behavioral improvements of assigned patients. Journal of Applied Behavior Analysis, 6,383-390.

Quilitch, H . R. (1975). A comparison of three staff-management procedures. Journal of Applied Behavior Analysis, 8, 59-66.

Realon, R. E., Wheeler, A. J., Spring, B., & Springer, M. (1986). Evaluating the quality of training delivered by direct-care staff in a state mental retardation center. Behavioral Residential Treatment, 1(3), 199-21 2.

Reid, D. H., Parsons, M. B., & Green, C. W. (1989). Staff Management in Human Services. Illinois: Charles C . Thomas.

Repp, A. C., & Deitz, D. E. D. (1979). Improving administrative related staff behaviors at a state institution. Mental Retardation, 17, 185192.

Schepis, M. M., & Reid, D. H. (1994). Training direct service staff in congregate settings to interact with people with severe disabilities: A quick, effective, and acceptable program. Behavioral Interventions, 9(1), 1326.

Welsh, W. V., Ludwig, C., Radiker, J. E., & Krapfl, J. E. (1973). Effects of feedback on daily completion of behavior modification projects. Mental Retardation, 11(4), 24-26.