Course Evaluation Report - Eppley...

Course Evaluation Report Capstone Study in Facility Management

Timothy Harvey Facility Management Officer WASO National Park Service

Elizabeth A. Dodson Training Manager

WASO-PFMD National Park Service

June 2009

Completed in accordance with Task Agreement J2420060049 of the National Park Service-Indiana University Cooperative Agreement CA: H2420060015

Stephen A. Wolter Executive Director

Christy McCormick Project Team

Zach Carnagey Project Team

Eppley Institute for Parks & Public Lands

Indiana University Research Park 501 N. Morton Street, Suite 100

Bloomington, IN 47404 812.855.3095

Acknowledgements The following individuals contributed to the course development or instruction of the Capstone Study for Facility Management

National Park Service Betsy Dodson Tim Harvey Steve Hastings Don Mannel Jeri Mihalic Don Hellmann Michelle Proce Debbie Simmons Peggy O’Dell Costa Dillon Deb Harvey Sean Kennealy

Eppley Institute Matthew Berry Allene Lowrey Christy McCormick Allison Parman Phyllicia White Zachary Carnagey Steve Wolter

Table of Contents Introduction ................................................................................................................................ 1

The Facility Manager Leaders Program .................................................................................. 1 The Problem and Need ....................................................................................................... 1 The Participants .................................................................................................................. 2

The Capstone Study in Facility Management ............................................................................. 3 Course Specifics ..................................................................................................................... 3

Process of Evaluation ................................................................................................................. 4 Methodology ........................................................................................................................... 4

Evaluations of the Instructors .............................................................................................. 4 Participant Course Evaluations ........................................................................................... 4

Evaluations of the Instructors ..................................................................................................... 5

The Audience ......................................................................................................................... 5 The Course Coordinators ....................................................................................................... 5

Guest Instructors .................................................................................................................... 5 Course Agenda and Corresponding Guest Speakers ............................................................. 6

Quantitative Instructor Feedback ............................................................................................ 7 Qualitative Instructor Feedback .............................................................................................. 8

Participant Course Evaluations .................................................................................................. 9

Quantitative Course Feedback ............................................................................................... 9 Qualitative Course Feedback ............................................................................................... 10

Most Useful Activities ........................................................................................................ 10 Least Useful Activities ....................................................................................................... 10

Suggestions for Improvements.......................................................................................... 11 Other Comments ............................................................................................................... 11

Summary and Recommendations ............................................................................................ 12 Summary .............................................................................................................................. 12

Course Content ................................................................................................................. 12

Course Logistics ................................................................................................................ 12 Instructors ......................................................................................................................... 12

Recommendations ................................................................................................................ 12 Appendix A: Instructor Evaluation ............................................................................................ 16 Appendix B: Course Evaluation ................................................................................................ 21

Table of Figures Figure 1: Table of job titles represented ..................................................................................... 5

Figure 2: Course Agenda and Guest Speakers .......................................................................... 6 Figure 3: Instructor Quantitative Feedback ................................................................................ 7 Figure 4: Mean Scores for the Quantitative Evaluation Statements ........................................... 9 Figure 5: Percentage of Responses to the Quantitative Evaluation Statements ...................... 10

Capstone Study in Facility Management Evaluation Report

June 2009 1

Introduction The Capstone Study in Facility Management, presented April 14, 2009 to April 17, 2009 in Washington, D.C., was the final course in a five-part series of courses in the Facility Manager Leaders Program (FMLP). This week-long course focused on specific competencies that future leaders in Facility Management need to develop. As the culminating course for the FMLP, students experienced the legislative realities of asset management, took the opportunity to meet with National Park Service (NPS) chiefs and policy makers, and were asked to present their best work from the year-long program. The main focus of the course was on the articulation and demonstration of new knowledge gained by successful completion of all prior FMLP coursework and a poster presentation with an accompanying portfolio. Additionally, students focused on life-long learning, leading change, and improving asset management in the NPS. This report reviews the need for the FMLP and includes a brief description of the course, the evaluation process, and the evaluation results for the Capstone Study in Facility Management.

The Facility Manager Leaders Program

The Problem and Need Over the past thirty-five years, the NPS facility management profession has evolved into a complex, mission-critical discipline. Being a facility manager in the twenty-first century requires insights, choices, and actions within a very competitive environment. Facility management in the federal sector enjoys a much higher profile than ever before. In-depth studies, such as the National Academy of Science’s Federal Facilities Council Report, acknowledge the importance of facility management:

…facilities managers can no longer be regarded only as caretakers who bring unwelcome news about deteriorating facilities and the need for investments. As facilities management has evolved from tactical, building-oriented activities to a strategic, portfolio-based approach, the skills required by facilities management organizations have similarly evolved. … [This evolution] requires not only the technical skills (e.g., engineering, architecture, mechanical, electrical, contracting) found in traditional facilities engineering organizations but also business acumen and communication skills.

Both the Department of the Interior (DOI) and the NPS have targeted the facility management profession as “mission-critical” because of the importance of maintaining the built environment and the high numbers of facility managers eligible to retire in the near future. In anticipation of the growing need for able facility managers, the NPS has created a comprehensive set of facility manager competencies to serve as a roadmap for development of the facility management workforce. FMLP was designed to prepare a core group of competent, up-and-coming facility managers to serve as leaders of facility management in the NPS in the years to come.


2 June 2009

The Participants The course audience included those charged with the facility function in the NPS. Students were geographically dispersed in parks across the country, representing various parks and experience levels. The course was designed for the following:

New facility managers and facility managers new to the Service.

Those in the Service who wanted to build experience in order to apply for a facility management position and/or those seeking promotion to a facility management position.

Individuals in other disciplines in the NPS who wish to pursue a career in facility management.


June 2009 3

The Capstone Study in Facility Management

Course Specifics The Capstone Study in Facility Management was a week-long course designed to be the culminating experience in FMLP. It provided classroom and field experiences on leadership, political realities, and professionalism as well as a trip to Capitol Hill. Additionally, the course provided students with the opportunity to demonstrate what they had learned throughout the previous year by giving poster presentations. The closing of the course was a goal-setting experience followed by a graduation ceremony. The course specifically afforded the students an opportunity to learn and practice elements of facility management, including the following:

Cultural and Natural Resource Issues as presented from the WASO perspective and their impact on the field of facility management.

Behavioral benchmarks for leading an organization or division.

Implications of political realities and their impact on facility management and the NPS as a whole.

Identification of elements that make up a profession and the ability to apply them to facility management.

Creation of a leadership team toolkit.

Demonstration of new knowledge gained as a result of the FMLP.


4 June 2009

Process of Evaluation The information in this report comes directly from student feedback received in course and instructor evaluations and from team member observations of the Capstone Study in Facility Management. A summary of the evaluations is provided in this report.

Methodology The course was analyzed using feedback from the seventeen student evaluations of the instructors and course. The details regarding how each of these components was implemented are included below.

Evaluations of the Instructors The students were provided with an evaluation form on which they could write their feedback and observations regarding each individual instructor who participated in the course. The instructor evaluation tool allowed for a quantitative measurement of each instructor’s effectiveness and provided space for qualitative comments about each instructor (see Appendix A). These evaluations were collected on the final day of the course. The results from this evaluation tool are included in this report.

Participant Course Evaluations At the conclusion of the course, the students were asked to complete a participant course evaluation. The evaluation tool, developed by the Eppley Institute, included both quantitative and qualitative questions regarding the course, the instructors, and improvements that could be made to the course (see Appendix B). The results from this evaluation tool are also included in this report.


June 2009 5

Evaluations of the Instructors

The Audience This course was designed to meet the needs of those charged with execution of the facility function in the NPS. Course participants came from parks throughout the country and represented a variety of park types and sizes, as well as a diversity of experience levels. The audience consisted of those accepted into the year-long Facility Manager Leaders Program (FMLP). Student participants had varied job descriptions, as shown by the table of current job titles shown below.

Asset Management Planning Coordinator Chief of Maintenance & Natural Resources Civil Engineer Facility Management Specialist Facility Operations Specialist Heavy Duty Equipment Mechanic Maintenance Mechanic Supervisor Maintenance Supervisor Maintenance Work Leader Maintenance Worker Maintenance Worker Supervisor Mason Mason, Historic Preservation Branch Supervisory Facility Management Specialist Supervisory Facility Operations Specialist Supervisory Maintenance Mechanic Trail Supervisor

Figure 1: Table of Job Titles Represented

The Course Coordinators The course coordinators provided the overall structure for the course as well as support and assistance for the guest instructors. The coordinators supplied the guest instructors with outlines, lesson plans, training aids, and instructional resources. They also provided the students with continuity and a common voice through which the key points for each session could be highlighted. In addition, the course coordinators presented a session on professionalism and lifelong learning.

Guest Instructors The guest instructors for the Capstone Study in Facility Management were selected based on their position of leadership in the NPS. Speakers from higher levels in the NPS were asked to provide their perspectives on leadership and educate the students about the challenges they faced regarding political realities.


6 June 2009

Course Agenda and Corresponding Guest Speakers The following table provides the course agenda and the guest speakers for each session.

Tu

esd

ay,

Ap

ril 1

4th

Time Session Guest Speaker(s)

8:30 – 9:30 am Course Introduction and Welcome Course Coordinators

9:30 – 10:45 pm Congressional and Legislative Affairs Don Hellmann

11:00 – 12:15 pm Concessions Jo Pendry 1:15 – 2:45 pm Professionalism Workshop (Part One) Christy McCormick

3:00 – 4:30 pm Professionalism Workshop (Part Two) Debbie Simmons

4:30 – 5:00 pm Technical Proficiency Test Zachary Carnagey

We

dn

esd

ay

, A

pri

l 1

5th


8:30 – 9:00 am Start of the Day Course Coordinators

9:00 – 10:00 am Leadership and Political Realities Peggy O’Dell

10:15 – 12:00 pm Leadership Case Study Costa Dillon

1:00 – 2:15 pm Leadership Development Steve Wolter

2:30-3:30 pm Park Improvement Program Michele Proce

3:30 – 3:45 pm Walk to Main Interior Building

3:45 – 4:45 pm Poster Presentation Practice and Critique Brian Strack

4:45 – 5:00 pm Conclusion of the Day Christy McCormick Jeri Mihalic

Th

urs

da

y,

Ap

ril

16

th Time Session Guest Speaker(s)

9:00 – 9:15 am Start of the Day Course Coordinators

9:15 – 10:00 am Travel to Main Interior and Poster Set Up

10:00 – 11:30 am Poster Presentation - Preview & Evaluation

FMLP students

1:00 – 3:00 pm Poster Presentation - General Audience FMLP students

3:30 – 5:00 pm Graduation Ceremony

6:00 – 8:00 pm Celebration Dinner and Awards Dinner (Mr. Smith’s of Georgetown, 3104 M St, NW)

Fri

day

, A

pri

l 1

7th


8:00 – 12:00 pm Course of Study Evaluation Planning Session Annual Candidate Evaluation

1:00 – 4:30 pm Tour of Lincoln Memorial and WW II Memorial

Sean Kennealy

Figure 2: Course Agenda and Guest Speakers


June 2009 7

Quantitative Instructor Feedback The course evaluation asked the participants to respond to three statements relating to the guest instructors: Statement 1: The instructor knew the subject matter well. Statement 2: The instructor made the subject matter interesting. Statement 3: The instructor encouraged student involvement. The evaluation rankings were scored on a Likert scale from 5 to 1, with 5 corresponding to Strongly Agree and 1 corresponding to Strongly Disagree. The following chart displays each instructor’s average scores for these statements. This chart reflects that the evaluations of each instructor were very high, with no response scoring below a 4.4. The majority of the scores were above 4.5, indicating the students were very satisfied with the instructors’ experience, their ability to make the subject matter interesting, and the way they involved students in the course.

Figure 3: Instructor Quantitative Feedback


8 June 2009

Qualitative Instructor Feedback The qualitative evaluation had only one question, which asked for comments on the overall effectiveness of the instructor and suggestions for improvement. The responses were positive on the whole, and what criticisms were offered were constructive. Almost every instructor received favorable comments on the quality of their class.

Sample comments related to course quality:

Great discussions and lots of insight

Great interaction!

Interesting, many students had different perspectives beyond FMLP!

Great issue/case study

Great sessions as usual. Good information and good involvement

Good to hear this perspective, but also a little scary. Lots of grim implications for park's coming from WASO and those outside NPS who have never worked in a park…

I liked the approach = more discussion than lecture Several instructors also received personal praise for their teaching.

Sample comments relating to instructors:

One of the best public speakers; a true inspiration of what the leadership standard should look like. One I hope to emulate.

Very down to earth

Great session. Great speaker - good examples - loved the story about Sup. and non-contradiction

Love hearing from Christy…good sessions

Michele is very engaging

Jeri is a shining light Finally, most instructors received at least some constructive criticism.

Sample comments relating to suggestions for improvement:

Need incorporation of FMSS data in concession-driven CAAS!

A little more familiarity with the slides would make the presentation better

Not enough time spent on articulating components as a team

[…]Although illustrative it would be most useful to see a highly controversial subject […]I thought showing that the "bad guys" had fallen on hard times was not useful

Perhaps just a tad slow to develop - would have left more time for Q + A

Too bad we had to rush at the end


June 2009 9

Course Evaluations The evaluation instrument for the Capstone Study in Facility management contained eight quantitative statements about the course. The statements included, among others, how clearly concepts were presented, the importance of the background of the speakers, and whether the facility met the needs of the participants. These statements were then followed by three qualitative questions designed to elicit comments from the participants about what were the most and least useful activities and suggestions for improvements. Space was also provided for other comments to be made.

Quantitative Course Feedback The quantitative section asked the students to rate the course according to eight statements about their learning, their leadership, the speakers, the political process of the NPS, and the training facility. The statements were rated on a Likert scale from 5 to 1, with 5 corresponding to Strongly Agree and 1 corresponding to Strongly Disagree.

STATEMENTS ABOUT THE COURSE Mean

1. The course was presented in an interesting manner 4.71 2. Concepts were clearly presented 4.57 3. It is important to me to have speakers who hold leadership positions in DOI 4.43 4. The sessions increased my knowledge of leadership 4.64 5. The sessions increased my knowledge about the political process related to the

NPS 4.57

6. The poster presentation increased my confidence in public speaking 4.50 7. I was encouraged to participate and answer questions 4.50 8. The training facility meets my needs 4.29

Figure 4: Mean Scores for the Quantitative Evaluation Statements The mean score for the statements was very high, with no category scoring lower than a 4.2. The lowest-scoring statement was “The training facility meets my needs,” at a 4.29. However, it is important to keep in mind this is only low in comparison with the other scores in this section. An average score of over 4 is still quite high and indicated most participants though the training facility was adequate. Interestingly, more participants responded “Agree” than “Strongly Agree” to this statement—the only statement for which this is the case. One participant commented that there was “too little room” in the training facility, so in future years this may be something to give further consideration to. Two respondents were neutral on the importance of the speakers’ leadership positions, and one each on the statements relating to the poster presentation, the encouragement to participate, and the training facility.


10 June 2009

Qualitative Course Feedback The course evaluation also asked students to respond to three open-ended questions about what were the most and least helpful aspects of the course and to provide suggestions for improvements. They were also given space to leave comments relating to other facets of the course. These comments will be used to help improve future courses in the FMLP.

Most Useful Activities The areas specifically identified as most useful were the poster presentation, the regional director, and political realities.

What did you find most useful about the course?

“Poster Presentation”

“Poster presentation was great”

“All of the leadership discussions and the presentation tip. I’ve been told I need to work on my delivery”

“Regional Director”

“The information conveyed by the speakers - all of them”

“All of it”

“Political realities”

Least Useful Activities Only three people made comments in this space.

What did you think was least useful in the course?

Figure 5: Percentage of Responses to the Quantitative Evaluation Statements


June 2009 11

“Having mentors there”

“The leadership case study was good but it could've been shorter”

“The disorganization”

Suggestions for Improvements There were only two suggestions made for ways to improve the course. One student commented that the “basic supervision” could be improved, and a second recommended better seating. A third student pointed to his critique paper for his ideas on how to improve the course.

Other Comments Two participants made “other” comments:

“Could use a larger facility for dinner”

“The program was all relevant and applicable to me”


12 June 2009

Summary and Recommendations

Summary The evaluative feedback can be summarized by categorizing the information into three areas: (1) course content, (2) course logistics, and (3) instructors. This summary appears below and forms the basis for the recommendations which follow.

Course Content This course was rated as very interesting, largely because of the strength of the

presenters, particularly the strength of the leadership presentations.

Other areas rated as extremely useful by the participants were the poster presentation, the presence of the Regional Director, and the session on political realities.

Students seemed frustrated by the disorganization of the course, which was caused by the compression of the course by one full day due to Easter weekend.

Students wanted to see the Supervisor’s Desk Reference more fully incorporated into the curriculum.

At least one student questioned the value of having mentors present for the course.

Course Logistics The training facility was the lowest-ranked portion of the course evaluation. It appears

that the majority of participants found the facility adequate for their needs but not excellent, as evidenced by the fact that over half of the participants agreed with the statement (in a reversal from all other aspects, in which the majority response was “Strongly Agree”). There seemed to be some dissatisfaction with the training space available and with the seating arrangements.

Instructors All of the instructors were very highly rated, although participants did make some small

changes that they believed would improve the individual sessions. These will be discussed in the recommendations section following.

Recommendations The recommendations which follow were based on the summaries above and the conclusions drawn from the feedback provided by the students.

Several students had suggestions for how to improve individual sessions. These included:

o Utilizing an organizational chart for comprehension in the Legislative and Congressional Affairs session.

o Allowing more time for or redistributing time within some sessions, including the Leadership session and the Professionalism workshop. Specifically, students felt


June 2009 13

that the Leadership Case Study could have been shortened to allow for more time for questions at the end. On the other hand, participants seemed disappointed about the apparent rush toward the end of the Leadership session.

o Instructors should be provided with their accompanying PowerPoint slides well in

advance to allow ample time for familiarizing themselves with the content and the format.

Based on comments received, it would be worthwhile to evaluate potential training facilities in advance based on layout and actual space available.

There was a general sense of dissatisfaction with the overall organization of the course. Since the organization was hampered this year due to the fact that it was scheduled the week following Easter Sunday, in future years every effort should be made to avoid scheduling courses in such close proximity to holiday weekends.

The NPS Supervisor’s Desk Reference should be incorporated more fully into the curriculum as a document participants will be required to be familiar with.


14 June 2009


June 2009 15

APPENDICES


16 June 2009

Appendix A: Instructor Evaluation

June 2009 17

National Park Service U.S. Department of the Interior Facility Manager Leaders Program

Capstone Instructor Evaluation As you progress through the week, please take a few moments to comment on the effectiveness of the instructors for the FMLP Principles course. Some instructors are presenting multiple sessions, so do not finalize their evaluation until the end of the week. All responses will be kept confidential and will only be used for future improvements in the Facility Manager Leaders Program (FMLP). Don Hellman Strongly

Disagree Neutral Strongly

Agree 1. The instructor knew the subject matter well. 1 2 3 4 5 2. The instructor made the subject matter interesting. 1 2 3 4 5 3. The instructor encouraged student involvement. 1 2 3 4 5

Please comment on the overall effectiveness of this instructor and any suggestions you have for improvements.

Michele Proce Strongly




18 June 2009


Christy McCormick Strongly




Debbie Simmons Strongly




Steve Hastings Strongly




June 2009 19


Peggy O’Dell Strongly Disagree

Neutral Strongly Agree

1. The instructor knew the subject matter well. 1 2 3 4 5 2. The instructor made the subject matter interesting. 1 2 3 4 5 3. The instructor encouraged student involvement. 1 2 3 4 5


Costa Dillon

Strongly Disagree




Steve Wolter Strongly




20 June 2009


Jo Pendry Strongly Disagree




Jeri Mihalic Strongly




Sean Kennealy

Strongly Disagree





June 2009 21

Appendix B: Course Evaluation

Capstone Study in Facility Management

22 June 2009

National Park Service

Park Facility Management Division Capstone Evaluation

Overall Course Evaluation

Strongly

Agree

Agree

Not Sure-

No Opinion

Disagree

Strongly

Disagree

The course was presented in an interesting

manner

Concepts were clearly presented

It is important to me to have speakers who

hold leadership positions in DOI

The sessions increased my knowledge of

leadership

The sessions increased my knowledge about

the political process related to the NPS

The poster presentation increased my

confidence in public speaking

I was encouraged to participate and answer

questions

The training facility met my needs

What did you find most useful about the

course?

What did you think was least useful in the

course?

What would you improve about the Capstone

Study as the fifth course in FMLP?

Other Comments:

Please give us your evaluation and opinions of the course, as well as your learning during the course by

checking the box below that most closely represents your opinion. All responses are confidential and will

only be used for the purpose of evaluating the Capstone Study in FMLP program.

Course Evaluation Report - Eppley...

Documents

Transcript of Course Evaluation Report - Eppley...