Hume Center Graduate Student Seminar Evaluation€¦ · Hume Center Graduate Student Seminar...

Hume Center Graduate Student Seminar Evaluation

Program & Product Evaluation Final Evaluation Project Report


Introduction

This report presents the results of the evaluation of the graduate student seminar implemented by the directors of education and outreach at the Ted and Karyn Hume Center for National Security and Technology (the Hume Center). The report describes the background of the program evaluated; the rationale for conducting the evaluation; the purpose of conducting the evaluation; key questions addressed; the stakeholders involved; sampling procedures, methodologies and instruments; logistics and timeline; limitations; and the results and their interpretation, and recommendations for the future of the program.

Background The Hume Center, at Virginia Tech, aims to cultivate the next generation of national security leaders by providing research, outreach, and education opportunities and programs addressing the challenges faced by the national security community. The Hume Center takes a student-centric approach, integrating students into all aspects of the research process; graduate students, in particular, sit at the intersection between the Hume Center’s research and educational practices. To assist graduate students in becoming competent members of their fields, the research community, and the national security community, the Hume Center established a series of graduate student seminars. These seminars are intended to help students associated with the Hume Center develop their presentation skills by providing opportunities to practice presenting in a safe space, where they can receive constructive feedback from peers. Additionally, these seminars are hoped to help students feel connected to their peers at the Hume Center, and to develop a sense of their place in their fields.

Rationale As described above, the Hume Center is dedicated to assisting graduate students in becoming competent members of a larger society, which includes developing presentation skills, forming working connections with peers, and having a concept of where they are situated within a field of study. The graduate seminar program was established to address these important facets of graduate education, but before this evaluation there was no reliable data to suggest that the program is working as intended. Purpose The purpose of the evaluation was to determine participants’ perceived usefulness of the seminar program; perceived change in presentation skills, and perceived ability to transfer these skills to other presentation situations; feeling of comradery generated by participation in the seminar program; and any change in participants’ sense of their field of study and their position within it. Specifically, the goals of this evaluation were to determine:



• how the seminar affects participating students’ appreciation of presentation skills. • the perceptions of participating students’ on their presentation skills following presenting during the seminar. • the impact of the graduate student seminar on Hume Center graduate student culture. • the impact of the graduate student seminar on graduate students’ appreciation and understanding of the nature of their field.

The evaluation was formative in nature, as the results will be used to modify the program for future use. Guiding/Key Questions The evaluation addressed the following questions:

o In what ways, if any, has the Graduate Student Seminar contributed to graduate students’ perception and appreciation of presentation skills?

o What impact does the Graduate Student Seminar have on Hume Center graduate student culture? o In what ways, if any, has the Graduate Student Seminar contributed to participants’ sense of academia and scholarship?

Stakeholders At the onset of the evaluation, primary stakeholders in the evaluation were identified as:

• Christine Callsen, Director of Outreach and Education, and • Dr. Kira Gantt, Associate Director of Outreach and Education.

During the evaluation process Dr. Gantt provided the background information, the intent for establishing the seminar, and the desired purpose of the evaluation. She was a key consult in the development of the evaluation plan, and she along with Christine Callsen approved the data collection methodologies and instrument before they were used.

Over the course of the evaluation, an additional primary stakeholder emerged. Deanna Clark, the Program Assistant and Administrative Support for the Research & Education Directorates, facilitated connections between the evaluator and research participants, and oversaw the administration of the seminars. Deanna was integral in the management of the graduate student seminar, and is expected to be an important figure in utilizing the recommendations given later in the paper. If a future evaluation is conducted, Deanna should be considered a primary stakeholder from the beginning of the evaluation process and involved in the initial planning and instrument development process to improve the results of the evaluation. Secondary stakeholders in the evaluation include any current and future undergraduate and graduate students, as well as faculty and staff, who participate in the seminar. These stakeholders were utilized in data collection, but not in the planning and development of the evaluation. Furthermore, these stakeholders will not be primary recipients of the evaluation’s results. If a future evaluation is conducted, the interest of these stakeholders can be taken into account when determining the purpose of the evaluation, in order to ensure the evaluation is addressing all concerns of the participants; this would be particularly important in a summative evaluation of the graduate student seminar.



Methods Sample/Sampling Strategy A combination of purposive sampling and convenience sampling was utilized for this evaluation. The sampling was purposeful in that it targeted select groups–graduate students, technicians, and faculty. It was also a matter of convenience, in that the participants selected were be those that are participating in the graduate student seminar during the evaluation period. Data was collected in two ways, observation and an online-survey. The survey was distributed online to all members of the Hume Center. This allowed members of the population of interest (current graduate students) to be evaluated, while also proving a more longitudinal perspective by surveying members of technical staff and faculty who could speak to any changes they have observed over time. Furthermore, it allowed members of the Hume Center who have attended seminars in the past, but were not available at the observed seminar periods, to respond. Twenty-four responses were collected in total: fifteen graduate students, two members of technical staff, six faculty members, and one undergraduate student responded. This sample was larger than expected, however not all respondents answered all the survey items; actual response totals are reported in the results section. Of the graduate students who responded, at least 5 students were perusing their master’s degree and 6 were perusing their doctorates; 4 students did not designate their current degree status. The graduate students who responded reported the following broad areas of academia as their field of interest: communications, aerospace, and electromagnetics. Moreover, between fifteen and twenty-five Hume Center personnel were present during each of the observed seminars. The majority of these participants were students. A small number of faculty and technical staff attended each seminar as well. Data Collection Methods and Instrumentation Data was collected through surveys completed by the participants, and observational data collected by the evaluator (me). The survey instrument was developed, entirely for this evaluation project, with Virginia Tech’s Qualtrics program. As mentioned earlier, the instrument was distributed and completed online, and responses are anonymous. The instrument consisted of Likert-type questions, and open-ended short-answer response questions; it is presented in Appendix A. Descriptive statistics were run on the data collected from the Likert-type questions to determine the minimum, maximum, mean, standard deviation, and variance of the responses. The short-answer responses were used to support the statistical descriptive data.



Observational data was collected as field-notes. Two seminars were attended by the evaluator. At each of these seminars, notes were taken to be as comprehensive as possible, so that no data is excluded because of a lack of sensitization to it. In this way, the observational data could be used as supplemental data to provide any context necessary for the evaluation—since observational data cannot address the perceptions of the participants (i.e., the key questions). The field notes were then reviewed and coded, with the key questions and results of the survey analysis in mind. These the codes were grouped into emerging themes. Logistics and Timeline The Hume Center held two seminars in November, the first on the 6th and the second on the 20th. Observational data was collected at both of these seminars. The survey was released following the seminar on November 6th, and data was be collected through November 14th. During this data collection period, the data was initially reviewed to determine the needs for further data collection. A reminder email was distributed to Hume Center employees on November 13th, which increased the number of responses obtained. A secondary data analysis period occurred the following week. The final report will be available to the HUME center following the returned final project for the class.

Event Completed Seminar observed and survey distributed November 6th Initial data analysis November 13th Survey data collection ended November 14th Secondary data analysis November 16th-18th Second seminar observation November 20th Completed analysis December 6th Completed report for class December 7th

Budget There were no necessary costs associated with this evaluation. Limitations This evaluation was limited by the timeframe imposed by the course; data could only be collected and analyzed over the course of November per the course specification. Although data was able to be collected and analyzed during this time period, a more thorough and involved evaluation could have taken place over a longer time frame.



Results Survey Results Presentation skills Students believe that seminar has improved their presentation skills (mean = 4.00, where 4.00 = agree). In particular, students find that giving presentations, receiving feedback and listening to feedback given to presenters contributed to an improvement in their presentation skills (mean = 4.38). Watching other students present during seminar (mean = 3.64, where 3.00 = neither agree nor disagree) and providing feedback to peers (mean = 3.67) are perceived to contribute less to the improvement of presentation skills by graduate students than presenting themselves and listening to feedback provided by others (mean = 4.38). The one undergraduate student who responded found providing feedback themselves to be as beneficial to increasing their presentation skills as listening to others provide feedback (mean = 4.00). Detailed descriptive statistics can be viewed in Table 1 and Table 2 below. Min Max Mean Variance Standard

Deviation Total Responses

Attending seminar has helped improve my presentation skills. 3.00 5.00 4.00 0.50 0.71 13 Giving a presentation in seminar has helped improve my presentation skills.

3.00 5.00 4.38 0.84 0.92 8

Feedback given to me after presenting in seminar has helped improve my presentation skills.

3.00 5.00 4.38 0.55 0.74 8

Watching others present during seminar has helped improve my presentation skills.

2.00 5.00 3.64 1.17 1.08 14

Providing critique and feedback to other presenters during the seminar has helped improve my presentation skills.

3.00 5.00 3.67 0.50 0.71 9

Listening to critique and feedback given to other presenters during seminar has helped improve my presentation skills.

4.00 5.00 4.21 0.18 0.43 14

Table 1. Graduate students’ perceptions of presentation skills as a result of participation in the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree.



Response Attending seminar has helped improve my presentation skills. 4.00 Giving a presentation in seminar has helped improve my presentation skills.

N/A

Feedback given to me after presenting in seminar has helped improve my presentation skills.

N/A

Watching others present during seminar has helped improve my presentation skills.

4.00

Providing critique and feedback to other presenters during the seminar has helped improve my presentation skills.

4.00

Listening to critique and feedback given to other presenters during seminar has helped improve my presentation skills.

4.00

Table 2. Undergraduate student’s perceptions of presentation skills as a result of participation in the Hume GRA Seminar. N = 1 Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. N/A = Not applicable. Graduate students who reflected on specific skills which they have learned as a result of seminar described improvements in formatting their slides, and interaction with the audience while presenting. The skills regarding formatting could have been learned through presenting themselves or through the observation of other’s presentations, such as “I’ve learned that highlighting words with certain colors mean different things.” However, the skills described regarding interacting with the audience were all gained as a result of presenting themselves. These insights are exemplified by the following statements “I have increased my skills with being able to ‘read’ the audience and adjust content accordingly,” “My enthusiasm [affects] others,” and “I learned how to structure and order my presentation for those who may be not familiar with my research. Also, the more I present, the less nervous I am doing it.” Following participation in the seminar, graduate students remain predominately neutral in their confidence in giving future presentations (mean = 3.86). Additionally, they are uncertain of their opportunities to present outside of seminar (mean = 3.80). The descriptive statistics can be viewed in Table 3 below. The undergraduate student who responded felt that their ability to present after attending seminar was not applicable, however, they did report being more confident in giving future presentations after attending seminar (mean = 4.00); this can be seen in Table 4.



Min Max Mean Variance Standard


After attending seminar, I have had opportunities to use my presentation skills.

2.00 5.00 3.80 0.84 0.92 10

I feel more confident in giving future presentations after participating in seminar.

3.00 5.00 3.86 0.59 0.77 14

Table 3. Graduate students’ perceptions of presentation skills and opportunities following participation in the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Response After attending seminar, I have had opportunities to use my presentation skills.

N/A

I feel more confident in giving future presentations after participating in seminar.

4.00

Table 4. Undergraduate student’s perceptions of presentation skills and opportunities following participation in the Hume GRA Seminar. N = 1 Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. N/A = Not applicable. To provide a more longitudinal picture of student presentation performance following the establishment of the Graduate Student Seminar, faculty and technical staff were asked to reflect on student abilities. Tables 5 and 6 show that faculty and technical staff agree that students’ presentation abilities and confidence has improved after participating in the seminar (means ≥ 4.00). The faculty and staff have differing opinions on the students’ ability to use their presentation skills at the Hume Center, outside of seminar; faculty believes students do have opportunities to use their skills (mean = 4.25), while the member of technical staff who responded neither agreed nor disagreed (mean = 3.0). These results can be seen in Tables 7 and 8, below. Min Max Mean Variance Standard


Student seem more confident in giving presentation after participating in seminar.

4.00 5.00 4.50 0.33 0.58 4

Students show improvement in their presentation abilities after participating in seminar.

4.00 5.00 4.50 0.33 0.58 4

Table 5. Faculty members’ perceptions of students’ presentation skills as a result of participation in the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree.





Student seem more confident in giving presentation after participating in seminar.

4.00 4.00 4.00 0.00 0.00 2

Students show improvement in their presentation abilities after participating in seminar.

4.00 5.00 4.50 0.50 0.71 2

Table 6. Technical staff members’ perceptions of students’ presentation skills as a result of participation in the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Min Max Mean Variance Standard


Students have opportunities to use presentation skills for Hume Center presentations, outside of seminar.

4.00 5.00 4.25 0.25 0.50 4

Table 7. Faculty members’ perceptions of students’ opportunities to outside of the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Response Students have opportunities to use presentation skills for Hume Center presentations, outside of seminar.

3.0

Table 8. Technical staff members’ perceptions of students’ opportunities to outside of the Hume GRA Seminar. N = 1 Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Hume Center culture All categories of respondents—students, faculty, and technical staff—agree that having a Graduate Student Seminar establishes values and expectations for the Hume Center. The specific means can be seen in Tables 9, 10, 11, and 12 below. The graduate student who strongly disagreed with the statement that “Having a Hume GRA Seminar means peer support among students, faculty, and staff is valued at the Hume Center” (response value = 1.00) commented that “Seminar isn’t inspiring support between students and faculty/staff.” They expressed that “Some staff members show up, but I have seen little to no participation from faculty,” implying that for at least some students, faculty and staff presence at seminar speaks more to how the Hume Center values peer support than just having a seminar.





Having a Hume GRA Seminar means presentation skills are valued at the Hume Center.

3.00 5.00 4.57 0.42 0.65 14

Having a Hume GRA Seminar establishes expectations for a graduate student scholarship at the Hume Center.

3.00 5.00 4.29 0.68 0.83 14

Having a Hume GRA Seminar means peer support among students, faculty, and staff is valued at the Hume Center.

1.00 5.00 4.36 1.17 1.08 14

Table 9. Graduate students’ perceptions of Hume Center Culture after the establishment of the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Response Having a Hume GRA Seminar means presentation skills are valued at the Hume Center.

5.00


5.00


5.00

Table 10. Undergraduate student’s perceptions of Hume Center Culture after the establishment of the Hume GRA Seminar. N = 1 Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Min Max Mean Variance Standard



4.00 5.00 4.50 0.33 0.58 4


4.00 5.00 4.50 0.33 0.58 4


4.00 5.00 4.75 0.25 0.50 4

Table 11. Faculty members’ perceptions of Hume Center Culture after the establishment of the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree.



Min Max Mean Variance Standard Deviation

Total Responses


4.00 5.00 4.50 0.50 0.71 2


4.00 5.00 4.50 0.5 0.71 2


4.00 5.00 4.50 0.5 0.71 2

Table 12. Technical staff members’ perceptions of Hume Center Culture after the establishment of the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. In addition to speaking to the culture of the Hume Center, in terms of collective values and expectations, the seminar plays a role in graduate student camaraderie. Graduate students feel that seminar allows members of the Hume Center to cooperate and support each other. One student remarked “Absolutely. The critique was often supportive, non-combative, and led to a sense of genuine support from the technical staff and other graduate students. This, in turn, led to greater social bonding among graduate students.” Furthermore, the students expressed that having seminar has provided an opportunity to be involved in peers work in a way that would otherwise not be as easily available. This feeling is exemplified by the following comments “It allows casual cooperation and keeps communication lines open that normally wouldn’t,” “We want each other to succeed and are more interested in supporting fellow grad students,” and “We are all trying to push each other to make strides in our work by teaching each other about what is being done through seminars.” One student suggested that the GRA seminar has contributed to a feeling of camaraderie because of a “mutual apprehension of having to do it.” Only one student expressed that the seminar was too short to contribute to a sense of camaraderie; they suggested that other Hume Center events—in particular, writing days—were more beneficial to developing camaraderie among graduate students, because students are there for a significant period of time. As shown in Table 13, faculty also was unsure if the seminar had contributed to a sense of camaraderie among the graduate students (mean = 3.50). However, the member of technical staff who responded to this question, as seen in Table 14, felt that the seminar has resulted in a change among the graduate students (response value = 4.00). A staff member commented: “I’ve noticed that students appear to be less divided by seniority/area of study, and seem more likely to know who to go for with specific questions. The graduate students also seem to be getting more social interaction with each other than previously seen.”



Min Max Mean Variance Standard Deviation

Total Responses

There has been a change in graduate student camaraderie since the Hume GRA Seminar was initiated

3.00 5.00 3.50 0.33 0.58 4

Table 13. Faculty members’ perceptions of graduate student camaraderie following the establishment of the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Response There has been a change in graduate student camaraderie since the Hume GRA Seminar was initiated

4.0

Table 14. Technical staff members’ perceptions of graduate student camaraderie following the establishment of the Hume GRA Seminar. N = 1 Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Academia and scholarship Seminar has contributed to graduate students’ appreciation of breadth and depth of their chosen field (mean = 4.08). However, graduate students are, on average, unsure of the other contributes of seminar to their scholarship, particularly the contributions of seminar to their understanding of research methodologies and how their research interests fit within the scope of national security research. The means for the graduate students’ responses can be seen in Table 15, below. The undergraduate felt that seminar contributed to not only their appreciation of the breadth and depth of their chosen field, but also how their research interests relate to commentary fields and their understanding of research methodologies, as shown in Table 16. The undergraduate student disagreed that seminar has enabled them to determine where their research interest fits in the scope of national security research (mean = 2.00). Min Max Mean Variance Standard Total



Deviation Responses Attending seminar has contributed to my understanding of research methodologies.

2.00 5.00 3.57 0.57 0.76 14

Attending seminar has contributed to my appreciation of the breadth and depth of my chosen field.

3.00 5.00 4.08 0.58 0.76 13

Attending seminar has contributed to my understanding of concepts related to my chosen field.

2.00 5.00 3.86 1.05 1.03 14

Attending seminar has helped me gain a sense of where my research interests fit within my chosen field.

1.00 5.00 3.93 1.30 1.14 14

Attending seminar has helped me gain a sense of how my research interest relate to complementary fields.

1.00 5.00 3.93 0.99 1.00 14

Attending seminar has helped me determine where my research interests fit within the scope of national security research.

1.00 5.00 3.43 1.34 1.16 14

Table 15. Graduate students’ sense of academia and scholarship following participation in the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Response Attending seminar has contributed to my understanding of research methodologies.

4.00

Attending seminar has contributed to my appreciation of the breadth and depth of my chosen field.

4.00

Attending seminar has contributed to my understanding of concepts related to my chosen field.

3.00

Attending seminar has helped me gain a sense of where my research interests fit within my chosen field.

3.00

Attending seminar has helped me gain a sense of how my research interest relate to complementary fields.

5.00

Attending seminar has helped me determine where my research interests fit within the scope of national security research.

2.00

Table 16. Undergraduate student’s sense of academia and scholarship following participation in the Hume GRA Seminar. N = 1 Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree.



As a whole, faculty members were more optimistic about the role of seminar in graduate students’ sense of scholarship and academia. Faculty members agreed that seminar contributed to graduate students appreciating the breadth and depth of their field (mean = 4.50); understanding of concepts related to their field (mean = 4.00); and how their research interests fit within their chosen field (mean = 4.75) and relates to complementary fields (mean = 4.50). In contrast, the members of technical staff are only in agreement with the following statement “Graduate students have an increased sense of how their research interests relate to complementary fields, since the Hume GRA Seminar was initiated.” (response value = 4.00). The means for all statements can be viewed in Tables 17 and 18, respectively. Min Max Mean Variance Standard


Graduate students have an increased understanding of research methodologies, since the Hume GRA Seminar was initiated.

3.00 5.00 3.50 0.33 0.58 4

Graduate students appreciate the breadth and depth of their chosen field more, since the Hume GRA Seminar was initiated.

4.00 5.00 4.50 0.33 0.58 4

Graduate students have an increased understanding of concepts related to their field, since the Hume GRA Seminar was initiated.

4.00 4.00 4.00 0.00 0.00 4

Graduate students have an increased sense of where their research interests fit within their chosen field, since the Hume GRA Seminar was initiated.

4.00 5.00 4.75 0.25 0.5 4

Graduate students have an increased sense of how their research interests relate to complementary fields, since the Hume GRA Seminar was initiated.

4.00 5.00 4.50 0.33 0.58 4

Graduate students are more aware of where their research interests fit within the scope of national security research, since the Hume GRA Seminar was initiated.

2.00 5.00 3.25 1.58 1.26 4

Table 17. Faculty members’ perceptions of graduate students’ sense of academia and scholarship following the establishment of the Hume GRA Seminar. Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree.



Response Graduate students have an increased understanding of research methodologies, since the Hume GRA Seminar was initiated.

3.0

Graduate students appreciate the breadth and depth of their chosen field more, since the Hume GRA Seminar was initiated.

3.0

Graduate students have an increased understanding of concepts related to their field, since the Hume GRA Seminar was initiated.

3.0

Graduate students have an increased sense of where their research interests fit within their chosen field, since the Hume GRA Seminar was initiated.

3.0

Graduate students have an increased sense of how their research interests relate to complementary fields, since the Hume GRA Seminar was initiated.

4.0

Graduate students are more aware of where their research interests fit within the scope of national security research, since the Hume GRA Seminar was initiated.

3.0

Table 18. Technical staff members’ perceptions of graduate students’ sense of academia and scholarship following the establishment of the Hume GRA Seminar. N = 1 Where 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree. Observational Data Results The observational data serves as a snapshot of the Graduate Student Seminar. From the two observed occasions the following themes emerged: Hume Center culture, purpose of seminar presentations, and instructional moments. Each of these themes are discussed in more detail below. Hume Center culture The major theme which arose during the qualitative analysis was “Hume Center culture.” These codes included examples of: camaraderie, expectations of Hume Center personnel’s involvement in seminar, and Hume Center values.

Evidence of camaraderie during the seminars was exemplified by joking, chatting and discussion among audience members, and discussion with the presenter regarding their presentation topic. The observed camaraderie extended across all personnel, not just among graduate students. This was especially true for incidents involving joking. Joking occurred among audience members, as well as between the presenters and audience. Similarly, all types of personnel engaged in discussion with the presenter regarding their



topic. This discussion occurred during the presentation—as interruptions, and through the question and answer portion—and at the end of the seminar. In contrast, chatting among the audience members primarily occurred between the students, and between faculty and/or staff, but not between students and faculty/staff. Expectations of graduate students involvement in seminar was made clear by the mandatory attendance. This expectation was reinforced through verbal reminders by staff before the seminar began, as well as through a physical sign-in sheet that students were required to fill-out before seminar was completed. Faculty and members of the technical staff were not required to sign in, but were verbally reminded that it was time for seminar. It was less clear what the expectations for faculty and technical staff’s involvement in seminar were. Different from expectations, values imply something which is considered important at the Hume Center, but is not something that is required. Codes involving attention, feedback, and support were categorized as representing Hume Center values, as they were behaviors which were repeatedly demonstrated by personnel. The audience was primarily attentive to the presentation. Attention was noted when the audience remained facing the presenter, responded to the presenter’s jokes or questions, and were not slumped into chairs or otherwise appearing disengaged. There was a subset of the population, particularly faculty and staff members, who split their attention between the presenter and an electronic device during the presentations, however, these individuals represented the minority of those in attendance. Furthermore, those that split their attention between the presenter and an electronic device did engage with the presenter at some point during the seminar. In addition to being attentive, the audience provided support and feedback to the presenter. During and after all three presentations, members of the audience provided feedback to the presenter. This feedback was constructive, and supportive of the presenter. There was no attacking the presenter or non-constructive critiques. While the feedback was always public, occurring in front of the entire audience, on more than one occasion an audience member would also offer the presenter encouragement, such as “you’re doing fine.” As alluded to earlier, members of the audience interrupted the presentations. These interruptions occurred for a number of reasons: to clarify a point the presenter had made, or ask the presenter for clarification; to offer constructive critique to the presenter; to teach a concept to the audience; to encourage the presenter; and, to make a joke. Therefore, the interruptions served as opportunities to provide feedback, support, and reinforce camaraderie. Purpose of seminar presentations The theme “purpose of the Graduate Student Seminar presentations” could be considered a subset of the “Hume Center culture” category, in that the purpose of seminar clearly relates to the expectations of, and for, Hume Center personnel. However, because



these codes were specific to the Graduate Student Seminar, and not the overall culture of the Hume Center, they were grouped as a separate theme. There were three primary types of presentations: interview preparation, lecturing peers regarding an academic topic, and lecturing peers about a National Security topic. It was evident that each presentation during the seminar had a different purpose but it was unclear if there was an established purpose for the Graduate Student Seminar. Moreover, if there is an established purpose for the Graduate Student Seminar that had been communicated to personnel, it is unclear if all participants were aware of it. During the seminar on November 20th there appeared to be differing expectations for the seminar presentation. One faculty member expressed the seminar should serve as a lecture for the audience, stating “we’ll all learn something here, even me.” and relaying to the presenter that they will work with them individually to re-work their presentation. Instructional moments Finally, a prevalent theme of the codes was “instructional moments.” This instruction occurred primarily in two forms (1) constructive critique and feedback from faculty on presentation skills, and (2) topic question and answer among the audience and the presenter. These instructional moment differ in their focus: the former focused on presentation skills, while the latter was content specific. Most prevalent to the purpose of the GRA seminar is constructive critique and feedback from faculty on presentation skills. These instructional moments included faculty relating information to the presenter and audience—such as advising students to explain all abbreviations—and, faculty engaging the audience and presenter in an activity. For example, following the November 6th presentation, one faculty member instructed the audience to tell the presenter three things they took away from the seminar. Then he asked the presenter to tell the audience the three things he wanted the audience to take from the presentation. Finally, the faculty member gave suggestions on how the presenter’s take-away goals could be met.

Discussion and Recommendations As outline previously in this report, the evaluation sought to determine:

• how the seminar affects participating students’ appreciation of presentation skills, • the perceptions of participating students’ on their presentation skills following presenting during the seminar, • the impact of the graduate student seminar on Hume Center graduate student culture, and • the impact of the graduate student seminar on graduate students’ appreciation and understanding of the nature of their field,

by addressing three key questions. The results will be discussed in the context of these questions. In what ways, if any, has the Graduate Student Seminar contributed to graduate students’ perception and appreciation of presentation skills? Graduate students perceive their presentation skills to have increased as a result of attending seminar. This increase was attributed to presenting, and to listening to feedback provided during seminars. However, graduate students did not feel more confident in their



presentation abilities because of their participation in seminar. Despite graduate students perceptions of their own skills, faculty and technical staff felt that the graduate students’ skills and confidence did improve as a result of attending seminar. What impact does the Graduate Student Seminar have on Hume Center graduate student culture? Establishing the Graduate Student Seminar enforces the cultural expectations and values for the Hume Center. Particularly, that the Hume Center is values presentation skills, and is supportive of its students. Moreover, the seminar and has contributed to camaraderie among students. However, not all students feel that seminar supports camaraderie between faculty/staff and students, because of the expectations for students to attend but no expectations for faculty and staff to attend and participate. Despite this student’s concern, during the two observed seminars, a high degree of faculty involvement was seen. In what ways, if any, has the Graduate Student Seminar contributed to participants’ sense of academia and scholarship? According to graduate students, the Graduate Student Seminar has contributed to graduate students’ sense of the depth and breadth of their chosen field. However, the seminar has helped students determine where their research interests fit within the scope of national security research. Furthermore, the Graduate Student Seminar has not provided students with increased understanding of methodologies, concepts related to their field, or how their research interests fit within their field and relate to complementary fields. Recommendations The Graduate Student Seminar increased students’ presentation skills, positively impacted Hume Center culture, and contributed to participants’ sense of academia and scholarship. However, the following recommendations can be followed to improve the Graduate Student Seminar in the future.

• Develop a concise purpose for the Graduate Student Seminar, and clarify the purpose of the Graduate Student Seminar among all personnel.

• Make sure all graduate students have the opportunity to present. • Encourage faculty and staff to attend and provide feedback on the presenter’s performance. • Provide increased opportunities for students to use their presentation skill outside of seminar.



Appendix A Note: The question numbers indicated in this appendix are not viewed by the survey respondents.

The Hume GRA Seminar was created to assist graduate students in becoming competent members in their fields, the research community, and the national security community by providing opportunities to practice presentation skills, form working connections with peers, and develop a sense of where they are situated within a field of study.

This evaluation is intended to determine how well the Hume GRA Seminar is serving this purpose, and determine if changes should be made to ensure this purpose is met.

We greatly appreciate you taking your time to complete this survey, your responses are valuable, and will remain anonymous. Please complete the survey by 11/14/15.



Q1 Please select your level of involvement with the Hume Center. Graduate Student (1) Technical Staff (2) Faculty (3) Undergraduate Student (4)



Answer If Select your level of involvement with the Hume Center Graduate Student Is Selected Q13 Provide a brief description of your program. Include the degree you are currently working on, and the years you have been in the program. For example: A second year Communications Masters student, with interest in signal processing.



Answer If Select your level of involvement with the Hume Center Graduate Student Is Selected Q4 Think about your participation with the Hume GRA Seminar. Select your level of agreement with the following statements.

Strongly Disagree

(1)

Disagree (2)

Neither Agree nor Disagree

(3)

Agree (4) Strongly Agree (5)

Not Applicable

(6)

Attending seminar has

helped improve my presentation

skills. (1)

Giving a presentation in seminar has


skills (2)

Feedback given to me

after presenting in seminar has


skills. (3)

Watching others present during seminar

has helped improve my presentation

skills. (4)



Providing critique and feedback to

others presenters

during seminar has helped improve my presentation

skills. (5)

Listening to critique and

feedback given to other

presenters during seminar


skills. (6)

After attending seminar, I have had

opportunities to use my

presentation skills. (7)

I feel more confident in giving future presentations

after participating in seminar. (8)

Attending



seminar has contributed to

my understanding

of research methodologies.

(9) Attending


my appreciation of

the breadth and depth of my chosen field. (10)

Attending seminar has contributed to

my understanding

of concepts related to my chosen field.

(11)

Attending seminar has helped me

gain a sense of where my research

interests fit within my

chosen field. (12)




gain a sense of how my research

interests relate to

complementary fields. (13)

Attending seminar has helped me determine where my research

interests fit within the scope of national security

research. (14)

Answer If Select your level of involvement with the Hume Center Graduate Student Is Selected Q5 Have you learned any presentation skills in attending seminar? If so, what skills have you learned?



Answer If Please select your level of involvement with the Hume Center. Graduate Student Is Selected Q31 Select your level of agreement with the following statements

Strongly Disagree (1)

Disagree (2) Neither Agree nor Disagree

(3)


Having a Hume GRA

Seminar means

presentation skills are

valued at the Hume

Center. (1)

Having a Hume GRA

Seminar establishes expectations for graduate

student scholarship at

the Hume Center. (2)

Having a Hume GRA

Seminar means peer

support among

students, faculty, and

staff is valued at the Hume Center. (3)



Answer If Select your level of involvement with the Hume Center Graduate Student Is Selected Q6 Do you feel seminar contributed to a sense of camaraderie among the graduate students at the Hume Center? If so, how do you feel it has done so?



Answer If Select your level of involvement with the Hume Center Undergraduate Student Is Selected Q17 What is your major?



Answer If Select your level of involvement with the Hume Center Undergraduate Student Is Selected Q21 What year are you? Freshman (1) sophomore (2) Junior (3) Senior (4)



Answer If Select your level of involvement with the Hume Center. Undergraduate Student Is Selected Q18 Think about your participation with the Hume GRA Seminar. Select your level of agreement with the following statements.

Strongly Disagree

(1)

Disagree (2)


(3)


Not Applicable

(6)

Attending seminar has


skills. (1)

Giving a presentation in seminar has


skills (2)

Feedback given to me

after presenting in seminar has


skills. (3)

Watching others present during seminar


skills. (4)



Providing critique and feedback to

others presenters

during seminar has helped improve my presentation

skills. (5)

Listening to critique and

feedback given to other

presenters during seminar


skills. (6)

After attending seminar, I have had

opportunities to use my

presentation skills. (7)

I feel more confident in giving future presentations


Attending




my understanding

of research methodologies.

(9) Attending


my appreciation of

the breadth and depth of my chosen field. (10)

Attending seminar has contributed to

my understanding

of concepts related to my chosen field.

(11)


gain a sense of where my research

interests fit within my

chosen field. (12)




gain a sense of how my research

interests relate to

complementary fields. (13)

Attending seminar has helped me determine where my research


research. (14)

Answer If Select your level of involvement with the Hume Center Undergraduate Student Is Selected Q19 Have you learned any presentation skills in attending seminar? If so, what skills have you learned?



Answer If Please select your level of involvement with the Hume Center. Undergraduate Student Is Selected Q30 Select your level of agreement with the following statements



(3)


Having a Hume GRA

Seminar means


valued at the Hume

Center. (1)

Having a Hume GRA




Having a Hume GRA

Seminar means peer

support among





Answer If Select your level of involvement with the Hume Center Undergraduate Student Is Selected Q20 Do you feel seminar contributed to a sense of camaraderie among the students at the Hume Center? If so, how do you feel it has done so?



Answer If Select your level of involvement with the Hume Center Faculty Is Selected Q15 How long have you worked with the Hume Center?



Answer If Select your level of involvement with the Hume Center. Faculty Is Selected Q16 Think about the graduate student population at the Hume Center, and select your level of agreement with the following statements.

Strongly Disagree

(1)

Disagree (2)


(3)


Not Applicable

(6)

Students seem more confident

in giving presentations


Students have opportunities

to use presentation

skills for Hume Center

presentations, outside of

seminar. (2)

Students show improvement in

their presentation abilities after

participating in seminar. (3)

There has been a change

in graduate student



camaraderie since the

Hume GRA Seminar was initiated. (4) Graduate

students have an increased

understanding of research

methodologies, since the

Hume GRA Seminar was initiated. (5)

Graduate students

appreciate of the breadth and depth of their chosen field more, since the


Graduate students have an increased

understanding of concepts

related to their field, since the

Hume GRA Seminar was



initiated. (7) Graduate


sense of where their research

interests fit within their

chosen field, since the


Graduate students have an increased sense of how their research interests relate

to complementary

fields, since the Hume GRA Seminar was initiated. (9)

Graduate students are

more aware of where their research




research, since the Hume GRA Seminar was initiated. (10)



Answer If Please select your level of involvement with the Hume Center. Faculty Is Selected Q28 Select your level of agreement with the following statements.



(3)


Having a Hume GRA

Seminar means


valued at the Hume

Center. (1)

Having a Hume GRA




Having a Hume GRA

Seminar means peer

support among





Answer If Please select your level of involvement with the Hume Center. Faculty Is Selected Q23 If you have noticed a change in graduate student camaraderie since the Hume GRA Seminar was initiated, please describe these changes. How do you think seminar has contributed to these changes?



Answer If Please select your level of involvement with the Hume Center. Technical Staff Is Selected Q24 How long have you worked with the Hume Center?



Answer If Please select your level of involvement with the Hume Center. Technical Staff Is Selected Q25 Think about the graduate student population at the Hume Center, and select your level of agreement with the following statements.

Strongly Disagree

(1)

Disagree (2)


(3)


Not Applicable

(6)

Students seem more confident

in giving presentations


Students have opportunities

to use presentation

skills for Hume Center

presentations, outside of

seminar. (2)

Students show improvement in

their presentation abilities after

participating in seminar. (3)

There has been a change

in graduate student



camaraderie since the

Hume GRA Seminar was initiated. (4) Graduate


understanding of research

methodologies, since the


Graduate students

appreciate of the breadth and depth of their chosen field more, since the


Graduate students have an increased

understanding of concepts

related to their field, since the

Hume GRA Seminar was



initiated. (7) Graduate


sense of where their research

interests fit within their

chosen field, since the


Graduate students have an increased sense of how their research interests relate

to complementary

fields, since the Hume GRA Seminar was initiated. (9)

Graduate students are

more aware of where their research




research, since the Hume GRA Seminar was initiated. (10)



Answer If Please select your level of involvement with the Hume Center. Technical Staff Is Selected Q29 Select your level of agreement with the following statements.



(3)


Having a Hume GRA

Seminar means


valued at the Hume

Center. (1)

Having a Hume GRA




Having a Hume GRA

Seminar means peer

support among





Answer If Please select your level of involvement with the Hume Center. Technical Staff Is Selected Q26 If you have noticed a change in graduate student camaraderie since the Hume GRA Seminar was initiated, please describe these changes. How do you think seminar has contributed to these changes?

Hume Center Graduate Student Seminar Evaluation€¦ · Hume Center Graduate Student Seminar...

Documents

Transcript of Hume Center Graduate Student Seminar Evaluation€¦ · Hume Center Graduate Student Seminar...