TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of...

29
2018 Penny Ma, Rajagopal Sankaranarayanan, Yeyi Ma, Tong Wang. 4/27/2018. Teaching Technology Lab Evaluation Report

Transcript of TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of...

Page 1: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 1 of 29

2018

Penny Ma,

Rajagopal Sankaranarayanan,

Yeyi Ma,

Tong Wang.

4/27/2018.

Teaching Technology Lab

Evaluation Report

Page 2: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 2 of 29

Executive Summary

The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based on

the major evaluation questions developed by the team after the initial project proposal meeting with

TTL Director – Dr. Anne Leftwich. This report will explain the major evaluation questions; outline

the evaluation method including data collection and analysis process, findings, and

recommendations.

Strengths

Two-thirds of the students responded they feel welcomed by TTL staff while using TTL.

More than half of the survey respondents said they are using TTL for group collaboration.

TTL staffs are more focused on solving the day to day problems in TTL.

TTL staffs are professional and keeping track of their time for tasks performed at TTL.

Only a few students responded they are unaware of TTL existence and think TTL is for AIs.

Areas of Improvements

Students responded to survey and interviews are not happy with TTL space layout and prefer

using collaboration room across TTL.

TTL staff is spending considerable time in organizing and managing inventory that leads to

delay in serving students at times.

TTL Staff is finding difficulty in prioritizing tasks – particularly ad-hoc tasks.

TTL Staff indicated that they are do not respected enough by students.

Recommendations

1. AI’s should advertise the TTL services more frequently to all the students.

2. Adding more working space such as Whiteboards, roundtables could promote TTL usage.

3. Preparing Job aids for newly added inventory could be helpful to facilitate the lesson plans.

4. Adding transparent and accessible cabinets could help organize TTL equipment effectively.

5. Setup clear task expectations for each TTL staff by making it explicit and transparent during the

start of the semester.

6. Help prioritize the TTL staffs daily tasks including ad-hoc tasks.

7. Create an environment for TTL staff that promotes recognition, respect, and appreciation by

students.

Page 3: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 3 of 29

Acknowledgments

We would like to thank the following people for supporting us in successfully completing this

evaluation project.

First, we would like to thank Dr. Leftwich, the Lab Director of the TTL for providing us this

valuable opportunity to conduct the TTL evaluation.

We would like to thank the Associate Instructors – Candace Buggs, Meize Guo and Victoria

Abramenka for helping us with their valuable support and inputs throughout this evaluation

project. We would like to thank the TTL staff members – Jennifer Park and Jui-Hsin Hung for

their co-operation and support throughout the project. We also want to thank the undergraduate

students who participated in the interviews and completed the surveys on our behalf. Their

opinions, time and effort are truly appreciated.

Finally, we would like to express our sincere gratitude to Dr. Ozogul for her expert guidance.

She provided us with great insights and constructive feedback, which helped us to focus on the

key elements and complete this evaluation project successfully.

Page 4: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 4 of 29

Table of Contents

Introduction ...................................................................................................................................5

Major Evaluation Questions ........................................................................................................5

Evaluation Method ................................................................................................................... 6

Project Initiation ..................................................................................................................7

Data Collection and Analysis ..............................................................................................7

Evaluation Results ...............................................................................................................9

Appendix A: Project Schedule ......................................................................................................16

Appendix B: Document Review Trends ......................................................................................17

Appendix C: Observation Checklist ............................................................................................18

Appendix D: Interview Protocol ..................................................................................................19

Appendix E: Survey Questionnaire .............................................................................................20

Appendix F: Descriptive Statistics ..............................................................................................25

Appendix G: Sample Interview Responses – Students ................................................................27

Appendix H: Sample Interview Responses – Staff .....................................................................28

Appendix I: Sample Status Report ...............................................................................................29

Page 5: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 5 of 29

1. Introduction

Teaching Technology Lab (TTL) is a multi-functional place for pre-service teaching students

(undergraduate students, mainly enrolled in W200 class) of the Indiana University School of

Education to learn and practice how to integrate instructional technologies into teaching and

learning. Dr. Anne Leftwich is the Lab director for the TTL who sets the vision and oversees the

daily activities of the TTL. TTL also has two supporting staff members, doctoral students with

experience in teaching and utilizing technologies in different classroom settings. TTL provides

following services:

Computing service

Technology integration assistance

Resources for students and staff/faculty

Equipment loan service

Technology workshop

Students experiments

In this evaluation project, the evaluation team evaluated the TTL based on the major evaluation

questions developed by the team after the initial project proposal meeting with TTL Director – Dr.

Anne Leftwich. The next section depicts the major evaluation questions.

2. Major Evaluation Questions

The purpose of this project is to evaluate the Teaching Technology Lab (TTL) in the school of

Education. The evaluation team had an initial project kick-off meeting with Dr. Anne Leftwich

on 22nd

February 2018, to understand her vision and goals for TTL and to develop our major

evaluation questions for this evaluation project. Below are the major goals of TTL:

TTL should be the place where W200 students feel warm, welcome, and comfortable to get

answers to all the technology questions they have.

TTL should be a place where W200 students check out and experiment with new technology

that is available.

TTL should be the learning collaboration space where W200 students work as a group.

Page 6: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 6 of 29

The following are the major evaluation questions our team has developed based on initial

meeting and TTL goals:

1. How are students utilizing the TTL?

2. What are expectations from students on TTL and TTL staff?

3. Is TTL being used as a space for learning collaboration?

4. How can TTL be rearranged to meet its goals?

5. How does TTL staff spend their time currently?

6. What does TTL staff need from Dr.Leftwich to take ownership of TTL day to day

operations?

7. What could Dr. Leftwich do to empower the TTL Staff? Operations or Human Resources?

8. How can TTL staff be best utilized to support students and Associate Instructors?

9. How can TTL staff be made aware of the best way to solve a problem before it becomes a

problem?

3. Evaluation Method

Below figure depicts the TTL evaluation process.

Figure 1. TTL Evaluation Process

Page 7: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 7 of 29

3.1. Project Initiation Phase

Project Kick-off Meeting:

The evaluation team had an initial project kick-off meeting with Dr. Anne Leftwich on 22nd

February 2018 at 8 PM to understand vision and goals for TTL and to develop major evaluation

questions for this project. Based on the initial project meeting, the project goals and major

evaluation questions are developed. The proposal had been sent to Dr. Leftwich to make sure all

the goals and the major evaluation questions are captured as expected. A project schedule was

developed and communicated to the client (See Appendix A).

Project Proposal:

The team then prepared the evaluation proposal with the major evaluation questions, proposed

evaluation method and time schedule to complete the evaluation. The project proposal was sent

to Dr. Leftwich and Dr. Ozogul for their review to obtain their feedback comments.

Status Reports:

The team has sent a status report to Dr. Leftwich on every Friday to share the project progress

for the work week to Dr. Leftwich. (See Appendix B for sample status report).

3.2. Data Collection and Analysis

Document Review:

The team started reviewing the documents provided by Dr. Leftwich to understand the

background information about TTL. The team has collected valuable data from the sign-up sheet

provided by Dr. L (data sheet from Fall 2016) to understand the TTL utilization by students and

to come up with the observation checklist, electronic survey options, and interview questions.

The evaluation team also referred this semester (Spring 2018) sign-up sheet to triangulate the

sign-up sheet data with onsite observation data, survey data and student interview data. Please

see Appendix B for the trends from Document review.

Page 8: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 8 of 29

Observation:

Based on the sign-up sheet data from Dr. Leftwich, we have created the items to be included in

the observation checklist as the first point of primary data collection (see Appendix C for the

observation checklist). Each member of the team observed the TTL for four hours for two full

days (3/20 and 3/22) using the observation checklist. The observation notes are reported in the

results section.

TTL Staff Interviews:

Once we collected the initial data through document review and onsite observations, an interview

protocol (see Appendix D) was developed to collect in-depth semi-structured interview with two

of the current TTL Staff members and three ex-TTL staff members to answer our evaluation

questions. These interviews gave us the opportunity to collect in-depth qualitative information

and to triangulate the data gathered from the onsite observations and document reviews. The

interview protocol was validated by Dr. Ozogul, who is not involved in the evaluation process to

get expert advice on the instrument thus ensuring ‘construct validity’. We also ensured

‘reliability’ (consistency) by forming two pairs to conduct interviews. Additionally to improve

‘credibility’ we did perform member checking with the interview participants for the accuracy of

responses.

Electronic Survey for Students:

Based on the evaluation questions, an electronic survey was designed using Qualtrics to collect

data from current W200 students. An electronic survey (see Appendix E) was sent to all students

enrolled in W200 course through Dr. Leftwich. We received 29 responses and only 26

respondents indicated that they are currently used TTL at least once. Out of 26 respondents, only

17 respondents answered all the questions related to TTL usage (valid responses – 70.58%). The

survey data were analyzed using the descriptive statistics. We added the highlighted survey

results (see Appendix E). The data from the survey is used for triangulation along with Onsite

observation and student follow-up interviews.

Page 9: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 9 of 29

Student Follow-up Interviews:

Based on voluntary participation, three students representing different sections are selected based

on voluntary participation to have an in-depth follow-up interview. The interview data were

analyzed using three different data analysis techniques – Constant comparison analysis,

Keyword-In-Context (KWIC) and classical content analysis. Themes and codes have been used

from the triangulated data from interviews, observations, and survey to come up with

recommendations. We use ‘inter-rater reliability’ – that is, codes and themes are verified by

different coders and consensus is reached for devising the recommendations. The next section

talks about the evaluation results and some of the sample responses from TTL staff members and

Students regarding TTL utilization.

3.3. Evaluation Results

Onsite TTL Observation Notes:

The Evaluation team members observed the TTL from 3/20 – 3/22 from 10 AM till 6 PM based

on observation checklist (see Appendix B). On the day (3/20 and 3/21) the TTL staff responsible

was on-time, opened the lab at 10 AM. On 3/22, the TTL staff responsible for that day arrived at

the lab 50 minutes late.

Based on our 20 hours of observations between 3/20 – 3/22, we did not see more than 3 students

using TTL. We have observed only once students coming in pairs and using TTL space for group

collaboration. TTL staffs do welcome the students and other visitors when they enter TTL. For

example, we heard them saying, “Hello, how can I help you?” When the student replied they

want to check out an equipment (voice recorder), they acted fast, procured the equipment from

the cabinet, checked if the equipment is powered on, changed the old battery and made sure it is

in working condition. They do care about resolving issues on time.

During our observations, a few students did check out tech equipment such as Dash robot, while

others explored the tech tools at the TTL space with help of AIs. AIs also took the whole class a

couple of instances to explain the arrangement of stations; some of the students show some

enthusiasm.

Page 10: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 10 of 29

TTL staff and AI’s explored on how to utilize new tech tools that arrived during the spring break

(For Ex., cubelets) and had a discussion on how to utilize them for their lesson plan. We are

observed that neither TTL staff nor AI’s had no prior knowledge about the tech tools available in

the TTL and were only exploring how to use it when they had to use it. Based on our observation

and data from the TTL staff interview, this seems to be the case most of the time (Please see

sample staff interview responses).

Other observations during those two days include; on Tuesday and Thursday, TTL staffs have

full 8 hr shift and they had lunch in the TTL during work. TTL staffs typically spend their day

updating the W200 website. We also observed they log their time in shared Google doc,

whenever they can. Apart from updating the website and coordinating with AIs, TTL staff do

clean the lab and organize the TTL space.

Overall TTL staffs are welcoming students, working on the issues fast and try to help the

students and AIs wherever they can. They utilize their free time by organizing things, clear the

TTL space and update the W200 website. We triangulated the data collected from onsite

observation, staff interview, and document review.

Survey results:

Please see Appendix F for the descriptive statistics based on the valid responses for the items in

the survey.

Interview results:

Please see Appendix G and Appendix H for the sample interview responses from TTL staffs and

the students. From the transcribed notes, we analyzed the data to come up with codes. Each team

member has verified the codes and created common themes using three data analysis methods –

constant comparison analysis (codes used to identify themes), keywords-in-context (keywords

before and after the codes are looked upon for the context) and classical content analysis – (count

how many times the codes are repeated). Each team member has verified and agreed on the

themes to answer the evaluation questions.

Page 11: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 11 of 29

Findings:

We relied on more than one data point and data analysis methods to come up with our findings.

For example, to answer the first evaluation question - How are students utilizing the TTL? We

relied on the onsite observation notes based on a checklist, student survey results from Qualtrics,

student sign-up sheets from document reviews, and in-depth student follow-up interviews.

Similarly, to answer the major evaluation question - How does TTL staff spend their time

currently? We relied on our onsite observation notes, document reviews (daily tasks Google doc,

sign-up sheets), TTL staff interviews (Current staff & Ex-staff) to come up with the findings.

The below section explains the findings of all our major evaluation questions.

Evaluation Question Findings

1. How are students utilizing the TTL? 41% of the survey respondents indicated that

they use TTL only once during the semester.

Based on the student interview and onsite

observations, we concur that students are not

using TTL very frequently during the semester.

The latest sign-up sheet also confirms this

finding.

2. What are expectations from students on

TTL and TTL staff?

Students who are using TTL reported they felt

the TTL staff is warm and welcoming. Having

said that, two-thirds of the survey respondents

told they are using TTL rarely (i.e., 41% using

only once in a semester and 35% using

sometimes in a semester. The interview data

revealed that students prefer to use the

collaboration room rather than TTL due to

space layout (See Appendix G for sample

student survey responses).

3. Is TTL being used as a space for

learning collaboration?

There is a scope for improvement to make the

TTL space for group collaboration. Interview

data revealed that most of the students use TTL

Page 12: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 12 of 29

only when they are asked to use by the

instructors and some students feel that using

collaboration room across TTL is better than

using TTL due to space constraints (See

Appendix G for sample student survey

responses).

4. How can TTL be rearranged to meet its

goals?

The interview data revealed that the TTL space

is not adequate for group collaboration. For

example, one of the students reported that there

are no whiteboards or open tables available for

students to use it as a workspace. Another

student said the space layout of TTL with

computers and equipment makes it look very

congested and prevents students to use it as a

learning space. Upgrading the TTL space

layout with more furniture and more open

space could help increase the TTL utilization.

5. How does TTL staff spend their time

currently?

Based on the onsite observations and the staff

interview data, the TTL staff is spending most

of their time in updating W200 website and

ETS co-ordination for iPad fixes. They are also

spending a considerable amount of time in ad-

hoc activities and supporting AIs answering

their questions related to their lesson plans.

TTL staff also spending a good amount of time

to learn about the new tech tools such as 3D

printers and robots in their remaining work

time (See Appendix H for sample staff

responses).

Page 13: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 13 of 29

6. What does TTL staff need from

Dr.Leftwich to take ownership of TTL

day to day operations?

The interview data revealed that TTL staff

needs help with prioritizing the tasks and

setting clear task expectations during the start

of the semester (See Appendix G for sample

staff responses).

7. What could Dr. Leftwich do to

empower the TTL Staff? Operations or

Human Resources?

Interview data revealed that TTL staff

members feel they are not respected and

getting enough recognition or appreciation for

the hard work they are doing from students.

Creating an environment that fosters mutual

respect, appreciation and recognition could

empower the TTL staff. (See Appendix G for

the sample staff interview responses).

8. How can TTL staff be best utilized to

support students and Associate

Instructors?

Based on the onsite observations and the

interviews, TTL staffs are providing enough

support to AIs and students.

9. How can TTL staff be made aware of

the best way to solve a problem before

it becomes a problem?

Based on the onsite observation and the staff

interviews, TTL staff is spending too much

time on the W200 website updates and Canvas

due to the curriculum change this semester.

Majority of their time is being spent in the

activities related to the W200 website updates

and Canvas updates. The remaining time is

spent in the ad-hoc activities such as answering

AIs and students on tech questions and helping

with checking out equipment etc., They are not

clear on how to prioritize the task at hand. If

they have a clear sense of urgency on tasks in

hand and focus on the important tasks, they can

be proactive. (See Appendix G for the sample

staff interview responses).

Page 14: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 14 of 29

Recommendations

The evaluation team has come up with following recommendations for the TTL.

1. AI’s should more frequently advertise the TTL services to all the students during the

start of the semester.

Based on the finding for the evaluation question 1, it is clear that majority of students (76%

of survey respondents, all of the interview respondents, and based on onsite observation) are

using TTL less frequently during the semester. The evaluation team recommends AIs to

advertise the TTL services more frequently to improve the TTL utilization.

2. More working space should be added to promote student group discussions and

collaborations.

Based on findings for the evaluation questions 2, 3, and 4, the interview data and the survey

data revealed that students want the TTL to be more open and inviting. The current space

layout is not promoting group collaborations. The evaluation team recommends rearranging

the TTL by adding more open spaces that may promote group collaborations between

students.

3. Preparing Job aids for newly added inventory could be helpful to facilitate the lesson plans.

Finding for the evaluation question 5 revealed that TTL staff members are spending a

considerable amount of time figuring out how to use new tech tools at the time of need to

help AIs incorporate them in the lesson plan. The evaluation team recommends creating a job

aid or manual as soon as new inventory arrives could be helpful to save time.

4. Adding transparent and accessible cabinets could help organize TTL equipment

effectively.

Student interviews revealed that students are spending 10 – 15 mins to check-out equipment

and feel sometimes frustrating. Staff interview data revealed that at times organizing

inventory is taking time. (See Appendix F and G for sample responses). The evaluation team

recommends adding transparent and accessible cabinets to reduce the student waiting time

given that survey respondents said that equipment checkout is the second most important

purpose of students using TTL.

Page 15: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 15 of 29

5. Setup clear task expectations for each TTL staff by making it explicit and transparent

during the start of the semester.

The evaluation team recommends the lab director to set up clear task expectations for each

TTL staff member by making tasks explicit and transparent during the start of the semester

based on the finding for answering the evaluation question 6 (see Appendix G for the sample

staff responses).

6. Help prioritize the TTL staffs daily tasks including ad-hoc tasks.

The finding for the evaluation question 6 revealed that TTL staff needs help with prioritizing

ad-hoc tasks day to day basis when there are in the middle of W200 website updates or fixing

an urgent issue (ex., 3D printer). The evaluation team recommends the lab director to help

prioritize the daily tasks for TTL staffs based on urgency.

7. Create an environment that promotes recognition, respect, and appreciation for TTL

Staff.

The finding for the evaluation question 7 revealed that the TTL staffs feel low morale and

want to be respected and appreciated for the hard work they are doing. Please see Appendix

G for the sample staff responses. The evaluation team recommends the lab director to create

an environment that promotes recognition, respect, and appreciation for TTL staff.

Page 16: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 16 of 29

Appendix A: Project Schedule

Project Tasks Due Date

Project Kick-off Meeting 2/22/18

Draft Project Proposal 1/3/18

Submitting proposal to Dr. Leftwich 3/8/18

Data Collection

Document Review

Survey Instrument design, development

Survey distribution

Onsite Observation (Tuesday – 3/20 & Thursday 3/22)

Interview protocol development

Interview with TTL Staff

Interview with students

3/8/18

3/15/18

3/22/18

3/22/18

3/29/18

4/5/18

4/12/18

Data Analysis

Content analysis of the interview data

Statistical analysis of the survey data

4/19/18

4/19/18

Draft Report Submission and Presentation 4/26/18

Final Report Submission to Dr. Leftwich 5/3/18

Page 17: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 17 of 29

Appendix B: Document Review Trends

Based on the sign-up sheet data provided by Dr. Leftwich for Fall 2016. Based on the sign-up

sheet data, we found the major purpose of TTL usage is ‘using computing devices’ followed with

‘equipment checkout’. Total record count – 335; Students – 212.

The evaluation team also analyzed the latest TTL sign-up sheet to see the trends in the

current semester. We could not find enough data to support TTL utilization by W200

students. The current sign-up sheet covers only the equipment borrows and returns and

doesn’t include the major purpose of the TTL visit by students. Total Record count: 42

Page 18: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 18 of 29

Appendix C: Observation Checklist

1. Students being welcomed when they enter?

Yes No

2. How do Students come to TTL

Individually In group

3. How TTL staff is spending time (every hour)

a. Clean up b. ETS consult

c. Organize TTL equipment d. Unpacking equipment

e. Equipment checklist f. Student assistance

g. 3D printer support h. iPad updates

i. Board games j. Purchase list

4. Are the students’ requests resolved by TTL Staff between this time?

Yes No

5. Is TTL staff is fully engaged on the task in hand?

Yes No

6. How responsive TTL staff when working on a task?

a. Very fast b. fast c. slow d. very slow

Done by : __________________

TTL Staff:__________________

Date: _____________________

Page 19: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 19 of 29

Appendix D: Interview Protocol

Staff Members

1. How do you spend a typical day in TTL?

2. Do you think you there are many challenges in managing time efficiently in TTL? Could you

share some examples?

3. Do you think you can manage all the day-to-day activities in TTL by yourself?

4. Is there any specific help or assistance you need form Dr. L for day-to-day operations in

TTL?

5. What additional support do you need from Dr. L?

6. How do you support students and AI currently? Could you share some examples?

7. Do you think you can fix the problems (if any) you currently see in TTL? If yes, give me

some examples. If no, why?

8. Any other comments/suggestions to improve TTL?

Students

1. Please tell me what you know about TTL?

2. Do you use TTL service? For what?

3. What is your experience of using TTL?

4. Do you have any expectation of TTL service?

5. What additional facilities (specific) you would like to see in TTL?

6. Do you use TTL as a group? Why? Why not?

7. What is preventing you to use TTL for group collaboration?

8. Are you satisfied with the current layout of TTL? Why?

9. Can you share your experiences with TTL staff?

10. Any other comments/suggestions to improve TTL?

Page 20: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 20 of 29

Appendix E: Survey Questionnaire

TTL Evaluation

Instructions:

The purpose of this survey is to assess your experience using the TTL (Teaching Technology Lab).

Through this survey, we hope to identify your expectations and challenges in using the TTL to inform key

decision makers regarding students' needs. Please complete the survey to reflect your opinions as

accurately as possible. Time taken to complete the survey will be no more than ten minutes. Your

response will be kept strictly confidential. However, you may provide your name and email address at the

end of the survey to participate in $10 Amazon gift card drawing. We sincerely appreciate your support in

taking time to help us with this evaluation. Please feel free to contact our team: Raj ([email protected])

and Yeyi ([email protected]) if you have any questions.

* Required

Section A: Background Information

1. Have you ever used TTL? *

o Yes (1)

o No (2)

Display This Question:

If 1. Have you ever used TTL? * = No

2. Could you please share us the reason why you did not use TTL? *

______________________________________________________

Display This Question:

If 1. Have you ever used TTL? * = No

We appreciate your feedback. Thanks for your time. Have a great day!

Page 21: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 21 of 29

Display This Question:

If 1. Have you ever used TTL? * = Yes

2. How often do you use TTL? *

o Rarely (at least once during the semester) (1)

o Sometimes ( at least once a month) (2)

o Often ( at least a week) (3)

o Very often (more than once a week) (4)

Display This Question:

If 1. Have you ever used TTL? * = Yes

3. What is your main purpose of using TTL? *

Please select all that apply

▢ Equipment Checkout (1)

▢ Technology Assistance (2)

▢ Help with assignments (3)

▢ Using computer devices (4)

▢ Using 3D Printers (5)

▢ Board Games (6)

▢ Group Collaboration (7)

▢ Other (Please specify) (8) ________________________________________________

Display This Question:

If 1. Have you ever used TTL? * = Yes

Page 22: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 22 of 29

Q9 Section B: Survey Questions

Display This Question:

If 1. Have you ever used TTL? * = Yes

4. Do you feel welcomed in TTL? *

o Yes (1)

o No (2)

Display This Question:

If 1. Have you ever used TTL? * = Yes

5. Please answer the below questions by choosing the option most suitable for you. *

Strong

disagree (1)

Somewhat

disagree (2)

Neither agree

nor disagree

(3)

Somewhat

agree (4)

Strongly agree

(5)

5.1. My

requests are

resolved in

time. (1) o o o o o

5.2. My

experience

with using

TTL services

is good. (2)

o o o o o

5.3. My

experience

with TTL staff

is good. (3) o o o o o

Display This Question:

If 1. Have you ever used TTL? * = Yes

Page 23: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 23 of 29

6. Do you use TTL as a space for group collaboration? *

o Yes (1)

o No (2)

Display This Question:

If 1. Have you ever used TTL? * = Yes

7. Are you happy with current space/floor layout of TTL? *

o Yes (1)

o No (2)

Display This Question:

If 1. Have you ever used TTL? * = Yes

8. What are your expectations from TTL? *

________________________________________________________________

Display This Question:

If 1. Have you ever used TTL? * = Yes

9. Do you have any other comments or suggestions for TTL? *

________________________________________________________________

Display This Question:

If 1. Have you ever used TTL? * = Yes

Section C: Additional Information

Display This Question:

If 1. Have you ever used TTL? * = Yes

Page 24: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 24 of 29

10. Can we contact you if we have follow-up questions about your survey responses? *

o Yes (1)

o No (2)

Display This Question:

If 1. Have you ever used TTL? * = Yes

11. Would you like to participate in the gift card drawing? *

o Yes (1)

o No (2)

Display This Question:

If 1. Have you ever used TTL? * = Yes

If we may contact you with follow-up questions, or if you would like to participate in the gift card

drawing, please leave your email (your survey responses will remain confidential)*:

Display This Question:

If 1. Have you ever used TTL? * = Yes

Q20 Email Address

________________________________________________________________

End of Block: Default Question Block

Page 25: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 25 of 29

Appendix F: Descriptive Statistics

# 1. TTL Usage

# 2. TTL Purpose

Page 26: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 26 of 29

# 3 – About TTL Staff and Space Layout.

Top 3 Student Expectations from TTL

1. Help with projects/assignments.

2. Additional Space for group/team collaboration.

3. Make TTL creative and more inviting to students.

Page 27: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 27 of 29

Appendix G: Sample Responses – Students

1. I always thought the room was just an office for graduate students. It doesn't look

open or inviting to students.

2. TTL staff knows how to operate the different equipment and the staff can help me with my

CEL assignments if I need help. Lastly, that they are nice. TTL almost always meets my

expectations.

3. Please provide enough space for collaborative work. Spaces to work like a

project area; it looks like a middle school computer lab, and it should be more

about creativity and advancement in tech.

4. I don’t spend too much time using it. I mainly use this room here the

printing room (which is at the opposite of the TTL) It got a lot of things

that TTL has as well. I think that it would serve a much larger purpose. I

think that space is kind of needed to serve so space was not here that

space would be more necessary. TTL kind of almost feels even though I

know it is open to us, it almost feels off limits.

5. I would say just have more collaborative spaces. I think the structure and the

floor plan to make it open and inviting. I think more students would want to be in

there. Or the room sometimes even like shut when there is no staff in there. So

maybe have space and equipment needs to be shut off for lock the way. Having

the ability to do so, that students still use that space. Really modeling the room

we are in. I think having a set up like this printer room or combining them or

doing something that makes the room accessible and opens like an environment

where people can or want to work could be

6. Maybe like boards, you can write on. Because is there a whiteboard hanging up the

computers I don’t remember. So maybe something we can write on like a whiteboard or

something ideally brainstorms in that area.

Page 28: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 28 of 29

Appendix H: Sample Responses – Staff

Help create a supportive environment where all the TTL staffs are respected by

students. Set the clear expectation at the start of the semester.

3D printer. Right now things are falling into place but it just takes too much time to

maintain it. People have to acknowledge it is not a printer but 3D printer and requires

special skills and effort to maintain it.

Set clear guidelines and responsibilities and who is responsible for

what. Make it explicit helps. It’s always nice to have sort of recognition

and appreciation for our job. it ’s really important to cultivate a sense

of belonging and culture, loyalty so you enjoy.

I can manage independently but I want some backup as there are no break

times even to go to the restroom.

Help prioritize the tasks for TTL. Set clear expectations by making it more explicit and

transparent.

Need confirmation on an understanding of tasks before start working on it.

No Fixed schedule and Adhoc issues. W200 updates are time-consuming.

Mostly of time interrupted by ad-hoc tasks as W200 as the lab is open to

everyone. No fixed schedule. Too much work to update W200 updates. 3D

printing issues consume a lot of time.

Page 29: TTL Evaluation - xiaomalaoshi.files.wordpress.com · Page 2 of 29 Executive Summary The purpose of this evaluation project was to evaluate the Teaching Technology Lab (TTL) based

Page 29 of 29

Appendix I: Sample Status Report