© 2021 NACADA: The Global Community for Academic Advising

The contents of all material in this presentation are copyrighted by NACADA: The Global Community for Academic Advising, unless otherwise indicated. Copyright is not claimed as to any part of an original work prepared by a U.S. or state government officer or employee as part of that person's official duties. All rights are reserved by NACADA, and content may not be reproduced, downloaded, disseminated, published, or transferred in any form or by any means, except with the prior written permission of NACADA, or as indicated below. Members of NACADA may download pages or other content for their own use, consistent with the mission and purpose of NACADA. However, no part of such content may be otherwise or subsequently be reproduced, downloaded, disseminated, published, or transferred, in any form or by any means, except with the prior written permission of, and with express attribution to NACADA. Copyright infringement is a violation of federal law and is subject to criminal and civil penalties. NACADA and NACADA: The Global Community for Academic Advising are service marks of the NACADA: The Global Community for Academic Advising

IMPROVING ENGAGEMENT IN REMOTE LEARNING ENVIRONMENTS

FACILITATING TRANSITION INTO HIGHER EDUCATION

JOHN WYATT, UNIVERSITY COLLEGE DUBLINDR. MAURICE KINSELLA, UNIVERSITY COLLEGE DUBLIN

OUTLINE• UCD & UCD LEAP

• DESIGN & IMPLEMENTATION

• COVID-19

• VLE DESIGN CHANGES

• KEY FINDINGS

• LESSONS LEARNED & FUTURE

UCD VET MEDICINE: THE SCHOOL

• UCD Vet Teaching Hospital open 24/7/365

• Top 25 QS World Subject Ranking

• AMVA, EAEVE, VCI accredited

• Requirements from UCD & accreditors

UCD VET MEDICINE: THE STUDENTS

• Approx. 300 1st year students

• 33% International Students (23% UCD Avg.)

• Classroom & practical learning components

• Student Adviser for support

UCD LEAP: SUPPORT DELIVERY ISSUES• Disengagement only apparent post-exams

• Difficult re-engaging students

• Existing supports under-used

• Negative impact on wellbeing

• Retention issues

• Social integration issues

UCD LEAP: CHANGES NEEDED

• Real-time engagement info sources

• Support interventions linked to data

• More immediate support for better outcomes

• Signposting both generic and tailored supports

• UCD Live Engagement & Attendance Project

• Bluetooth attendance data smartphone app

• At-risk students contacted

• Underpinned by Self-Determination Theory

• Self-populated

INITIAL DESIGN

2019• Student Feedback

2020• Student & SA Feedback

2021• Student & Research Team Feedback

LEAP DESIGN

AttendanceData

Reporting

Intervention

Progression

IMPLEMENTATION: INITIAL ROLLOUT

• More attendance data visibility

• Real-time interventions commenced

• Preliminary findings confirmed relationship

• Setup issues (accuracy & timetabling)

• Embedding issues (student & staff buy-in)

• High-attendance support gap

IMPLEMENTATION: FEEDBACK & CHANGES

• “Trusted Persons” format

• Light touch first intervention

• Stage 0 creation

• VLE identified as key engagement source

COVID-19

“My learning is nothing like it was

and I have never felt worse about my

performance”

“It’s a lot harder to engage in such a clinical program

remotely”

Classes(1162)

Students (70x avg)

80,000+ data points

lost

“My appreciation for the teaching staff has grown significantly for the supports and work

they put in for us”

1. Login Frequency

2. Quality of interaction

VLE DESIGN: CRITERIA

VLE DESIGN: PROGRAMME VIEW

VLE DESIGN: STUDENT LOG EXAMPLE

KEY FINDINGS: VLE DATA

N: MVB1 (94), MVB2 (89), GE1 (52), VNUR1 (44)

KEY FINDINGS: VLE DATA

N: MVB1 (94), MVB2 (89), GE1 (52), VNUR1 (44)

KEY FINDINGS: VLE DATA

N: MVB1 (94), MVB2 (89), GE1 (52), VNUR1 (44)

KEY FINDINGS: VLE DATA

N: MVB1 (94), MVB2 (89), GE1 (52), VNUR1 (44)

VLE DESIGN: STUDENT LOG EXAMPLE

Of students who failed modules, 54.5% were flagged, 45.5% were unflagged

Flag Info Autumn Spring

Total Flags 95 161

Unique Students flagged 37 43

Avg Flags per student 2.57 3.74

Avg Flags by week 7.92 13.42

KEY FINDINGS: VLE DATA

3.3

3.4

3.5

3.6

3.7

3.8

3.9

4

4.1

-6 -4 -2 0 2 4 6

SEAtS Usage & GPA VET10060

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

0 100 200 300 400

VLE Topic AccessAccess %

TopicsSEAtS Usage

GPA

KEY FINDINGS: ASSESSMENT DATA2020/21 Assessment Component Type 2019/20 Assessment Component Type

RESEARCH AND FEEDBACKSite:School of Veterinary Medicine, University College Dublin

Participants:Students: 2018:n=13 2019: n=18; 2020: Interviews n=14; 2021: n=21 SAs: 2021: n=10

Methodology:Mixed-method approach

Instruments:i.Questionnaire – Writtenii.Qualitative Interview – Phone and Written

Analysis: Reflexive Thematic Analysis(Braun & Clarke, 2014; Clarke & Braun, 2018)

• F2F instruction is missed

• Student Advisers seen as vital

• Support for early flagging

KEY FINDINGS: 2020 STUDENT FEEDBACK

“Professors are very available for help and

questions”

“Physical attendance is important so they

can explain fully what they mean”

“I would not be here today without them”“Really helped with my personal growth”

“Helps you try to solve the problem”

“If its not helping every person but it is helping one person,

you like that”

“There’s that 1% that you maybe need to

keep an eye so reaching out is nice”

• F2F instruction is still missed

• Student Advisers still seen as vital

• Challenges of online learning

KEY FINDINGS: 2021 STUDENT FEEDBACK

““(Advisor Name) is a great help”“She is amazing and so helpful”

“Great to know that there is a readily available advisor always there for you”

“Online learning makes my studies

seem more like chores”

“I don’t feel like a student in university without any practical

work”

“Lack of organization of lecture content”

“Bombarded with work”

“The balance of college work and personal time has

been lost”

• F2F support is still needed

• Student Advisers foster engagement

• Case for blended approach

KEY FINDINGS: 2021 ADVISER FEEDBACK

“Supporting students who may feel disconnected”

“Key element of role is supporting student integration to third level”

“Difficult to support students remotely, in

particular when students are upset”

“My student cohort are finding remote learning difficult”

“Online space has a place going into the

next iteration of student services”

“Tasks can be completed at a

distance but some face to face contact

is desired”

LESSONS LEARNED

•Scalability•Actionable intervention

data.•Accurate, but limitations

(ie: asynchronous downloading).

•Off-site architecture needed.•Low construction and maintenance costs.

•Address VLE Module ‘Siloing’.•Exists within UCD’s digital

infrastructure.•Ready integration into

stakeholder practice.

•VLEs capacity to foster multi-dimensional engagement.•Ongoing role of on-site student engagement.

Conceptual Operational

TechnicalEconomic

RECOMMENDATIONS: KEY INSIGHTS

• VLE data can enable Advisers to facilitate interventions

• Digital and in-person supports are interlinked

• Try to capture relative, not absolute engagement

• Remote learning has changed support delivery

RECOMMENDATIONS: FUTURE ACTIVITY

• Continue assessing VLE engagement model

• Implement ‘blended’ engagement monitoring tools

• Disseminate academic & internal lessons learned

• Identify value-add activity areas for continuation

CONTACT US• john.wyatt@ucd.ie

• maurice.kinsella@ucd.ie

• niamh.nestor@ucd.ie

REFERENCES1. Akçapınar, G., Mohammad, N. H., Majumdar, R., Flanagan, B., & Ogata, H. (2019). Developing an early-warning system for spotting at-risk students by using eBook interaction logs. Smart

Learning Environments, 6(1), 1-15. doi:http://dx.doi.org.ucd.idm.oclc.org/10.1186/s40561-019-0083-4 2. Alves, P., L, M., & Morais, C. (2017). The Influence of Virtual Learning Environments in Students' Performance. Universal Journal of Educational Research, 5, 517-527.

doi:10.13189/ujer.2017.0503253. Alves, P., Miranda, L. & Morais, C. (2016). Learning styles and access to virtual learning environments in the academic performance, Academic Conferences International Limited, Kidmore

End, pp. 25.4. Ayala, J. C., & Manzano, G. (2018). Academic performance of first-year university students: the influence of resilience and engagement. Higher Education Research & Development, 37(7),

1321-1335. doi:10.1080/07294360.2018.15022585. Black, A. E., & Deci, E. L. (2000). The effects of instructors' autonomy support and students' autonomous motivation on learning organic chemistry: A self-determination theory perspective.

Science Education, 84(6), 740–756. 6. Brooks, C., Thompson, C., Teasley, S. (2015) A time series interaction analysis method for building predictive models of learners using log data. In: Proceedings of the Fifth International

Conference on Learning Analytics and Knowledge, pp. 126–135. ACM, New York (2015a)7. Brophy, J. E. (1988). On motivating students. In D. Berliner & B. Rosenshine (Eds.), Talks to teachers (pp. 201–45). New York: Random House.8. Chipchase, L., et al. (2017). ‘Conceptualising and Measuring Student Disengagement in Higher Education: A Synthesis of the Literature’. International Journal of Higher Education. 6(2), 31-

42. 9. Cook, David & Artino, Anthony. (2016). Motivation to learn: an overview of contemporary theories. Medical Education. 50. 997-1014. 10.1111/medu.13074. 10. Cooper, M., Ferguson, R., & Wolff, A. (2016). What can analytics contribute to accessibility in e-learning systems and to disabled students' learning? Paper presented at the 99-103.

doi:10.1145/2883851.288394611. Credé, M., et al. (2010). Class Attendance in College: A Meta-Analytic Review of the Relationship of Class Attendance with Grades and Student Characteristics. Review of Educational

Research. 80(2), 272-295.12. Fuller, M. B., Wilson, M.A., & Tobin, R. M. (2011) The national survey of student engagement as a predictor of undergraduate GPA: a cross-sectional and longitudinal

examination, Assessment & Evaluation in Higher Education, 36:6, 735-748, DOI: 10.1080/02602938.2010.48879113. Gardner, J., & Brooks, C. (2018;2017;). Student success prediction in MOOCs. User Modeling and User-Adapted Interaction, 28(2), 127-203. doi:10.1007/s11257-018-9203-z14. Gasevic, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success.

Internet and Higher Education, 28,68–84.15. Hlosta, M., Zdrahal, Z., & Zendulka, J. (2017). Ouroboros: Early identification of at-risk students without models based on legacy data. Paper presented at the 6-15.

doi:10.1145/3027385.302744916. Jayaprakash, S. M., Moody, E. W., Laurıa, E. J., Regan, J. R., and Baron. J. D. Early alert of academically at-risk students: An open source analytics initiative. Journal of Learning Analytics,

1(1), 6-47.17. Kinsella, M., Nestor, N., Rackard, S., Last, J. & Wyatt, J. Using Attendance Analytics as a Motivational Tool for First-year University Students: The Live Engagement and Attendance Project

(LEAP). The European Conference on Education 2020 (ECE2020), 16th-19th July 2020 University of London. IAFOR.

18. Kinsella, M., Wyatt, J. & Nestor, N. 2021c. Responding to Acute Changes in Higher-education Engagement: Insights from UCD ‘Live Engagement and Attendance Project’. Ireland International Conference on Education (IICE 2021). Dun Laoghaire, Ireland.

19. Kelly, Anna M., and Padden, Lisa (2018). Toolkit for Inclusive Higher Education Institutions: From Vision to Practice. Dublin: UCD Access & Lifelong Learning https://www.ucd.ie/all/t4media/0274_UCD_TOOLKIT_1118_ONLINE_LR.pdf

20. Morra, T., & Reynolds, J. (2010). Universal Design for Learning: Application for Technology-Enhanced Learning. Inquiry: The Journal of the Virginia Community Colleges, 15 (1). Retrieved from https://commons.vccs.edu/inquiry/vol15/iss1/5

21. Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. http://dx.doi.org/10.1016/j.compedu.2009.09.008.

22. Mubarak, A. A., Cao, H., & Zhang, W. (2020). Prediction of students’ early dropout based on their interaction logs in online learning environment. Interactive Learning Environments, , 1-20. doi:10.1080/10494820.2020.1727529

23. Nistor, N. & Neubauer, K., From participation to dropout: quantitative participation patterns in online university courses. Computers & Education, 55, 2, (2010), 663-672.24. National Centre for Learning Disabilities (2020). Promise and Peril: Examining the Role of Ed Tech for Students With Disabilities https://www.ncld.org/wp-

content/uploads/2020/07/Promise-and-Peril-Examining-the-Role-of-Ed-Tech-for-Students-with-Disabilities.07212020.pdf25. Nik Nurul Hafzan, Mat Yaacob, Safaai, D., Asiah, M., Mohd Saberi, M., & Siti Syuhaida, S. (2019). Review on predictive modelling techniques for identifying students at risk in university

environment. MATEC Web of Conferences, 255, 3002. doi:10.1051/matecconf/20192550300226. Ormond Simpson (2013) Student retention in distance education: are we failing our students?, Open Learning: The Journal of Open, Distance and e-Learning, 28:2, 105-

119, DOI: 10.1080/02680513.2013.84736327. QS. 2020a. The Impact of the Coronavirus on Global Higher Education. QS. Accessed at: http://info.qs.com/rs/335-VIN-535/images/The-Impact-of-the-Coronavirus-on-Global-Higher-

Education.pdf28. Sclater, N. 2013. Learning Analytics: The Current State of Play in UK Higher and Further Education. JISC29. Simpson, S. (2013). ‘Student retention in distance education: are we failing our students?’ Open Learning: The Journal of Open, Distance and e-Learning. 28(2), 105-119.30. UCD Agile: Mapping our Student’s Experience of UCD – Executive Summary: https://www.ucd.ie/agile/whatagiledoes/mappingourstudentsexperience/thestudentperspective/31. UCD Index Survey 2019: https://www.ucd.ie/itservices/t4media/UCD%20INDEx%202019%20Student%20Survey%20Quantitative%20Analysis%20.pdf32. Valle, A., Núñez, J. C., Cabanach, R. G., González-Pienda, J. A., Rodríguez, S., Rosário, P., et al. (2009). Academic goals and learning quality in higher education students. The Spanish

Journal of Psychology, 12(1), 96-105.33. Webber, Karen L., et al. Does Involvement Really Matter? Indicators of College Student Success and Satisfaction. Journal of College Student Development, vol. 54 no. 6, 2013, p. 591-

611. Project MUSE, doi:10.1353/csd.2013.0090.34. Wolff, A., Zdrahal, Z., Nikolov, A. & Pantucek, M. (2013) Improving retention: predicting at-risk students by analysing clicking behaviour in a virtual learning environment, LAK13, Leuven,

Belgium.

REFERENCES

THANK YOU!