1

HEPPP Funded Initiatives Theories of Change and Program Logic Models UNSW’s HEPPP-funded initiatives for 2021 include:

1. Gateway Program and Pathway (Outreach, Pre-Access & Access) 2. Scholarships (Access) 3. Gateway Scholars Program (Participation) 4. Academic Tutoring Scheme (Attainment/Success)

In line with the HEPPP funding model, all of these programs aim to increase the access, participation and success of under-represented students in higher education. Because these programs share an overarching impact goal, their measures of long-term success will be shared (and are outlined in the impact measurement table on page 1). However, each program will approach producing this impact differently. This document outlines how each program intends to produce the stated impact (the theory of change) and specifies the inputs, outputs and outcomes of the programs in in logic models. Finally, the indicator banks included in this document outline how each program will measure the intended outcomes. Impact Measurement Indicators

Access • Increased applications, offers and enrolments to university and UNSW for students from: o Students from Gateway schools o Students form low SES backgrounds o Students from regional and remote backgrounds o First in family students

• Expanded breadth of schools represented at university and UNSW

Participation • Increased retention and success rates for students from: o Students from Gateway schools o Students form low SES backgrounds o Students from regional and remote backgrounds o First in family students

• Increased retention and success rates for students who participant in

programs compared to those who do not

Attainment • Increased academic attainment (WAM) for students from: o Students from Gateway schools o Students form low SES backgrounds o Students from regional and remote backgrounds o First in family students

• Increased academic attainment (WAM) for student who participate in

programs compared to those who do not

2

1. Gateway Program and Pathway (Pre-Access & Access)

Theory of Change

In 2020, The Gateway Admissions Pathway and Program (GAPP) was introduced as UNSW’s primary widening access admission strategy to address the under-representation of students from low socio-economic, regional and remote, and Aboriginal or Torres Strait Islander backgrounds. The program involves an on-campus experience for Year 10, online school holiday programs for Year 11 and 12 students and an early conditional offer scheme with adjusted ATAR requirements for access to UNSW. The theory of change for how the intended impact will be generated is outlined below. The Map: Knowledge Although students from under-represented backgrounds often have aspirations for further education (Harrison & Waller, 2018), they may not have access to accurate knowledge in their social networks (Andrews, 1999; Harvey-beavis & Robinson, 2000; Young, 2004). If we provide a reliable source of information about further education and career possibilities (an information map), then students will be better placed to make informed choices. The Compass: Navigational Capacity Students from under-represented backgrounds face complex challenges when navigating unfamiliar further education and workplace environments. If we can support students to develop their learner identity (their understanding of their learning strengths and weaknesses and how they relate to future study and career options), and their confidence, then they will better be able to successfully navigate their education journey (McFarlane, 2018). The Key: Academic Attainment There is a strong relationship between socioeconomic status and educational achievement and opportunity in Australia (Lamb et al., 2020). If we can support under-represented students to achieve a higher ATAR, then more tertiary study options will be available to them. Further, ATAR has been shown to be a very poor predictor of success at university for students from socio-educationally disadvantaged backgrounds (Messinis & Sheehan, 2015). If we provide an alternative pathway into UNSW with a reduced ATAR requirement for students from nearly 400 of the most disadvantaged schools in NSW, then more students from under-represented backgrounds will be able to access higher education at UNSW. References

Andrews, L. (1999). Does HECS deter? Factors affecting university participation by low SES groups (No. 99F Occasional Paper Series). Canberra: Higher Education Division, Department of Education, Training and Youth Affairs.

Harrison, N., & Waller, R. (2018). Challenging discourses of aspiration: The role of expectations and attainment in access to higher education. British Educational Research Journal, 44(5), 914-938.

Harvey-Beavis, A., & Robinson, L. (2000). Views and influences: Tertiary education, second- ary students and their advisers. Canberra: Commonwealth Department of Education, Training and Youth Affairs.

3

Lamb, S., Huo, S., Walstab, A., Wade, A., Maire, Q., Doecke, E., Jackson, J. & Endekov, Z. (2020). Educational opportunity in Australia 2020: Who succeeds and who misses out. Centre for International Research on Education Systems, Victoria University, for the Mitchell Institute: Melbourne.

MacFarlane, K. (2018). Higher education learner identity for successful student transitions. Higher Education Research & Development, 37(6), 1201-1215. Messinis, G., & Sheehan, P. (2015). The academic performance of first year students at Victoria University by entry score and SES, 2009-2013. Melbourne: Victoria Institute of Strategic Economic Studies.

Young, J. (2004). ‘Becoming different’: Accessing university from a low socio-economic community – barriers and motivators. Systemic Practice and Action Research, 17(5), 425–469.

4

Inputs and

activities Participants Outputs Short term outcomes Medium term outcomes

Year 10 Gateway Program Inputs

• NIDA contribution

• University Ambassadors

• A & E Staff Activities

• Develop program content

• Train UAs Develop Inside Out app

400 Yr 10 students (50 from 8 schools)

• Inside Out: Values/degrees/careers app

• Write Now workshop (1 hour): Students write their personal elevator pitch, linked to their future career goals (drawing on Inside Out app)

• Pitch Perfect workshop (1 hr 45 mins): Students work with NIDA specialists to practice presenting their personal pitchUsing dramatic techniques

• Lunch with corporate mentors (1 hour): incorporating an opportunity for live networking to put into practice the skills developed in Write Now and Pitch Perfect

THE MAP Possibilities • Students understand the opportunities and benefits of higher

education • Students develop an understanding of what studying at

university is like and the expectation of university • Students develop their knowledge of available degrees THE COMPASS Navigational Capacity • Students develop their ability to write about and present

themselves persuasively and with confidence • Students develop an understanding of their learning

strengths, interests and values and how these link to higher education and career options

• Students begin to map out their path from school to higher education to career

• Students develop a sense of belonging at university and at UNSW

• Students identify as a future university student THE KEY Academic Attainment • Students develop their motivation for HSC success

THE MAP Possibilities • Students feel prepared to make

decisions about their post-school future

• Students write strong ECO applications

THE COMPASS Navigational Capacity • Students have increased motivation

and confidence to continue their journey to higher education and graduate employment

• Students can tell the story of their own learner identity and how it links to their education journey

• Students choose post-school options that suit their learning strengths, interests, values and career goals

THE KEY Academic Attainment • Students have increased attainment

at school • Students have increased academic

capacity and preparedness for higher education

5

Year 11 Gateway Program

Inputs • InspirED

• University Ambassadors

• A & E Staff Activities • Develop

program content

• Train UAs

• Develop online platform

Spring Into • 1 workshop for 50+

students in each partner or highly engaged school

• 70% registration rate

Spring Program 1500 registrations • 20 registrations

from partner and highly engaged schools

• 8 registrations per schools for other schools with progression to uni 40% or higher

• 30 registrations per NGO partner

• 700 participants

Spring Into • Face-to-face workshops in

schools: post-school goals, current learning strengths and link to degree and careers (1 hr)

Early Conditional Offer Support • Gateway Admission Pathway info

(1 hr) • Start Here– start writing your

personal statement session (1 hr)

Faculty Experiences • 6-9 Faculty Experiences (1.5

hours) Academic Skills • Note Taking workshop (1 hr) • Thinking Critically workshop (1hr) HSC Content • HSC Subject masterclasses:

InspirED English Std/Adv Paper 1 (1.5 hrs)

• InspirED English Std/Advc Paper 2 (1 hr)

• InspirED Maths Std/Adv harder preliminary topics (1 hr)

University Ambassador • Real talk student stories session

(1 hr) • Q and A session (1 hr)

Follow up Academic support • Access to HSC4ME app

THE MAP Possibilities

• Students understand the opportunities and benefits of higher education

• Students develop an understanding of what studying at university is like and the expectations of university

• Students develop their knowledge of available degrees

• Students develop their knowledge of the nature and focus of each UNSW faculty

• Students develop an understanding of how to apply for the ECO

• Students develop an understanding of what makes a good personal statement

• Students know how to access further support for their ECO application

THE COMPASS Navigational Capacity

• Students develop an understanding of their learning strengths, interests and values and how these link to higher education and career options

• Students begin to map out their path from school to higher education to career

• Students develop a sense of belonging at university and at UNSW

• Students identify as a future university student

• Students feel more confident about their transition to university THE KEY Academic Attainment

• Students deepen their understanding of HSC subject specific content

• Students develop their understanding of how to perform well in HSC exams

Students develop strategies to manage the challenges of the HSC (stress, motivation, study)

THE MAP Possibilities • Students feel prepared to make

decisions about their post-school future

• Students write strong ECO applications

THE COMPASS Navigational Capacity • Students have increased

motivation and confidence to continue their journey to higher education and graduate employment

• Students can tell the story of their own learner identity and how it links to their education journey

• Students choose post-school options that suit their learning strengths, interests, values and career goals

THE KEY Academic Attainment • Students have increased

attainment at school • Students have increased

academic capacity and preparedness for higher education

6

Yr 12 Gateway Program

Inputs • InspirED

• University Ambassadors

• Engagement officers

Activities • Develop

content

• Train UAs Develop online platform

Kick Winter 1 workshop for 50+ students in each partner or highly engaged school • 70% registration

rate Winter Program 2000 registrations • 20 registrations

from partner and highly engaged schools (tiers??)

• 8 registrations per schools for other schools with progression to uni 40% or higher

• 30 registrations per NGO partner

1200 participants

Kick Winter • Face-to-face workshops in

schools: affirming motivationg passions, learning strengths and profiling Gateway Admissions Pathway and Program (1 hr)

Early Conditional Offer • Gateway Admission Pathway

info session • Go Write Ahead: Writing your

personal statement workshop

HSC Content • Inspired Masteclasses:

English Sd, Adv, Ext 1, Maths Std, Adv, Ext 1, Chemistry, Biology (1 hr sessions)

• Online Practice exams (marked): English Sd, Adv, Ext 1, Maths Std, Adv, Ext 1, Chemistry, Biology (1 hr sessions)

Faculty Sessions • Series of faculty welcome

info sessions (45 mins) University Ambassadors

• Study Share: Managing challenges of the HSC (1 hr)

• Burning Questions (1 hr) Follow up Academic support

• Access to HSC4ME app ( all subjects for one term)

• Supported 6- week online tutoring packages ina single subject (limited)

THE MAP Possibilities

• Students understand the opportunities and benefits of higher education

• Students develop an understanding of what studying at university is like and the expectations of university

• Students develop their knowledge of available degrees

• Students develop their knowledge of the nature and focus of each UNSW faculty

• Students develop an understanding of how to apply for the ECO

• Students develop an understanding of what makes a good personal statement

• Students know how to access further support for their ECO application

THE COMPASS Navigational Capacity

• Students develop an understanding of their learning strengths, interests and values and how these link to higher education and career options

• Students begin to map out their path from school to higher education to career

• Students develop a sense of belonging at university and at UNSW

• Students identify as a future university student

• Students feel more confident about their transition to university THE KEY Academic Attainment

• Students deepen their understanding of HSC subject specific content

• Students develop their understanding of how to perform well in HSC exams

• Students develop strategies to manage the challenges of the HSC (stress, motivation, study)

THE MAP Possibilities • Students feel prepared to make

decisions about their post-school future

• Students write strong ECO applications

THE COMPASS Navigational Capacity • Students have increased

motivation and confidence to continue their journey to higher education and graduate employment

• Students can tell the story of their own learner identity and how it links to their education journey

• Students choose post-school options that suit their learning strengths, interests, values and career goals

THE KEY Academic Attainment • Students have increased

attainment at school • Students have increased

academic capacity and preparedness for higher education

7

Inputs and

activities Participants Outputs Short term outcomes Medium term outcomes

UNSW Gateway Pathway

Inputs • Access &

Equity Staff

• UAC Activities • Conduct post

implementation review of pilot and refine processes

• Streamline by moving to SRS UAC portal

• 1500 applicants

• 840 from the Gateway Winer Program

(70% conversions rate from GWP)

• 1275 early conditional offers

• (85% conversion rate from applicant to offer, up from 80% in 2020)

• 956 students meet the ATAR requirements of their ECO course (75% conversion from ECO to firm offer, up from 68% in 2020)

• 408 enrolments (35% conversion rate from ECO to enrolment, up from 27% in 2020)

THE KEY Academic Attainment Students have retention and success rates similar to whole UNSW community

8

Methods Both quantitative and qualitative data will be used to assess whether the intended outcomes of the program have been met. The analysis methods for these data are outlined below. All indicators (quantitative and qualitative) are mapped against each of the outcomes in Indicator Bank table below. Quantitative Data: Post-test/Retrospective Pre-test For several of the outcomes listed below, we will use a quantitative post-test/retrospective pre-test survey design, sent to participating students at the conclusion of the program. It is important to note that although this method can be used to observe whether there are any changes across the time period of the program, it cannot be used to infer that those changes were a result of the program. In other words, because there is no counterfactual in this design, it is impossible to say whether those changes would have occurred regardless of the program being in place. We have chosen a post-test/retrospective pre-test design over a traditional pre-test/post-test design for several reasons. First, when time for a program is limited, pre-test/post-test questionnaires consume time that would be better spent on the program delivery (Marshak, deSilva & Silberstein, 1998). Further, changes in knowledge and attitudes over the program time period can be masked if participants over-estimate their knowledge and skills at pre-test. Pre-test overestimation most often occurs when participants lack a clear understanding of the constructs (attitudes, behaviours, skills) that the program is aiming to affect. Indeed, it is participants’ inexperience with these constructs that necessitates the program in the first place. Often, taking part in the program can show participants that they actually know much less than they reported they did in the pre-test (Pratt et al., 2000). In such cases, pre-test/post-test comparisons are misleading because the participants’ frame of reference is different at the two testing points, known as a response shift bias (Howard et al., 1979). To avoid response shift bias and obtain a more valid comparison of knowledge and attitude change, researchers have suggested collecting both contemporary and retrospective information at the conclusion of the program (Goedhart & Hoogstraten, 1992). Qualitative Data Some outcomes of the program will be assessed using qualitative data, and some using a combination of the above-described quantitative analyses and qualitative data. Qualitative questions will be included in post-event surveys and a focus group will be run at the beginning of each year for students who have come to UNSW through the Gateway pathway and who participated in the Program in high school. Thomas (2020) argues that changes in attitudes and behaviours aimed for in widening participation [WP] programs can often be evidenced most effectively thought qualitative methods such as individual or group interviews with participants. In the WP literature, the value of incorporating richer qualitative data into evaluation for a more complete understanding of the complex impacts of WP interventions on students’ decisions about university has been recognised (Holland et al., 2017; Raven, 2015). Despite the value, the incorporation of qualitative data into evaluation is currently rare (Harrison & Waller, 2017). Aside from assessing whether some of our intended outcomes have been met, the collection of qualitative data also allows evidence of unintended outcomes of the program to emerge. As such, the evaluation question will not simply be “did we achieve our predetermined outcomes?” but “what were all of the outcomes of this program?”. This approach will give us a more holistic understanding of the effects of our program, and we hope it will reveal opportunities that we otherwise might miss.

9

Gateway Program Indicator Bank

THE MAP: Possibilities

Outcomes

Quantitative Indicators

(Pre/Retrospective Post) Qualitative Indicators

Year 10 Students develop their knowledge of available degrees

I had/have a good understanding of what degrees are available to study at university

Did you learn about what degrees are available to study at university? Were you introduced to any degrees you didn’t know about before?

Year 11-12

Students develop their knowledge of the nature and focus of each UNSW faculty

I had/have a good understanding of what each UNSW Faculty has to offer

As a result of the program, what do you now know about the faculties at UNSW?

Students develop an understanding of how to apply for the ECO

I had/have a good understanding of how to apply for the UNSW early conditional offer

Did you learn anything about the early conditional offer? Students develop an understanding of what makes a good personal statement

I had/have a good understanding of what makes a good UNSW early conditional offer application

Students know how to access further support for their ECO application

I knew/know where to go to access further support for my early conditional offer application

Year 10-12

Students understand the opportunities and benefits of higher education

I had/have a good understanding of what opportunities are available at university Did you learn anything about what university has to offer?

Students develop an understanding of what studying at university is like and the expectations of university

I had/have a good understanding of what studying at university is like I had/have a good understanding of the expectations of university study

Did you learn anything about what studying at university is like?

All All What information do you have now that you did not have before the program?

10

THE COMPASS: Navigational Capacity

Outcomes Quantitative Indicators Qualitative Indicators

Year 10 Students develop their ability to write about and present themselves persuasively and with confidence

I felt/feel confident in presenting myself to others How do you feel about the idea of talking about yourself to others?

Year 11-12 Students feel more confident about their transition to university I felt/feel confident about my transition to university

How do you feel about transitioning from high school to university? In what ways, if any, did the program affect how you feel about it?

Year 10-12

Students develop an understanding of their learning strengths, interests and values and how these link to higher education and career options

I had/have a good understanding of my learning strengths and weaknesses I had/have a good understanding of what degrees might suit me

In what ways did the program impact your understanding about yourself?

Students begin to map out their pathway from high school to higher education and career What would you like to do after school? What about after

university?

Students develop a sense of belonging at university and UNSW I could/can see myself studying at UNSW How do you feel when you think about being at

university? What about being at UNSW? Students identify as a future university student I could/can see myself studying at university

11

THE KEY: Academic Attainment

Outcomes Quantitative Indicators Qualitative Indicators

Year 10 Students develop their motivation for HSC success through an understanding of the value and purpose of higher education

I felt/feel motivated to achieve my academic potential in the HSC

How motivated do you feel motivated to succeed in your HSC? In what ways has the program changed how motivated you feel?

Year 11-12

Students deepen their understanding of HSC subject specific content

I felt/feel confident in my understanding of the HSC content for ***

Has the *** Program supported your preparation for the HSC? If so, how?

Students develop their understanding of how to perform well in HSC exams

I have/had a good understand how to do well in the HSC

Students develop strategies to manage the challenges of the HSC (stress, motivation, study)

The *** Program has provided me with strategies to manage stress that will work for me The *** Program has provided me with strategies to maintain my motivation throughout the HSC The *** Program has provided me with study strategies that will work for me

12

2. Scholarships (Access)

Theory of Change

The Compass: Navigational Capacity

If we provide students with financial assistance, then they will feel they belong at UNSW and that their potential to contribute to the UNSW community is being recognised

The Key: Academic Attainment

If we reduce financial barriers to study, then students will be better able to access, participate and succeed at UNSW.

Assumptions

• That recipients of scholarships do face financial barriers to participation • We recognise that financial barriers represent only one type of barrier that students

face and that a scholarship can not ameloriate other barriers.

Methods

At the end of each year, students who received HEPPP funded scholarships will be surveyed to determine whether intended outcomes have been met (questions in indicator bank on page 14). In the survey, students will also be asked a qualitative question asking them to talk about what the scholarship has meant to them, to determine whether there have been any unanticipated outcomes.

13

Program Logic

Inputs and

activities Outputs Short term outcomes Medium term outcomes

Gateway Scholars

Inputs • HEPPP budget • UNSW

Scholarships Staff

• UAC Equity Scholarship and Educational Access Scheme applications

Activities • Equity

scholarship allocation meetings with selection panel (includes UNSW Scholarships, Access & Equity and Nura Gili staff)

UNSW Accommodation Scholarship

• 10 new scholarships per year • Value: Up to $25,000 pa towards residential

fees for the standard duration of the program

UNSW Access Scholarship

• 20 new scholarships per year • Value: $10,000 pa stipend for the standard

duration of the program

UNSW President and Vice-Chancellors Equity Scholarship

• 30 new scholarships per year • Value: $5,000 pa stipend for the standard

duration of the program

UNSW Pre-Entry and Enabling Programs Grant

• 10 new scholarships per year • Value: $5,000 pa stipend for one year • Must be completing an approved enabling or

pre-entry program at UNSW

UNSW Equity Award

• 90 new scholarships per year • Value: $5,000 pa stipend for one year

The Compass

• Students feel welcomed and supported by the UNSW community

• Students feel that their potential is being recognised by UNSW

The Key • Students access UNSW who may

otherwise not have had the financial means to

• Students’ financial stress is reduced, allowing them to better reach their academic potential at UNSW

The Key • Students have lower attrition

rates

14

Indicator Bank

Outcome Quantitative Indicator Qualitative Indicator

All

Please tell us below what receiving a scholarship meant to you personally i.e. how has it made you feel? what has it allowed you to you?

Students feel welcomed and supported by the UNSW community

Receiving the scholarship made me feel welcomed by the UNSW community

Students feel that their potential is being recognised by UNSW

Receiving the scholarship made me feel that my academic potential has been recognised by UNSW

Students access UNSW who may otherwise not have had the financial means to Without the scholarship, I may not have come to UNSW

Students’ financial stress is reduced, allowing them to better reach their academic potential at UNSW

I feel less financial stress as a result of receiving the scholarship I feel better able to reach my academic potential as a result of receiving the scholarship

Students have lower attrition rates Attrition rates

15

3. Gateway Scholars Program (Participation)

Theory of Change

The Gateway Scholars Program, piloted in T1 2021, is designed to provide holistic support to first year students entering UNSW via the Gateway Early Conditional Offer Pathway. This program provides wrap-around support and care for students across the social, academic, personal and professional areas of their lives. The program draws upon leading transition pedagogy (Gale & Parker, 2011, 2014, Kift, 2015, Meuleman et al, 2015) and is particularly invested in adaptive, relevant and capacity-building approaches to student transition into higher education. Underpinned by strengths-based, intersectional and social model principles of support and success this program builds upon students’ existing strengths and capabilities while also offering support in areas of heightened need. The theory of change for how the intended impact will be delivered is highlighted below:

The Map: Knowledge

Students, upon admission to UNSW, are likely to have general knowledge about university, and UNSW, from their involvement in the Gateway Programs to date. However, ensuring that this knowledge is then enhanced and tailored to be specific, accessible and memorable is critical to ensuring ongoing success and engagement. The Gateway Scholars Program aims, therefore, to provide students with specific knowledge around UNSW’s systems and processes, support services and available resources. The Program imparts this knowledge by providing tailored communication, resources, academic and peer support and building students’ awareness of, and exposure to, UNSW’s structure. If we equip students with knowledge relevant to their UNSW experience and their personal, professional and academic goals,then students will be set up to meaningfully engage with the UNSW community.

The Compass: Navigational Capacity

Navigating higher education institutions, with their complex structures and bureaucracy, may pose particular challenges for students from equity backgrounds. This is especially the case in institutions who have not typically had higher numbers of equity students, and for students who have none, or few, individuals who have attended higher education in their personal networks. Ensuring that students not only understand the support services which are available to them but also build trust, confidence and agency in accessing these services is critical to their ongoing relationship with UNSW, beyond the support of the Gateway Scholars Program. In addition to formalised support, providing opportunities for students to navigate the social spheres at UNSW, and build meaningful connections with their peers, is an essential component of building students’ sense of self-efficacy and belonging at UNSW. If we provide students with safe and kind exposure to support services and peer groups early in their UNSW experience, then students will feel confident immersing themselves in these environments and build on own self-efficacy and sense of belonging.

The Key: Success

Students from equity backgrounds may have internalised the deficit discourse which often surrounds their engagement with higher education, skills and future possibilities. Ensuring that students have the resources and support, through peer support and mentoring, academic support, workshops and community building, to thrive at UNSW is an important part of challenging this discourse. Academic progression and achievement are one marker of success at UNSW. Directing students to targeted academic support including live, online and asynchronous resources will support their academic

16

attainment. Students are also given the opportunity to explore their career choices, learn to articulate their skills, build meaningful relationships and build their sense of confidence and belonging as tertiary students. If we provide students with explicit academic, social, personal and professional support, then they will be better able to achieve success at UNSW. Methods Both quantitative and qualitative data, longitudinal and short-term are used to assess outcomes listed in the program logic below. This style of assessment is considered best practice for programs of this nature as it allows for the capturing of students’ experiences specifically within the program and their overall sense of belonging, preparedness and confidence in higher education over time. As such two principal methods of data collection are used, as outlined below:

Longitudinal Quantitative Questionnaire

Students who signed up for the Gateway Scholars Program are invited to participate in a longitudinal study which seeks to capture how students’ attitudes, experiences and sense of university preparedness and belonging may shift over time. The questionnaire uses Likert style questions to query students’ sentiments across a range of areas considered key to student transition in higher education. This evaluation does not ask questions specific to students’ experience in the Gateway Scholars Program nor any evaluation of the program itself. The main goal of this data collection is to understand how student sentiment may shift over time and to highlight any patterns in students’ sense of belonging, confidence and preparedness for higher education over the course of a year. It must be noted that, as is the case with other data collection for the Gateway Program, that while this method allows for observation of changes over time, we cannot use it to infer that those changes were from the Gateway Scholars Program. Regardless, this method provides an important snapshot in time of these students’ sentiments and enables us to better understand their feelings at different stages of their first-year journey to inform program support and approaches taken in later years.

Three program evaluations will be sent out at set points during the 2021 academic year via Qualtrics (Week One, T1, the end of T2 and the end of T3) to capture any shifts. The questionnaire will be anonymous but linked via a unique code and will be analysed in Qualtrics. A final report will be provided at the close of the academic year based on the results of this longitudinal study with any recommendations.

Program-Specific Evaluation

In addition to capturing longitudinal data on students’ sense of belonging and their sentiments towards tertiary study overall (see above) the Gateway Scholars Program also conducts an evaluation on the program itself. As the Gateway Scholars Program is a pilot program for 2021, it is important to receive feedback from students who have participated in the program at the end of each term. This reflects the student-led model of the Peer Connections team and contemporary best practice in equity student support in higher education.

Engagement in this program, especially across mentoring and academic support, will be tracked. This serves as a helpful gauge of students’ capacity and interest in engaging with the program and will inform how the program runs following on from the T1 pilot. It is important to note that there are numerous reasons that students may not engage with program elements including competing demands, isolation or withdrawal, prioritisation, interest and sense of capability and belonging. While all students now face increasing demands on their time, equity students may find it difficult to allocate

17

time to programs like Gateway Scholars given the additional demands of commuting, work and family responsibilities.

As such, towards the end of T1, students who signed up to the Gateway Scholars Program will be invited to complete a program evaluation about their experiences with the program. This questionnaire will specifically ask about the social, mentoring and academic components of the program and student feedback and suggestions for the program moving forwards. Students will also be invited to attend in-person or online focus groups which will allow for rich qualitative data to be collated. This approach reflects best practice as it provides more nuanced and interactive approach to capturing feedback and students can see their feedback embedded which builds confidence and trust in the processes of giving feedback. The results from both the questionnaire and the focus groups will be used to refine program offerings for future terms.

18

Program Logic Inputs and

Activities Participants Outputs Short term outcomes Medium term outcomes

Gateway Scholars

Inputs

* Peer Connections Staff

* University Ambassadors

Activities * Design and

launch of Gateway Scholars Program

Target: 60% of students who came through Gateway ECO sign up for Gateway Scholars Program

* Gateway Open Day, Info Day and O-Week events

* Welcome mail and onboarding pack

* Gateway-specific Live Chat and enrolment assistance

* Social Events * Twice a term check in calls by

student ambassadors to provide guidance and support

* Weekly peer mentoring group led by a PhD student within their faculty T1

* PASS style academic support provided for faculty-identified courses/areas of concern T1

* Supported places in UNSW or the TAFE Essential Maths Bridging Courses for students who are from a low-SES background

THE MAP: Knowledge * Students have an increased understanding of how to navigate

the acceptance, enrolment and new term preparation process * Students have an increased understanding of support

services and systems at UNSW and understand how to access them

* Students can connect with information that is specific, accessible and applicable to their journey at UNSW as they move beyond generalised knowledge of tertiary education

* Students understand the purpose of Gateway Scholars and what support is available to them

THE COMPASS: Navigational Capacity * Students develop a sense of belonging in the UNSW

community and make friends * Students feel more comfortable to seek support when they

need it * Students feel increased confidence to participate in university

lectures and tutorials * Students feel more confident to manage the challenges of

university study (e.g time management, stress etc)

THE KEY: Success * Students feel supported to reach their academic

potential at university * Students feel more confident to meet the expectations

of university study * Students can identify their skills and opportunities for

development * Students can identify future career aspiration or goals

and how their path of study and opportunities at UNSW will progress them towards these goals

*

THE MAP: KNOWLEDGE Students can outline the support services that are available to them and how to access them THE COMPASS: Students have successfully navigated to a support service and accessed it and/or participated meaningfully in lectures/tutorials and/or established friendships at UNSW THE KEY: Success Students have retention and success rates similar to whole UNSW community Students continue to feel a sense of belonging to UNSW Students feel confident continuing to explore career options and skills and practice these skills within the Gateway Scholars Community

19

Indicator Bank

Outcomes Quantitative Indicators (Asked 3 times over first year of engagement)

Qualitative Indicators (Asked in different terms over first year of engagement in focus

groups and qual survey questions)

The Map: Knowledge

Students have an increased understanding of how to navigate the acceptance, enrolment and new term preparation process

“I feel confident that I can successfully enrol and access the information I need to start a new term at UNSW”

“How did you find the enrolment process?” (T1) “Was the support from Gateway Scholars e.g. Live Chat, welcome packs helpful in navigating the enrolment process? If so, how so?” (T1)

Students have an increased understanding of support services and systems at UNSW and understand how to access them

“I understand where I can go for help if I need it at UNSW”

How would you describe your understanding of what support is available to you at UNSW? (T1,T3) Do you feel as though the Gateway Scholars Program has helped you understand what support is available at UNSW and how to access it? (T1)

Students can connect with information that is specific, accessible and applicable to their journey at UNSW as they move beyond generalised knowledge of tertiary education

“I understand where I can go for help if I need it at UNSW”

How would you describe your understanding of university study and UNSW now compared to when you first started? (T1, T3)

Students understand the purpose of Gateway Scholars and what support is available to them

"Why did you sign up for the Gateway Scholars Program?” (tick box options) “What were some barriers to participating in the program?” (tick box options)

What made you want to sign up to the Gateway Scholars Program? (T1) Did you understand what the Gateway Scholars Program was? (T1) What, if any, were some barriers to participating in the program? (T1)

The Compass: Navigational Capacity

Students develop a sense of belonging in the UNSW community and make friends

“I am sure that my degree was the right choice for me” “I feel confident that I will make friends at UNSW” (T1)/ “I feel confident that I have made friends at UNSW” (T2, T3)

“Do you feel as if you’ve made some friends at UNSW?” (T1, T3) “Have you made any friends due to the Gateway Scholars Program?” (T1) “Could you please provide feedback about your mentoring experience in the Gateway Scholars Program?” (T1)

20

“I believe that I will easily feel like part of the UNSW community” (T1)/ “I feel like part of the UNSW community” (T2, T3) “I have made at least one friend from the Gateway Scholars Program”

Students feel more comfortable to seek support when they need it

“I understand where I can go for help if I need it at UNSW” “I feel comfortable asking for help at UNSW” “I believe I could, if needed, comfortable explain to UNSW staff why a personal circumstance might affect my study at UNSW”

“How would you feel asking for help at UNSW?” (T1T3)

Students feel increased confidence to participate in university lectures and tutorials

“I believe that I can confidently answer a question or volunteer opinions in class”

“Could you tell me about how you’re finding your lecturers and tutorials?” “Has the Gateway Scholars program impacted how you feel in your classes? If so, how so? ? (T1)

Students feel more confident to manage the challenges of university study (e.g time management, stress etc)

“I am worried about how I will manage emotionally at UNSW” “I am worried about how I will manage financially at UNSW” “I am worried about how I will manage my time at UNSW”

“How would you describe your time management at UNSW?” (T3) “How would you describe your wellbeing at UNSW?” (T3)

The Key: Success

Students feel supported to reach their academic potential at university

“I believe I will receive adequate support from UNSW to successfully complete my studies” “I feel confident that I am adequately prepared for university study”

“How are you feeling about your academic results and or progress at UNSW?” (T1T3) “Could you please provide feedback about things you enjoyed or suggestions for improvement for the academic support groups” (T1) Has the Gateway Scholars Program impacted how you feel academically at UNSW? (T1)

Students feel more confident to meet the expectations of university study

“I feel confident that I will get good academic results at UNSW”

“How do you feel about your ability to succeed academically at university now compared to before you started at UNSW?” (T3)

21

Students can identify their skills and opportunities for development

“I feel confident that I can identify key skills and opportunities for development for myself”

“Could you tell me about some skills you think you’re good at?” ( T3) “Could you tell me about some areas you’d like to develop?” (T3) “Has the Gateway Program helped you discover or refine these at all? If so how so?” (T3)

Students can identify future career aspiration or goals and how their path of study and opportunities at UNSW will progress them towards these goals

“I understand what kinds of careers I may be able to pursue with my degree and other skills and/or experiences”

“Can you tell me a bit about some career goals you might have at this stage?” (T3) “How might your time at UNSW help you achieve this?” (T3) “Has the Gateway Scholars Program impacted how you understand your future career opportunities?” ( T3)

22

4. Attainment: Academic Tutoring for students from Low-SES backgrounds in UPP (Success)

Theory of Change

The Compass

If we provide small tutoring groups to pathway students, then they will feel better supported and more confident in their enabling programs at UNSW.

The Key

If we provide tailored academic support, then students will be better equipped in participating in their enabling programs, leading to increased academic success and eligibility and progression into UNSW programs.

Methods

Students will be surveyed in order to gauge their confidence levels at the beginning and end of the tutorials. At the end of the tutorials students will also be asked to reflect on how they think the tutorials have impacted on their performance. Performance date (grades and academic progression) of the target group will also be compared against groups that did not receive tutorial assistance.

23

Program Logic Inputs and

Activities Participants Outputs Short term outcomes Medium term outcomes

Academic Tutoring for UPPLow-SES Students

Inputs • Academic

Skills staff • Casual

tutors

Activities • Delivery of

small tutorials

Target Enabling Pathway students (UPP and UNSW Prep) who have demonstrated at admissions LSES status, Indigenous background and/or are from Regional and Remote areas.

Weekly small tutorials for students, related to study skills and subject matter of courses

Students feel supported to reach their academic potential

Students feel increased confidence in course content Students feel increased confidence to participate in lectures / tutorials

Students develop a connection with others in the small tutorial group setting

Students have improved academic success rates Students feel more confident they are able meet the expectations of a university degree Students feel more connected to the university community Students can identify their strengths and where they may need assistance

24

Indicator Bank

Outcomes Quantitative Indicators Qualitative Indicators

Students feel supported to reach their academic potential

End of term: “The tutorials have made me feel supported to be able to do my best in my studies.” (Likert scale from Strongly disagree to strongly agree)

Students feel increased confidence in course content

Start of term: “I feel confident about how I will perform in my course/courses.” (Likert scale from Strongly disagree to strongly agree) End of term: ““I feel confident about how I am performing in my course/courses.” (Likert scale from Strongly disagree to strongly agree)

End of term: “Can you describe how the tutorials affected your confidence studying in individual courses and in the UPP/UNSW Prep more broadly?”

Students feel increased confidence to participate in lectures / tutorials

Start of term: “I feel confident about asking Qs and talking in class.” (Likert scale from Strongly disagree to strongly agree) End of term: “I feel confident about asking Qs and talking in class.” (Likert scale from Strongly disagree to strongly agree) Relevant assessment grades (eg. Presentation, group work) and tutorial participation marks (where appropriate)

End of term: “Can you describe how the tutorials affected your confidence asking questions and talking in class?”

Students develop a connection with others in the small tutorial group setting

“I made a connection with another student in my tutorial.” (Yes/No)

Students have improved academic success rates

End of term: “I think the tutorials helped me perform better in my course/s.” (Likert scale from Strongly disagree to strongly agree) Comparisons of data between academic performance of program recipients and academic performance of

• LSES, ATSI and RR students in previous study periods, when tutoring of this kind was not available, and

• Other T2 and T3 2021 enrolled students who do not have access to this tutoring.

Students feel more connected to the university community

Start of term: “I feel connected to UNSW.” (Likert scale from Strongly disagree to strongly agree) End of term: “I feel connected to UNSW.” (Likert scale from Strongly disagree to strongly agree) “The tutorials helped me feel connected to UNSW.” (Likert scale from Strongly disagree to strongly agree)

For “The tutorials helped me feel connected to UNSW.” – “Could you please explain your rating?”

Students feel more confident they are able meet the expectations of a university degree

Comparisons of data between progression rates of program recipients into UNSW degree programs and progression rates of

25

• LSES, ATSI and RR students in previous study periods, when tutoring of this kind was not available, and

• Other T2 and T3 2021 enrolled students who do not have access to this tutoring.

End of term: “Do you think the tutorials helped you gain more confidence about doing a university degree program?” (Likert scale from Strongly disagree to strongly agree)

Students can identify their strengths and where they may need assistance

End of term: “Do you think the tutorials helped you identify your strengths and where you can improve?” (Likert scale from Strongly disagree to strongly agree)

"Please explain your answers”