CISEREUEvaluationToolkitAudrey Rorrer, PhD
University of North Carolina at CharlotteCollege of Computing & Informatics
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
Evaluation&AssessmentConsiderations
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
ToolkitResources
Evaluation101
Recruiting
StudentOutcomes
Etc.
IntentionalEvaluation• Whatisthevision?
• Projectgoals• Studentimpact• Resources&support• Milestones
• Whatevidenceisneeded?
NSFCISEREUPIMeeting,Orlando,FL,March2017
Eval 101:REUProgramOutcomes
• Applicants• AppliedvsAccept• Demographics• Limitedresearchaccess
• Participants• Attitudes,Behaviors,Cognition(skills,knowledge)
• Experiences,Activities• Longitudinaloutcomes
• ProgramOutcomes• Publications• Dissemination• Products
• Specialty• Mentoring?• Graduatestudentdevelopment?
• OtherContributions
NSFCISEREUPIMeeting,Orlando,FL,March2017
RecruitingResources:CommonApplication
NSFCISEREUPIMeeting,Orlando,FL,March2017
• 56itemscoveringcontactinformation,academicpreparation,graduateschoolplans,demographicinformation
• Customizable(theme,items,nextstepsinformation)• Immediateaccesstoyourapplicants• Managesstudentdatainaform
• Notoolforreferences,resumes,ortranscripts
WhatPISees
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
RecruitingResources:CommonApplicationWhatApplicant
Sees
1.AdvertiseonGoogleDocStudentsaredirectedtoyou2.SharedApplicantPoolYoucontactstudentsfromothersites
NSFCISEREUPIMeeting,Orlando,FL,March2017
RecruitingResources:ApplicantSharing
Fall:advertise
Winter:select
Spring:share
Results:CommonApplicationTrends
Site Descriptors2010
(N=13) 2011 (N=20)2012
(N=22)2013
(N= 26)2014
(N=25)2015
(N= 23)2016
(N= 27)
Range of # Applicants 29-152 4-176 18-212 20-299 29-232 15- 349 27- 286
Avg Applications per Site 77 79 93 112 102 122 108
Largest # of Sites Applied to by
Individual 30 (n=1) 6 (n=1) 7 (n=2) 18 (n=1) 14 (n=1) 16 (n=1) 9 (n=3)
% of Female Applicants 27% 26% 24% 28% 27% 26% 32%
% of URM Applicants 28% 34% 36% 31% 31% 35% 33%
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
StudentOutcomes:AlaCarteStudentSurvey
NSFCISEREUPIMeeting,Orlando,FL,March2017
Post-TestOnlyPreandPost-Test
Self-Efficacy
GraduateSchoolIntent
Attitudes
Grit
Help-Seeking
ProfessionalIdentity
Leadership
ResearchSkills
Mentoring
ProgramSatisfaction
StudentOutcomes:HowSurveyWorks
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
DeploymentOptions
ToolkitAdministered➕ Datacollectionandanalysisdone
foryou➕ Datapackage=yoursitedata(raw
andcoded),codebook,results➕ Anonymizeddata
PIAdministered• Sitecollectsandanalyzes• Identified
StudentOutcomes:DataPackage
• RawandCodeddata ofpreandpostresponses• MatchedPairsdata• CodeBook• Analysisofpre/postdata• Dataisde-identified
NSFCISEREUPIMeeting,Orlando,FL,March2017
Results:AlaCarte2016
Construct Pre (SD)N= 305
Post (SD)N= 268
Self-Efficacy* 3.67 (.61) 4.21 (.53)Intent to Grad School* 3.83 (.75) 3.7 (.88) Attitudes* 4.33 (.51) 4.09 (.74) Help-Seeking/Coping
4.27 (.61) 4.35 (.64)
Research Knowledge* 3.15 (.66) 3.92 (.59)
Grit 3.36 (.35) 3.39 (.40)
Scientific Identity* 3.42 (.91) 3.68 (1.0)
Leadership* 4.18 (.60) 4.30 (.65)
Mentorship Not applicable 4.38 (.84)
Note: 5 pt Likert scale
Significant Effects forTime(pre- andpost- surveys,*(p<.05)
Descriptives:• 43% Female • 35% multi-ethnic
minority groups
• Predominantly Rising Juniors and Seniors
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
Etc.:AdditionalConsiderations
NSFCISEREUPIMeeting,Orlando,FL,March2017
IRB LongitudinalTracking
OutcomesofInterest
CampusResources
•EvaluationToolkit:reu.uncc.edu/cise-reu-toolkit•HowTovideos•EvaluationmaterialsandresourcestailoredtoCISEREU
OnlineEvaluationToolkit
•GoogleFormapplicationUNIQUEtosite•Standardized&customizable•AggregatedescriptivestatisticsacrossCISEdirectorate
CommonApplication
•ManagedviaGoogleDrive&CommonApplications•SitePI“releases”unselectedcandidates•AllPIshaveaccesstoonlinefolder
SharedApplicantPool
•StudentOutcomes:modulated,valid/reliable•FacultyImpact:Surveydeployingsummer2016
Surveys:AlaCarteand Faculty
•Underconstruction•JointheworkinggroupAlumniTracking
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
reu.uncc.edu/cise-reu-toolkitThankyou
Appendices
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
FutureToolkit:AlumniTracking
WorkingGroup
March2016
Discussion&Ideas
DevelopPrototypes
Feedback &Launch
WorkingGroupMembers:• Lazaros Gallos,Rutgers• ClaireDuggan,NortheasternU.• BonnieSwan,U.CentralFlorida• StephenGilbert,IowaState• TiffanyReardon,Berkeley• Huirong Fu,Oakland• JamiePayton,UNCCharlotte• RajaKushalnagar,RIT
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
ApplicantCounts:SitesAppliedTo
=1006
=1105
=1934
=2905
=2445
=2827
=2335
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
ApplicantDemographicTrends:UnderrepresentedMinorityDistribution
28%
34%
36%
31%
31%
35%
33%
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
ApplicantDemographicTrends:GenderDistribution
27%
26%
24%
28%
27%
26%
32%
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
CommonApplications:GraduateSchoolPlans
• Majorityplantopursuegraduatedegrees• 79%in2013• 80%in2014• 84%in2015• 79%in2016
• Fewarefirstgenerationcollegestudents• Lessthan20%acrossall6years• 17%arefirstpersoninfamilytoattendcollege(2015)• 19%arefirstpersoninfamilytoattendcollege(2016)
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
2016CommonApplicants:Unique
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
2016CommonApplicants:Consent
NSFCISEREUPIMeeting,Arlington,VA,March2017
• Percentageofapplicantswhoanswered'Yes'tohavetheirapplicationconsideredbyadditionalCISEREUprograms
¨ Percentageofapplicantswhoanswered'yes'tooptoutofparticipatingintheresearchstudy
AlaCarteStudentSurveyModular, pre/post assessment of student outcomes
• Self Efficacy – I can formulate a research problem
• Intent to attend graduate school – I plan to apply to graduate school in a
computing discipline• Attitudes towards computing
– I like to use computer science to solve problems
• Help seeking/coping skills – When I do poorly on an exam,
typically I….skip class• GRIT
• I am a hard worker
• Research Skills – Formulate a research hypothesis– Write a research paper for
publication• Leadership in Science
- I know how to be good team member
- I know how to encourage team performance
• Professional Identity as Scientist- I feel like I belong in science
• Mentoring Satisfaction (post-only)• My mentor was helpful in providing
direction and guidance on research project issues
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
SurveyInstrumentsAvailable
• SURE- SurveyofUndergraduateResearchExperiences
• D.Lapatto atWashingtonUniversity
• URSSA- UndergraduateResearchStudentSelfAssessment
• UniversityofColoradoBoulder,NSF
• Botharefreeforresearchprogramsandcourses• AdaptationmadefortheAlaCarte
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
HistoryofEvaluationProject2009 Working Group Members
Guy Alain Amousou Chris Aberson Wendy Cooper Teresa Dahlberg Andy Fagg Stephen Gilbert Manfred Huber Niels Lobo Sanjay Madrias Joan Peckham Eric Wong Yu-Dong Yoa
Kevin Zeng
NeedsAssessment&Study
EstablishedCommonIndicators&Tools
ResearchModuleInclusiveTerminologySharedApplicantPool
NewModulesFacultyStudyAlumniTracker
InstructionalVideos
2009Humbolt State
2010UNCCharlotte
20132014-16Toolkit
ExpansionProject
NSFCISEREUNewPIMeeting,Arlington,VA,March2017
Top Related