Every thing you want to know about surveys… And aren't afraid to ask! Carl Berger January 30, 2001...
-
date post
19-Dec-2015 -
Category
Documents
-
view
215 -
download
1
Transcript of Every thing you want to know about surveys… And aren't afraid to ask! Carl Berger January 30, 2001...
Every thing you want to know about surveys…
And aren't afraid to ask! Carl Berger
January 30, 2001
All sources are available, look on the last slide.
But it’s really not about surveys…
It’s about helping Presidents, Provosts, and CIO's to make decisions!
But… you've got to know the territory!
(Gratuitous comments and/or insights…on the bottom of some slides)
Why?
Institutional ReadinessBasis of strategic plan for transformationFaculty interviews and MERLOTNot e-mail survey
Paul Hagner, Interesting Practices and Best Systems in Faculty Engagement and Support
NLII White Paper January 25, 2001
Paul's paper is must reading!
NLII Readiness
Readiness and Bridges Task Forces Entrepreneurial Faculty and “Second-Wave” Faculty Differences in
EngagementThese two groups of faculty, while united in their commitment to quality learning
environments, are very different in both their technical capabilities and their attitudinal readiness to embrace these new technologies. It would be a serious mistake for administrators to make allocation decisions based solely on the characteristics of the “entrepreneurs,” since their needs and their motivations can differ greatly from the “second-wave” faculty.
… an “enabling environment” is a precondition to institutional change.
universal student access,reliable networks,multiple opportunities for training and consulting, and“a faculty ethos which values experimentation and toleration of falters.”
But how do you know?
Guess Traditional Sources Conventional Wisdom Anecdotes Or…
You could ask them!
Even more important…
Helping decision makers Those who have little time to
develop deep knowledge Present in ways that decisions seem
to pop up Or don’t!
Lots of time its realizing that this data snipit can't help a decision!
Topics Asking the right question Asking the question right! Getting a great response (What is a
great response?) Decision Graphics Hidden meanings An example and some surprising?
results
Advantages to asking (survey)
Real data, almost always dumps myths
Understand all groups, early adopters, early majority, late majority, luddites
Faculty, Students, Administrators…
The real pay-off is that they understand that they (faculty, students, administrators) are part of the process!
Disadvantages of asking
Expensive Time consuming Will they understand it when were
done? And worse yet… you may find out that which you
don’t want to know!
About two years ago we invested heavily in NetG. (an on-line training program) Good idea?… stay tuned…
Take the plunge, needs, wants and reality
Enough to give credence Representative Need to get luddites Keep cost down Understandable And… Getting the right questions but even more… Getting the right analysis and presentation!
We chose paper surveys and non-respondent follow-up to make sure weweren't favoring techie types!
On the shoulders of …
Flashlighthttp://www.tltgroup.org/programs/flashlight.html
UCLAhttp://www.uncwil.edu/oir/faculty_folder/ucla_survey_99/survey_results.htm
Michiganhttp://sitemaker.med.umich.edu/cberger/reports
Berkeley, Cal State System and Others on the way, Stanford, Penn State, Minnesota but best…
YOU!Beg, borrow or steal good items and question styles. But give credit!
The 12 Step Program to Success
1. Select audience 2. Categories 3. Initial development with faculty,
students 4. Test with small group 5. Contact 6. Follow-up 7. Data entry
8. Analysis9. Presentation
10. Distribution11. Feedback
12. They want next versionNumber 12 is the measure of success. (Along with using the results for decisions)
Asking the right questionDon't jump too quickly to a survey:The Ehrmann technique (Open focus groups and listen,
listen, listen)
A majority of three (If you hear the same from three folks, unsolicited then it's a survey item)
Now build a surveySmall sample trial (Try out some survey types)
Single page (A few 1-pagers with several groups)
Check, check and check again for interpretation and errors
Try for 1 big paper survey every 2 years to prevent "surveyed to death"
Steve Ehrmann of Flashlight has the best question development technique!Gary Gatien of UM develops excellent questions
Asking the question right
Don’t ask “Do you use…?” Ask “How often do you use…” Don’t ask “Do you use either a or b?” Ask “Check all that apply” Don’t ask “How often did you use it last
week…” Ask “When you used it the most how
often did you use…”
Make questions do multiple duty. "How often" also tells yes, no, + lots more
Underlying constructs
Think of an underlying scale Try to avoid yes or no Use common or natural intervals Not…
Often…Frequently…sometimes…once…never
But…1/day…1/week…1/month…1/term…1/yr
With a little creativity you can create a log scale from the last one!
Non respondents1. Make a non respondent list.2. Compare against a representative list. We used our LDAP.3. Look for over or under represented groups.4. Over sample them in relation to the population. (make
comparable)5. Then follow up as above. Send e-mail, campus mail, and finally
call.6. Don't use phone or personal calls to get promises of response.7. Either carry out the survey over the phone or go to their office.8. Stop when you have a 'good feeling.’9. Try long enough to get a significant group or collapse trying.10. Finally, include, successful non respondents to check out
some unusual claims.Well... not quite a 12 step program but that's about two weeks of work
in my evaluation course (without the interactive lab to give a feeling for the frustration of tracking down and analyzing results from nonrespondents).
Data entry Enter for analysis and presentation Use different codes for missing, not
applicable, or filled out incorrectly Look out for multiple missing data
codes. Try out a sample set analysis before
it is too late. Do a small amount (25%)of double
entry to get reliability measures.
Make all missing data codes 9999. Data entry persons like 9, 99, 999 But you then can't do a simple search and replace for missing data.
Data analysis
Look out if you average. Medians may be better.
Report stats sparingly. But look for variation. Look for out of range. (stats can do) Simple stats, and complex ones
(factor analysis, MDS, etc)
Outlying data can really move averages. Also don't be afraid of advanced stats. New computer programs can help you visualize complex data. Try StatView for some real eye openers.
Presentation
Minimize tables for comparison Use graphs.
http://www.uncwil.edu/oir/faculty_folder/ucla_survey_99/survey_results.htm
You could spend hours looking for conclusions…
Same data as a graph…(unmodified Microsoft graph…bleh…We'll fix this later.)
Just converting it to a chart isn't more helpful…
0
5
10
15
20
25
30
UNC
Men
UNC
Total
4yr
Women
daily
2-3/week
1/week
1-3/month
never
Chart junk…
False 3DTerrible background
<-- Lousy ColorsPoor Layout
Worse…No way to see real differences
These errors are caused by Accepting Excel defaults.
With a little work the results-->
are cleanand clear
Tufte would love this one!
Student Computer Ownership
26
3
36
6
27
27
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Recommended (89) Required (36)
Undergraduates (206) MBA (202) EMBA (121)
Decision graphics
Making data clear with graphics and..
Using graphics to help decision makers
Combines complex chart data Uses visual design theory Uses perception theory
Decision Graphics started in the 80's to help parents understandIndividualize Learning Programs for special education students.
From tables to a decision graphic (7 steps) Step one:… Start with a table of data Question 26 Use Tech
for…
What a table to try to figure out! But at least it is sorted by the total ofAlready use, would like very much, would like somewhat.
118323961768284757412961not at all
3720223230383745752796don't knowabout it
275270576845464540251717would likesomewhat
999064492932232327151310would likevery much
513362437123143already use
e-maillist
Coursewebpage
E-discuss
group
AudioVisual
MMPresentation
CompSimulation
Writingtutorials
TutorialsSelfpaced
Textanalysis
DistTeach
StreamingVideo
Question 26 Use Tech for…
Step 2… Convert table to an Excel Graph
I'll never know why MS builds starts with such terrible charts!
0
20
40
60
80
100
120
140
already use would like very much would like somewhat
don't know about it not at all
Step 3: Convert to a bar chart (set legend below the chart) Question 26 Use Tech
for…
0 50 100 150
Streaming Video
Dist Teach
Text analysis
Self paced
Tutorials
Writing tutorials
Comp Simulation
MM Presentation
AV
E-discuss
Course Web Page
E-mail list
already use would like very much would like somewhat don't know about it not at all
Step 4: Change to a stacked bar chart… Question 26 Use Tech
for…
Aha…now that sorting makes sense as you look along the dark blue bars.
0 50 100 150 200 250
Streaming Video
Dist Teach
Text analysis
Self paced
Tutorials
Writing tutorials
Comp Simulation
MM Presentation
AV
E-discuss
Course Web Page
E-mail list
already use would like very much would like somewhat don't know about it not at all
Step 5: Expand bars (100% wide and less space between)…
Question 26 Use Tech for…
0% 20% 40% 60% 80% 100%
Streaming Video
Dist Teach
Text analysis
Self paced
Tutorials
Writing tutorials
Comp Simulation
MM Presentation
AV
E-discuss
Course Web Page
E-mail list
already use would like very much would like somewhat don't know about it not at all
Step 6:Change colors to flow from cool to warm Question 26 Use Tech
for…
0% 20% 40% 60% 80% 100%
Streaming Video
Dist Teach
Text analysis
Self paced
Tutorials
Writing tutorials
Comp Simulation
MM Presentation
AV
E-discuss
Course Web Page
E-mail list
already use would like very much would like somewhat don't know about it not at all
Step 7: What the heck.. Shade those colors… Question 26 Use Tech
for…
Bottom Line… Our faculty want course web pages but no distance teaching!
0% 20% 40% 60% 80% 100%
Streaming Video
Dist Teach
Text analysis
Self paced
Tutorials
Writing tutorials
Comp Simulation
MM Presentation
AV
E-discuss
Course Web Page
E-mail list
already use would like very much would like somewhat don't know about it not at all
Remember the UNC data?(unmodified Microsoft graph)
0
5
10
15
20
25
30
UNC
Men
UNC
Total
4yr
Women
daily
2-3/week
1/week
1-3/month
never
With color metrics…
Conclusion… Not much difference!
0% 20% 40% 60% 80% 100%
UNC Men
UNC Women
4yr Men
4yr Women
daily 2-3/week 1/week 1-3/month never
Ranking versus top three…
Ranking works with few choices.Selecting top three will take care of
ranking 4 choicesSelecting top three is easier to takeResults easier to analyze and display
Easier to take
But the results are revealing…
What method do you like to use to learn technology?
And guess which University just spent big bucks for on-line computer classes?
0 100 200 300 400 500 600
On-line computer classes
Telephone consultants
Computer seminars
Univ computer classes
On the job training
Self taught from Books etc.
Self taught from exploring, experimenting
Help from friends, families, colleagues
Most preferred way of learning 2nd preferred way of learning 3rd preferred way of learning
A little factor analysis
Yep, we've got leading and second wave faculty plus the good old AV types
Oblique Solution Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6 Factor 7 Number
Web editor 0.70 0.01 0.05 0.02 0.00 0.05 0.07 234
Web text/graphics 0.67 0.04 0.02 0.06 0.12 0.10 0.01 296
Software tools for Web 0.56 0.06 0.06 0.08 0.01 0.00 0.13 180
Publishing 0.47 0.22 0.19 0.11 0.05 0.15 0.11 329
Web audio/video 0.42 0.02 0.03 0.06 0.00 0.30 0.10 135
Multimedia 0.31 0.16 0.11 0.12 0.10 0.03 0.21 190
Presentation 0.03 0.68 0.00 0.06 0.06 0.07 0.07 494
Statistics 0.17 0.66 0.05 0.12 0.04 0.09 0.06 306
Spreadsheets 0.04 0.64 0.05 0.10 0.10 0.06 0.07 558
Databases 0.19 0.40 0.10 0.12 0.03 0.06 0.19 327
Comp Proj 0.14 0.36 0.06 0.04 0.02 0.22 0.06 354
3-D modeling 0.03 0.01 0.75 0.01 0.00 0.03 0.02 91
Modeling 0.04 0.11 0.70 0.01 0.02 0.09 0.13 167
High performance computing 0.05 0.17 0.60 0.04 0.12 0.15 0.02 54
Computer anim/graphics 0.15 0.03 0.58 0.06 0.04 0.07 0.08 175
Graphics 0.22 0.37 0.40 0.10 0.03 0.10 0.10 363
Movies/filmstrips 0.04 0.00 0.05 0.78 0.00 0.06 0.00 176
Video tape/Audio tape 0.03 0.10 0.06 0.70 0.09 0.23 0.05 424
Overheads/transparencies 0.00 0.39 0.11 0.54 0.03 0.05 0.19 600
E-mail 0.02 0.14 0.04 0.10 0.82 0.02 0.03 728
Browser 0.12 0.09 0.03 0.17 0.69 0.03 0.00 708
Word processing 0.05 0.19 0.02 0.18 0.58 0.04 0.05 723
Video Conferencing 0.09 0.10 0.03 0.02 0.04 0.71 0.00 94
Web Conferencing 0.23 0.03 0.00 0.03 0.05 0.59 0.11 78
Digital Video 0.04 0.04 0.09 0.05 0.06 0.52 0.15 94
Project Management 0.00 0.05 0.03 0.02 0.03 0.09 0.64 119
Calendar 0.03 0.15 0.09 0.21 0.12 0.10 0.63 321
Courseware 0.23 0.14 0.19 0.30 0.03 0.26 0.44 78
The U of Michigan case study
The 1999 Faculty Survey Distributed in February 1999 1500 faculty, stratified random sample 19 Schools and Colleges 743 responses Results to CIO in August 1999 Released in to the public in March
2000
Survey Categories
Use Resources Support
Q29 Base Academic Unit
Had to use a log scale. Med School rules! (We had one school with more responses than faculty, what a story!)
1 10 100 1000
Medical School
Literature, Science & Arts
College of Engineering
none
Dental School
Library
School of Nursing
School of Business
School of Music
School of Art and Design
School of Natural Resources
School of Social Work
College of Pharmacy
Law School
School of Education
School of Information
School of Public Health
Architect & Urban Planning
Kinesiology
ISR
MERIT
School of Public Policy
Questions 29-35Demographics…
Q30Appointment
Q33Job classification
Q34Tenure
Regular600
Full professor176
Tenure289
Adjunct70
Associate professor127
Non tenure434
Temp50
Assistant professor161
Q31Full or Part
Instructor24
Q 35Gender
Full time640
Lecturer96
Male467
Part time86
Other140
Female280
Question 27Would you use the web for…
Very little use now but… pent up demand for next wave use.
Q10 How often do you use…
Surveyed just as our CourseTools was coming on line. Next survey???
0% 20% 40% 60% 80% 100%
Word processing
Browser
Spreadsheet
Presentation
Graphics
Publishing
Web editor
Database
Stats
Calendar
Web software
Multimedia
Modeling
Project Management
Courseware
every day 1-2 week several/month 1/month several/year 1/year never
Question 15 Concerns…
Bottom Line… It's time, reliability and support!
Adding students
<-Faculty1999 UM Faculty Survey
Students ->2000 UMStudent Survey
0 100 200 300 400 500 600
On-line computer classes
Telephone consultants
Computer seminars
Univ computer classes
On the job training
Self taught from Books etc.
Self taught from exploring, experimenting
Help from friends, families, colleagues
Most preferred way of learning 2nd preferred way of learning 3rd preferred way of learning
Not too different, second wave for both?
0 50 100 150 200 250
Telephone consultants
Online computer classes
Computer seminars
On the job training
UM computer classes
Self taught from books
Self taught from exploring
Help from friends
First choice Second choice Third choice
Credits
CIO and SACUACIO StaffITD StaffOIT StaffISR743 faculty members!
Special thanks to:Kati BauerSteven BurdickJose Marie GriffithsGary GatienKaren KostNicole KirgisKathleen McClatcheyEric Rabkin
Flashlight, part of the TLT group
Online database In depth rather than broad Excellent source Part of a broad program
http://www.tltgroup.org/programs/flashlight.html
The Full MontyThe Michigan 1999 Faculty Survey and the 2000 Student Survey Form
Original blank survey (.doc) [Faculty and Student]
and…Initial results (.pdf) [Faculty]
but wait…there’s more…This presentation (.ppt)
are available at www.carat.umich.eduFollow links to projects and scroll to faculty
survey…
Or…if you want a special question answered [email protected]
Thanks for coming. We can help decisions and improve learning and teaching!