Post on 24-Feb-2016
description
DRAFT:Opinion Space Pilot Study
Topic: Reputation of the American Automotive Industry
Participants: Automotive Industry Online Community
Research Period: Fall 2010 – Spring 2011
May 2011Rick Wagner, Joyce Salisbury, Ken Goldberg, David WongUC Berkeley Logo
Pilot Study Goals1) Sample Existing Perceptions of the
Image of the American Automotive Industry
2) Collect Ideas about How to Improve the Image of the American Automotive Industry and Identify the Most Insightful Ideas
3) Study how Opinion Space compares with Existing Survey Methods in Terms of Participant Engagement
4) Consider Demographic Variations among Participants
5) Consider Applications for gm.com
Executive Summary1) Sample Size (# Participants):
1198 2) Avg Number of Minutes Spent on Site:
363) Avg Number of Peer-to-Peer Evals: 904) % Who are “Extremely Likely”
to Participate Again: 95
5) Average Age of Least Active Participants: 44
6) Average Age of Most Active Participants: 50 (!)
7) Key Issues:8) 20 Most Insightful Ideas
Executive SummaryDiscussion Question: Improving Industry
Image1) Integrity: Honesty, Responsiveness to
Flaws2) Value: Quality, Price, Safety, Warranty3) Only 2 of top 20 mention “bailout”4) Listen more to consumers5) Don’t over-power vehicles6) Fewer “Gee-Whiz” features7) Become leaders in innovation8) Don’t charge too much for GPS9) LED Headlights10)Develop Autos that run on Natural Gas!
Executive Summary• Opinion Space was very effective in engaging AIOC participants
– Average total time spent on the system: 36 minutes.– Average number of ratings per user: 90 insight and agreement ratings.– Much excitement expressed to participate again (see figure).
• Top Insights– AIOC members suggested many things GMis already working on.
• Next steps– Host another Opinion Space on GM.com toengage the public in generating ideas.– Reengage the AIOC to better communicate what GM is working on and solicit ideas on how toimprove.
Q150. How willing would you be to participate in other versions of this
system (on related topics)?
Not likely
Extremely likely
Opinion Space
Our Approach
1. Visualization 2. Level Playing Field
3. Wisdom of the Crowds 4. Game Structure
"Opinion Space will harness the power of connection technologies to provide a unique forum for international dialogue. This is...an opportunity to extend our engagement beyond the halls of government directly to the people of the world.”
- U.S. Secretary of State Hillary Clinton
“The world spends over 110 billion minutes per month on social networks and blog sites.”
- NielsenWire, June 2010
“We’re moving from an Information Age to an Opinion Age.” - Warren Sack, UCSC
20 sec. per comment X 35,387 comments
= 8 days
goalsFor Organizations• Understand the diversity of their
community• Engage their community
– Solicit feedback and creative suggestions
– Rapidly identify patterns, insightful ideas
For Community Members• Understand relationships with other community members• Engage with a diversity of viewpoints
and ideas• Express ideas, and be heard hybrid vigor
opinion.berkeley.edu
Step 1: Enter your opinions and response
Step 2: Visualize your position
Step 3: Read and rate others users
System Design (Aug-Dec 2010)
Baseline Questions:
• When deciding on which major product to buy, I often consider where it was made in making my decision.
• Reducing oil consumption should be a higher priority in the U.S.
• I am willing to pay more for a product that comes from a socially responsible corporation.
• The U.S. government should not give loans to corporations.
• New technologies are causing more problems than they are solving.
Discussion Question:What is the Most Important Step or Steps that US
Automotive Manufacturers could take to Improve Their Image?
System Stats• Duration 20 Dec 2010 – 21 Jan 2011• Announcement, 2 Reminders
• 1198 participants (57% response rate)• 1,148 Responses • 96,000 Insight Ratings• 96,000 Agreement Ratings
Opinion Space
Users and sessions per day
Unique visits per
day
Active users per
day
Session duration
Zoomed out
Zoomed in
Ratings per day
Distribution of ratingsAll responses are rated on two scales: 1) How much do
you agree with the response?
2) How insightful is this response?
Ratings and views per comment
Insight ratings per user
Zoomed out
Zoomed in
Agreement ratings per user
Zoomed out
Zoomed in
Statement Statistics1) When deciding on which major
product to buy, I often consider where it was made in making my decision.
2) Reducing oil consumption should be a higher priority in the U.S.
3) I am willing to pay more for a product that comes from a socially responsible corporation.
4) The U.S. government should not give loans to corporations.
5) New technologies are causing more problems than they are solving.
Statement 1: When deciding on which major product to buy, I often consider where it was made in making my
decision.
Statement 2: Reducing oil consumption should be a higher priority in the U.S.
Statement 3: I am willing to pay more for a product that comes from a socially responsible corporation.
Statement 4: The U.S. government should not give loans to corporations.
Statement 5: New technologies are causing more problems than they are solving.
Statements in relation to position in Opinion Space
For each statement, a participant’s point will move in the direction and angle indicated by the arrow as they drag the slider.
Participant Responses• Participant responses were ranked using the
insightfulness ratings from other participants. • Two main algorithms were used to rank responses.
– Spatial ranking: ratings of a response are weighted by the spatial distance between the author of the response and the rater.
– Confidence interval ranking: responses are ranked by the lower
bound of a 95% confidence interval of the mean.
• See Appendix for a verbatim list of the top 10 responses
Topics and Insights from the Top 18 Participant Responses
Topic: US Automotive Corporations
- Achieve more integrity- Streamline production to reduce costs- Renegotiate union contracts to keep
prices low- Find out what the customer really
wants- Engage in product development with
the goal of long term use- Market value to consumers using
measurable and verifiable methods- Pay back government loans- Lessen environmental impact
Topic: Price
- Remove extraneous, meaningless features that increase the price and don’t add significant value.- GPS cited as an example*
- Keep prices low and competitive with foreign companies
- Affordability is the bottom line for most people
*participant1329’s response (rank #1): “When offering new technology, they must keep the price unchanged or slightly higher. Newer technology is often exponentially more than its value. For example, remember the GPS? Automakers were charging $1500-2500 for an installed GPS when you could buy one for $300. This makes the automaker look greedy…”
Topics and Insights from the Top 18 Participant Responses
Topic: What participants want in quality and value
- Durable, reliable, dependable, well-built, no manufacturing defects, built with stronger and lighter materials- Safety - Fuel-saving, fuel-efficient, and good
fuel-economy and mileage- Increased warranties that reflect
higher quality cars- Performance- Comfortable for larger families
Topic: Electric cars
- Stop building electric cars and focus on increasing gas mileage
- Make electric cars with higher performance that can go longer distances
- Make cars large enough to fit larger families*
*participant2053’s response (rank #5): “…Make vehicles that contain fuel efficient technologies in more then just the small car genre. We are a family of 5 and need to transport a wheelchair for our middle son. Most super economical cars wont carry us as a family. I like the idea of electric vehicles but would need to see them able to go greater distances before needing to be charged…”
Top terms from participant responses
Top co-occurring terms from participant responses
Post Survey Analysis
Q105. What was your overall opinion of the Opinion Space site?
Q120. How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the
Opinion Space activity?
Q130_1. I was motivated to consider responses from people far from my own response on the opinion map
Q130_2. The size of each point was a reasonable measure of how insightful the corresponding response was
Q135. How do you feel about using two sliders (agreement, insightfulness) to rate responses?
Q140_1. I would have liked to comment on other participants' responses
Q140_2. I would have liked to have been notified when my response was rated by other participants
Q150. How willing would you be to participate in other versions of this system (on related topics)?
Q110. What aspects of the Opinion Space interface did you like?
Main Ideas: - Ability to explore others’ ideas and see
how similar or different you are from others
- Learning a lot from reading the opinions of others; opened up from previous way of thinking
- The ability to rate others opinions- Allowed ongoing participation over
time so participants could come back- Sense of being heard - Interesting change of pace; fun, fast,
different, unique, and easy to use
- Enjoyed the combination of statements and free form answer box
- Enjoyed the anonymity- The UI is very visually pleasing,
uncluttered, and intuitive - The ability to see who has rated you- Topics were of high interest - The star metaphor with larger points
meaning more insightful idea- Engaging and made excited to rate
the opinions of others
“I really enjoyed being able to read and rate other participants' responses. I found that there were many with which I didn't necessarialy agree with but were well thought out and concisely argued. That gave me the chance to rate highly an opinion other than my own and also opened my eyes to other sides of "the argument" so to speak.”
Q110. What aspects of the Opinion Space interface did you like?
“I liked the way we could change screens to get to the ones we didn't rate yet, instead of it all being on one page in some kind of a continuous boring list…”
“I loved reading other members ideas and/or opinions - some were very interesting and I even learned something from some! Other ideas/opinions were unique and things that I never thought about.”
“…the idea of an incentive and/or prize (not to mention just the morale-boost of having others treat my own opinions with respect and interest) was itself an incentive to read as many entries as I could. It was a contest of sorts, but a most supportive and kind one, not the cutthroat, nasty kind that I try to steer clear of. I was so sorry when it ended!...”
Q115. What aspects of the Opinion Space interface could be improved (and how)?
Main Ideas: - Include a tutorial, more instructions,
and/or a demo video on how to use the system and interface
- See more details on each participant- Allow commenting on your response- Ability to see who has rated you and
get feedback on your ideas- Ability to create categories and
clusters- Want a dedicated page to visit as
opposed to finding link through email- Keep the system online longer- Add a spell checker- Include background info on
discussion topics- More real time information and
interaction with other participants- Want a way to view the higher level
trends in the system
- Use a mouse-over as opposed to clicking to view a participant’s response
- Organization of who you’ve rated and who rated you; a list here makes sense
- Have system give suggestions of whose ideas are similar/different than your own
- Add additional questions to make conversation more in-depth
- Have more reminder emails with notifications of the status of the system (e.g. new responses to be rated).
- Increase size of the stars- Increased features for navigation
(see all stars, zoom in and out).- Rewards for everyone who
participates- Larger viewing box for long
responses
Q145. What new features would you like to see in Opinion Space?
Subgroup Analysis of Participation
100 Least Active vs. 100 Most Active Reviewers
Age of 100 least active reviewers
Mean Median Stdev
43.48 40.5 13.29772913
100 Least Active vs. 100 Most Active Reviewers
Age
Age of 100 most active reviewers
Mean Median Stdev
49.76 51 11.17149945
100 Least Active vs. 100 Most Active Reviewers
Gender
Gender of 100 least active reviewers
Male Female Ratio (M:F)
49 51 0.96:1
Gender of 100 most active reviewers
Male Female Ratio
56 44 1.27:1
100 Least Active vs. 100 Most Active ReviewersEducation
Education of 100 least active reviewers Education of 100 most active reviewers
Median MaxCollege Graduate (4
years)College Graduate (4
years)
Median MaxCollege Graduate (4
years)College Graduate (4
years)
100 Least Active vs. 100 Most Active Reviewers
Income
Income of 100 least active reviewers
Median
$75,000 to $99,999
Income of 100 most active reviewers
Median
$75,000 to $99,999
100 Least Active vs. 100 Most Active Reviewers
Region
Region of 100 least active reviewers Region of 100 most active reviewers
100 Most Active vs. 50 Most Active Reviewers
Subgroup Analysis of Participation
100 Most Active vs. 50 Most Active ReviewersAge
Age of 100 most active reviewers
Mean Median Stdev
49.76 51 11.17149945
Age of 50 most active reviewers
Mean Median Stdev
50.88 53 10.69885975
100 Most Active vs. 50 Most Active ReviewersGender
Gender of 100 most active reviewers
Male Female Ratio
56 44 1.27:1
Gender of 50 most active reviewers
Male Female Ratio
30 20 1.5:1
100 Most Active vs. 50 Most Active ReviewersEducation
Education of 100 most active reviewers
Education of 50 most active reviewers
Median MaxCollege Graduate (4
years)College Graduate (4
years)
Median MaxCollege Graduate (4
years)College Graduate (4
years)
100 Most Active vs. 50 Most Active ReviewersIncome
Income of 100 most active reviewers
Median
$75,000 to $99,999
Income of 50 most active reviewers
Median
$75,000 to $99,999
100 Least Active vs. 100 Most Active Reviewers
Region
Region of 100 most active reviewers Region of 50 most active reviewers
50 Most Active Reviewers vs. 50 Top Authors
Subgroup Analysis of Participation
50 Most Active Reviewers vs. 50 Top AuthorsAge
Age of 50 most active reviewers
Mean Median Stdev
50.88 53 10.69885975
Age of 50 top authors
Mean Median Stdev
47.7 49 9.428149341
Gender of 50 most active reviewers
Male Female Ratio
30 20 1.5:1
50 Most Active Reviewers vs. 50 Top AuthorsGender
Gender of 50 top authors
Male Female Ratio
28 22 1.27:1
Education of 50 most active reviewers
50 Most Active Reviewers vs. 50 Top AuthorsEducation
Education of 50 top authors
Median MaxCollege Graduate (4
years)College Graduate (4
years)
Median MaxCollege Graduate (4
years) Postgraduate College
Income of 50 most active reviewers
Median
$75,000 to $99,999
50 Most Active Reviewers vs. 50 Top AuthorsIncome
Income of 50 top authors
Median
$75,000 to $99,999
Region of 50 most active reviewers
50 Most Active Reviewers vs. 50 Top AuthorsRegion
Region of 50 top authors
Subgroup Analysis of ParticipationPost Survey Results
*red text indicates statistically significant differences
100 Least Active Reviewers100 Most Active Reviewers50 Most Active Reviewers
50 Top Authors
Aggregate Results
Q120. How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the Opinion Space
activity?
100 Least Active Reviewers100 Most Active Reviewers50 Most Active Reviewers
50 Top Authors
Aggregate Results
Q105. What was your overall opinion of the Opinion Space site?
Q130(1). I was motivated to consider responses from people far from my own response on the opinion map
100 Least Active Reviewers100 Most Active Reviewers50 Most Active Reviewers
50 Top Authors
Aggregate Results
Q130 (2) The size of each point was a reasonable measure of how insightful the corresponding response was
100 Least Active Reviewers100 Most Active Reviewers50 Most Active Reviewers
50 Top Authors
Aggregate Results
Subgroup Analysis of ParticipationPost Survey Results
*red text indicates statistically significant differences
Q150. How willing would you be to participate in other versions of this system (on related topics)?
100 Least Active Reviewers100 Most Active Reviewers50 Most Active Reviewers
50 Top Authors
Aggregate Results
Subgroup Analysis of ParticipationPost Survey Results
*red text indicates statistically significant differences
Of all statistically significant differences (95% confidence interval)
Subgroup Analysis of Statement Ratings
Subgroup Analysis of Statement Ratings(statistically significant differences)
Statement 1: When deciding on which major product to buy, I often consider where it was made in making my decision.
Education = PostgraduateIncome = $125k - $150k
Aggregate Results
Statement 2: Reducing oil consumption should be a higher priority in the US
Income = $15k - $25kRegion = Northeast
Aggregate Results
Region = West
Statement 3: I am willing to pay more for a product that comes from a socially responsible corporation.
Age = 30 - 39Age = 40 - 49
Aggregate Results
Education = College GraduateGender = MaleRegion = West
YAMYEF
Subgroup Analysis of Statement Ratings(statistically significant differences)
Statement 4: The U.S. government should not give loans to corporations.
Education = High SchoolEducation = Some College
Aggregate Results
YAM
Income = $25k - $35kIncome = $35k - $50k
Subgroup Analysis of Statement Ratings(statistically significant differences)
Statement 5: New technologies are causing more problems than they are solving.
Aggregate ResultsAge = 18 - 29Age = 50 - 59
Education = High SchoolGender = Female
Gender = Male
Income = $125k - $150kIncome = $150k - $200k
Income = $25k - $35kIncome = $50k - $75k
Subgroup Analysis of Statement Ratings(statistically significant differences)
Subgroup Analysis of Post Survey Ratings
Of all statistically significant differences (95% confidence interval)
Subgroup Analysis of Post Survey Ratings(statistically significant differences)
Q105. What was your overall opinion of the Opinion Space site?
Aggregate ResultsEducation = Postgraduate
Education = Trade/Vocational SchoolIncome = $100k - $125k
Region = NortheastYAM
Subgroup Analysis of Post Survey Ratings(statistically significant differences)
Q120. How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the Opinion Space
activity?Aggregate Results
Age = 50 - 59Education = Postgraduate
Region = West
Q130(1). I was motivated to consider responses from people far from my own response on the opinion mapAggregate Results
Age = 60 - 70Education = College Graduate
Subgroup Analysis of Post Survey Ratings(statistically significant differences)
Q130 (2) The size of each point was a reasonable measure of how insightful the corresponding response was
Aggregate ResultsAge = 30 - 39Age = 60 – 70
YAM
Q135. How do you feel about using two sliders (agreement, insightfulness) to rate responses?
Aggregate Results
Age = 60 - 70Age = 30 - 39
Subgroup Analysis of Post Survey Ratings(statistically significant differences)
Q150. How willing would you be to participate in other versions of this system (on related topics)?
Aggregate ResultsGender = Female
Gender = Male
Income = $100k - $125k
Income = $15k - $25kIncome = $50k - $75k
Summary of Subgroup Findings
Young Affluent Males (YAM)
Young Affluent Males (YAM)
Aggregate rating statistics
Subgroup rating statistics Statements:
1) When deciding on which major product to buy, I often consider where it was made in making my decision.
2) Reducing oil consumption should be a higher priority in the U.S.
3) I am willing to pay more for a product that comes from a socially responsible corporation.
4) The U.S. government should not give loans to corporations.
5) New technologies are causing more problems than they are solving.
Young Educated Females (YEF)
Aggregate rating statistics
Subgroup rating statistics Statements:
1) When deciding on which major product to buy, I often consider where it was made in making my decision.
2) Reducing oil consumption should be a higher priority in the U.S.
3) I am willing to pay more for a product that comes from a socially responsible corporation.
4) The U.S. government should not give loans to corporations.
5) New technologies are causing more problems than they are solving.
Young Educated Females (YEF)
Incentive Analysis
Subgroup Response to Q120:“How much were the raffle entries (NOT incentive points) a motivating factor for you to participate in the Opinion Space activity?”
Subgroup Q120 Mean Q120 StdevTop 100 Reviewers 3.450549451 1.206791553
Top 50 Reviewers 3.369565217 1.340971349
income-$125,000 to $149,999 3.038461538 1.181506076
age-18-29 3.012658228 1.237531955
Top 50 Authors 3 1.268761639
YEF 2.948905109 1.234073238
YAM 2.939393939 1.179095543
age-30-39 2.933014354 1.21604544
income-$50,000 to $74,999 2.909090909 1.231660693
region-Northeast 2.894736842 1.263556556
100 least active reviewers 2.626666667 1.230429013
…
Measurements of Participant Activity
Mean Stdev
Insight ratings per user 45.56 142.52
Agreement ratings per user 46.03 142.57
Session Length (min) 12.69 11.25
Average Session Count 3.00 5.89
User Events 46.46
Total comments rated 105,861.00
Total comments viewed 118,737.00
Total Sessions 3,147.00Comments viewed per
session 37.73
Insight ratings per viewed comment 0.81
Agreement ratings per viewed comment 0.82
AIOC Opinion Space – All Users
Mean Stdev
Insight ratings per user 42.48 132.37
Agreement ratings per user 42.95 132.44
Session Length (min) 12.59 11.13
Average Session Count 3.00 5.90
User Events 44.68
Total comments rated 90,017.00
Total comments viewed 110,804.00
Total Sessions 3,101.00Comments viewed per
session 35.73
Insight ratings per viewed comment 0.80
Agreement ratings per viewed comment 0.81
AIOC Opinion Space – Rogue Raters Removed
Measurements of Participant Activity
Mean Stdev
Insight ratings per user 3.20 18.84
Agreement ratings per user 3.34 18.89
Session Length (min) 16.11 11.41
Average Session Count 2.00 1.54
User Events 15.21
Total comments rated 7,520.00
Total comments viewed 20,326.00
Total Sessions 1,755.00Comments viewed per
session 11.58
Insight ratings per viewed comment 0.31
Agreement ratings per viewed comment 0.32
Unilever Opinion Space – 1939 Total users
Mean Stdev
Insight ratings per user 5.04 10.78
Agreement ratings per user 5.11 10.67
Session Length (min) 12.24 11.08
Average Session Count 2.00 1.29
User Events 43.99
Total comments rated 2,072.00
Total comments viewed 9,881.00
Total Sessions 390.00Comments viewed per
session 25.34
Insight ratings per viewed comment 0.19
Agreement ratings per viewed comment 0.19
Opinion Space 3.0 (state.gov/opinionspace) – 370 Total users
Measurements of Participant Activity
Mean Stdev
Insight ratings per user 4.89 22.05
Agreement ratings per user 4.85 21.58
Insight ratings per viewed comment 0.19
Agreement ratings per viewed comment 0.18
Opinion Space 2.0 previously at state.gov/opinionspace – 5711 Total users
Rogue Rater Analysis
Summary
Appendix
Google Moderator