New Ways of Deliberating Online: An Empirical Comparison of Network and

Threaded Interfaces for Online Discussion

Anna De Liddo & Simon Buckingham Shum Knowledge Media Institute

The Open University, UK

6th International Conference on e-Participation, ePart2014

Sept 1-3, 2014 Trinity Collage, Dublin, Ireland

Social Media, Community Ideation and Question-Answering is proliferating on the Web

Setting the Problem •  Poor Debate: No tools to identify were ideas

contrast, where people disagree and why... popularity vs critical thinking

Setting the Problem fundamentally chronological views which offer: •  no insight into the logical structure of the ideas, such as the

coherence or evidential basis of an argument. •  No support for idea refinement and improvement

LINK to PETITION: http://www.change.org/en-GB/petitions/stand-against-russia-s-brutal-crackdown-on-gay-rights-urge-winter-olympics-2014-sponsors-to-condemn-anti-gay-laws

Setting the Problem •  No ways to assess the quality of any given idea

LINK to QUORA: http://www.quora.com/Physics/Do-wormholes-always-have-black-holes-at-the-beginning#answers

•  Poor Commitment to Action •  Poor Summarization •  Poor Visualization

Very High

•  Lack of Participation •  Poor Idea Evaluation •  Shallow Contribution

High

•  Cognitive Clutters •  Lack of Innovation Moderate

•  Platform Island and Balkanization •  Non-representative decisions Minor

Pain Point Prioritization

Setting the Problem

This hampers both:

•  quality of users’ participation and •  The quality of proposed ideas •  effective assessment of the state of the debate.

A new class of Collective Intelligence and Online Deliberation Platforms That make the structure and status of a dialogue or debate visible Coming from research on Argumentation and CSAV, these tools make visually explicit users’ lines of reasoning and (dis)agreements. •  Deliberatorium •  Debategraph •  Cohere •  CoPe_it! •  Problem&Proposals •  YourView •  The Evidence Hub

A Common Data Model: simplified IBIS

!

MIT Deliberatorium

Cohere

Debategraph (as CoPe_it! and The Evidence Hub)

!

Motivation

•  No systematic evaluation of the merits or otherwise of network vs. threaded interfaces.

•  the bigger the number of participants to the discussion, and the higher the complexity of the discourse ontology, the more clumsy and less usable graph visualizations become

•  resistance to the use of graph visualization of arguments in certain domains of application.

•  It therefore remains unclear what are the advantages and affordances of different graphical representation of arguments to support online discussion

Research Question

•  the focus of the study is on reading and searching the online deliberation platforms

RQ: •  Does an interactive, self-organizing network

visualization of arguments provide advantages over a more conventional threaded interface for reading and search?

The Study

•  exploratory user study to observe users’ performance under three information-seeking tasks, and

•  compare their performances using two different user interfaces for arguments visualization (threaded vs network visualization of arguments)

•  A grounded theory analysis of the experimentation’s video to show how different graphical interfaces for the representation of arguments affect the way in which users read and understand the online discourse.

Participants

•  10 subjects (two groups of 5) •  members of different Open University departments •  with widely mixed IT expertise •  randomly allocated to the two different groups •  IT expert/non expert ratio in each group was

approximately the same. •  median age 40 (with range from 32 to 48) •  native or near-native English speakers

The Online Environment: The Evidence Hub

•  The Evidence Hub is a collective intelligence and online deliberation tool to support argumentative knowledge construction by crowdsourcing contributions of:

issues, potential solutions, research claims and the related evidence in favor or against those

http://evidence-hub.net/

Study Conditions

•  two different versions of the evidence Hub •  different user interfaces for arguments visualization •  same database to make sure that participants in the two

groups would receive exactly the same quantity and type of information

The Linear-threaded Interface for Arguments Visualization

http://evidence-hub.net/

Network Interface for Argument Visualization

http://evidence-hub.net/

Tasks

•  Identifying solutions to an issue (Task1) •  Identifying synergies between solutions (Task 2) •  Identifying contrasts in the wider debate (Task 3)

Emerging metrics

Metrics used to compare user’s performance across these three tasks: •  Task Accomplishment; •  Data Model Interpretation and •  Emotional Reactions to

Task accomplishment Main reason of task failure is task understanding Accomplished by incorrect” mostly in the linear interface group •  the linear structure of the interface does not help to interconnect

and compare content •  users focused more on the content rather than on the

argumentation process •  lead to digressions and incorrect responses.

“Accomplished easily” mostly in the network visualization interface group. This is mainly due to the •  visual hints provided by the network representation. •  links labels and colors particularly predominant in determining

success. •  less attention paid to the iconography of the nodes. •  connections density very effective to provide answer to the task.

Data Model Interpretation •  DMI was better supported by network visualization of

arguments.

•  Category misinterpretation and uncertainty in data model interpretation tend to occur more frequently in Group1 than in Group2,

•  network visualization of arguments support a complete and correct understanding of both categories and data model.

•  learn by example mechanisms: exploration of a new argument map they reinforce their understanding of the data model

Emotional Reactions •  a general sense of surprise and positivity toward the network

visualization is recorded, which easily sparks into likeness and even excitement This also increase users confidence with the tool (“I feel confident, I’m pretty sure this is the answer”-P7).

•  main objects of surprise due to the self-arranging graph (“it is like a jelly, it is so fantastic!”-P7; “it is all shifting! It is interesting… I quite like that!”-P8).

•  emotional reactions to the linear interface regards general skepticism which can also decay into confusion and feeling lost (“I think I am lost”-P1; I am now burnign.. because I haven't worked out how to do it” ”that is the all page I am looking at…ohhh ok I give up!”-P5).

•  The main reason the need of “too many clicks” to seek information and frequent “change of context” which often provokes disorientation.

Conclusions •  Linear interfaces perform worse especially when the

information is nested into more articulated argumentation chains (Task 2).

•  Network-like representation and the visual hints such as network structure, iconography and links’ labels and colors facilitate the identification of argumentation chains, thus supporting indirect connection and higher-level inferences of how the content connects.

Conclusions •  Data model interpretation is also improved by

argument visualization. Notably, exposing the data model in form of argument maps appears to enable a learning by example mechanism, whereby users reinforce their understanding of the data as they navigate through the user interface.

•  Effectiveness of network visualization of arguments and the positive impact it has on arguments reading and comprehension increases as information complexity increases: the bigger the discussion, the better network visualizations performs compared to linear-threaded visualizations.

Conclusion •  There is an element of fun and excitement associated to

dynamic network visualization of arguments.

•  Network visualization of arguments augment online discussion by providing a layer of structure that helps to improve human comprehension of the argumentation structure behind the online discourse.

•  These findings open new avenues for combining social media discourse with advanced network visualizations of discourse elements (issues, solutions and arguments) to both improve users’ engagement, sensemaking and appreciation of large-scale online deliberation processes.

Catalyst’s Online Deliberation Tools

Collaborative Knowledge Production

Collaborative Web Annotation and Knowledge mapping

Social Network Analysis and Visualization

Structured Online Discussion and Argumentation

Advanced Analytics for: Attention mediation & Deliberation diagnostic

Collaborative Web Annotation and Knowledge mapping

http://litemap.open.ac.uk

Internationalization to English and German

Connect and Map out the key issues and arguments visually with LiteMap

Get the LiteMap bookmarklet

Harvest, annotate and classify contributions from the Utopia’s discussion forum

1

2

3

Structured Online Discussion and Argumentation

debatehub.net

Get involved

Follow us on Twitter @CATALYST_FP7

Watch the Demos on Youtube CATALYST FP7

Thanks for your time!

Dr. Anna De Liddo Research Fellow in Collective Intelligence Infrastructures

KMi, Open University, UK

email: anna.deliddo@open.ac.uk Twitter: Anna_De_Liddo

HomePage: http://kmi.open.ac.uk/people/member/anna-de-liddo

Evidence Hub Website: http://evidence-hub.net/

Projects: Catalyst: http://catalyst-fp7.eu/

EDV – Election Debate Visualizations: http://edv-project.net/

Tools: evidence-hub.net, debatehub.net, litemap.open.ac.uk