Source management for Ushahidi and SwiftRiver

Sources, collaboration and verification

A summary of relevant findings for SwiftRiver from the Ushahidi Wikipedia Understanding Sources project

Heather Ford. SwiftRiver/Ushahidi ethnographer.

A reminder on the goals of the project

• Attempting to answer the following questions:

• How are Wikipedia editors choosing, verifying, editing sources?

• How are they collaborating to do this?

• What could we apply to Ushahidi/SwiftRiver?

Key differences between WP and Ushahidi/SwiftRiver

Wikipedia Ushahidi/SwiftRiver

Editors make summarized statements about what has happened linked /backed

up by sources

Editors do not summarize what has happened but provide other editorial

functions like verifying statements and developing categories

Editors choose which sources to include All sources are generally included

Chosen sources must be what Wikipedians define as “reliable” which often does not include social media

sources

Social media sources are often the primary source of Ushahidi reports

All statements must be “verified” to be included

Verified and unverified statements are both (usually) reflected

Key findings relevant to SwiftRiver

1. Relevance is the primary reason for choosing a source (not accuracy);

2. Verification is a two-step process of reading/watching the cited source and then finding corroborating sources;

3. Important to authority and transparency is an understanding of who the authors are, and what the policy/bar for entry is;

4. Talk pages are critical places to alert, discuss and collaborate;

5. There was incentive for collaboration around editing in breaking news stories.

1. Relevance not accuracy

A number of factors are being weighed when deciding to choose a source beyond the so-called “reliability” or accuracy of a source, including:

• the balance of secondary to primary sources in the article (Wikipedia policy calls for editors to use a majority of secondary sources),

• whether the article is based on a breaking news story (in which case the majority of sources might have to be primary, eyewitness sources), or an article on a person (in which case their official Twitter feed, for example, may be considered a reliable source on that person);

• whether the source is notable in the context of the article (misinformation can also be relevant if it is widely reported and significant to the course of events as Judith Miller’s NYT stories were for the Iraq War).

Thus relevance is more appropriate a factor in deciding whether to use a source than accuracy. This becomes important when one considers how to filter results in sources searches.

2. The verification process

1.Verification is a two-step process of a) reading/watching the cited source in order to check that it is being accurately reflected in the text, and b) finding sources that corroborate or oppose the statements.

2.For example, the image to the right was used inthe article with the caption that reflected thatthese were “police in civilian clothing beating aprotester” until one editor watched the footagefrom where the screenshot originated and asserted that there was nothing in thecommentary from Al Jazeera that backed up thisstatement about the identity of those in the image.

3.Design that supports this special activity could enable editors/curators to:highlight a questionable source/statement --> further investigate by curators carefully reading/watching the original source + checking whether it is accurately reflected in the story --> checking whether the information is corroborated by other sources --> marking it as complete within a certain category (and then allowing later review)

3. Who are the authors? What are their policies?

• Without an understanding of who edited a page and what the policies on verification or source use are, there is no way for readers to accurately gauge the veracity of the article;

• On Wikipedia, we know that editors are using the principles of “verifiability, not truth”, “reliable sources” and “neutral point ofview” (NPOV) to curate and edit pages and although we couldknow more about individual authorship, users have a general idea of veracity because of the principles in force. SwiftRiver and Ushahidi require curators to develop and reflect similar principles to readers to answer questions such as:

• What are your principles for deciding which information you include/do not include? What is your POV? How can I check up on the information provided here? Are all sources going to be included? Only reliable ones? Only accurate ones? How will rumor/misinformation be represented?

4. Talk pages for collaboration

• Talk pages are critical places for collaboration between editors;

• It is here that they are able to reach consensus, provide space for leadership, develop specific guidelines regarding style, format, etc for the article, alert others as to breaking news, ask for help, show what they’ve done, verify sources, provide corroboration, and provide moral support (amongst others);

• Could SwiftRiver/Ushahidi similarly provide a place for editors to discuss the shape of the “story” they are telling and/or discuss the the verifiability of a source?

5. Collaboration around breaking news alerts

1.One of the most popular items offered on talk pages was for breaking news in which an editor would bring a new item to the page to achieve one or more of the following goals:

a) a way to alert other editors to recently-released information relevant to the article (important especially since editors were working in different time zones and the talk page would be one of the first places a collaborating editor would go when they woke up or starting editing);

b) an ask for others to add the source to the article for them if they didn’t have the time or the access privileges, for example, if they were editing as an anonymous IP during the periods when the page was semi-protected (some editors saw the discovery of sources and posting to talk as their single task rather than editing);

c) a place to “hold” sources while they were waiting for other sources or were unsure as to the relevance of the article for inclusion and were asking for opinions of other editors;

d) a way to ask other editors to find references for unsourced or incorrectly sourced information on the article.

For example

Here, an editor brings a new article to the talk page, where s/he picks out the relevant facts (this was new at the time because it had been unclear up until then what the Egyptian army’s role was in the 2011 Revolution). S/he follows it up with a corroborating source and then Lihaas edits the page and marks the talk page as “done”.

Tasks

Tasks generally accorded with the following workflow:

design implications

• SwiftRiver could enable collaboration by providing a space for individuals to bring breaking news to a common page for discussion regarding relevance/summarizing/verification/corroboration etc.

• In this way, nuggets grow into larger stories that can then be connected with one another. Rather than seeing verification as a separate process happening after curatorship which is the case with the Standby Task Force, for example, verification and curating/editing happen together in small teams of people who adopt tasks, rather than predefined and continuous roles.

3 key takeaways for Ushahidi/SwiftRiver

1. Use relevance rather than accuracy to filter results;

2. Distinguish between statements and sources/authors;

3. Decide what form/s a “report” will take.

1. Use relevance rather than accuracy to filter results

• We were originally thinking of collaboration in terms of the same people doing the same work and then for the algorithm to check whether there were similarities/inconsistencies, but this requires scale and doesn’t accord with the process by which people actually collaborate;

• People collaborated by checking up on cited sources, summarizing, finding relevant news, and deciding where the news fitted.

Statement

Source 1

Source 2

Source 3

For example:

Statement

Source 1

Source 2

Source 3

Aleppo_Alissa: “Bombs are going off all around us in central Aleppo”

For example:

Statement

Source 1

Source 2

Source 3


Reuters: “The Syrian government just released a statement saying that they

were initiating a campaign to push back the “terrorists” in Aleppo.”

For example:

Statement

Source 1

Source 2

Source 3




Free_Syria: “Government forces just bombed the Central Children’s Hospital

in Aleppo.”

For example:

Statement

Source 1

Source 2

Source 3

Witness Syria“Aleppo is under

siege by government forces.”





in Aleppo.”

For example:

Statement

Source 1

Source 2

Source 3







in Aleppo.”

For example:

(!)

Statement

Source 1

Source 2

Source 3







in Aleppo.”

For example:

(!)(!) Aleppo_Alissa has a history

of telling untruths.

Statement

Source 1

Source 2

Source 3







in Aleppo.”

For example:

(!)

(!)

(!) Aleppo_Alissa has a history of telling untruths.

Statement

Source 1

Source 2

Source 3







in Aleppo.”

For example:

(!)

(!)


Can we really take this as a reliable source? Need more sources to corroborate this.

Statement

Source 1

Source 2

Source 3







in Aleppo.”

For example:

(!)

(!)

(!)



Statement

Source 1

Source 2

Source 3







in Aleppo.”

For example:

(!)

(!)

(!)

(!) Sources are unreliable -- it isn’t clear who is doing the

shooting



Distinguishing between the author and the source/refWikipedia:

Ushahidi:

Author (of statement) | StatementVerified/Under dispute/Unverified

ORUnreviewed/Reviewed

A history of contribution that is disputed/undisputed

reviewed/unreviewed (i.e. new)

3. Decide what form a “report” will take

• Is it just a long list? Or a map? Or a textual summary? Or a live blog-style report?

• This is important because this will determine what tasks people are performing, how/whether sources should be linked into clusters, what are the most important variables/categories for filtering results etc.

• If it’s a map, we might want to focus verification activities on making sure the location (of the source/author and/or event) is accurate; if it’s a live blog, then we want to enable users to provide feedback on the sources the editor has used in the story, as well as to provide the editor with ways to characterise her/his sources.

the end.

Source management for Ushahidi and SwiftRiver

Technology

Transcript of Source management for Ushahidi and SwiftRiver