Tasking the Tweeters: Obtaining Actionable Information from Human Sensors

Alun Preece, Will Webberley (Cardiff)

Dave Braines (IBM UK)

Introduction

Social Media for Real-time Intel

Social Media streams as sources of actionable intelligence for Situation Awareness (SA)

• Acknowledging a human-based sensor network• A good source today: Twitter

– Real-time characteristics– Follower-based model– Open APIs

• Real-world examples from recent events:– Boston marathon bombing (US)– Lee Rigby murder (UK)

Some SA platforms emerging:• Twitcident, Apollo, ReDites, Sentinel

Mapping Social Media to DCPD

Direction - what data to collect from where

Collection• e.g. Twitter: APIs for streaming, searching sampling

(other platforms available too)• Post-filtering for noise reduction

Processing• e.g. probabilistic, NLP, sentiment, event detection• Provide semantic enrichment (contextual) for Shared

Understanding.• Detect trends, clusters, anomalies etc

Dissemination• Visualisation, alerting,

summarisation• Further querying• Direct further collections

A generic social media processing pipeline mapped to DCPD steps

Dynamic ISR asset management

Missions-and-means framework formalised as a collection of ontologies

Tasks characterised by the data needed to achieve them

type of data (visual, IR, radar etc )

“quality” rating 0 to 9 Assets rated by the data they

provide

MMF framework

NIIRS-based approach

Software tool for agile sensor-task assignment Extensible knowledge-base of sensor-task

suitability Uses existing models and frameworks to map

capabilities

Sensor Assignment to Missions (SAM)

In previous work we have defined a framework for

dynamic ISR asset management:

A Pilot Study

July 26th, 2014: Cardiff protest march

• Planned protest march in Cardiff, UK• Against Israeli incursions into Gaza• Potential for public order disruption• Approximately 2,000

people• Some limited local trouble• Evidence of protest and

activities found on Twitter– Real-time during the event– In various stages afterwards

Source: Wales online – www.walesonline.co.uk

Social Media Timeline

Timeline of the July 26th 2014 protest and its aftermath

Social Media Timeline

Timeline of the July 26th 2014 protest and its aftermath

UK-wide tweets

Social Media Timeline

Timeline of the July 26th 2014 protest and its aftermath

Verbal and physical abuse at bars [15:15]

UK-wide tweets

Social Media Timeline

Timeline of the July 26th 2014 protest and its aftermath

Verbal and physical abuse at bars [15:15]

March ends [15:40]

UK-wide tweets

Social Media Timeline

Timeline of the July 26th 2014 protest and its aftermath

Verbal and physical abuse at bars [15:15]

March ends [15:40]

Tweeting after the march

UK-wide tweets

Social Media Timeline

Timeline of the July 26th 2014 protest and its aftermath

Verbal and physical abuse at bars [15:15]

March ends [15:40]

Tweeting after the march

Police mentionsincrease after broadcast news

UK-wide tweets

Social Media Timeline

Timeline of the July 26th 2014 protest and its aftermath

Verbal and physical abuse at bars [15:15]

March ends [15:40]

Tweeting after the march

Police mentionsincrease after broadcast news

UK-wide tweets

Important:We are observing perception of the event, not the

event itself…

Practical details

Sentinel Twitter Stream Analysis• Geo-tagged tweets• Topical search terms• Mentions of local places

People on the ground• Access to live twitter (+ search)• Manually identify “key” tweets

Some issues• Generality of tweets• Crowd size estimation: “a few hundred”,

“thousands”• Very few tweets geo-tagged

The Sentinel application

Observations from the pilot

• Sweet spot for initial relevancy: Search terms + geo-spatial

• Social Media reflects perception, not reality• We are not claiming that this simple study is

representative.• Key events and activities can be detected:

– …but how early can these be found through “small signals”?

• Some issues with Social Media:– Propagation of misinformation– Detection of bias– Quantification of contextual factors

• There is the potential to inform action viathis kind of situation awareness

Modeling Tweets and Tweeters

Background: CNL for conversation

Need an appropriate form for human-machine interaction:

humans prefer natural language (NL) or images these forms are difficult for machines to

process, leading to ambiguity and miscommunication

Compromise: controlled natural language (CNL)there is a person named p1 that is known as ‘John Smith’ and is a person of

interest.

low complexity | no ambiguityITA Controlled English (CE)

Defining sources and peopleSources, e.g. a Twitter account:conceptualise a ~ twitter account ~ A that

is an online identity andis a temporal thing andhas the value L as ~ location ~ andhas the value NT as ~ number of tweets

~ andhas the web image PP as ~ profile

picture ~ andhas the value NT as ~ number of tweets

~ andhas the value NFR as ~ number of

friends ~ andhas the value NFO as ~ number of

followers ~.

there is a journalist named ‘Paul Heaney’ thatuses the twitter account ‘paulheaney67’

andworks for the media organization ‘bbc’.

People (and their derivation from a source):

…we are actually building profiles of “human sensors”.

Human Sensor profilesThe following information is available for inclusion in the human sensor profile:• All data from their Twitter profile (including location)• Who they frequently interact with• Who they talk about• Who are their influencers• Recently posted media (photos, videos)• Terms names from recent tweets• Locations from recent tweets

– Including travel to/from locations• Sentiment analysis for tweets and terms

The use of our human friendly CNL means that additional “local knowledge” can easily be added too.

e.g. “stance” – to capture some importance contextual detail

This is a dynamic social network

Talking to Moira

An example Moira query showing some elements of the tweeter model

All this information (people, sources, tweets, terms, events etc) is available in a CNL knowledge base.

The Moira agent is able to access this and support conversation with human team members…

Tasking Tweeters

Defining ISR tasks

From our previous work:

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

Defining ISR tasks

From our previous work:

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

The “action” – what you are trying to achieve

Defining ISR tasks

From our previous work:

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

The “action” – what you are trying to achieve

What you are trying to do, e.g. “detect”, “localize”

Defining ISR tasks

From our previous work:

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

The “action” – what you are trying to achieve

What you are trying to do, e.g. “detect”, “localize”

From a predefined ISR ontology

Defining ISR tasks

From our previous work:

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

The “action” – what you are trying to achieve

What you are trying to do, e.g. “detect”, “localize”

From a predefined ISR ontology

From a gazetteer or similar

Defining ISR tasks

From our previous work:

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

The “action” – what you are trying to achieve

What you are trying to do, e.g. “detect”, “localize”

From a predefined ISR ontology

From a gazetteer or similar To establish temporal

bounds

Defining ISR tasks

From our previous work:

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

The “action” – what you are trying to achieve

What you are trying to do, e.g. “detect”, “localize”

From a predefined ISR ontology

From a gazetteer or similar To establish temporal

bounds

For simple resource scheduling

Defining Social Media ISR tasks

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

Defining Social Media ISR tasks

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

Direction:• The search terms (topics)

are derived from the “detectable”

• The spatial extent from the “spatial area”

Defining Social Media ISR tasks

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

Direction:• The search terms (topics)

are derived from the “detectable”

• The spatial extent from the “spatial area”

Collection:Stream-processing of tweets based on “direction” phase.

Defining Social Media ISR tasks

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

Direction:• The search terms (topics)

are derived from the “detectable”

• The spatial extent from the “spatial area” Processing:

The required “intelligence capability” determines the type of processing:• “localization” – derive

location data from tweets or tweeter

• “detection” – use existing event detection algorithms.

Collection:Stream-processing of tweets based on “direction” phase.

Defining Social Media ISR tasks

conceptualise the task T~ requires ~ the intelligence

capability IC and ~ is looking for ~ the detectable thing DT and

~ operates in ~ the spatial area SA and

~ operates during ~ the time period TP and

~ is ranked with ~ the task priority PR.

Direction:• The search terms (topics)

are derived from the “detectable”

• The spatial extent from the “spatial area” Processing:

The required “intelligence capability” determines the type of processing:• “localization” – derive

location data from tweets or tweeter

• “detection” – use existing event detection algorithms.

Collection:Stream-processing of tweets based on “direction” phase.

Dissemination:Alerting (or otherwise) via contextual application such as Sentinel, or agent such as Moira.

Identifying “key tweeters”

• In practice “key tweeters” emerge:– Use spatial terms: they want people to know where they

are– Use terms/hashtags: they want their tweets to be found– Social network: who are they and who they connect to

• From these we can determine:– Whether they are in a “position to know”– Their skills in Twitter usage– Their influence and reach

• All of this helps buildknowledge of trust andinformation quality

Findings so far• Existing ISR task representation can drive Twitter collection• Human & machine agents can use this information in

many ways• The Moira agent helps us to interact with the knowledge

base:– Engage the system in a conversation– Assert new local knowledge– Extend the model– Invoke additional functions such as

“fact extraction”

Use of the “stance” relationship in a conversation with Moira

An example of fact extraction from tweet text using Moira

Wrapping up

Related work

• Conversational interaction:– Bi-directional chains for

ISR pipelines– Humans and machines

in collaboration

• Experiments with Human subjects:– Using the Moira interface– Crowd-sourced Situational Understanding– Combine Human input and physical sensors– Handling incomplete and conflicting information– Use of relevancy criteria to minimise resource utility

Some conclusions

• Streamed insight from Social Media could be incorporated into traditional ISR asset management.

• This could be streamlined through:– Automatic assignment of assets (for stream processing)– Automatic identification of Social Media collections

• Lots of issues:– e.g. misinformation and coordinated rumours

• Awareness improves potential for action:– Early countering strategies, opportunities for

community intervention

• Limitations and opportunities:– We have focused on text-based analysis– Imagery potential: image processing,

face detection, object recognition etc

Tasking the Tweeters: Obtaining Actionable Information from Human Sensors

SPIE DSS 2015 – Ground/Air Multisensor Interoperability, Integration & Networking for Persistent ISR IV

Any questions?dave_braines@uk.ibm.com

Research was sponsored by US Army Research Laboratory and the UK Ministry of Defence and was accomplished under Agreement Number W911NF-06-3-0001. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the US Army Research Laboratory, the U.S. Government, the UK Ministry of Defense, or the UK Government. The US and UK Governments are authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon.

Development of the Sentinel platform was funded by the European Commission under the project “Tackling Radicalisation in Dispersed Societies (TaRDiS)”, and the ESRC via the project “After Woolwich: Social Re- actions on Social Media” (ES/L008181/1). Cardiff University provided funding for the pilot study examining community impacts of the NATO Summit.

We thank Kieran Evans and David Rogers (Cardiff University) for setting up the data collection pipeline for the pilot study in Section 2 and assistance with the data analysis. We thank Darren Shaw (IBM Emerging Technology Services, UK) for creating the tweeter locator service in Section 3. Valuable insights on policing and community reaction to events such as the ones featured in our pilot study were provided by Martin Innes, Colin Roberts and Sarah Tucker (Cardiff Universities Police Science Institute, http://www.upsi.org.uk).