Team Narrative MindStanford University

Team Info

Sponsor: US Army Cyber Command (ARCYBER)

The Narrative Mind team contains experts in software engineering, social media design, and

web-based information operations (IO). We seek to develop tools that will optimize discovery

and investigation of communication trends on social media.

Weekly Total: 10 interviews

Users: 14

Experts: 38

Buyers: 4

Cumulative Total: 56 interviews

Problem

Statements,

NOT RFPs

Creates

“Innovation

Challenges”

Organizational Chart

CW3 “Fitzgerald”

MAJ “Kramer”

Reports analysis

COL “Tim”

LTC “Zeke”Director & Deputy SIG

Assesses threats 1-

3yrs out

Proposes

CONOPs

BG FrostDeputy

Commander Ops

ARCYBER(Service Component)

NSAADM Rogers

Internal ARCYBER officeProgram Managers

LTG CardonCommanding General

BG (O8-level)Undergoing Leadership

Change

Contains own cyber

troops that carry out cyber

missions.

JOINT FORCES HQ-CYBER

USCYBERCOMADM Rogers

USSTRATCOM

COCOMs

CENTCOM,

AFRICOM,

NORTHCOM

Proposes

concepts

Secretary Carter

ARCYBER

Acquisition

Process

Mr. R. PontiusDeputy to Commanding

General, Civilian

COL “John”G3 Operations

Subordinate

Units:

- 1st IO Cmnd.

- 780th Military

Intelligence

Brigade

- NETCOM

Fully-staffed Joint

Force IO/SIGINT

troops.

Provides

specialized

analysis on ad

hoc basis.

Proposes

concepts

to be

implemented

C5 ConsortiumPresident Charlie McBride

Board of Directors

Companies/Academics (by application)

Traditional DoD

Contractors

Non-traditional

Contractors

DIUXDevelops relationships

with commercial sector.

Submit White Papers

& Demos

Necessary

Partnership

Started 2015

PEO-EIS

ACTAdvanced

Concepts and

Technology

Directorate

Manages

CYBER

capabilities/

requirements.

Other Transactional Authorities

Key Objectives of OTAs:

● Reduce barriers to entry.

● Incentivize commercial

participation through expedited

awards and negotiable IP

terms.

Necessary Conditions of OTAs:

● Prototype projects only.

● Enhances mission effectiveness.

● Costs are fair/reasonable.At least one criterion must be met:

● All significant participants are

small businesses or non-traditional

defense contractors.

● Non-traditional defense contractor

participates to a significant extent.

● One-third cost share for traditional

defense contractor

Example of Actual Cyber OTA Micro Cloud Computing System

OTA Timeline

All the Hypotheses

Global

Tweet-level

Awareness Response

Adversary IO Org Chart

No baseline for monitoring/aggregating

adversary use of tech; cyber targeting

Automatic Narrative Detection

Language/culture experts aren’t able

to work at scale

Virality Predictor

Understand which narratives are the most salient

Important Event Predictor

Preempt real world events

Counter-Narrative Generator

DoD/Gvt. social media

presence is weak

Persistent ID-Alias tracker

Account bans and multiple aliases across

different networks make IDs hard to track

Poison Link Generator

Monitor who views

adversary dark-web

content

Filesharing Site Scraper

Need more cached information

access

Expedited Content Categorization

Scale of social media makes manual efforts

painful

Bot Detector

Can’t determine actual scale of

adversary following and support.

Hypotheses ❏ Three previously proposed narrative units would be valuable to track.

- Co-occurring hashtags

- Ranked list of engaged content/tweets over time

- Geospatial sentiment maps

Experiments ❏ Ask users to define their own narrative units.

❏ Ask users to rank the salience of different narrative units.

❏ Individual trawling around twitter feeds of presidential candidates.

Results ❏ A ranked list of most popular content over time is the most valuable, along with information

on co-occurring hashtags.

❏ Sentiment data is not as helpful because it tends to be inaccurate.

❏ There are tools that exist to do any one of these things, but they are not stitched together for

easy use by operators.

Actions 1. Analyze political campaign dataset for hashtag co-occurrence frequency.

2. Discuss refined MVP and continue to develop new narrative units.

3. Research communication link between awareness and response groups.

Customer Discovery

Responses to Narrative Units Presented Previously

A ranked list of

most popular

content over time

A list of

hashtags that

appear with the

same message

with extreme

frequency.

Positive or negative opinion among

geographic or demographic

population.

Narrative Type #1

Narrative Type #2

Narrative Type #3

E.g

Popula

rity

with u

sers

Sentiment analysis is thought

to be too crude to provide

useful input. May pollute

signal-noise.

Unit #2 “popular content”

(commonly described as

media like photos or videos)

seemed to be the most

credible. Unit #1 co-occurring

hashtags might provide more

granularity and context than

just single hashtag trends.

MVP - 3 Types of “Narratives” Outputted by Our System

A list of hashtags that

appear with the same

message with extreme

frequency.

Positive or negative

opinion among geographic

or demographic

population.

Narrative Type #1 Narrative Type #2Narrative Type #3

Displayed

to Analysts

Analysts associate

each “narrative unit”

with a real-world,

human-readable

description of a

narrative.

Gain creator: Software automatically

tracks activity of a real-world narratives by

tracking the activity of associated “units”.

Real World NarrativeLong-form, unstructured human description or

collection of info.

A ranked list of most

popular multi-media

content over time

Pain reliever: Social media firehoses

scanned for list of narrative units that match

predefined “schemas”, ranked by frequency.

Problems with this MVP:● What are the right “units”?

● How is a narrative’s overall growth

calculated from the multiple units with

different sizes? (e.g. relative

“weights?)

Benefits of this MVP:● Aside from schema, specific units

don’t need to be defined a priori.

● Model allows retroactive reindexing of

a single narrative’s growth based on

new associations.

● Builds a “training set” for more

sophisticated efforts.

Mission Model Canvas

- Understand viral potential of

social media posts in real-time.

- Track how groups use

technology over time.

- Gnip/Twitter/Facebook

- CrowdFlower,

Samasource, or

Mechanical Turk

- Pre-existing social

media service and micro-

labor aggregators

- Third-party access

platforms for social

media

-Data visualization (Zoic)

-Content analysis

platforms (Sens.ai,

Leidos)

Primary

ARCYBER

-Bg. General (decision

maker)

-MAJ/LTC/COL

(operational plan)

-Analysts/Operators

(actionable insights)

COCOMs

-General (decision

maker)

-MAJ/LTC/COL

-Analyst/Operator

Secondary

Political Campaigns

-Campaign managers

-Supporters

Consumer Brands

-CMO

-Public Relations Team

- Optimize workflow for

social media analysts.

-Deliver insights to

commanders about

online environment.

-Insights into responses

against specific

broadcasting narratives

-Detect opposition

narratives emerging in

real time

-Early warning on

emerging brand threats

- Customized UI

- Testing with analysts

-Enable faster problem awareness to problem response times for

decision-makers across organizations

- MechanicalTurk or crowdsourcing labor (microtasks)

- UI Development/Testing with ARCYBER analysts.

- Software Development

- Research aggregation? (e.g. Import and reformat existing

knowledge about extremist organizational structure)

- Access to Twitter

firehose or API

- Local language

speaking crowdsourcing

staff.

- Accurate testing for

intercoder reliability

- ARCYBER: Bg. General,

LTC, Strategic Initiatives

Group, OTA, Purchasing PMs,

End-User operator)

-COCOMs: OTA, Operators,

Purchasing PMs

-Political Campaign:

Opposition research team, ???

-Private Sector: CMO, ???

Beneficiaries

Mission AchievementMission Budget/Costs

Buy-In/Support

Deployment

Value

PropositionKey Activities

Key Resources

Key Partners

Value Proposition Canvas - ARCYBER

Products

& Services

Web/Desktop

Application Update research

Customer

Jobs

- New mission priorities.

- Little familiarity with overall

environment.

- No repository for high-level

learnings

Gains

Pains

Gain

Creators

Pain

Relievers

- New opportunities for cyber-

ops

- Social network graph?

-Illustrates evolution of

networks

- Provides insight for

mapping “online terrain”

Visualize information about

how organizations are

using social media

ARCYBER - Operator/Analyst (Actionable Insights)

Value Proposition Canvas - ARCYBER

Products

& Services

Web/Desktop

Application

Aggregate insights

from analysts,

compile into plan

passed to decision

maker

- New mission priorities.

- New social media platforms

- Ability to prioritize focus

Customer

Jobs

Gains

Pains

Gain

Creators

Pain

Relievers

-Faster turnaround time with

better data gained from

information awareness software

and technology

- Clear hierarchy of

priorities

ARCYBER - MAJ/LTC/COL (Operational Plans)

-Better data in informing

operational plans provided

to decision makers

-Reducing uncertainty in

operational planning

Value Proposition Canvas - ARCYBER

Products

& Services

Web/Desktop

Application

Act on reports and

plans generated

about various cyber

issues across globe- Little understanding of

ground-level nuances of

challenges

- Little understanding of

proliferating platforms

Customer

Jobs

Gains

Pains

Gain

Creators

Pain

Relievers

- Organizational chart of

technology broken down by

functional purpose

- Timeline-based viewer

- Social network graph?

-Reduce uncertainty of

decision making

-Add methodological rigor

to ARCYBER’s operations

- Accomplish mission

objectives

ARCYBER - Brigadier General (Decision Maker)

Value Proposition Canvas - COCOM

Products

& Services

Web/Desktop

Application

Target audience

analysis, finding

opportunities to

optimize influence

operations

Customer

Jobs

-Loss of visibility on key

communicators

- No repository for high-level

learnings

Gains

Pains

Gain

Creators

Pain

Relievers

-Auto-populated list & real-time

visualization

-Illustrates evolution of info

dissemination networks

- Provides insight for mapping

“online terrain”

Closure of visibility gap

COCOM - Operator/Analyst (Actionable Insights)

Value Proposition Canvas - COCOM

Products

& Services

Web/Desktop

Application

-Long term

operational planning

- Measuring

effectiveness

Customer

Jobs

- Unclear picture of how

networks evolve, adapt over

time

Gains

Pains

Gain

Creators

Pain

Relievers

- Organizational chart of

technology broken down by

functional purpose

- Visualization of dissemination

networks over time

-Illustrates evolution of info

dissemination networks

- Provides insight for

measuring operations,

mapping “online terrain”

Greater awareness of

network evolution improves

planning, indicators of

operational effectiveness

COCOM - Maj/LTC/Col. (O4-O6) (Operational Plans)

Value Proposition Canvas - COCOM

Products

& Services

Web/Desktop

Application

- Judge potential

effect and risk of

operations

- Approve/reject

plans

Customer

Jobs

- Less familiarity with

internet as an operational

environment

- Needs high level summary

of technology use

Gains

Pains

Gain

Creators

Pain

Relievers

- Organizational chart of

technology broken down by

functional purpose

- Timeline-based viewer

-Illustrates evolution of info

dissemination networks

- Provides insight for

mapping “online terrain”

Visualize high-level

information about use

of technology.

COCOM - Brigadier General (O7) (Decision Maker)

Value Proposition Canvas - Political Campaign

Products

& Services

Web/Desktop

Application

-Identify opposition

narratives on SM

channels

-Craft responses to

respond w/short

turnaround

Customer

Jobs

-Limited lead time in

anticipating threats from

opposition campaigns

-New stories/themes can

derail campaign momentum

without a targeted response

Gains

Pains

Gain

Creators

Pain

Relievers

-Faster response time to

opposition’s campaigns

-Maintain the initiative in media

environment by rapidly

responding to threats

-Early detection of

emerging opposition

narratives on SM

-Insight into evolving threat

environment

Counter opposition

themes quickly, identify

newly spreading

rumors that could

undermine campaign.

Political Campaigns - Opposition Analysis/Campaign Director

Value Proposition Canvas - Consumer Product Company

Products

& Services

Web/Desktop

Application

-Monitor brand

integrity and

customer themes

-Identify and address

emerging problems

Customer

Jobs

-Identify themes about

corporate brand, pinpoint

problem areas early

-Avoid negative narratives

from becoming costly PR or

marketing problems

Gains

Pains

Gain

Creators

Pain

Relievers

-List of customer narratives

about brand that are being

posted online

-Proactive response to

emerging brand threats, better

customer satisfaction

-Early warning about

negative themes

-Identify problem areas,

protect corporate brand

from costly problems

-Protect brand integrity,

create greater

shareholder value by

addressing customer

concerns

Consumer Product Company - CMO/Marketing Organization