Automating Frame AnalysisAntonio Sanfilippo, Lyndsey Franklin, Stephen Tratz, Gary Danielson, Nick Mileson, Rick Riensche, Liam

McGrath

Pacific Northwest National Laboratory

Overview

Review of Frame Analysis Automating frame annotation An application of automated Frame Analysis Ongoing and future developments

Frame Analysis: What is it? What is it for?

Frame Analysis focuses on how people understand situations through the analysis of communicative and mental processes to explain How communication sources

construct issues to influence target audiences, e.g. framing suicide bombing as “martyrdom”

How the target audiences respond to framing, e.g. degree of resonance

Recognizing framing intent leads to an understanding of the goals of the communication source

A brief history of Frame Analysis

Approach pioneered by Goffman in 1974 has become an important analytical components across the social sciences

Strongest impact on the study of social movements Renewed interest in the social psychology of collective action in the early 1980s led to further work which strengthened Goffman’s initial insights

Frame Analysis is now a main component in theories of social movement

Our objectives in automating Frame Analysis

Despite great recent theoretical advances, there still is no systematic method to identify and marshal frame evidence in a time/cost effective manner

Address current limitations in the representation, acquisition and analysis of frame evidence Leverage complementary approaches to Frame Analysis Combine theoretical insights from Frame Analysis and

Linguistics with Information Extraction capabilities and Content Analysis methods

Frame Analysis components

Collective action frames Social movement entrepreneurs offer a strategic

interpretation of issues, e.g.“Islam is the solution” proclaimed Akef

Create inter-subjective meaning to recruit and mobilize people for the promotion of movement goals, e.g. establish Shar’iah law

Frame resonance (not the focus of this talk) Describes the relationship between a collective action frame,

the target audience, and the broader cultural context E.g. credibility of the frame and its promoter, relevance of the

frame to the target audience, frame consistency

Theories of collective action frames

Gamson Snow and Benford Injustice: identify individuals or institution to blame for grievances Identity: specify aggrieved group with reference to shared interests and values Agency: recognize that grieving conditions can be changed through activism

Diagnostic frame: tell new recruits what is wrong and why Prognostic frame: present a solution to the diagnosed problem Motivational frame: give people a reason to join collective action

EntmanSubstantive frame functions Substantive frame foci

Defining effects or conditions as problematic Identifying causes Conveying moral judgment Endorsing remedies or improvements

Political events Issues Actors

Frame representation: “intelligent union” approach

PROMOTER used by Snow and Benford corresponds to the result of

Gamson’s identity frame function overlaps with Entman’s notion of

actors COMMUNICATIVE INTENT

implicit in the frame classification of Gamson (injustice, identity, agency) and Snow and Benford (diagnostic, prognostic, motivational)

TARGET corresponds to the result of

Gamson’s injustice frame function ISSUES

as in Entman

GROUP-X denounces the insistence of the security apparatus on terrorizing innocent people and on using the emergency law against honest COUNTRY-X citizens, through its campaign of raids and detentions against GROUP-X in the governorates of LOCATIONS A, B, C, and D.

PROMOTER

INTENT

TARGET

ISSUES POLITICS SOCIAL LAW SECURITY

A methodology that promotes objectivity and automation

INTENT is further broken down into 15 speech act classes ASSERT, BELIEVE, CRITICIZE, EXPLAIN, REQUEST, … Each INTENT class has various lexical realizations (from

WordNet)

We distinguish 9 types of ISSUES SECURITY, RELIGION, POLITICS, SOCIAL, LAW, MILITARY, … Each ISSUE has a list of lexical realizations (from WordNet

Domains)

INTENT CRITICIZE

Lexical realizations

accuse, blame, calumniate, charge, condemn, criticize, denigrate, deplore, impeach, incriminate, lambast, malign, …

A methodology which can be effectively evaluated

Four human subjects edited frame annotations automatically assigned to 30 documents

The annotation judgments of the four annotators were compared and assessed for agreement using the kappa test

Fleiss kappa test: group of four annotators

Ratings Kappa z-score

1660 0.499 46.2

Cohen kappa test: six pairs or annotators

Average Ratings Average Kappa Average z-score

1700 0.70 28.68

Frame extraction

Designed and implemented fully automatic extraction algorithm to find frames in naturally occurring text See Sanfilippo et al. (2007) for details

Evaluating automatic frame extraction

Used kappa and precision/recall tests to evaluate of manually and automatically assigned annotations to 30 documentsCohen kappa test: human vs. computer (four pairs)

Average Ratings Average Kappa Average z-score

1674 0.52 z = 21

Fleiss kappa test: four human annotators plus computer

Ratings Kappa z-score

1433 0.422 50.5

#Correct #Incorrect Precision Recall F1157 43 0.785 0.698 0.739

(frame detection)

Analyzing frame evidence

Developed a semantically-driven and visually interactive search environment to query and quantify frame evidence

Using Frame Analysis to support predictive intelligence

At the year X elections Group-X (a radical religious group) tripled the number of seats previously occupied to become the largest opposition bloc in the parliament

Assess whether Group-X will adopt more secular views, or the recent electoral success will lead to increased

radicalization

Process

Harvested 619 documents from Group Y official website for years X and X+1

Processed documents with frame extraction pipeline Loaded the results into the frame search environment Issued semantic queries to identify

Negotiation frames: accept, explain, support, etc. Contentious frames: accuse, criticize, correct, reject, etc.

Frame query results

Contentious Frames Negotiation Frames z-score

Year-X 24% 14% 2.28

Year-X+1 35% 10% 6.16

z-score 2.40 1.22

Conclusions and further work

Our approach enables the analysis of messaging strategies from document collections in a time and cost effective fashion

Current and future work Frame Analysis with direct unreported speech (ongoing) Frame Resonance (planned)

Thanks!

Antonio SanfilippoPacific Northwest National LaboratoryTel.: 509-375-2677antonio.sanfilippo@pnl.gov