A Social Media Based Examination of the Effects of Counseling RecommendationsAfter Student Deaths on College Campuses

Koustuv SahaGeorgia Tech

koustuv.saha@gatech.edu

Ingmar WeberQatar Computing Research Institute, HBKU

iweber@hbku.edu.qa

Munmun De ChoudhuryGeorgia Tech

munmund@gatech.edu

Abstract

Student deaths on college campuses, whether brought aboutby a suicide or an uncontrollable incident, have serious reper-cussions for the mental wellbeing of students. Consequently,many campus administrators implement post-crisis interven-tion measures to promote student-centric mental health sup-port. Information about these measures, which we refer toas “counseling recommendations”, are often shared via elec-tronic channels, including social media. However, the currentability to assess the effects of these recommendations on post-crisis psychological states is limited. We propose a causalanalysis framework to examine the effects of these counselingrecommendations after student deaths. We leverage a datasetfrom 174 Reddit campus communities and ∼400M posts of∼350K users. Then we employ statistical modeling and nat-ural language analysis to quantify the psychosocial shifts inbehavioral, cognitive, and affective expression of grief in in-dividuals who are “exposed” to (comment on) the counselingrecommendations, compared to that in a matched control co-hort. Drawing on crisis and psychology research, we find thatthe exposed individuals show greater grief, psycholinguistic,and social expressiveness, providing evidence of a healingresponse to crisis and thereby positive psychological effectsof the counseling recommendations. We discuss the implica-tions of our work in supporting post-crisis rehabilitation andintervention efforts on college campuses.

IntroductionCollege campuses are close-knit, largely geographicallycolocated communities where a crisis event can have a pro-found negative impact on the overall wellbeing of the cam-pus community (Swan and Hamilton 2017). One such cri-sis that is frequently encountered is the death of a student.Recent statistics report that two in every 1000 U.S. collegestudents die every year, because of accidental, suicidal, andacute and chronic illness reasons (Turner, Leno, and Keller2013). Among these, campus suicides have almost tripledwithin the last fifty years, and about 18% of undergradu-ates and 5% of graduate students have had lifetime thoughtsof attempting a suicide (Collegian 2017). These alarmingstatistics not only hint at the strains of campus and aca-demic life, every such tragic incident also has widespreadrepercussions by affecting the general psychological well-being of the campus (Wrenn 1999). In fact, some of the

Copyright © 2018, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

most dangerous consequences of such crises include “copy-cat suicides” (when student suicides come in clusters due tosocial contagion effects) and mental health challenges likepost-traumatic stress disorder. Given students already under-utilize mental health care resources due to social stigma,lack of awareness, and the pressures of academic life (Eisen-berg, Golberstein, and Gollust 2007), unanticipated criseslike student deaths bring additional challenges to the mentalhealth amelioration efforts on campuses.

Crisis events on college campuses, such as student deaths,therefore, underscore the necessity to reinforce existing in-tervention programs or undertake new initiatives toward re-ducing the psychological effects of the crisis in the stu-dent community (Blanco et al. 2008). A common approachadopted by campus administrators involves public commu-nication and outreach, promoting information about variousstudent-centric support, coping resources, and counselingservices. Given the pervasive use of web-based technologiesin the college student demography (Pew 2016), these rec-ommendations are often shared via email and social media,also because such communication channels bear the poten-tial to provide a common, stigma-free platform to commentand discuss about the event itself, as well as to grieve andcope. Figure 1 shows an excerpt of one such post shared bya campus administration on Reddit. In this paper, we refer tosuch posts as “counseling recommendations.”

However, significant methodological gaps exist in mea-suring the effectiveness of these post-crisis interventionalrecommendations shared by campus officials (Schwartz andWhitaker 1990). These range from a reliance on retrospec-tive self-reports, to the difficulty in causally determining thelink between exposure to these recommendations and thepsychological states of students following a crisis (DeSte-fano, Mellott, and Petersen 2001).

Our Work. We address the above gaps in examining theefficacy of counseling recommendations following a crisis,in the specific context of student death incidents on collegecampuses, targeting two innovations. First, we use unobtru-sively gathered social media data of college Reddit commu-nities, where these recommendations are shared by campusofficials. Social media helps us track individuals who engagewith these recommendations and what effects they have ontheir psychological states. Then, as a second innovation, wedevelop a causal analysis framework that statistically mod-els the shifts in psychological states characterizing individ-

Figure 1: An excerpt of a counseling recommendation post sharedon r/gatech following the death of a Georgia Tech student.

uals who are exposed to these recommendations, and thosein a control group. As indicators of these changes, drawingfrom natural language analysis (word embeddings), the cri-sis literature, and psychological theories like the “grief workhypothesis” (Schut 1999), we develop the following cate-gories of measures: a) affective changes, specifically aroundthe expression of grief (we model a new “grief lexicon”), b)behavioral changes, and c) cognitive changes.

Focusing on a dataset of ∼400M posts and ∼350K usersspanning 174 college communities on Reddit, our findingsshow that, compared to baseline scenarios, in the after-math of student death incidents, individuals who are ac-tively exposed to the recommendation (via commentary)tend to show statistically significant shifts in their psychoso-cial attributes compared to a matched control cohort who donot engage with the recommendations in the same manner.Examining these changes, we find that the exposed groupdemonstrates greater expressivity of grief, shows signalsof social integration and diversity in interactions, and ex-hibits improved cognitive processing as well as linguisticand stylistic complexity. We situate our findings in the crisisand mental health literatures that associate such shifts with ahealing response, which in turn are indicative of benefits toone’s psychological state. Our work thus provides the firstlarge-scale, (social media) data-driven study of the effectsof post-crisis counseling interventions. We conclude by dis-cussing the implications of our work in supporting improvedmental health policy decisions with social media, followinga devastating crisis in a community.Privacy and Ethics. Given the sensitive nature of this work,despite working with public de-identified data, we are com-mitted to securing the privacy of individuals and the cam-puses included in our dataset. We do not report any informa-tion that can identify a specific person or a student death.

Related WorkCrisis and Mental Health Interventions. According to thesocial amplification theory of risk, crises affect the psycho-logical, social, institutional, and cultural normalcy of lifeamong the exposed individuals and their close ones (Kasper-son et al. 1988). Crisis intervention teams regularly under-take assignments to tackle and prevent mental health prob-

lems in the aftermath of crises (Reijneveld et al. 2003).For example, grief being a natural response to intense sad-ness and distress that ensues many crises, particularly, thedeath of someone close, working with the framework of“grief work hypothesis” (Schut 1999), psychologists oftenrecommend trauma and bereavement intervention therapiesto overcome the emotional upheaval of loss (Cable 1996;Saltzman et al. 2001). However, several studies in psychol-ogy have argued about the effectiveness of such outreach-ing interventions. Some observed that routine referral tocounseling resources following loss lowered anxiety, sup-ported coping and regaining self-esteem, and enabled theindividuals to relate better and look to the future (Currier,Neimeyer, and Berman 2008). In contrast, prior researchalso found that the majority of bereaved people are re-silient enough to adapt without the need of counselors andtherapists, questioning whether the intervention measuresat all have beneficial effects post-crisis (Bonanno 2004;Jordan and Neimeyer 2003).

In addition to this apparent dichotomy regarding the ef-fects of post-crisis interventions, commonly adopted meth-ods, like surveys on mental health service utilization furthersuffer from limitations. They do not capture the short-termdynamics and context of the situation–critical during a crisis,are prone to retrospective recall bias (Tourangeau, Rips, andRasinski 2000), and suffer from compliance, implementa-tion, and scalability issues (Scollon et al. 2009). Specificallyafter a student death, employing a psychological assessmentsurvey that asks delicate questions is difficult due to the sen-sitivity of the situation (De Choudhury et al. 2014).

A sound study design examining the effects of post-crisis counseling interventions should include the possi-bility to differentiate natural change due to coping andresilience from changes attributable to the interventionsthemselves (Schut and Stroebe 2010). Further, to establishwhether an intervention has benefits for an individual’s psy-chological state, requires a comparison between an interven-tion and a non-intervention control group. However, so far,such studies have been severely limited due to the challengein collecting adequate pre- and post-intervention data. Ourwork addresses these gaps by: 1) appropriating a naturalisticsource of data before and after student death crises—socialmedia; and 2) using causal inference techniques, that can in-fer the effects of exposure to counseling recommendationsthat are shared after such crises on college campuses.Social Media, Crisis, and Mental Health. Several stud-ies have demonstrated that analyzing language can helpus understand psychological states relating to an individ-ual’s mental health (Pennebaker and Chung 2007). In re-cent years, linguistic patterns observed on social media havebeen examined in the context of inferring and eventuallyimproving wellbeing (De Choudhury et al. 2013). Com-plementarily, the crisis literature has also found promisingevidence of supporting the potential of web and social me-dia language in better understanding the impacts of naturaland man-made disasters (Mark et al. 2012; De Choudhury,Monroy-Hernandez, and Mark 2014).

Contextually related to our problem, Brubaker et al.(2012) and Glasgow et al. (2014) analyzed social mediadata to understand community grieving following personaland societal tragic events. Specific to college communities,

Saha and De Choudhury recently examined the evolution ofstress following a gun violence incident on campus (Sahaand De Choudhury 2017). This rich body of work moti-vates our choice of social media as a data source and a “pas-sive sensor” to examine the psychosocial changes that ensuestudent deaths on college campuses, and to what extent stu-dents are affected by exposure to outreaching interventionmeans like counseling recommendations.

These studies have, however, largely employed correla-tional techniques, and although they are very insightful,causal approaches are critical to tease out specific influenceson one’s psychological state that are attributable to a treat-ment of interest, in our case, it being counseling recommen-dations. Recently, researchers have drawn on the causal liter-ature to study the impacts of social support and online com-munity participations in helping weight loss (Cunha, Weber,and Pappa 2017) and reducing suicidal risk (De Choud-hury and Kıcıman 2017). Our adoption of a causal infer-ence framework to assess the psychological effects of coun-seling recommendations advances these investigations in anew, unexplored context.

DataFor our study, we use Reddit as our data source. Reddit(reddit.com) is a social news aggregation and discussionwebsite consisting of diverse communities, known as “sub-reddits”, which offer demographical, topical, or interest-specific discussion boards. Subreddits dedicated to collegesare widely prevalent and provide a common portal for stu-dents on the same campus to discuss and share about a vari-ety of issues related to their personal, social, and academiclife. Bagroy et al. (2017) demonstrated that campus subred-dit data well represents the campus population for over 100U.S. colleges and may be utilized as a reliable source of datafor inferring mental wellbeing.

To collect data, with the help of the websites “US News”(usnews.com which lists the top U.S. universities) and“SnoopSnoo” (snoopsnoo.com which groups subredditsinto several categories, one of which is “Universities andColleges”), we first compiled a list of 174 college subred-dits. The largest subreddits on this list, based on subscribercount, include r/UIUC, r/berkeley, r/gatech, and r/UTAustin,which had 13K-19K subscribers as of January 2018.Counseling Recommendations (CR) Dataset. Next, start-ing with a seed list of generic and campus-specific key-words, we first used an iterative snowballing technique tobuild a list of search queries to identify counseling recom-mendation posts in our 174 subreddits: 1) Generic Keywordsare related to death and counseling, such as “death”, “sui-cide”, “counsel*”, “rip”, “therapy”. This list also includesphrases related to email, and positions of responsibility, like“email”, “email dean”, “president”, 2) Campus-specificKeywords are specific to a campus, which we compiled byconsulting the official college websites to obtain names ofthe campus administrators (e.g., president or dean) and thecounseling body. Using these keywords, we queried Red-dit’s search interface for counseling recommendation posts,and manually inspected the returned posts for correctnessin terms of our definition of counseling recommendations.This gave us 88 counseling recommendation posts across 46subreddits, which we denote as the CR dataset.

Baseline Datasets. Additionally, for our research goal—quantifying the psychosocial changes attributable to thecounseling recommendations following student death eventsinstead of other hidden factors (e.g., changes associatedwith active participation in any content shared by campusofficials, exposure to content around non-crisis events, orgeneral interest in counseling-related content), we considerthree other baseline datasets (ref. Table 1).

SD ¬SDC CR B2¬C B3 B1

Table 1: Datasets onstudent death (SD)and counseling rec-ommendation (C).

Baseline Dataset B1 includes an-nouncements from campus officialsunassociated with a crisis (studentdeath) event and without any point-ers to counseling or support resources.E.g., B1 contains posts about non-crisisor non-critical campus events, and ap-pointments or resignations of officials.

Baseline Dataset B2 consists of campus announcementsunassociated with a student death but includes counselingrecommendations that are either routine, or about socio-political issues and policies (e.g., immigration).

Baseline Dataset B3 includes posts that are campus an-nouncements acknowledging a student death but withoutpointers to counseling information.

We acquired these datasets employing similar techniqueas in the case of CR posts—identifying keywords itera-tively (e.g., “sexual”, “violence”, “immigration”, “policy”,or “student affairs”), querying and manually inspecting thecorrectness of returned posts. Eventually, B1 had 229 posts,B2 had 30 posts, and B3 had 1 post across the 46 subredditsin which at least one CR post was present.

Next, using nested queries on the cloud platform, GoogleBigQuery which hosts an entire archive of Reddit data(Bagroy et al. 2017), we obtained the usernames of thoseusers who commented on theCR,B1,B2, andB3 posts. Wealso collected these users’ historical archives (or“timelines”)with all posts. Our paper uses “posts” as one term for postsand comments unless specified otherwise. Additionally, wecollected similar data of 358,871 other users (378,381,052timeline posts), who posted on the campus subreddits, out-side of the CR, B1, B2, and B3 posts. As a measure to re-strict our corpus among those individuals who belong to thesame campus per subreddit, we further pruned our dataset ofany users who posted on more than one campus subreddit.Finally, we identified 842 users and 3,167,266 timeline postsfor theCR dataset, 2,215 users and 6,818,873 timeline postsfor the B1 dataset, 321 users and 1,231,784 timeline postsfor B2, and no users in B3.

MatchingOur ultimate goal is to quantify to what extent a counselingrecommendation shared on Reddit following a student deathincident impacts the psychological states of individuals whoare exposed to it. Answering this question necessitates test-ing for causality in order to eliminate any confounds associ-ated with the observed effects (that is, psychosocial changesof individuals) following a post-crisis intervention (that is, acounseling recommendation shared by campus official aftera student death incident). Causal analysis is also importantbecause the observed effects could simply be a result of thepassing of time, or of people’s ability to heal and cope withthe crisis and gain resilience, and therefore may have little

0.00 0.05 0.10 0.15 0.20—Cohen’s d—

PostsKarma

AffectiveCognitive

PerceptionInterpersonal

TemporalLexical

BiologicalPersonal

Social Act.Act.+Lang.

(a) CR dataset

0.00 0.05 0.10 0.15 0.20—Cohen’s d—

PostsKarma

AffectiveCognitive

PerceptionInterpersonal

TemporalLexical

BiologicalPersonal

Social Act.Act.+Lang.

(b) B1 dataset

0.0 0.1 0.2 0.3 0.4 0.5—Cohen’s d—

PostsKarma

AffectiveCognitive

PerceptionInterpersonal

TemporalLexical

BiologicalPersonal

Social Act.Act.+Lang.

(c) B2 datasetFigure 2: Cohen’s d for balance between Treatment and Controlgroups. Absolute values are mean-aggregated per type.

to do with the counseling recommendations. Therefore, thecrux of our approach is to tease apart the effects that areattributable to the counseling recommendations instead ofother psychosocial changes that follow crisis events.

Ideally, such problems are tackled using RandomizedControlled Trials (RCTs). However, given our data is ob-servational and an RCT is impractical and unethical in ourspecific context involving crises (student deaths) and psy-chological states of individuals, we adopt a causal analysisframework based on statistical matching, which “simulates”an RCT by controlling for as many observed covariates aspossible (Imbens and Rubin 2015). In our case, given thescale of our large dataset (∼400M posts from ∼350K users)and the high dimensionality of the covariates along whichwe intend to match the users, we adopt a two-tier approachthat optimizes for computational efficiency. This includes: 1)Propensity score matching, conditioned on offline and on-line behaviors of users, and 2) Mahalanobis distance com-putation, measured on the linguistic attributes of user posts.Both of these matching techniques are widely adopted in thecausal inference literature (Rosenbaum and Rubin 1985).

Defining Treatment and Control Groups Any causal in-ference framework involves first defining a “treatment”, andthen constructing cohorts which would constitute “treat-ment” and “control” groups. In our problem, treatment con-stitutes exposure to a counseling recommendation. We oper-ationalize it as commenting on a Reddit post that is a coun-seling recommendation. We note that while commentary isa limited way to identify CR exposure and lurkers may alsobe considered exposed, it is a high precision method (thatis, the commenting individuals were definitely exposed tothe counseling recommendation) and is readily measurablefrom our data. We adopt this definition of treatment for allposts in our CR and baseline datasets (B1, B2, and B3).

Next, causal analysis literature (Rosenbaum and Rubin1985) recommends that effects can be appropriately inferredon “treated” users only when we do not observe comparableresults for another randomized group of “control” users un-der similar setting. Accordingly, for each of the datasets, wecategorize two groups of users based on the above-definedtreatment – 1) Treatment group who were commenters intheir respective CR, B1, B2, or B3 posts, and were activeon Reddit before and after it, 2) Control group as a subsetof all other users in the same subreddit, where each memberis a statistical match of one from the Treatment group.

Statistical Matching Approach Our matching strategycontrols for a variety of covariates such that the effects(psychosocial changes) are examined between two groupsof users showing similar overall offline and online behav-ioral and linguistic patterns. 1) First, assuming that our user

Group→ Treatment Control t-testDataset↓ Before After Before After t p

Main (CR) 0.13 0.02 -0.16 0.21 0.97 0.33Baseline (B1) -0.41 0.38 -0.57 0.43 1.07 0.28Baseline (B2) -0.05 0.02 -0.14 -0.04 0.52 0.60

Table 2: Mean z-scores of the number of words posted Before andAfter exposure for the Treatment and Control Groups.pool consists primarily of college students as shown in priorwork (Saha and De Choudhury 2017), we control for userswithin the same campus subreddit. This mostly accounts forany offline behavioral changes attributable to regional, sea-sonal, academic calendar, or other local factors. For onlinebehavioral patterns, we include as covariates the number ofcomments and posts, ‘karma’, and tenure on the platform—similar covariates were used in recent work (Chandrasekha-ran et al. 2017). 2) Second, controlling for the linguistic at-tributes, we use the 50 categories given in the LinguisticInquiry and Word Count (LIWC) lexicon as covariates inour matching model. These categories span across affective,cognitive, lexical, stylistic, and social attributes (Chung andPennebaker 2007). Next, for each dataset CR, B1, B2, B3,our two-tier matching framework proceeds as follows:

1) In the first step, with the offline (subreddit participa-tion) and online behavioral covariates introduced above, wetrained logistic regression classifiers estimating the propen-sity to receive a treatment, called propensity scores (p). Forevery Treatment (Tri) user and their exposure date, wematched on users commenting on the same subreddit withat least one post before and after that exposure date. Next,we obtained the top k (k = 3) most similar users per (Tri)user, conditioning to a maximum caliper distance (c) (withα = 0.2), i.e., | Tri(p)−¬Tri(p) |≤ c, where c = α∗σpooled(σpooled is the pooled standard deviation, and α ≤ 0.2 is rec-ommended for “tight matching”). 2) In the second step ofmatching, per Tri user, we identified the most similar user(Cti) among the top k users, based on the 50 LIWC lexi-cal categories as covariates and adopting the Mahalanobisdistance metric (Rosenbaum and Rubin 1985). Finally, weobtained 821 matched pairs in the CR and 1,754 and 295 inthe B1 and B2 datasets respectively. Note that, since B3 hadno user, we did not include it in our approach and analyses.

Assessing Balance between Groups In order to ensurethat our matching techniques eliminated any imbalance ofcovariates, we used effect size (Cohen’s d) metric to quan-tify the differences in the Treatment and the Control groupsacross each of the covariates. This was performed for theCR dataset as well as the baseline datasets B1 and B2.Lower values of Cohen’s d imply better similarity betweenthe groups, and a value lower than 0.2 indicates “small”differences between the groups (Cohen 1992). Overall, wefound that the two-tier matching approach significantly im-proves covariate imbalance by over 35%, 9%, and 61% afterthe addition of the lexical covariates in the three datasetsCR, B1, and B2 respectively (see Figure 2). This justi-fies the choice of our matching approach that optimizes forcomputational efficiency, at the same time controls for be-havioral and linguistic differences across the Treatment andControl groups in the CR, B1, B2 datasets.

Validating Temporal Confounds. We also assessed thelikelihood of temporal differences in activities of our

matched cohorts. For example, it could be possible that onegroup posts at a higher frequency than the other, whichwould distort the time-aggregated analysis of effects (i.e.,psychosocial changes) we observe across them after the stu-dent death events. For this purpose, we compared the z-scores of the number of words shared by Treatment andControl individuals Before and After the CR (or B1, B2)posts. Quantifying the standardized variation around themean value of a distribution, z-scores, that do not rely on ab-solute values, estimate the relative changes in a time series.Using paired two-tail t-tests, we find that the daily z-scoresfor our Treatment and Control groups in any of the CR,B1, B2 datasets show no statistically significant differences(p > 0.05, see Table 2), revealing that temporal confoundsare unlikely in our ensuing analysis.

Measuring EfficacyNow, we present the measures via which we quantify thepsychological effects of counseling recommendations. Ourmeasures are based on the three core psychosocial constructselucidated in the psychology literature: a) Affective, b) Be-havioral, and c) Cognitive attributes (Breckler 1984). In-spired from the widely adopted “difference in difference”technique in the causal-inference research (Abadie 2005),we estimate the effects of counseling recommendations interms of the changes corresponding to all our psychosocialmeasures in the Treatment and Control groups Before andAfter the date of a specific CR, B1, or B2 post.

Affective Changes Researchers have demonstrated affec-tive variability in individuals following crisis events (Market al. 2012). Our work models affect from the perspectiveof “grief”. Grief is a “response” and a mix of conflictingfeelings and a wide range of strong emotions (James andFriedman 2009). When someone dies, alongside bringingshock, disbelief, and numbness, it leaves friends and rela-tives feeling lost, anxious, depressed, or physically unwell.Grief is the process by which we adjust to the death of some-one close (Saltzman et al. 2001; Wrenn 1999). A rich bodyof literature in psychology, by way of the “grief work hy-pothesis” (Schut 1999) therefore has identified the copingand healing benefits of grieving (Cable 1996), which in turnare associated with achieving timely resilience and returnto normalcy and day-to-day activities following crises. Thusexamining grief as a measure of psychological change fol-lowing CR exposure is extremely relevant in our setting.

While prior work has developed methods to iden-tify affective attributes like mood, emotion, and senti-ment (De Choudhury, Counts, and Gamon 2012; Saha etal. 2017), currently, there are no computational means to in-fer grief from language. Moreover, due to the complexityof grief as an affective construct (note the definition above),gathering high-quality ground truth is challenging. Further-more, in assessing psychosocial changes among individu-als, particularly in response to an environmental stimulus(such as crisis), psychology literature and theories advocatea grounded representation of affect, comprising of not onlythe commonly used valence (pleasantness dimension), butalso the intensity of affect, known as activation. To addressthese challenges, and to obtain a theoretically valid assess-ment of grief around the sharing of counseling recommen-

Word tf -idf Word tf -idf Word tf -idf

thank 14.5 loved 4.65 help 2.82sorry 13.0 husband 3.54 memories 2.72loss 8.95 support 3.34 feelings 2.52remove 6.77 passed 3.25 easier 2.51hope 6.73 hugs 3.21 miss 2.37lost 6.00 beautiful 3.13 son 2.36grief 5.98 sharing 3.15 peace 2.34death 5.84 glad 3.00 cancer 2.30died 5.46 suicide 2.96 comfort 1.91pain 4.94 heart 2.88 sucks 1.79

Table 3: Top 30 n-grams used discriminatingly in reddit grief com-munities. These n-grams were obtained by ranking their Log Like-lihood Ratio (LLR) measures with generic non-mental health com-munities (−1 ≤ LLR < 0), tf -idf values are scaled at 10−2.

Anger: .1

Afraid: .2

Lost: .3

Regretful: .2

Pain: .1

Dead: .1

Time: .5

Suicide: .1

Grief: .4

Death: .1

Sad: .5

Funeral: .1

Alone: .3

Pity: .2

Love: .2Friend: .1Heart: .1Wife: .1Imagine: .5Improve: .1Hug: .1Wish: .2 Wonder: .3Inspire: .2Family: .1Hope: .6Thought: .7Grateful: .4Month: .2Kind: 1

Figure 3: Weighted distribution of affect categories (ANEW) forgrief lexicon on Russel’s circumplex model. Top ANEW categoriesand their standardized tf -idf ([0, 1]) are labeled.

dations, we employ a novel open vocabulary approach of 1)building a grief lexicon; and 2) mapping the words in thegrief lexicon to two affective dimensions, valence and acti-vation, drawing on the established Russell circumplex modelof affect (Posner, Russell, and Peterson 2005).Building a Grief Lexicon. To build a grief lexicon, weadopted an open-vocabulary based transfer learning ap-proach. Transfer learning approaches have been employedrecently in social media studies of health, wherein thedataset under question did not contain labeled data on a tar-get variable of interest (Saha and De Choudhury 2017).In our approach, we leveraged data from 15 subredditsaround the topic of grief, such as r/grief, r/GriefSupport,or r/bereavement, where people engage in sharing their sor-row and grieve about the loss of their loved ones. Fromthese subreddits, we obtained over 50K posts (DG), basedon the archives available on Google’s Big Query. Addition-ally, we obtained a generic Reddit corpus,DR of posts unre-lated to any grief or mental health issues, also used in priorwork (Bagroy, Kumaraguru, and De Choudhury 2017).

Thereafter, we extracted all n-grams (n = 2) from theabove two datasets DG and DR, along with their tf -idfscores. Then, we used Log Likelihood Ratio (LLR) mea-sures to obtain a ranked list of most distinguishing n-gramsacross the two corpuses. LLR for an n-gram is determinedby calculating the logarithm (base 2) of the ratio of its twoprobabilities, following add-1 smoothing. Based on the LLRmeasures, when an n-gram is comparably frequent in boththe datasets, itsLLR is close to 0; it is< 0, when the n-gram

is more frequent inDG, and> 0 for the opposite. Among the4,714 n-grams exhibiting negative LLR, we obtained a listof those 50% of n-grams with the most negative values—weused median as the measure of central tendency here. These2,357 n-grams with a big negative skew in LLR are mostdistinctive of DG, and we refer to them as the “Grief Lex-icon”, LG. Table 3 reports a sample of the top 30 of thesen-grams ranked on their tf -idf scores.Modeling the Affective Dimensions of Grief. Next, to char-acterize the valence and activation dimensions of words inthe above grief lexicon based on the circumplex model, weemployed the widely used word embedding technique to de-rive latent semantic relatedness between words (Mikolov etal. 2013) and the Affective Norms for English Words (ANEW)lexicon (Nielsen 2011). ANEW is an affect dictionary, cu-rated after extensive and rigorous psychometric studies, con-taining a list of over 1,000 affect categories and their quan-tified measures of valence and activation. Prior research hassuccessfully used ANEW to understand expression of moodand affect (De Choudhury, Counts, and Gamon 2012).

For every affect category in ANEW, we obtained its vectorrepresentation in a 300 dimensional word-embedding spaceusing the word2vec model (pre-trained on Google Newsdataset of ∼100B words). Within the word-vector space, se-mantic similarity between any two words can be estimatedwith cosine similarity, using which we mapped all the n-grams in our grief lexicon (LG) to the most similar ANEWcategory (if any, threshold = 0.69 (Rekabsaz, Lupu, andHanbury 2017)) and obtained their valence and activationvalues. Accordingly, 2,357 n-grams from our grief lexiconwere mapped to 459 unique ANEW categories. With theirvalence and activation values as coordinates on an x-y frameand tf -idf as the magnitudes, we modeled our grief lexi-con in the two-dimensional circumplex space of affect (seeFigure 3). We find that expressions across a range of va-lence and activation values occur frequently in grief, e.g.,“kind”, “inspire”, “love”, “anger”, “sad”, “afraid”, and soon. This aligns with the definition of grief (James and Fried-man 2009), and justifies our lexically induced open-datastrategy of modeling grief in the circumplex model of affect.Characterizing Treatment & Control with Grief. With theabove grief lexicon and its 2-dimensional affective model,we quantify the affective expression of grief in the Treat-ment and Control groups around the date of the CR, B1,or B2 posts in their respective datasets. Specifically, withineach of these groups, we obtain all the n-grams and theirtf -idf values before and after the date of post. Applying thesame word-vector based similarity metric described above,we map these n-grams to the most similar grief word and itsvalence and activation value. Then, we compute the meanpercentage change of valence and activation of grief in ourTreatment and Control groups in our datasets.

Behavioral Changes Next, we measure psychosocialchanges in behavior around the date of counseling recom-mendation posts. In the psychology, mental health, and cri-sis literature (De Choudhury, Monroy-Hernandez, and Mark2014), many behaviors including changes in social function-ing and shift of interests can be indicative of an individual’schanging psychological trajectory. We are interested in ob-serving the following changes as effects of exposure to coun-

seling recommendations: Does the user become more activeon Reddit, indicating improved extroversion? Do they par-ticipate in more subreddits, indicating a diversity of interestsand interactions? Do they involve themselves in more dis-cussion threads on Reddit, indicating social engagement? In-spired from prior work (Wise, Hamman, and Thorson 2006),we answer these questions with three metrics, a) activity, orfrequency of posting, b) interaction diversity, that is, numberof unique subreddits they participate in, and c) interactivity,given by computing the number of comments to post ratio.

Cognitive Changes Literature in psychology identifiescognitive attributes as another indicator of an individual’spsychological state (Bandura 1993) —an uptick in wellbe-ing is known to be associated with reduced cognitive im-pairment and improved perceptual processing. Further, psy-cholinguistics literature has revealed the association of lin-guistic structural and stylistic patterns in written communi-cation with cognition (Pennebaker and Chung 2007). Bor-rowing from prior work (Ernala et al. 2017), we adopt thefollowing techniques to examine cognitive changes throughlinguistic syntax, structure, and stylistic vocabulary usage:Coleman-Liau Index (CLI) is a measure of linguistic struc-ture and provides a readability assessment test based oncharacter and word structure within a sentence (Pitler andNenkova 2008). This measure approximates a U.S. gradelevel required to understand the content, and can be calcu-lated with the formula: CLI = 0.0588L − 0.296S − 15.8,where L is the average number of letters per 100 words andS equals the average number of sentences per 100 words.Complexity and Repeatability are syntactic measures thatindicate an individual’s cognitive state in the form of plan-ning, execution, and memory, and are in turn, linked to psy-chological states (Ernala et al. 2017). We quantify complex-ity as the average length of words per sentence, and repeata-bility as the normalized occurrence of non-unique words.LIWC. Linguistic Inquiry and Word Count (LIWC) is awell-validated lexicon that groups words into psycholinguis-tic categories (Pennebaker and Chung 2007). We specifi-cally focus on the normalized occurrences of Cognition &Perception, Linguistic Style, and Social Context categories.

ResultsWe present our results starting with an overview compar-ing the differences between the changes in Before and Aftersamples per dataset, CR, B1, and B2. To evaluate statisti-cal significance of these differences, we conducted Welch’st-test, and adjusted the p-values using False Discovery Rate(FDR) correction. Table 5 gives a summary. We find that formost of the measures, the Treatment and Control groups inB1 andB2 show no statistically significant differences in theBefore and After periods, but that all other measures bar-ring one (Activity) show significant differences in the Treat-ment and Control groups in the CR dataset. This datasetalso shows revealing changes in magnitude for the Treat-ment group, for example – a) for affect, grief expressionsignificantly increases, b) for behavior, increased social en-gagement, interactiveness, and diversity of interests, and c)for cognition, improved cognitive and linguistic processing.

Several studies in psychology and the crisis literature haveassociated greater expressivity whether in terms of the posi-

Data→ CR B1 B2

Measure ↓ ∆Tr ∆Ct ∆Tr ∆Ct ∆Tr ∆Ct

Affective Changes

Grief: Activation 15 -1 – – – –Grief: Valence 9 -1 – – – –Behavioral Changes

Activity – – – – – –Interaction Diversity 9 8 34 27 – –Interactivity 29 -1 – – – –Cognitive Changes

Readability 14 11 3 -1 11 11Complexity 1.3 .7 5 6 .4 .6Repeatability -3 9 1 1.5 .5 3Linguistic Style 481 92 – – – –Cognition & Perception 457 70 – – – –Social Context 382 49 – – – –Table 5: Comparing the mean percentage difference between Be-fore and After periods in the Treatment (Tr) and Control (Ct)groups. Bar lengths represent relative and numbers denote absolutemagnitudes. Blank entries convey no statistical significance.

tivity or intensity of emotionality, bereavement and grief ex-pression, or language with an improvement in their psycho-logical wellbeing status (Klein and Boals 2001). Situatingour results within these studies, we observe that comparedto baseline scenarios, counseling recommendations follow-ing student deaths are succeeded by effects indicative of im-proved wellbeing. In the following subsections we discussin further detail our results on the CR dataset.

Affective ChangesFirst, we examine the affective changes that characterize theTreatment group’s exposure to counseling recommendation.Employing the circumplex representation of grief words, wefind that grief expressions increase considerably (15% forvalence, 9% for activation) in Treatment as compared to amarginal (-1%) decrease in Control (t = 2.68, p < 0.05).Figure 4 plots these changes from the Before to the Afterperiod on the same circumplex model, where larger circlesindicate greater differences for those corresponding grief ex-pression. A closer look at Figure 4(a) reveals that higher dif-ferences are more prominent in the cases where a specificgrief expression increased in the After period. These ex-pressions which show significant changes, belong to all fourquadrants in the circumplex model, such as “friend”, “hope”,“sad”, and “lost”. In contrast, although drawn on the samescale, large circles are scarce in Figure 4(b), suggesting min-imal changes in grief expression in the Control group. Thisobservation affirms that individuals exposed to counselingrecommendations in the CR dataset become more expres-sive from an affective perspective, and this affective expres-sion illustrates grieving as a positive psychological responseto the crisis (Pennebaker and Chung 2007).

Behavioral ChangesWe find that counseling recommendations are associatedwith no significant differences in terms of a user’s post-ing frequency (activity). An alternative interpretation of thisfinding backs our causal analysis that, despite all users con-tinuing usual social media activity before and after the ex-posure to the CR post, the outcome varies for the Treatmentand Control groups for “every” other measure.

Valence

Act

ivat

ion

afraid

alone

anger

dead

death

family

friend

funeral

good

gratefulgrief

heart

hopeinspire

kind

lifelost

love

month

pain

personpity

presentregretful

sad

salutesuicide

thankfulthought

wifewish

Af.>Bf.Bf.>Af.

(a) Treatment

Valence

Act

ivat

ion

afraid

alone

anger

dead

death

family

friend

funeral

good

gratefulgrief

heart

hopeinspire

kind

lifelost

love

month

pain

personpity

presentregretful

sad

salutesuicide

thankfulthought

wife

wish

Af.>Bf.Bf.>Af.

(b) ControlFigure 4: Differences in grief words (from the proposed grief lexi-con) in the Treatment and Control groups, plotted on Russel’s cir-cumplex model of affect. The radius of the circles are proportionalto the mean differences in occurrences of the grief words betweenthe Before and After periods around the date of CR post.

−100 0 100 200 300 400∆ Interaction Diversity %

0.000

0.002

0.004

0.006

0.008

0.010

Pro

port

ion

ofU

sers

ControlTreatment

(a) Interaction Diversity

−100 0 100 200 300 400Interactivity

0.000

0.001

0.002

0.003

0.004

0.005

0.006

Pro

port

ion

ofU

sers

ControlTreatment

(b) InteractivityFigure 5: Distribution of differences of interactivity (comments toposts ratio) and interaction diversity (unique subreddits).

Next, Figure 5 shows the behavioral changes in usersaround the date of sharing of the CR posts. For interactiondiversity, that is, the measure of a user’s engagement acrossmultiple communities, we find similar changes in the Treat-ment and Control group, the former being marginally higherby 1% (t = 4.0, p < 0.05). However for interactivity, a ma-jor increase by 29% occurs in the Treatment cohort, as com-pared to a small -1% change in Control (t = 4.1, p < 0.05).These measures support positive social functioning effectsof CR posts, in turn known to have coping benefits follow-ing loss of someone close (Pennebaker and Chung 2007).

Cognitive ChangesReadability. Within the Treatment group in the CR dataset,we find a mean increase of 14% in the Coleman-Liau Index(CLI) following exposure to the counseling recommenda-tions. Although this number is close to the changes in Con-trol group (11%), we observe statistically significant differ-ences (t = −81, p < 0.05) between the two groups. Sinceboth groups of users were statistically matched on their over-all linguistic usage, and are alike in their educational quali-fication (college students), a comparable overall increase inreadability is unsurprising, especially because this measuretypically increases with writing over the years (Pitler andNenkova 2008). To illustrate this observation further, we ob-tained the probability density function (with Gaussian ker-nel) of CLI in the Before and After periods of exposure toCR posts, for the Treatment and Control cohorts (Figure 6).

0 2 4 6 8 10 12 14 16 18Coleman Liau Index

0.00

0.05

0.10

0.15

0.20

0.25P

ropo

rtio

nof

Use

rsBeforeAfter

0 2 4 6 8 10 12 14 16 18Coleman Liau Index

0.00

0.05

0.10

0.15

0.20

0.25

Pro

port

ion

ofU

sers

BeforeAfter

Figure 6: Distribution of Readability (CLI) in the Treatment (left)and Control groups (right), Before and After the CR post.

This figure shows that the distribution of the CLI measurechanged considerably for the Treatment group, and no sucheffect is observable in the Control group. Specifically, thevariance of distribution in Treatment cohort reduced sub-stantially by 90% (σ decreased from 6.1 to 1.9) after CRpost exposure. Increased readability of written speech isknown to indicate better control over the train of thought,better coherence in expressing ideas, and better discourse or-ganization (Thorndyke 1977). That such increases manifestin the Treatment group after exposure to CR posts furtherindicate psychological effects around improved wellbeing.Repeatability and Complexity. Figure 7 shows the Afterand Before differences in linguistic repeatability and com-plexity in the Treatment and Control groups following expo-sure to CR posts. For repeatability, the figure reveals that agreater fraction of Treatment users show negative and near-zero changes (MdnTreatment = −2 vs. MdnControl = 8), thatis their linguistic repeatability decreases. In addition to sta-tistically significant differences (t = 11.3, p < 0.05), wefind that while repeatability decreases by 3% for Treatmentusers, it increases by 9% for Control users. For complex-ity, Treatment users demonstrate over 80% increase com-pared to the Control users (1.3% vs. 0.7%). Although nu-merically the change is small, statistical significance tests(t = 18.6, p < 0.05) show compared to a linguisticallymatched Control population, the Treatment users show agreater increase in the usage of longer words. Mental healthchallenges can manifest in the form of poverty of speech, areaccompanied by a reduction in syntactic complexity, and animpairment in syntactic comprehension (Ernala et al. 2017).Such tendencies typically result from an overall cognitivedeficit, difficulty concentrating, distraction, or a preferencefor expressing simpler ideas. As repeatability and complex-ity capture such syntactic attributes in Reddit posts, reduc-tion in repeatability and increase in complexity followingCR post exposure are, therefore, indicative of positive psy-chological changes in the Treatment cohort.Cognition & Perception, Linguistic Style, Social Context.Finally, analyzing the normalized occurrences of LIWC cat-egories for linguistic style, cognition, and social context,we observe interesting patterns. Figure 8 shows the vari-ability (95% confidence interval) of differences for statisti-cally significant LIWC categories. We find that for all of thecategories, the Treatment dataset shows significantly highervariability than the Control . As all of these plots lie on thepositive y-axis, we further infer that levels of cognitive mea-sures increased following exposure to the CR posts.

−100 0 100 200 300∆ Repeatability %

0.000

0.002

0.004

0.006

0.008

0.010

0.012

Pro

port

ion

ofU

sers

ControlTreatment

(a) Repeatability

−10 0 10 20 30∆ Complexity %

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

Pro

port

ion

ofU

sers

ControlTreatment

(b) ComplexityFigure 7: Distribution of differences in repeatability and complex-ity in Treatment and Control groups.

Cognition and Perception Linguistic Style Social Context

***C

ausa

tion

*Cer

tain

ty*C

ogni

tive

Mec

h.**

*Inh

ibiti

on**

*Dis

crep

anci

es**

*Neg

atio

n**

*Ten

tativ

enes

s*F

eel

***H

ear

***I

nsig

ht**

*Per

cept

*See

***1

stP.

Sin

gula

r**

*1st

P.P

lura

l*2

ndP.

*3rd

P.*I

ndef

.P

rono

un**

*Fut

ure

Tens

e**

*Pas

tTen

se*P

rese

ntTe

nse

*Adv

erbs

*Art

icle

*Ver

bs**

*Aux

.Ve

rbs

*Con

junc

tion

***E

xclu

sive

*Inc

lusi

ve*P

repo

sitio

n*Q

uant

ifier

***R

elat

ive

*Fam

ily**

*Frie

nds

***H

uman

s*S

exua

l**

*Soc

ial

***W

ork

0

5

10

15

Mea

ndi

ff.((

Af.-

Bf.)

/Bf.)⇤1

00 ControlTreatment

Figure 8: Differences in cognitive measures between the Treatmentand Control groups following CR exposure, based on usage ofLIWC categories. The vertical lines represents 95% confidence in-terval range, and the dot shows the mean. Statistical significance isreported based on Welch t-test. p-values are adjusted using FDRcorrection (∗p < 0.05, ∗ ∗ 0.001 < p < 0.01, ∗ ∗ ∗p < 0.001).

We find that cognitive measures, such as “causation”,“cognitive mechanics” and “tentativeness” significantly in-crease after the exposure to CR posts. Per prior work, thisindicates an improvement in an individual’s cognitive func-tioning (Pennebaker and Chung 2007). Additionally, greaterusage of “negation”, and words relating to “feel” and “per-cept” indicate greater perceptual expressiveness, known tobe associated with first-hand accounts of the real world hap-penings, events, and experiences (Brubaker et al. 2012).

Likewise, within linguistic style measures, we find reveal-ing changes, such as pronouns (1st, 2nd, and 3rd) and tempo-ral attributes increase considerably (mean difference=∼5) inthe Treatment dataset. Both psycholinguistics and crisis lit-erature note that 1st person and past tense usage relate withnarrating personal or collective experiences of upheavals,which seems likely in our case (Mark et al. 2012). Priorwork also notes the higher usage of 2nd person pronounsin the aftermath of crises and 3rd person pronoun use is as-sociated with the language of adaptive and coping relatedhealth benefits following crises. Further, the increased usageof lexical density features such as “adverbs”, “articles”, and“quantifiers” indicate that Treatment users express via morecomplex narratives (Chung and Pennebaker 2007)—a signalof better psychological health (Ernala et al. 2017). Amongthe social context measures, treated users use more “family”and “friends” words. Based on prior work, this is a known

behavior for individuals coping with grief and trauma, andreference to socialization has therapeutic benefits for an in-dividual’s psychological state (Seeman 1996).

Discussion and ConclusionSummary. We demonstrate that with a novel causal analysisframework and unobtrusively gathered social media data, itis possible to quantify, to what extent exposure to counsel-ing recommendations following a student death on a col-lege campus positively impacts an individual’s psychologi-cal state. Our work, therefore, bears the potential to comple-ment existing techniques of assessing the effectiveness ofintervention measures deployed after crises. In this way, weadvance the growing body of research in social media andhealth, opening up new avenues of addressing health chal-lenges by employing social media as a mechanism of sup-portive mental health and crisis intervention delivery.

Using a Reddit dataset of 174 campus communities and∼400M posts from ∼350K users, we observe statisticallysignificant psychosocial (affective, behavioral, cognitive) ef-fects of exposure to counseling recommendations on thetreated population as compared to a statistically matchedcontrol cohort. In assessing these psychosocial effects, ourcausal inference framework allowed us to account for be-havioral and linguistic covariates across the treatment andcontrol groups, also eliminating confounds due to tempo-ral variability in their Reddit activity. Further, by comparingagainst baseline scenarios, our approach reveals that the ob-served effects were characteristic of the specific context ofstudent death related crises, instead of other latent factors.

A contribution of our work is a “grief lexicon” and a trans-fer learning based methodology to build it. Drawing on re-cent advances in computational linguistics research, we ex-panded a validated affect dictionary with word embeddingsand employed it on public social media data. Our techniquecan be used in other social media and health research thatinvolves extracting domain-specific information, but whereground truth data is limited and unlabeled data is plenty.Implications. Our findings provide support for the “griefwork hypothesis” (Schut 1999), that situates grief counsel-ing and therapy as a way of working through loss. In ourtreatment group, following student death incidents, we findevidence of greater affective expressivity of grief, the greaterdesire for social connectedness and diversity in interactions,improved cognitive and perceptual processing, and emergentlinguistic and stylistic complexity. Based on psychology andcrisis literature around the healing and coping benefits ofgrieving (James and Friedman 2009), our results indicatethat exposure to counseling recommendations on social me-dia after crisis events, signals effects associated with positivebenefits for one’s psychological state.

We believe our findings are not only useful in helpinggauge whether sharing counseling recommendations on so-cial media are at all effective, but also can support crisis re-habilitation efforts on college campuses. Campus officialscan utilize the outcomes of our work as a way to identifyindividuals who are not benefiting from these counselingrecommendations. This can help them employ other proac-tive intervention measures to support their mental health.Broadly, our work can inform campus policy decisionsaround mental health outreach. Our work also sheds light

into the role of communication technologies like social me-dia, in supporting these efforts, both during crises as well asto tackle college student mental health challenges.Limitations And Future Work. While our findings are in-dicative of the positive benefits of exposure to counselingrecommendations, we cannot make broad claims about theefficacy of these recommendations in improving the men-tal wellbeing of the entire college campus. Our findings arelimited to only those individuals who chose to explicitly en-gage with the CR posts via Reddit commentary. Thus ourobservations suffer from a self-selection bias. It is possiblethat students were exposed to the information via alternativemeans (e.g., word-of-mouth) and that some availed coun-seling services independent of exposure to such post-crisisoutreach. Since these are not observable to us, our resultsshould be interpreted with caution. Multi-prong data gath-ering approaches used in prior crisis informatics work (DeChoudhury et al. 2014) are a potential solution.

Our results do not indicate whether the individuals whoengaged with the counseling recommendations actuallyavailed counseling services. We cannot be certain if the pos-itive psychosocial shifts we see are a consequence of someform of therapy or other measures they adopted to cope withthe impacts of the events. Nevertheless, our causal analysisdoes indicate positive effects on psychological wellbeing inthe treatment cohort compared to a control. This suggeststhat irrespective of the mechanisms of counseling or supportadopted, exposure to counseling recommendations on socialmedia largely yields positive psychological outcomes.

Another limitation of the work is the lack of data ona true control that encompasses student death incidents incampuses without any shared counseling recommendation.While the creation of such a true baseline is ethically ques-tionable (debarring some students from help resources whilesome others benefit from it), future work can investigateother means to create an appropriate control through part-nerships with student health services on a campus.

Finally, in the individuals exposed to the counseling rec-ommendations, it is promising to see signs of healing andcoping, which in turn indicate that they might be returningto normalcy and achieving resilience in the aftermath of thestudent death incidents. However, in the absence of groundtruth clinical assessments, we cannot claim that these psy-chosocial shifts imply clinically meaningful changes in themental health of the exposed individuals. Future work canaugment our analyses with self-reported or counseling ser-vice utilization data to assess the post-crisis clinical efficacyof the counseling recommendations on college campuses.

AcknowledgementsWe thank Sindhu Ernala and Eshwar Chandrasekharan fortheir valuable feedback. Saha and De Choudhury were partlysupported by NIH grant #R01GM112697.

ReferencesAbadie, A. 2005. Semiparametric difference-in-differences esti-mators. The Review of Economic Studies 72(1):1–19.Bagroy, S.; Kumaraguru, P.; and De Choudhury, M. 2017. A socialmedia based index of mental well-being in college campuses.Bandura, A. 1993. Perceived self-efficacy in cognitive develop-ment and functioning. Educational psychologist.

Blanco, C.; Okuda, M.; Wright, C.; Hasin, D. S.; Grant, B. F.; Liu,S.-M.; and Olfson, M. 2008. Mental health of college students andtheir non–college-attending peers: results from the national epi-demiologic study on alcohol and related conditions. Archives ofgeneral psychiatry.Bonanno, G. A. 2004. Loss, trauma, and human resilience: havewe underestimated the human capacity to thrive after extremelyaversive events? American psychologist 59(1):20.Breckler, S. J. 1984. Empirical validation of affect, behavior, andcognition as distinct components of attitude. J. Pers. Soc. Psychol.Brubaker, J. R.; Kivran-Swaine, F.; Taber, L.; and Hayes, G. R.2012. Grief-stricken in a crowd: The language of bereavement anddistress in social media. In ICWSM.Cable, D. 1996. Grief counseling for survivors of traumatic loss.Chandrasekharan, E.; Pavalanathan, U.; Srinivasan, A.; Glynn, A.;Eisenstein, J.; and Gilbert, E. 2017. You can’t stay here: The ef-ficacy of reddit’s 2015 ban examined through hate speech. Proc.ACM Hum.-Comput. Interact. (CSCW):31:1–31:22.Chung, C., and Pennebaker, J. W. 2007. The psychological func-tions of function words. Social communication 343–359.Cohen, J. 1992. Statistical power analysis. Curr. Dir. Psychol. Sci.Collegian. 2017. tinyurl.com/yb4hhus4. Acc: 2017-10-26.Cunha, T.; Weber, I.; and Pappa, G. 2017. A warm welcome mat-ters!: The link between social feedback and weight loss in/r/loseit.In WWW. IW3C2.Currier, J. M.; Neimeyer, R. A.; and Berman, J. S. 2008. The effec-tiveness of psychotherapeutic interventions for bereaved persons: acomprehensive quantitative review. Psychological bulletin.De Choudhury, M., and Kıcıman, E. 2017. The language of socialsupport in social media and its effect on suicidal ideation risk. InICWSM, volume 2017. NIH Public Access.De Choudhury, M.; Gamon, M.; Counts, S.; and Horvitz, E. 2013.Predicting depression via social media. In ICWSM.De Choudhury, M.; Counts, S.; and Gamon, M. 2012. Not allmoods are created equal! exploring human emotional states in so-cial media. In ICWSM.De Choudhury, M.; Monroy-Hernandez, A.; and Mark, G. 2014.Narco emotions: affect and desensitization in social media duringthe mexican drug war. In CHI. ACM.DeStefano, T. J.; Mellott, R. N.; and Petersen, J. D. 2001. A prelim-inary assessment of the impact of counseling on student adjustmentto college. Journal of College Counseling 4(2).Eisenberg, D.; Golberstein, E.; and Gollust, S. E. 2007. Help-seeking and access to mental health care in a university studentpopulation. Medical care 45(7):594–601.Ernala, S. K.; Rizvi, A. F.; Birnbaum, M. L.; Kane, J. M.; andDe Choudhury, M. 2017. Linguistic markers indicating therapeuticoutcomes of social media disclosures of schizophrenia. Proc. ACMHum.-Comput. Interact. (CSCW):43:1–43:27.Glasgow, K.; Fink, C.; and Boyd-Graber, J. L. 2014. “ our grief isunspeakable”: Automatically measuring the community impact ofa tragedy. In ICWSM.Imbens, G. W., and Rubin, D. B. 2015. Causal inference in statis-tics, social, and biomedical sciences. Cambridge University Press.James, J. W., and Friedman, R. 2009. The Grief Recovery Hand-book, The Action Program for Moving Beyond Death, Divorce, andOther Losses including Health, Career, and Faith. Harper Collins.Jordan, J. R., and Neimeyer, R. A. 2003. Does grief counselingwork? Death studies 27(9):765–786.Kasperson, R. E.; Renn, O.; Slovic, P.; Brown, H. S.; Emel, J.;Goble, R.; Kasperson, J. X.; and Ratick, S. 1988. The social am-plification of risk: A conceptual framework. Risk analysis.

Klein, K., and Boals, A. 2001. Expressive writing can increaseworking memory capacity. J. Exp. Psychol. Gen.Mark, G.; Bagdouri, M.; Palen, L.; Martin, J.; Al-Ani, B.; and An-derson, K. 2012. Blogs as a collective war diary. In CSCW.Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J.2013. Distributed representations of words and phrases and theircompositionality. In NIPS.Nielsen, F. A. 2011. A new ANEW: evaluation of a word listfor sentiment analysis in microblogs. In Proceedings of the 2011ESWC Workshop on ‘Making Sense of Microposts’, 93–98.Pennebaker, J. W., and Chung, C. K. 2007. Expressive writing,emotional upheavals, and health. Handbook of health psychology.Pew. 2016. tinyurl.com/y6v8j6p8. Acc: 2017-10-30.Pitler, E., and Nenkova, A. 2008. Revisiting readability: A unifiedframework for predicting text quality. In EMNLP. ACL.Posner, J.; Russell, J. A.; and Peterson, B. S. 2005. The circumplexmodel of affect: An integrative approach to affective neuroscience,cognitive development, and psychopathology. Dev. Psychopathol.Reijneveld, S. A.; Crone, M. R.; Verhulst, F. C.; and Verloove-Vanhorick, S. P. 2003. The effect of a severe disaster on the mentalhealth of adolescents: a controlled study. The Lancet.Rekabsaz, N.; Lupu, M.; and Hanbury, A. 2017. Exploration of athreshold for similarity based on uncertainty in word embedding.In ECIR. Springer.Rosenbaum, P. R., and Rubin, D. B. 1985. Constructing a controlgroup using multivariate matched sampling methods that incorpo-rate the propensity score. The American Statistician.Saha, K., and De Choudhury, M. 2017. Modeling stress with socialmedia around incidents of gun violence on college campuses. Proc.ACM Hum.-Comput. Interact. (CSCW):92:1–92:27.Saha, K.; Chan, L.; De Barbaro, K.; Abowd, G. D.; and De Choud-hury, M. 2017. Inferring mood instability on social media by lever-aging ecological momentary assessments. Proc. ACM IMWUT.Saltzman, W. R.; Pynoos, R. S.; Layne, C. M.; Steinberg, A. M.;and Aisenberg, E. 2001. Trauma-and grief-focused intervention foradolescents exposed to community violence: Results of a school-based screening and group treatment protocol.Schut, H., and Stroebe, M. 2010. Effects of support, counsellingand therapy before and after the loss: can we really help bereavedpeople? Psychologica Belgica.Schut, Margaret Stroebe, H. 1999. The dual process model of cop-ing with bereavement: Rationale and description. Death studies.Schwartz, A. J., and Whitaker, L. C. 1990. Suicide among collegestudents: Assessment, treatment, and intervention.Scollon, C. N.; Prieto, C.-K.; and Diener, E. 2009. Experience sam-pling: promises and pitfalls, strength and weaknesses. In Assessingwell-being.Seeman, T. E. 1996. Social ties and health: The benefits of socialintegration. Annals of epidemiology 6(5):442–451.Swan, J., and Hamilton, P. M. 2017. Mental health crises.Thorndyke, P. W. 1977. Cognitive structures in comprehension andmemory of narrative discourse. Cognitive psychology.Tourangeau, R.; Rips, L. J.; and Rasinski, K. 2000. The psychologyof survey response. Cambridge University Press.Turner, J. C.; Leno, E. V.; and Keller, A. 2013. Causes of mortalityamong american college students: a pilot study. J. col. student psy.Wise, K.; Hamman, B.; and Thorson, K. 2006. Moderation, re-sponse rate, and message interactivity: Features of online commu-nities and their effects on intent to participate. JCMC.Wrenn, R. 1999. The grieving college student. Living with grief:At work, at school, at worship 131–141.