SplunkLive! New York Dec 2012 - SNAP Interactive

High Velocity IntelligenceApplication Monitoring with Splunk

SNAP Interactive, Inc.

Presented by:

Nicholas DiSantoArchitecture Team Lead

www.snap-interactive.com

Company Overview•SNAP Interactive, Inc.

•www.AreYouInterested.com

•Believes it is one of the largest social discovery platforms on the web (based on monthly active users)

•More than 5 million monthly active users

•Over 1 billion total pieces of structured data from its users

•Synced to millions of Facebook profiles

•Receives over 1,000 real-time updates per minute on like actions from Facebook

•Subscription-based business model

•SNAP is publicly traded - Ticker: STVI


About Nick

•Developing on LAMP stack at tech startups for 10 years

•Leading a team of core engineers

•Passionate about experimentation & data driven iteration

•Striving to eliminate all technical blockers to speed and innovation

•@NicholasDiSanto


Summary

•We use splunk for many, many things!

•Today, I will share some of our more interesting applications

•How we get data into splunk

•What we do with that data

•Various types of monitoring

•Extensive user behavior analysis


What We Give Splunk

•Custom application logs

•Structured, minified, event data

•De-normalized user demographics

•Application profiling data

•Error logs


Sending Splunk Data

•Centralize logging functions and:

•Format arbitrary structured data into splunk extractable field/value pairs: field=”value”

•Normalize and minify field names

•Detect user_id and augment logs

•Optionally log a percent of events

•Target different log files (error, info, analytics)


User Demographics

•Our analytics log contains application events, triggered by real users

•We augment these event logs with useful demographic data to classify the events

✦ Gender

✦ Seeking gender

✦ Country

✦ Ethnicity

✦ Date of birth

✦ Date last email click

✦ Date joined

✦ Date last email open

✦ Date last site visit

✦ AB test variant


Demographic Power

•By augmenting event logs with user demographics we can perform powerful and detailed analysis of user behavior

•Target analysis at countries, genders, or age ranges

•Classify events by days since: registration, login, email open, etc.

•...and much more


Performance Metrics

•We time key algorithms in our application, and log:

•server name

•query name

•time spent working

•This lets us graph the average, min and max times of these algorithms per server

•We also dark launch features, benchmarking performance prior to official launch.


Performance

•Average query time for key algorithms by server


What Splunk Gives Us

•Monitoring - to measure application health

•Analysis - to drive future product decisions

•AB test evaluation - to validate hypotheses

•Detection - to find patterns & classify users


Monitoring

•With continuous deployment, detailed monitoring is absolutely essential.

•Each deploy we watch changes in:

•Realtime classified error graphs

•Core event stat graphs

•We also monitor email deliverability, revenue, and performance (although not every deploy)


Error & Event Monitoring•Alert us immediately after deploy if something has gone wrong

•Use realtime background searches

•Single dashboard with multiple graphs and tables

•We are exploring realtime sms alerts to the ‘developer on call’

•Use historial data to identify min/max expected thresholds(weighted averages: same time of day, same day of week)

•Detect consistent deviations and alert


Error Monitoring

•Count of all errors : past 30 seconds


Error Monitoring

•All errors: past 5 minutes w/deploys


Error Monitoring

•Rolled up errors: past 5 minutes


Error Monitoring

•Rolled up filtered errors: past 5 minutes


Error Monitoring

•Rolled up JS errors: past 3 hours


Event Monitoring•We monitor ~20 event stats, in realtime, each deploy


Event Monitoring

•Overview and detail views, powered by a single realtime background search


Monitor Email & Performance

•Email

•Deliverability is essential to business

•Need to maximize engagement

•Performance

•What async jobs may be contributing to high DB load?

•What performance are end users experiencing?

•Are particular servers overloaded?


Email Deliverability

•Overview of key metrics


Email Monitoring

•Inserts into email scheduled send queue


Performance

•Asynchronous process timers

•Can correlate spikes with site issues


Analysis

•We heavily leverage summary indexing for performance gains

•Daily rollups are grouped judiciously, giving us fast, flexible, analysis over long periods

•We summarize: revenue, email deliverability, core KPI, and general stats data

•Custom dashboards facilitate easy searching

•Lots of ad hoc searching by product team


Email Analysis•Sends opens, clicks, bounces & FBL rates by email type


Email Analysis

•Monitor changes in open & click rates by email, ISP, country, etc.


Email Analysis

•Analysis dashboard *Sample data


Core KPI Dashboard

•Powerful targeted cohort analysis*Sample data


AB Test Results

•We are constantly running a variety of AB experiments on our live users

•We divide our user population in to nine 10% segments and ten 1% segments

•Each segment can be targeted with an experiment

•All event logs are annotated with the appropriate AB experiment name

•This allows us to measure behavior changes between experiment and control groups


Easy AB Analysis

•All event logs contain the an AB field

•This identifies the experiment group of the user at that point in time

•Fully integrated into core analysis dashboards

•Ad hoc analysis becomes simple

`my_search` (AB=my_test OR AB=ctrl)| `my_reporting_command` by AB


AB Dashboards

•key metrics: experiment vs control group


Latest Splunking - Detection

•The way our users interact with one another is insightful

•We can use this data to classify users:

•Identifying “attractive” users

•Identifying spammers & scammers

•We test hypotheses with ad hoc searches

•Find reliable patterns then setup scheduled searches that interface with MySQL

•This data then feeds into our application in various ways


Contact Us•SNAP Interactive, Inc.


•Nicholas DiSantoArchitecture Team [email protected] @NicholasDiSanto

•Lindsay BubbicoFinn Partners (Public Relations Associate)[email protected]@LindsayBubbico

SplunkLive! New York Dec 2012 - SNAP Interactive

Business

Transcript of SplunkLive! New York Dec 2012 - SNAP Interactive