Copyright © 2012 Splunk, Inc.

Data-Driven Security:Managing Risk at EtsyNick Galbreath @ngalbreathDirector of Engineering - EtsyGartner Security SummitNational Harbor, MDJune 12, 2012

Agenda

Who am I? Who is Etsy? Splunk at Etsy?

Web Application Security

Account Takeover

Payments and PCI

Credits, Data, Further Reading

2

3

@ngalbreath

http://slidesha.re/KPvHYu

Whois Nick GalbreathDirector of Engineering at Etsy covering:– Fraud– Security– Support Engineering– (and other stuff outside of this talk)

Software Development background inE-Commerce and Social MediaBooks, Patents, Oh My… http://client9.com/

4

2012-06-12

is my

two year

anniversary

at Etsy

2012-06-12

is my

two year

anniversary

at Etsy

5

$525,000,000 in community sales875,000 active sellers41MM unique visitors15MM registered members150 countries

What Could Possibly Go Wrong?• Marketplace Risk like Big Auction Site• Payment Risk like Payments Company• Social Risk like that Big Social Network

With a member base frequently:• New to Etsy• New to Running a Business• New to the Internet

6

Photo Credit: Rod Ramsey http://bit.ly/KnI8uB

To Make It More Interesting:Continuous Deployment

On average, there are 50+ production code changes per day. So when we have a problem:Is it an operations problem?Is it a development problem?Is it a product problem causing

complaints to come in?Or is it an attack?

7

Learn more http://bit.ly/KFYYlZ

Learn more http://bit.ly/KFYYlZ

Old Workflow: #notwinningLogging into production network (!)Finding the right fileUnzipping the right fileGrepping

• Writing very clever scripts to extract data• Writing more clever scripts to merge data• Making a report – in plain text• Alerting

8

34 minutes for one day’s log for nothing!

34 minutes for one day’s log for nothing!

Splunk installed at Etsy mid-2010

9

Serious New Technology FatigueWhy don’t we use a Real Database with SQL?Grep technology works&^#*&@^*#^%^ YAQL – Yet Another Query Language

"Hey. .. let's go try this NEW thing!" (door slamming shut)

"Sorry.... we're closed.”Steve Martin. Comedy is not pretty. 1979. Track 8 ~2:45

L’OutrageThen a colleague:• Didn’t know Etsy’s stack (new)• Remote and out of office• Didn’t have production access• Didn’t know any of my very clever scripts• Not experienced with Splunk• In about 30 minutes

whips up a real-time email alert for a velocity check on a particular URL

10

I only have one thing to say about this…..

OH, YEAAHH!400+GB indexed per day30+ TB total storage60+ data sources from“hundreds of servers” (via central syslog aggregation)

11

Data-Driven Security

12

Data-Driven By Mat Edelson. John Hopkins Engineering Magazine, Fall 2011http://eng.jhu.edu/wse/magazine-fall-11/item/data-driven/Illustration by Mark McGinnis No association, just a great article & illustration

Three examples of how we use data and Splunk to help make Etsy a safer place to conduct business.

•Web Application Security•Account Takeover•Payments and PCI

That said we are barely scratching the surface of Splunk!

WebApp Security

Make Security Visible

14

Your peers actually are interested in security. But are you letting them?

Turn security from a binary event intoa continuous event.

Detect the Steps

15

A journey of a thousand miles begins with a single step. Lao-tzu, China 600BC

A single breach begins with a journey of a thousand steps.Nick Galbreath, USA 2012AD

SQLi, XSS, CSRF

16

source=“info.log” log_name_space=“SECURITY” attacktype=“XSS”

That was easy

That was easy

SQLi, XSS, CRSF

17

source=“info.log” log_name_space=“SECURITY” attacktype=“SQL”| geoip ip

Paints a different picture

Paints a different picture

The Dumbest Check Possible for SQLiWe have some snazzy technology for detecting SQLi in Splunk, but you don’t need it to get started:

source=access.log

(uri="*UNION+ALL*" OR uri="*UNION%20ALL*”)

Will wildly undercount but also low false positive rateWill detect scans from various toolsWill get you started in making security visible

18

SQLi and Database Errors

19

• We use Splunk to alert on any database syntax errors too.• SQLi attacks and probes will likely trigger a bust of syntax errors

if code doesn’t properly sanitize data

That was

closeThat was

close

source="error.log" ( "syntax error" NOT "smarty" NOT "ClientLogger" ) | eval event=_raw | table event'

Do the same with server 500 errors,

core dumps

Investigating Rent-A-CPU Trafficsource=“access.log”| lookup datacenter-cidrs provider_cidr AS true_client_ip OUTPUTNEW provider_name | where isnotnull(provider_name) | top provider_name

20

Public DataSee Appendix

Public DataSee Appendix

SANS ISC 10K Sources

21

source=“access.log” | where isnotnull(true_client_ip) | lookup isc-bad-ips src_ip AS true_client_ip | where isnotnull(rank) | table true_client_ip, rank, reports, attacks, last_seen | stats count by true_client_ip,rank | sort rank

Public DataSee Appendix

Public DataSee Appendix

Attacker-Driven Testing

Attacker-driver testing augments Etsy’s proactive security measuresSplunk alerts us on potential attacks using a number of parametersWhat URLs are being targeted? Maybe they found something?Can it be reproduced? (sometimes completely automated validation)Fixes can be pushed out that day, if not within minutes.

22

“I thought I found something but then it stopped working…” Email to security-reports@etsy.com from ethical hacker

Security Post-MortemsFor any security vulnerability, found either external or internal, exploited or not, we hold “blameless post-mortems”Use to teach about security issues

e.g. review OWASP Top 10 http://bit.ly/fXsJg6Can we make it so this mistake doesn’t happen again or can be automatically detected?A Key to post-mortem is know when something started and when it ended. Logs “at your fingertips” via Splunk helps greatly (and absolutely essential for actual incidents)

23

Account Takeover

Account Takeover

25

• Stolen credentials• Brute forcing of credentials• Using account takeover of email to

further takeover other accounts

Horrible for victim and really slow to clean up

Many Users Failing to Sign-in from One IP

26

'source=“info.log” log_namespace=“login” reason="wrong password” true_client_ip!=38.117.156.XXX| dedup etsy_username,true_client_ip | transaction true_client_ip | where eventcount > XXXX| table true_client_ip,etsy_username| geoip true_client_ip | table true_client_ip,true_client_ip_countryname,etsy_username'

Brute Forcing Passwords?source=”info.log”

log_namespace="login”

reason="wrong password"

true_client_ip!=38.117.156.XXX

| transaction etsy_username

| where eventcount > XXXX

| table etsy_username,true_client_ip,eventcount

| sort -eventcount

27

Frequency Buckets set in Splunk Dashboard

People will try 100 passwordsmanually

People will try 100 passwordsmanually

I Forgot My Password x1000

28

source=“/web/access.log” request_uri=/forgot_password.php http_method=POST | transaction true_client_ip | where eventcount > XXX| table true_client_ip,eventcount| sort –eventcount Hello from

Serbia!Hello from

Serbia!

Not just fraud… has disclosed problems in email transport and product problems with our reset flow

Apply the same analysis to other things that should not change much

– Payment cards– Email addresses– Passwords (successful change)– Regular physical addresses

29

CAPTCHA

30

Splunk 2x2 dashboard keeps us in-the-know on how often CAPTCHAs are being shown, to whom, and how often they pass.

reCAPTCHA http://www.google.com/recaptcha

Integrated into Support ToolsSplunk is glued into our internal tools used by General Support and MITS (Marketplace Integrity / Trust & Safety) teams.

31

Payments and PCI

Payments @ EtsyRamping up on our own payments platformFull PCI EnvironmentWith separate Splunk installation

33

This space intentionally left blank.

Alerting on Unusual Payment ActivityAll the WebApp security and account take- over rules apply, along with special checks for payment activityAbnormally large paymentsPayment velocityVery small payments (skimming?)The usual IP address checks.

34

Part of a largerpayment risk solution

Part of a largerpayment risk solution

Compliance and Reporting

Instead of building custom applications with fuzzy requirements“Log it, let Splunk figure it out later” Even the business guys can use it for ad-hoc queries. Unexpected side effect: removing and/or changing data is really hard. This is good. Compare to SQL. (Splunk also has a secure log system)Easy to make reportsPCI QSA so far says this meets PCI requirements.

35

Internal RiskAgain, instead of build out of new application (with fuzzy requirements) Log It, Splunk it later.Who, is what making what changesWho is looking at potentially sensitive dataAnd alert on it.Used in payments and main support applications

36

Etsy Support and MITS 2012100% Good Eggs

Team Etsy 2012

Credits

Acknowledgements

This presentation would not be possible without the hard work by:

Big thanks to everyone at Etsy in Engineering, Payments, Operations, Support and MITSAnd of course, the fine folks at Splunk!

38

Jerry SoungFraud and Risk Engineering

Zane LackeySecurity Engineering

Marcus BarczakOperations

Data and ReferencesDatacenter IP List:

https://github.com/client9/ipcat

ISC Top Troublemaker IPs:http://isc.sans.edu/ipsascii.htmlhttp://isc.sans.edu/sources.html

On Security and Continuous Deployment:http://bit.ly/KFYYlZ

Other presentations on Etsy and Security/Fraud/DevOps:http://slidesha.re/Kw5zdV http://slidesha.re/IMaavq http://slidesha.re/JGaU2s

39

Security Engineering and “Just Culture”Treating security mistakes as “accidents” (whether exploited or not)Based originally on health care initiativesPatient Safety and “Just Culture”, David Marx JD– http://psnet.ahrq.gov/resource.aspx?resourceID=1582– http://bit.ly/LhRHaT (presentation)

John Allspaw on Blameless Post-Mortems:http://codeascraft.etsy.com/2012/05/22/blameless-postmortems/

40

It’s time for questions!

Nick Galbreath@ngalbreath

www.etsy.com

http://slidesha.re/KPvHYu