#BbWorld14#BbWorld14

Security Automation at Blackboard: What are we Doing?

Matthew Saltzman

7-16-2014

Matthew.saltzman@blackboard.com

#BbWorld142

Introduction

• Security at Blackboard has been becoming more and more important in the past couple of years

• Number of notable improvements in the security of the Learn platform:– Whitelist replaces Blacklist in Content Editor– Added security logging– Improved password storage (SHA-512 with HMAC-SHA-512 Salt)– Etc.

• Additionally, we’ve been able to target our resources better– Assessments of Blackboard B2s through development process– Embedding security into design process of new features/products

#BbWorld143

Introduction

• All these improvements require many, many more hours of testing– Without needing that many more man-hours of work

• To make this happen, we needed to automate as much of our testing as possible– Penetration Testing– Static code analysis– Reporting on results and problem areas

#BbWorld144

Infrastructure for Security Testing Automation

#BbWorld145

Tools

• Jenkins– Continuous Build Automation tool

• Chef/Ansible– Configuration management platform– Chef is better at continuous state management, in other words, pushing

changes to the server when updates occur– Ansible is more useful for one-off configuration changes

• URL Database– This is a database that has a listing of most URLs in the Learn

application, which we use to dictate what to scan

#BbWorld146

Jenkins

• Build management tool used for continuous delivery – http://jenkins-ci.org/

• Errors in Jenkins signify a failed build that needs to be corrected immediately– New software won’t be released with build failures

• Can also be used to run scripts/tools each time the code is updated

#BbWorld147

Jenkins Appearance

#BbWorld148

Chef

• Centralized configuration management system– Centralized, in that a single server contains the information about chef

nodes (servers)/configuration scripts (cookbooks), and manages the state of the servers

• Very useful for managing state that will change often, since the state is managed from the configuration management system

• Not as useful for one-off changes, since there is a lot of overhead

#BbWorld149

Chef Workflow

• Taken directly from the Chef website, http://www.getchef.com/chef

#BbWorld1410

Ansible

• Core product is essentially a remotely executing shell script

• De-centralized, the invoking server pushes information to the remote host which then runs

• Does not manage state automatically

• However, very useful for configuration changes that maintain for long periods of time, or for remotely executing scripts

#BbWorld1411

Ansible Script Example

• From http://www.ansible.com/how-ansible-works

#BbWorld1412

URL Database

• Listing of Learn URLs to target for security testing– Obtained through all of our QA testing/integration testing, manual

browsing of the application, and performance testing

• Logs all requests made to the system

• Log with request data gets piped through a parser, and put into a relational database to power all penetration testing tools– Blackboard Learn is difficult to navigate via spider, due to searches and

content creation/editing– Was even more difficult in earlier versions of learn that relied on

framesets and iframes

• Attempts to correlate URL to Bb source files (work in progress)

#BbWorld1413

Penetration Testing Automation

#BbWorld1414

Tools

• Penetration testing is very much a tool-assisted process

• AppScan Standard– Stand-Alone penetration testing tool

• All activity takes place internal to the tool itself

• Burp– Security Proxy Tool

• Meaning a user will proxy traffic to the target site through it to trigger scanning

#BbWorld1415

AppScan Standard Automation – Overview

• AppScan Standard is our primary penetration testing tool– Has a wide variety of different types of issues to look for– Has many different methods for penetration testing– Results are somewhat accurate, for high confidence rules

• Designed to be used manually once, and then repeat the same tests over and over again

• Requires a significant time investment to go through and validate results

#BbWorld1416

AppScan Standard – UI For Manual Use

#BbWorld1417

AppScan Standard Automation – Dependencies

• URL Database– Uses the request information to create explore files, which determine

what to scan

• AppScan Standard applicatoin

• Automation Tool (Ansible in development)– Used to run the AppScan Application

#BbWorld1418

AppScan Standard Automation – Components

• Has 3 components– Login Script (logs the tool back in if the logout page is tested or

reached)– Explore File (list of pages, complete with headers, cookies, and

parameters)– Policy Files (determines which violations to look for, and determines

the various methods of penetration testing that appscan can use)

• Has a command line interface to use automatically

#BbWorld1419

AppScan Standard Automation – Process

1. Create a login script for each server being targeted

2. Create a process to convert data from the URL database into an Explore File– We are currently using a Stored Procedure

3. Create a policy file determining which rules to run and which to ignore

4. Create shell script to run AppScan and create XML output file for consumption later– It isn’t enough to automatically run scans, the results need to be

reviewed as well– Without processing the results before reviewing, this can be a long and

arduous process

#BbWorld1420

AppScan Standard Automation – Demo

#BbWorld1421

Burp Automation – Overview

• Accurate XSS testing tool– Blackboard has had a significant number of XSS issues in the product

• Proxy system, so requires a tool, such as a browser, to navigate through Burp to a target site

• Does not easily re-scan existing sites– Is designed for a Security professional to use to spot-check the system

#BbWorld1422

Burp – UI For Manual Use

#BbWorld1423

Burp Automation – Dependencies

• URL Database – Need to set up a number of requests to send through Burp

automatically – Uses a very different format than AppScan Standard

• Automation Tool (Ansible coming)– Used to run Burp

#BbWorld1424

Burp Automation – Components

• Has three main components:– Feed of RAW request data– Java based Burp Plugin– Burp State file

• The saved state of Burp, which contains a macro for logging into Learn, as well as the types of issues and scanning techniques to be used

• Has the ability to be run from the command line in java’s headless mode

#BbWorld1425

Burp Automation – Process

• Combine the Burp Plugin Jar file and the Burp Jar file into a single executable Jar file

• Obtain a set of pages in RAW form to be scanned

• Create burp state file– Run the standard Burp UI– Add a macro for logging into the server being targeted

• Run Burp command, passing in required parameters

#BbWorld1426

Static Analysis

#BbWorld1427

Tools

• Sonar– Tool for running custom PMD rules– Custom plugin for B2 permissions issues

• In development now• Based on the rule of least privilege

• AppScan Source– IBM Security Scanning tool– Gradle plugin to run this– Store output for use later

• Sonatype Component Lifecycle Management (CLM)– 3rd party library vulnerability scanning tool

#BbWorld1428

Sonar – Overview

• Sonar is an Open Source tool for static analysis– Can be used for code quality, security, unit test analysis, etc.

• For the Security Team, this is where we run PMD rules, and custom security plugins from

• Dependent on Jenkins– Each B2 is scanned by Sonar as part of the build process

#BbWorld1429

PMD Rules – Overview

• Runs in Sonar (http://www.sonarqube.org/)

• A number of Security Rules were created to inform us of security API misuse– Use of the Blacklist XSS filter instead of SafeHTML– Use of insecure escaping algorithms– Missing calls to validate nonce tokens on form submission

• Constantly in flux as the APIs in the product change

• Example:

#BbWorld1430

PMD Rules – Process

• Each Rule contains one Java file representing the rule, and one unit test

• The Sonar PMD plugin needs to be updated to include the rule

• Each rule needs to be assigned a certain severity– Blocker and Critical issues prevent the build from being released– Major, Minor, and Info are used for information purposes

• Won’t block the build• Issues in these categories are added to the backlog if isn’t fixed

#BbWorld1431

Sonar B2 Permissions Plugin – Overview

• B2s should never request more “permissions” than they use– Aside from a few custom Learn specific permissions, the list is here:

http://docs.oracle.com/javase/7/docs/technotes/guides/security/permissions.html

– Limiting permissions prevents exploits by limiting the impact area

Severity of security issues can most often be contained by limited by the proper application of Java permissions

#BbWorld1432

Sonar B2 Permissions Plugin - Components

• XML parsing class to determine which permissions are requested

• PMD rules to determine which permissions are used

• Class to find out if any permissions were requested but not used

• Sonar module to display excessive permissions (In Development)

#BbWorld1433

Sonar B2 Permissions Plugin – Process

1. Configure Sonar-Runner to point to the sonar instance being used

2. Make sure your sonar instance is configured and turned on

3. Add the plugin to the sonar instance if this hasn’t already been done

4. Navigate to the B2 being tested

5. Run the sonar runner

#BbWorld1434

Sonar B2 Permissions Plugin - Demo

#BbWorld1435

AppScan Source – Overview

• Security Source Code Scanning tool– Inspects class files and source files for vulnerabilities in the code itself– Source of the largest subset of vulnerabilities found by our tools

• List of findings is enormous, actual vulnerabilities are less than 5% of these

• Designed for automation, unlike dynamic penetration testing

#BbWorld1436

AppScan Source – Manual Use UI

#BbWorld1437

AppScan Source – Dependencies

• Jenkins– AppScan is used to scan each individual B2 as an independent

application– Can scan each B2 on each check-in by integrating with Jenkins

#BbWorld1438

AppScan Source – Process

1. Compile the project to be scanned into class files

2. Run the AppScan Source script to scan a project– Creates a new application record in AppScan Source– Adds each project with classes and dependencies to that application– Scans all projects inside the application

#BbWorld1439

AppScan Source – Demo

#BbWorld1440

Sonatype CLM – Overview

• 3rd party library scanning tool– Examples: Struts/Spring– Checks for vulnerabilities listed in vulnerability database like

NIST/MITRE only

• Jenkins plugin comes standard

• Removes burden of finding library vulnerabilities from Engineers/Security Team

#BbWorld1441

Sonatype CLM Report – Issue Appearance

#BbWorld1442

Sonatype CLM – Dependencies

• Jenkins– Each B2 package needs to be scanned for new libraries containing

vulnerabilities or new vulnerabilities in existing libraries– Running as part of the build process helps ensure that newly found

issues get fixed quickly

#BbWorld1443

Security Operations and Reporting Console

#BbWorld1444

Operations Console – Overview

• Issue repository for various scanning tools– AppScan Standard results– Burp Results– AppScan Source results

• Internally developed technology– https://github.com/blackboard/security-operations-console

• Allows for finding review

• Contains reports on those findings

#BbWorld1445

Dynamic Analysis Findings Display

#BbWorld1446

Operations Console – Reviewing a Dynamic Finding

1. Navigate to the URL presented

2. Enter a known vulnerable string into the entity

3. If issue is valid, select the valid radio button– Enter bug ticket number in below text box

4. If issue is not valid, select the false positive radio button– Enter a reason that the issue is not valid below, so when the issue is

reviewed, it’s understood why False Positive was selected

#BbWorld1447

Static Analysis Findings Display

#BbWorld1448

Operations Console – Reviewing Specific B2s (Static Analysis)

• Static analysis is directly tied to individual projects– So we can filter down to individual B2s as well

• Can also specify the severity and issue quality– Severity of issues is determined by AppScan source, rather than our

team– Quality is how confident AppScan Source is with its’ finding

• Can also select issues found during a specific time frame– Allows us to review findings from particular builds

#BbWorld1449

Static Analysis Issue Display

#BbWorld1450

Operations Console - Reporting

• 3 Reports– Dynamic Analysis CWE Reporting

• Valid vs False Positives

– Static Analysis Vulnerability Type Report• Valid vs False Positives

– Issues found By Date• New Issues vs Reviewed for both Static Issues and Dynamic Issues

#BbWorld1451

Operations Console – CWE Report

#BbWorld1452

Operations Console – Vulnerability Type Report

#BbWorld1453

Operations Console – Issues By Date Report

#BbWorld1454

Self Service Penetration Testing Tool

#BbWorld1455

Self Service XSS Tool – Overview

• Tool that allows developers to test their own code on their system without a security testing tool installed

• Based on Burp Proxy

• Internally developed– Currently in progress

• Does not log data about findings– Meant to be used prior to checking new code in, not as something for

the security team to evaluate issues on

#BbWorld1456

Self Service XSS Tool – Current UI Appearance

#BbWorld1457

Self Service XSS Tool – Components

• Burp Plugin– Passes all URIs sent in into the proxy to the active scanner– Sends results back to the self service tool’s web server to display to the

users

• Web Application– Application that functions as a proxy for whichever system the

developer wants to target• Which is how we can target users’ development builds

– Also reports issue data via web sockets

#BbWorld1458

Self Service XSS Tool – Process

1. Enter the full URL of the system being targeted, and your username and password (tells the system who you are)

2. Browse to the area of the application being tested

3. View the results for the area of the application being tested

4. Correct the issues found

5. Navigate back to that area of the application and retest

#BbWorld1459

Self Service XSS Tool - Demonstration

#BbWorld1460

Questions?

#BbWorld1461

URL Database – Schema Diagram

#BbWorld1462

AppScan Standard Explore File Format

<?xml version="1.0" encoding="utf-8"?><requests><request method="GET" host="appsec-atarg02.pd.local" path="/webapps/assessment/do/authoring/viewAssessmentManager" scheme="https" SessionRequestType="Login" ordinal="156" port="8443"><raw encoding="none">GET /webapps/assessment/do/authoring/viewAssessmentManager HTTP/1.1Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8Accept-Encoding: gzip, deflateAccept-Language: en-us,en;q=0.5Connection: keep-aliveCookie: session_id=F40578BAC62379AB617E0CF3705E3AA7; JSESSIONID=8F181C8BEE9867B127538607E6914F15.rootHost: fitnesse-patch02.pd.localReferer: http://fitnesse-patch02.pd.local/webapps/blackboard/landingPage.jsp?navItem=cp_test_survey_pool%26course_id=_2_1%26sortItems=falseUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2</raw><cookie name="session_id" value="F40578BAC62379AB617E0CF3705E3AA7" path="/" domain="appsec-atarg02.pd.local" secure="False" expires="1/1/0001 12:00:00 AM"/> <cookie name="JSESSIONID" value="8F181C8BEE9867B127538607E6914F15.root" path="/" domain="appsec-atarg02.pd.local" secure="False" expires="1/1/0001 12:00:00 AM"/> <parameter name="assessmentType" captureIndex="0" value="Test" type="QUERY" linkParamType="simpleLink" separator="&amp;" operator="=" reportName="assessmentType"/><parameter name="course_id" captureIndex="0" value="_2_1" type="QUERY" link ParamType="simpleLink" separator="&amp;" operator="=" reportName="course_id"/><sessionCookies><cookie name="session_id" value="F40578BAC62379AB617E0CF3705E3AA7" path="/" domain="appsec-atarg02.pd.local" secure="False" expires="1/1/0001 12:00:00 AM"/> <cookie name="JSESSIONID" value="8F181C8BEE9867B127538607E6914F15.root" path="/" domain="appsec-atarg02.pd.local" secure="False" expires="1/1/0001 12:00:00 AM"/> </sessionCookies></request></requests>

#BbWorld1463

AppScan Standard Automation Command

• appscancmd /test_only /starting_url http://${target_machine} /dest_scan Z:\Security_Automation\Scan_Files\${file_name}.scan /scan_template Z:\Security_Automation\template\parameter.scant /manual_explore_file Z:\Security_Automation\Explore_Data_Files\${file_name}.exd /policy_file Z:\Security_Automation\Policy_Files\${policy_file}.policy /report_type rc_ase

#BbWorld1464

Burp Login Macro

#BbWorld1465

Burp Command

/usr/local/jdk7/bin/java -Dorg.owasp.esapi.resources=<project-directory>/lib -Xms2048m -Xmx2048m -XX:+ExplicitGCInvokesConcurrent -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseTLAB -Djava.awt.headless=true -classpath <jar directory>/BurpExtender.jar:<project-directory>/lib/esapi-2.0.1.jar:<project-directory>/lib/burpsuite_pro_v1.4.12.jar:<project-directory>/lib/ojdbc6.jar burp.StartBurp <target-machine> <project-directory>/login_files/<target-machine>.burp.login <location of scan files>/<output file name> <xml_output_directory>/<output file name> <blackboard version>

#BbWorld1466

AppScan Source Gradle Plugin Execution Command

/usr/local/jenkins/tools/hudson.plugins.gradle.GradleInstallation/Gradle_Default_Bb-custom_/gradle-1.12/bin/gradle --stacktrace --refresh-dependencies -PappscanBinDir=/usr/local/AppScanSrc/bin -Pmongodb=appsec-db.pd.local AppScanSource -b assignment/build.gradle

#BbWorld1467

Operations Console Code Coverage Report

#BbWorld1468

Dynamic Analysis Issue Display Key

Key Description

ID Unique identifier for the issue

URL The url in Learn that is vulnerable

Url Parameters The set of GET parameters

Body (POST) Parameters The set of POST parameters

Entity Type The vulnerable entity (Could be Parameter, Cookie, Header, etc.)

Entity Name If applicable, the name of the parameter

Reasoning The reason that the entity is flagged as vulnerable

CWE Common Weakness Enumerationhttp://cwe.mitre.org/

Entity Change What, if anything, was altered in the entity to show the vulnerability