SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies...

121
SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat Analysis Deliverable D4.2: Demonstrators evaluation Abstract: This deliverable reports the results of the case studies. It mainly contains the specifications of the testing environments, descriptions of the case studies, assessment indicators, and experiment results. It will cover all the case studies, allowing a comprehensive assessment of our de- signs. Contractual Date of Delivery March 31, 2016 Actual Date of Delivery April 20, 2016 Deliverable Dissemination Level Public Editor Gregory Blanc and Yuji Sekiya Contributors All NECOMA partners The NECOMA consortium consists of: Institut Mines-Telecom Coordinator France ATOS SPAIN SA Principal Contractor Spain FORTH-ICS Principal Contractor Greece NASK Principal Contractor Poland 6CURE SAS Principal Contractor France Nara Institute of Science and Technology Coordinator Japan IIJ - Innovation Institute Principal Contractor Japan National Institute of Informatics Principal Contractor Japan Keio University Principal Contractor Japan The University of Tokyo Principal Contractor Japan The research leading to these results has received funding from the European Union Sev- enth Framework Programme (FP7-ICT-2013-EU-Japan) under grant agreement n° 608533.

Transcript of SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies...

Page 1: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

SEVENTH FRAMEWORK PROGRAMMEInformation & Communication Technologies

ICT

Cooperation Programme

Nippon-European Cyberdefense-Oriented Multilayer threat Analysis†

Deliverable D4.2: Demonstrators evaluationAbstract: This deliverable reports the results of the case studies. It

mainly contains the specifications of the testing environments, descriptionsof the case studies, assessment indicators, and experiment results. It willcover all the case studies, allowing a comprehensive assessment of our de-signs.

Contractual Date of Delivery March 31, 2016Actual Date of Delivery April 20, 2016Deliverable Dissemination Level PublicEditor Gregory Blanc and Yuji SekiyaContributors All NECOMA partners

The NECOMA consortium consists of:

Institut Mines-Telecom Coordinator FranceATOS SPAIN SA Principal Contractor SpainFORTH-ICS Principal Contractor GreeceNASK Principal Contractor Poland6CURE SAS Principal Contractor FranceNara Institute of Science andTechnology

Coordinator Japan

IIJ - Innovation Institute Principal Contractor JapanNational Institute of Informatics Principal Contractor JapanKeio University Principal Contractor JapanThe University of Tokyo Principal Contractor Japan

† The research leading to these results has received funding from the European Union Sev-enth Framework Programme (FP7-ICT-2013-EU-Japan) under grant agreement n° 608533.

Page 2: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

www.necoma-project.eu 2 March 31, 2016

Page 3: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

Contents

1 Introduction 9

2 Platforms Specifications 112.1 Cloud-based Testbed . . . . . . . . . . . . . . . . . . . . . . . 13

2.1.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . 132.1.2 Specifications . . . . . . . . . . . . . . . . . . . . . . . 13

2.2 MPLS-based Testbed . . . . . . . . . . . . . . . . . . . . . . . 172.2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . 172.2.2 Specifications . . . . . . . . . . . . . . . . . . . . . . . 17

2.3 SDN-based Testbed . . . . . . . . . . . . . . . . . . . . . . . . 212.3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . 212.3.2 Specifications . . . . . . . . . . . . . . . . . . . . . . . 22

2.4 Tablet-based Testbed . . . . . . . . . . . . . . . . . . . . . . . 242.4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . 242.4.2 Specifications . . . . . . . . . . . . . . . . . . . . . . . 25

2.5 Smartphone Emulator . . . . . . . . . . . . . . . . . . . . . . 262.5.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . 262.5.2 Specifications . . . . . . . . . . . . . . . . . . . . . . . 26

3 Case Studies 293.1 DDoS Mitigation . . . . . . . . . . . . . . . . . . . . . . . . . 29

3.1.1 Evaluation Framework . . . . . . . . . . . . . . . . . . 313.1.2 SDN-based DDoS Mitigation . . . . . . . . . . . . . . . 353.1.3 DDoS Mitigation as a Service . . . . . . . . . . . . . . 413.1.4 Pushing Defenses Upstream . . . . . . . . . . . . . . . 503.1.5 Conclusion and Lessons Learned . . . . . . . . . . . . 63

3.2 Botnet Introspection . . . . . . . . . . . . . . . . . . . . . . . 663.2.1 Evaluation Framework . . . . . . . . . . . . . . . . . . 66

3

Page 4: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.2.2 C&C Server Introspection with DNS Queries . . . . . . 693.2.3 Conclusion and Lessons Learned . . . . . . . . . . . . 77

3.3 Smartphone User Protection . . . . . . . . . . . . . . . . . . . 783.3.1 Evaluation Framework . . . . . . . . . . . . . . . . . . 793.3.2 Drive by Download Prevention . . . . . . . . . . . . . 823.3.3 Phishing Protection . . . . . . . . . . . . . . . . . . . . 873.3.4 Offloading Smartphone’s FW to OpenFlow-capable AP 943.3.5 SMS Fraud Protection . . . . . . . . . . . . . . . . . . 993.3.6 Conclusion and Lessons Learned . . . . . . . . . . . . 104

3.4 Malware Campaign Mitigation . . . . . . . . . . . . . . . . . . 1063.4.1 Evaluation Framework – Datasets . . . . . . . . . . . . 1073.4.2 Conclusion and Lessons Learned . . . . . . . . . . . . 116

4 Conclusion 119

www.necoma-project.eu 4 March 31, 2016

Page 5: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

List of Figures

2.1 UTokyo Cloud-based Testbed Overview . . . . . . . . . . . . 132.2 The MPLS testbed. Black, filled lines show IP connectivity,

hashed lines depict information flow. . . . . . . . . . . . . . . 172.3 IMT platform topology . . . . . . . . . . . . . . . . . . . . . . 212.4 A topology of smartphone firewall . . . . . . . . . . . . . . . 24

3.1 The amplification attack concept . . . . . . . . . . . . . . . . 353.2 Experimental topology of the scenario . . . . . . . . . . . . . 363.3 Scenario workflow . . . . . . . . . . . . . . . . . . . . . . . . 383.4 Time to Recover . . . . . . . . . . . . . . . . . . . . . . . . . . 403.5 The proposed framework enabling DDoS mitigation as a service 423.6 The toplogy used in our experiments: the applications of La-

bels and FlowID are also illustrated, where action FWD meanspacket/flow forwarding. . . . . . . . . . . . . . . . . . . . . . 43

3.7 Time to rebuffer . . . . . . . . . . . . . . . . . . . . . . . . . 453.8 Status of the player . . . . . . . . . . . . . . . . . . . . . . . . 463.9 Average buffer length . . . . . . . . . . . . . . . . . . . . . . 473.10 Duration of the player in paused mode . . . . . . . . . . . . . 473.11 Average Goodput . . . . . . . . . . . . . . . . . . . . . . . . . 483.12 The demonstrator on the MPLS testbed. Black, filled lines

show IP connectivity, green and red filled lines show MPLStunnels, hashed lines depict information flow - green for at-tack and threat related, red for configuration orders, and bluefor measurement collection . . . . . . . . . . . . . . . . . . . 51

3.13 Traffic volumetry in bps between R1 and R3 . . . . . . . . . . 543.14 Traffic volumetry in pps between R1 and R3 . . . . . . . . . . 543.15 Traffic volumetry in bps between R2 and R3 . . . . . . . . . . 54

5

Page 6: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

LIST OF FIGURES

3.16 Traffic volumetry in pps between R2 and R3 . . . . . . . . . . 553.17 Round trip time for ping messages on the platform before the

attack, during the attack (before and after mitigation) . . . . 563.18 Ping drop rate on the platform before attack, during attack

before and after mitigation . . . . . . . . . . . . . . . . . . . 563.19 Precision, recall, and specificity; based on packet drops for

R1 and 6000 attack flows as the function of attack volume,without mitigation . . . . . . . . . . . . . . . . . . . . . . . . 57

3.20 Precision, recall, and specificity; based on packet drops forR1 and 6000 attack flows as the function of attack volume,with mitigation . . . . . . . . . . . . . . . . . . . . . . . . . . 57

3.21 Precision, recall, and specificity; based on packet drops forR2 and 11000 attack flows as the function of attack volume,without mitigation . . . . . . . . . . . . . . . . . . . . . . . . 58

3.22 Precision, recall, and specificity; based on packet drops forR2 and 11000 attack flows as the function of attack volume,with mitigation . . . . . . . . . . . . . . . . . . . . . . . . . . 58

3.23 ACL loading time as the function of the number of insertedrules on a router with no rules loaded . . . . . . . . . . . . . 60

3.24 ACL loading time; based on the cumulative insertion of abatches of 10000 rules . . . . . . . . . . . . . . . . . . . . . . 61

3.25 Topology and sequence of the scenario: the sandboxed net-work (right side) learns malicious domain names based fromthe sandboxed traffic, the user network (left side) employsthis malicious domain lists to filter out the traffic in the wild. . 70

3.26 A pipeline from data processing to filtering . . . . . . . . . . . 713.27 Successful IP address resolutions observed in DNS traffic . . . 733.28 HTTP traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . 743.29 Observed malicious domain names . . . . . . . . . . . . . . . 753.30 Scenario workflow for drive-by-download prevention . . . . . 833.31 Sequence diagram of the proposed mechanism as a Chrome

extension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 843.32 Rendering time with/without the extension . . . . . . . . . . 853.33 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . 883.34 Stress Test Analysis of the Analysis Modules using ApacheBench 903.35 Analysis of Usability Experiment . . . . . . . . . . . . . . . . 923.36 Operational sequence for the smartphone protection scenario

using OpenFlow-capable APs . . . . . . . . . . . . . . . . . . 953.37 Stress Test Analysis of the OpenFlow APIs using Siegeh . . . . 963.38 Effects on Battery Consumption with and without an AP-based

Firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 973.39 Flow of the real-world scenario. . . . . . . . . . . . . . . . . . 100

www.necoma-project.eu 6 March 31, 2016

Page 7: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

LIST OF FIGURES

3.40 Number of potential victims identified in network traffic datasets,based on indicators obtained through analysis of two mal-ware campaigns. . . . . . . . . . . . . . . . . . . . . . . . . . 115

www.necoma-project.eu 7 March 31, 2016

Page 8: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

LIST OF FIGURES

www.necoma-project.eu 8 March 31, 2016

Page 9: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

1Introduction

As one of the latest deliverables produced by the NECOMA project, thisparticular report supports the other deliverable produced by the works inworkpackage 4 (Deliverable D4.3), the actual demonstrators. This deliv-erable will cover the implementation details of the testing environments,including the probes set to measure the assessment metrics defined in De-liverable D4.1, and the deployment of the use case scenarios and the resultsobtained from the experiments to assess the performance and resilience ofthe NECOMA designs. These two deliverables are essential to measure theproject objectives, as well as the progress of the project from the perspec-tive of the state of the art. As it was the case in other deliverables, we willrefer, in the remainder of this deliverable, to “case studies” or “use cases”interchangeably, and to individual instances of use cases as “scenarios”.

In this deliverable, we proudly include the latest outcomes of most workscarried out in the previous workpackages, as we pledged to demonstratesome of the mechanisms designed in workpackages 1, 2 and 3, including atleast two infrastructure-level resilience mechanisms. In particular, we wereable to demonstrate a complete chain in pipelining security information insome of the scenarios, effectively progressing from past research projectswhere most processes existed in isolation. Here, some scenarios effectivelyshowcase the pipelining of data acquisition into analysis and cyberdefense,eventually enabling the automated defense or mitigation of certain classesof attack patterns.

This deliverable not only deals with the experimental results of the saidscenarios through test and validation of the methods, mechanisms and tech-niques resulting from the related workpackages, but it also achieves a greaterobjective for each case study that we considered:

• building an evaluation framework: each case study comes with one orseveral related benchmark datasets (which are detailed in DeliverableD1.4), a set of evaluation metrics or indicators specific to the threat (as

9

Page 10: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 1. INTRODUCTION

defined in Deliverable D4.1) and a set of experimental platforms andtools as described in the present deliverable, as per the requirementsexpressed in Deliverable D4.1;

• laying down a foundation and guidelines for testing and experiment-ing with the designs of workpackages 2 and 3: the scenarios ultimatelyallow us to experiment with particular settings with respect to the casestudies, but the above-mentioned framework actually comprises met-rics allowing the assessment of the strengths and weaknesses of ourdesigns in a reflective manner, leading to further improvements;

• improving the designs of workpackages 2 and 3 and existing mech-anisms: as expressed above, the insights obtained from testing thedesigns, with respects to the given metrics, will allow us to make rec-ommendations on not only the improvements of the said designs butalso to existing mechanisms used in different scenarios.

In this deliverable, we report on the implementation and demonstrationof most scenarios proposed in Deliverable D4.1. For most of them, we wereable to demonstrate a partial or full integration of the threat informationpipeline, as defined in NECOMA’s DoW. The results we present in this de-liverable also support our claims that we achieved reconfigurable defensemechanisms that improve the resilience of both the infrastructure and end-point layers. Demonstrators in Deliverable D4.3 complement the results pre-sented in this deliverable, and show that the integration of different analysismodules from WP2 went successfully, also leveraging the datasets that wereintroduced in WP1. Finally, a certain degree of automation was achieved insome of the DDoS mitigation and botnet introspection scenarios, while themalware campaign mitigation scenario closely supports and partially auto-mates the work of CERT/CSIRT analysts, thanks to technologies developedin NECOMA.

www.necoma-project.eu 10 March 31, 2016

Page 11: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

2Platforms Specifications

In this chapter, we present the different testing platforms that we devel-oped to host the demonstrators. The demonstrators implement a diverseset of scenarios with respect to the four NECOMA use cases: DDoS mitiga-tion, botnet introspection, smartphone user protection, and malware campaignmitigation. Table 2.1 shows the coverage of the platforms on the use cases.

Note that the malware campaign mitigation scenario is performed en-tirely using real data in the real operational environment of a CSIRT team(CERT Polska). Therefore no separate testing platform was created for thescenario.

11

Page 12: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 2. PLATFORMS SPECIFICATIONS

Table 2.1: Platforms hosting the NECOMA demonstratorsDDoS Mitigation Botnet Introspection Smartphone User Pro-

tectioncore edge centralized distributed browser filter

Cloud-basedtestbed(Sect. 2.1)

SDN-basedDDoSMitigation(Sect. 3.1.2)

C&C ServerIntro-spectionwith DNSQueries(Sect. 3.2.2)

MPLS-basedtestbed(Sect. 2.2)

PushingDefensesUpstream(Sect. 3.1.4)

SDN-basedtestbed(Sect. 2.3)

DDoS Mit-igation asa Service(Sect. 3.1.3)

Tablet-basedtestbed(Sect. 2.4)

Drive-byDownloadPrevention(Sect. 3.3.2),PhishingProtection(Sect. 3.3.3)

OffloadingSmart-phone FW’sto OF-capable AP(Sect. 3.3.4)

Smartphoneemulator(Sect. 2.5)

SMS FraudProtection(Sect. 3.3.5)

www.necoma-project.eu 12 March 31, 2016

Page 13: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

2.1. CLOUD-BASED TESTBED

2.1 Cloud-based Testbed

This platform is a testbed to accommodate Virtual Machines using host virtu-alization and network virtualization technologies. Most of the componentsin the platform are built using open-source software. The platform providesan easy, flexible, sealed, and reproducible environment for its users to em-ulate real network environments. In the NECOMA project, the platform isused to evaluate the DDoS mitigation and botnet introspection case studiesin Sections 3.1.2 and 3.2.2, respectively.

2.1.1 Overview

The platform is mainly composed of virtual machines, virtual switches run-ning inside hypervisors, physical network switches, and storages. Thanks tothe software, the platform can be configured flexibly to meet various kindsof use case analyses.

Figure 2.1 shows an overview of this platform. The hypervisors are lo-cated at two campuses and a 10Gbps Ethernet link connects the campuses.All of the hypervisors connect to the physical network switches and the net-works inside the switches are isolated using VLANs.

HONGO Campus KOMABA Campus

Power Edge R420

Power Edge R320

R515

Power Edge R320

Power Edge R320

Power Edge R320

Power Edge R320

Power Edge R320

S4810 S4048

Catalyst 3750-24TS-1U Catalyst 3750-24TS-1U

Controller, Hypervisor

Hypervisor

Hypervisor

Hypervisor

Hypervisor

Hypervisor

Hypervisor

Storge

Demonstration and Experiment NetworkManagement Network

Figure 2.1: UTokyo Cloud-based Testbed Overview

2.1.2 Specifications

We designed the platform to configure and execute reproducible use caseanalyses. Seven physical servers as hypervisors, four physical switches, andone storage server are deployed. Four hypervisors and two physical networkswitches are located at the Komaba campus. Three hypervisors, one storageserver, and two physical network switches are located at the Hongo campus.

www.necoma-project.eu 13 March 31, 2016

Page 14: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 2. PLATFORMS SPECIFICATIONS

The software for building test environments and performing the scenariosare installed in the hypervisor, named “necoma-ansible”.

Users can manipulate deployment configurations and experiment sce-narios using the Ansible tool as described later. The configurations andscenarios are called “playbook” and are expressed in the YAML1 format.

When a user wants to deploy VMs and virtual switches on the platformand configure them for an experiment, the user can write playbooks forthe deployment and configuration and execute them. If the deployment orconfiguration fails due to a misconfiguration of the playbooks, the user canrestore the original environment by executing the pre-defined playbook inthe platform.

2.1.2.1 Hardware

The following compute, network, and storage devices are used to composethe platform.

2.1.2.1.1 Compute

• Six DELL PowerEdge R320 servers working as hypervisors

CPU Intel Xeon E5-1410v2 (2.80GHz, 4-cores)

Memory 32GB

HDD Four 1TB

Network Intel X520 two 10Gbps Ethernet Card

• One DELL PowerEdge R420 server working as a controller and hyper-visor

CPU Intel Xeon E5-2430L (2.00GHz, 6-cores)

Memory 64GB

HDD Two 4TB

Network Intel X520 two 10Gbps Ethernet Card

2.1.2.1.2 Network

• One DELL S4810 switch

– 48 ports 10Gbps SFP+

– 4 ports 40Gbps QSFP+

• One DELL S4048 switch1http://yaml.org/

www.necoma-project.eu 14 March 31, 2016

Page 15: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

2.1. CLOUD-BASED TESTBED

– 48 ports 1Gbps/10Gbps SFP+

– 4 ports 40Gbps QSFP+

• Two Cisco Catalyst 3750-24TS-1U switches

– 24 ports 10/100/1000Mbps UTP

– 4 ports 1Gbps SFP

2.1.2.1.3 Storage

• One DELL R515 server working as a storage

CPU AMD Opteron 4386 (3.1GHz, 8-cores)

Memory 32GB

HDD Twelve 3TB

Network Intel X520 two 10Gbps Ethernet Card

2.1.2.2 Software

All of the following software are open-source software and some of themare developed by the NECOMA members.

2.1.2.2.1 Management

• KVM: Kernel Virtual MachineA hypervisor implementation on the Linux kernel. It enables us to runvirtual machines(VMs) on a physical server. The software is availableat http://www.linux-kvm.org/.

• libvirt: The virtualization APIA library to control KVM, virtual switch, network configuration, andstorage allocation by providing Application Programming Interfaces(APIs). The software is available at http://libvirt.org/.

• MAAS: Metal as a ServiceSoftware for installing OS into phyical or virtual machines automati-cally. In this platform, MAAS is used for installing a Linux OS (Ubuntu14.04) into physical machines (hypervisors) and VMs running on thehypervisors. When the administrator of this platform wants to deploya new physical machine into the plartform and configure it as a hy-pervisor, he must connect the new machine to the network providedby MAAS and power up the machine. The MAAS installs a Linux OSand required toolchain to the machine automatically. In case of de-ploying a new VM, the MAAS install a Linux OS into the VM as the

www.necoma-project.eu 15 March 31, 2016

Page 16: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 2. PLATFORMS SPECIFICATIONS

same manner. If the user wants to deploy a new VM on the platform,he must create a new VM and connect it to the network. The softwareis available at http://maas.ubuntu.com/.

• Ansible: Simple IT automationAn automation tool to configure servers based on pre-defined rules.In this platform, Ansible is used for building experimental environ-ments such as building a VM via libvirt, configuring the OS installedin the VM, and configuring network topologies with virtual and phys-ical switches. The software is available at http://www.ansible.com/.

2.1.2.2.2 Traffic Generation

• srcgen: multiple source traffic generator using netmapThis software can generate UDP DDoS traffic for dummy attacks. It isavailable at https://gist.github.com/upa/d6ff72b0306945a0284f.

• CreateZones: creating dummy DNS zones and queriesThis software can generate dummy DNS zones for emulating actualDNS delegations and dummy queries based on a ruleset or actual cap-tured data. It is available at https://github.com/shored/createzones/.

2.1.2.2.3 Monitoring/Probing

• Cacti: The Complete RRD Tool-based Graphing SolutionThis software retrieves network stats, usage of resources, and work-loads by using SNMP. The software is available at http://www.cacti.net/.

• virtsnmp: SNMP agent for monitoring hypervisor and virtual machinesThis software works as an agent of an SNMP server and provides work-load information regarding the hypervisor and virtual machines run-ning inside the hypervisor. The software is available at http://draft.scyphus.co.jp/cloud/virtsnmp.html.

www.necoma-project.eu 16 March 31, 2016

Page 17: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

2.2. MPLS-BASED TESTBED

2.2 MPLS-based Testbed

The platform consists of a set of routers forming a small MPLS networkplaying the role of an ISP network. Some nodes represent an attacker anda legitimate user on one side and services on the other side, communicatingover the network.

The objective is to create conditions under which a volumetric DDoS at-tack causes resource exhaustion at the endpoint level (e.g., at the datacenterentrance) for two reasons:

1. the core network having more capacity than the edge, and

2. the funneling effect for traffic coming from distributed sources to-wards the target of the attack.

2.2.1 Overview

An overview of the testbed is shown in Fig. 2.2. The routers R1, R2, andR3 form an MPLS domain; on the left we have both the attacker node andthe legitimate client; on the right the node providing the targeted/protectedservice, behind a local mitigation solution (6cure TP).

Figure 2.2: The MPLS testbed. Black, filled lines show IP connectivity,hashed lines depict information flow.

2.2.2 Specifications

In the following subsections, we will describe the hardware and softwarerunning on the platform in order to meet the requirements defined in D4.1,Sect. 2.3.1.2.

www.necoma-project.eu 17 March 31, 2016

Page 18: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 2. PLATFORMS SPECIFICATIONS

2.2.2.1 Hardware

2.2.2.1.1 Compute The different nodes on the platform execute on thefollowing hardware:

target HP DL360 server, 4 cores Xeon 3.20GHz, 1GB RAM

attacker Dell R610, 2 * X5687 (8 cores, 3.60 GHz, HT in use), 24 GB RAM

6cure TP Dell R710, 2 * X5690 (12 cores, 3.47 GHz), 24 GB RAM

hypervisor Dell R620, 2 * E5-2660 v2 (20 cores, 2.20 GHz, HT in use), 64 GBRAM (shared among all demonstration VMs)

We have a separate target node with dated hardware, as this machinedoes not need any particular resources. The attacker is a separate, physicalserver, as the attack traffic generation requires a lot of resources. 6cure TPappliance runs on dedicated hardware as well. Other nodes do not haveany particular performance requirements and execute on the hypervisor asvirtual machines.

2.2.2.1.2 Network The platform uses the following networking equip-ment:

R1 Cisco 7204 VXR

R2 Cisco 7204 VXR

R3 Cisco 3660

sw-dell Dell 8024F, 10G switch

sw-ng Netgear GS748Tv3, 1G switch

sw-cisco Cisco Catalyst 2970, 1G switch

Using three routers allows us to construct two alternative paths towardsthe target as indicated in our requirements. The networking hardware wasthe limiting factor in terms of performance for our testing environment andlimited the scale of attacks we could run on the testbed.

2.2.2.1.3 Storage The storage is local, directly attached storage usingrotational disks (mostly 15 krpm SAS drives).

2.2.2.2 Software

2.2.2.2.1 Management This paragraph deals with how scenarios are de-ployed on the platform.

www.necoma-project.eu 18 March 31, 2016

Page 19: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

2.2. MPLS-BASED TESTBED

2.2.2.2.2 Traffic Generation The legitimate traffic used on the platformwas obtained from a MAWI capture2 from which a subset covering traffic to-wards two /24 subnets receiving the largest amount of traffic was extractedand split in separate pcaps for each /24.

The traffic characteristics of the two /24 destinations are somewhat dif-ferent, the first one having an average packet size of 1200 bytes and thesecond one with a smaller average size. The first /24 destination is alwaysreinjected towards R1 and the second towards R2. The different types oftraffic allow us to observe if the the packet size impacts the behavior.

The attack traffic is generated by a script that produces DNS replies to-wards given target for a given volume of requests by modifying and dupli-cating real responses using scapy3. The script uses a pcap containing a DNSresponse on UDP to a type ANY request for isc.org as a starting point andallows the modification of source (n sources randomly selected from a givensubnet, corresponding to the amplifiers) and destination IP addresses (theattack targets), destination port (randomized), and the DNS query identifi-fier (randomized). This results in a pcap with attack traffic from n sourcessending all the responses.

Both legitimate and attack traffic are replayed using either tcpreplay4 orthe dagflood utility for Endace DAG cards5 from the attacker node.

The legitimate traffic is replayed at the same constant bandwidth rate(and thus at different packet rates due to differences in the packet sizesin the two captures) towards both routers R1 and R2 in every test run.The attack traffic intensity varies between different test runs, but always insuch a way that the bandwidth used by the attack traffic remains the sametowards both routers R1 and R2.

2.2.2.2.3 Monitoring/Probing We use ping to measure the latency be-tween the attacker host and the target host. We send 10 ICMP echo requestsin a short burst (ping -q -W 5 -i 0.2 -c 10) and take the average roundtrip time as a measurement of latency. We measure also the drop rate if echorequest messages, which are dropped on the platform only due to conges-tion.

In addition, we use httperf6 for a simple measure of application levelperformance on the testbed. We run 100 HTTP GET requests, 20 requestsper second, and observe the number of requests failing (httperf --hog

--server $ip --uri $url --num-conn 100 --rate 20 --timeout 5).

2http://www.fukuda-lab.org/mawilab/v1.1/2014/02/24/20140224.html3http://www.secdev.org/projects/scapy/4https://github.com/appneta/tcpreplay5http://www.endace.com/endace-dag-high-speed-packet-capture-

cards.html6https://github.com/httperf/httperf

www.necoma-project.eu 19 March 31, 2016

Page 20: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 2. PLATFORMS SPECIFICATIONS

Various performance and volumetry related metrics such as CPU andmemory, network interface counters are retrieved via SNMP from differentcomponents on the testbed. We use collectd7 for collecting this informa-tion with SNMP and storing it. The collected data is visualized and accessedvia grafana8.

We capture traffic with tcpdump at MPLS ingress and egress points onthe platform to observe both legitimate and malicious traffic. These capturesare used to establish volumes and drop rates for both types of traffic.

7https://collectd.org/8http://grafana.org/

www.necoma-project.eu 20 March 31, 2016

Page 21: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

2.3. SDN-BASED TESTBED

2.3 SDN-based Testbed

The IMT platform aims at hosting realistic use cases by leveraging SDN tech-nology. It offers a diverse set of properties to the hosted scenarios, such ashigh-speed Internet environment. This is achieved through optic fiber con-nectivity, as well as 1 Gbps links between the forwarding equipments andendpoint terminals (i.e., servers). The platform is highly versatile as it canbe configured to represent diverse topologies. It leverages SDN programma-bility to specify paths flows must follow. Virtualization is also perused tospawn a number of services at the edge of the SDN network, and orches-tration of all resources will be later provided as a service to the platformusers.

2.3.1 Overview

While the platform is expected to host many more devices (hosting andforwarding combined) and with a high level of interconnection (i.e., full-meshed network), the actual implementation relies on a handful of SDNswitches and servers. As mentioned previously, the switches allow for di-verse topologies traversed by flows generated by the servers placed at theedges. These servers usually run virtualization software to spawn virtualmachines offering a diverse set of services, e.g., video streaming or Webservices.

Figure 2.3: IMT platform topology

As shown in Figure 2.3, the global structure of the platform separatesthe switches to form a cloud provider infrastructure on one side, and an

www.necoma-project.eu 21 March 31, 2016

Page 22: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 2. PLATFORMS SPECIFICATIONS

end-user infrastructure (customer network) on the other side. Therefore,most of the SDN switches belong to the provider side. The client and thecloud provider infrastructure are separated by the black lines which repre-sent which equipment belongs to the provider and the client.

2.3.2 Specifications

This subsection will describe the final hardware and software specificationsof the IMT platform.

2.3.2.1 Hardware

2.3.2.1.1 Compute Computational ressources are provided by the dif-ferent servers used for virtualization. There are several DELL R620 servers,with Xeon E5-2690 v2 (@3Ghz) CPU, 32 GB DDR3 RAM. These servers hostthe different virtual machines used in the DDoS Mitigation using SDN UseCase.

2.3.2.1.2 Network The network components are composed of one man-agement switch and seven SDN-enabled switches. The management switchis an IBM G8052, 48 ports. Its current role is to provide Internet connectiv-ity to the other components of the platform. In addition to that, firewallingand network monitoring will be enforced by the management switch as it isthe ingress point for hosts external to the platform. All of the SDN switchesare Alcatel OS6860-24 ports.

2.3.2.1.3 Storage The DELL R620 servers are equiped with 1.35 TB HardDrives (15000 rpm). They are used to store both virtual machines as wellas ISO files used to create the virtual machines.

2.3.2.2 Software

Virtulization is achieved with the VMWare solution ESXi version 6. Thanksto this, it is possible to easily instantiate Virtual Machines or allocate ressourcesbased on the needs for each machine. It is also possible to handle accountsand permission, so partners can have access to the platform to deploy theirown experiment.

2.3.2.2.1 Management In addition to ESXi version 6, the platform hostsa VCenter client to ease management of platform ressources. ESXi allowsfor user and permissions usage, so some users can create virtual machinesand other only fill the datastore. It is an efficient tool that can be used togive partners an access to the platform.

www.necoma-project.eu 22 March 31, 2016

Page 23: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

2.3. SDN-BASED TESTBED

2.3.2.2.2 Traffic Generation In this section, we present the differenttools used to generate traffic during the experiments. We have used threemain tools, and we will present a fourth one that could have been used inother circumstances.

• TAPAS The purpose of the DDoS mitigation scenario is, among other things,ensuring QoE (quality of experience) for customers. Therefore, havinga solution allowing the measurement of the efficiency of the solutionwhile providing a visual output was considered. This is the reason wechose TAPAS, an adaptative streaming prototyping tool. It is meant tosimulate the activity of a video streaming player, without adding theoverhead of video computation. In addition to that, TAPAS automati-cally adapts the bandwidth and the quality of the video according tothe network conditions. TAPAS is a client side only software. It doesnot require any configuration on the server hosting the video content.However, the video files require a little editing, as the configurationfile included with the videos needs to match the IP address requestedfor the files, i.e., the IP address of the Web server.

• iPerf While TAPAS provides a visual output to assess the efficiency of theDDoS mitigation solution, it does not generate enough traffic to fill thebandwidth of the differents links inside the platform. For this reason,we use iPerf which is designed to test the maximum bandwidth usableby the links. It has a client-server structure, where the server awaitsthe connection of a client and processes the requests transmitted fromthe client. Once the test is done, the output shows the quantity of datasent per second, i.e., the maximum transfer rate it could reach duringthe test.

• hping3 hping3 has one basic function: sending TCP/IP packets. Packetscan be configured to either have a specific length, a specific source/destinationport, etc. Since the rate of packet sending can be configured and attainthe maximum speed the computer can reach, it is an excellent tool forTCP/ICMP flooding.

2.3.2.2.3 Monitoring/Probing As described earlier, we use VCenter tohave a global overview of the usage of the virtualization servers. It is possi-ble to monitor ressource consumption in terms of CPU, Memory and Storage.There is no tool to efficiently monitor the switches resources, as each switchneeds to be accessed indepedently to retrieve data for monitoring purposes.

www.necoma-project.eu 23 March 31, 2016

Page 24: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 2. PLATFORMS SPECIFICATIONS

2.4 Tablet-based Testbed

This testing environment allows the demonstration of a number of smart-phone user protection scenarios. In particular, the tablet testbed allowsusers to connect to websites hosted in servers at the edge of the platform us-ing a smartphone and/or a tablet device. Defense mechanisms are providedas middleboxes between the simulated Internet and the user devices.

2.4.1 Overview

Phishing Server

LegitimateServer

A smartphone

Browser Extension

NECOMAtterSystems

NECOMAtterBots

WirelessAccess Point

OpenFlowController

Service Network

Wireless Network

Control Network

(pseudo)bot master

(pseudo)victim node

Figure 2.4: A topology of smartphone firewall

Figure 2.4 shows the topologies used in the scenarios described in Sec-tion 3.3.3 and 3.3.4. There are three networks as follows.

• Service Network allows us to connect the hosts to services. NECOMAt-ter bots are also connected to this network since several bot programshave to analyze cyber threats.

• Control Network is used as a communication channel for the defensemechanism.

• Wireless Network is needed for connecting a smartphone device to thetest environment.

www.necoma-project.eu 24 March 31, 2016

Page 25: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

2.4. TABLET-BASED TESTBED

2.4.2 Specifications

2.4.2.1 Hardware

The platform is composed of several equipments:

2.4.2.1.1 Compute

Google Nexus 7 A smartphone/table powered by Android OS.

Windows Tablet A smartphone/table powered by Microsoft Windows 8.1.

DELL Servers 8-core 2.9 Ghz, 16 Go RAM, 10 Gb/s Ethernet.

EyeTribe An eye-tracking camera.

2.4.2.1.2 Network

Buffalo WZR-HP-AG300H An OpenWRT capable wireless AP.

2.4.2.2 Software

2.4.2.2.1 Management In this scenario, it is not necessary to use anymanagement software.

2.4.2.2.2 Traffic Generation For the smartphone resilience firewall sce-nario, we developed an Android application which can launch a set of threadsand send packets at designated intervals. Additionally, no other traffic isgenerated in this environment.

2.4.2.2.3 Monitoring/Probing In the personalized phishing preventionscenario, we measure detection precision/recall, delay, and user experience.Please refer to Deliverable D2.1 for how we measure precision and recall.As for the delay, it can be measured through performance evaluation with aWeb stress test tool. The user experience is a qualitative variable, collectedthrough a questionnaire. The participants assess the defense mechanismaccording to a five-grade evaluation system (e.g., strongly agree, weaklyagree, neither, weakly disagree, and strongly disagree).

In the smartphone firewall scenario, we measure delay and energy con-sumption. The delay is also measured through performance evaluation witha Web stress test tool. Energy consumption is measured directly on thesmartphone device during specific time period.

www.necoma-project.eu 25 March 31, 2016

Page 26: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 2. PLATFORMS SPECIFICATIONS

2.5 Smartphone Emulator

In the SMS Fraud Protection scenario, a malware author uploads a maliciousapplication to a mobile app marketplace and a user attempts to downloadthis application using a smartphone or or other mobile device. The applica-tion is then trying to send SMS or call premium numbers that fraudulouslycharge the user. The protection mechanism is based on CAPTCHAs and isrun is an Android Smartphone Emulator, instead of a physical device.

2.5.1 Overview

This subsection will give a general overview of the platform with a figure tobe described.

2.5.2 Specifications

This subsection will describe the final hardware and software specificationsof your platform.

2.5.2.1 Hardware

The following are needed in order for the platform to be operational andtested.

2.5.2.1.1 Compute The mobile app marketplace server can be a copy ofa real marketplace or a simplified form of it (e.g., a web server that containsa list of Android applications.) Native mobile devices running Android orAndroid Emulator instances. The Android Emulator instances should runthe modified Android OS.

2.5.2.1.2 Network Wireless or wired network in order for the device orthe Android Emulator to be able to access the app marketplace.

2.5.2.1.3 Storage Storage will be needed in case a full copy of an appmarketplace is used.

2.5.2.2 Software

The modified Android OS.

2.5.2.2.1 Management This paragraph deals with how scenarios are de-ployed on the platform

www.necoma-project.eu 26 March 31, 2016

Page 27: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

2.5. SMARTPHONE EMULATOR

2.5.2.2.2 Traffic Generation No specific traffic is generated.

2.5.2.2.3 Monitoring/Probing The modified version of the Android OSis able to check the permissions of each application installed in the mobiledevice. Whenever an application is trying to access the SMS sending systemcall, it is reported to the user. The user is then presented with a list ofoptions on how to handle this activity (e.g whitelist, blacklist, or check lateroptions).

www.necoma-project.eu 27 March 31, 2016

Page 28: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 2. PLATFORMS SPECIFICATIONS

www.necoma-project.eu 28 March 31, 2016

Page 29: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3Case Studies

3.1 DDoS Mitigation

In this use case, we focus on several scenarios that attempt to address thelimitations of existing DDoS mitigation schemes. In particular, the schemespresented below do not resort to manual configuration to enforce reaction.To that end, the technologies they leverage – either widespread (e.g., MPLS)or emerging (e.g., OpenFlow) – allow to reconfigure PEPs dynamically.

We present three scenarios, that differ from the threat they address, thetechnologies they leverage, or the environment they are deployed into:

• an OpenFlow controller defends Internet Exchanges (IXs) against DNSamplification attacks by reconfiguring the IX SDN-capable switch. Thisscenario also demonstrates how these attacks are detected early throughan sFlow monitoring system that feeds alerts about anomalous trafficto cyberdefense-oriented information sharing platform. Resilience isachieved as the controller consumes alerts and blocks the detectedflows.

• an SDN-based framework facilitate the integration of reactions in anautonomic security loop on top of an ISP network, exposing a servicethat allow customers to contribute to the defense by detecting anoma-lously large traffic directed to them. The resilience mechanism sitson top of the SDN controller as a modular application that managesflows, switches, paths, policies and alerts.

• an MPLS-based technique allows an overwhelmed network to distributethe defenses upstream in the face of amplification attacks that attemptto saturate network resources close to the target. In this scenario, theresilience mechanism is placed between the Internet gateway and thecustomer network.

29

Page 30: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

These different scenarios represent diverse instantiations of DDoS miti-gation mechanisms set across a broad range of deployment environments,in order to tackle distinct DDoS threats.

www.necoma-project.eu 30 March 31, 2016

Page 31: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

3.1.1 Evaluation Framework

3.1.1.1 Datasets

DDoS attack detection scenarios usually assume the presence of at least twotypes of traffic: namely, legitimate traffic and malicious traffic. The charac-teristics of both traffic types are not accurately defined, and vary accordingto what application flows are transported, how well the target will scaleto the demand, how the bandwidth is provisioned across the network andother mostly environmental parameters. As new services and new attacksappear, these characteristics are bound to change, preventing the systematicreuse of traffic, be it real traffic collected for replay purposes, or generatedtraffic, based on an arbitrary vector of characteristics.

Both approaches have their advantages and suffer from drawbacks. Us-ing real traffic necessitates the deployment of collecting probes, the avail-ability of storage space, and periodic update. It allows replaying traffic thatlooks the closest to real-life traffic, providing the ability to perform realisticand convincing evaluation of mitigation schemes, particularly when assess-ing their effectiveness. However, the testing environment needs to replicatethe one in which the replayed traffic has been collected, to some extent, toprevent any error during the experiments.

On the other hand, generating synthetic traffic may be the output ofa learning process, in which the storage space size will vary according tothe method used. Additionnaly, using an incremental learning or miningtechnique contributes to both updating the model (i.e., flow characteristics)and reducing storage space. This approach, though, is still far from beingable to reproduce the complex flow mixtures that traverse our networks.

In our scenarios, we have attempted to balance the specificity of theflows used and the ability to precisely reproduce their characteristics. In-deed, when focusing on a limited set of flows, we are able to better modelthem, achieving a realistic generation. But the traffic generated remainsmostly homogeneous. This makes the interpretation of false positives dif-ficult, if not anecdotic. On the other hand, it does not prevent us fromassessing the scalability and overhead of our schemes.

3.1.1.2 Metrics

Depending on the considered scenario, the metrics, as introduced in Deliver-able D4.1, may be more or less relevant. However, the list of metrics definedthen constitutes a framework, from which evaluators can pick a subset of ap-propriate metrics to assess specific aspects of the target scheme. The mainobstacle is usually the ability to deploy probes at specific locations.

In our scenarios, we strive to provide as many real or emulated compo-nents as possible in order to obtain a realistic behaviour, therefore increasingthe potential locations for probe deployment. In general, we are concerned

www.necoma-project.eu 31 March 31, 2016

Page 32: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

by the effectiveness of the resilience schemes we deploy, the overhead theycause, and their overall performance. In some scenarios, we have consid-ered a broader set of metrics when the impact on the legitimate traffic canbe measured in a more relevant manner.

Referring to what has been described in Section 2.2 of Deliverable D4.1,our scenarios deal with the following metrics:

effectiveness recall and precision allow to evaluate how correctly the re-silience mechanims are able to correctly infer legitimate and maliciouspackets, respectively. When applicable, i.e., when dealing with detec-tion mechanisms, packets were captured at both ends of the enforce-ment domain, and the two sets were compared. On the other hand,specificity measures the resilience of our schemes, i.e., its ability topreserve the legitimate traffic, in the face of major disabling attack.This is generally implemented using traffic capture tools set on net-working devices (e.g., tcpdump) or traffic statitics collection mecha-nisms such as sFlow.

overhead and performance latency measures how much delay is intro-duced by the resilience mechanism while CPU load and memory us-age allow to assess the impact of the mechanism in terms of com-puting resources. While latency is mostly relevant for in-line mech-anisms, the other two metrics can only be considered when dealingwith non-dedicated devices. In our scenarios, our mechanisms arealways born by a dedicated PDP or PEP, the controller, that decideswhere to forward the flows. More precisely, when dealing with theSDN controller, the latency induced by the operation of our schemesis an expected overhead of the application layer operation in the SDNparadigm. However, our mechanisms being provided as services, theiroperational overhead should not be of concern, as they will be onlytriggered when receiving a message of interest (e.g., an alert, or aninteresting feed). Similar to the effectivenss metrics, latency measure-ment relies on traffic capture points being set at the edges of the re-silience domain, or can be directly computed from inter-arrival times.

Time to recover considers how fast our schemes are able to enforcemitigation, and thus getting back to a normal state of the target re-sources. We have considered the times at which an alert is generatedand the time when the PEP gets reconfigured. This is easily measuredat the ISP controller, which receives the security alert from the cus-tomer, and pushes the reconfiguration rules to the switches.

Maximum flow capacity, simultaneous flow capacity and throuhputwere not really considered as these performance metrics are usuallyinherent to the technologies and infrastructure used. For example, inSDN scenarios, we can not avoid the ingress switch of the resilience

www.necoma-project.eu 32 March 31, 2016

Page 33: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

domain to issue packet-in messages each time a packet from a newflow enters the network. This inherently slow downs the first packetof a flow, but once the flow has its corresponding flow rule installed onthe ingress switch, all subsequent packets of the same flow will not bestalled and sent to the SDN controller. Additionnaly, label-switchingmethods used in the MPLS- and SDN-based scenarios allow to min-imize the network state maintained at each switch, and reduce theprocessing time compared with legacy forwarding. Therefore, thesemetrics were not considered in our scenarios, as our mechanisms usu-ally make them irrelevant.

cost these qualitative metrics deal with several impacts with respect to in-stallation, operation and recovery of our mechanisms. By design, theproposed mechanisms should be easy to install, since they usuallydo not require additional hardware to be deploy. For example, theMPLS-based scenario relies on the widespread MPLS protocol to en-force mitigation, and is usually available in most off-the-shelf networkequipment. On the other hand, with the assumption that the targetnetworks are SDN-enabled, our SDN-based scenarios will only requiredeploying our schemes at the control plane, whether it is a customcontroller or applications running on top of the controller.

deployment these metrics are not really measurable and actually constitutethe assumptions of our deployment environments.

quality of experience these metrics, although quantitative, are not appro-priate for all scenarios. Indeed, their scenario-specific characteristicaccommodates situations in which application flows directed to cus-tomers are the target of the attacks. In the SDN-based DDoS mitigationas a Service scenario, video streaming is the target of flooding attacks.We are able to measure the impact of the attack directly through thevideo streaming client software.

3.1.1.3 Experiments

DDoS mitigation is a well-traveled use case, but testing these mechanismsrequire specific environments to be more convincing. In particular, a well-defined scenario including a target network and specific attacks to be miti-gated are generally provided in the three scenarios presented in NECOMA’sDDoS mitigation use case. Moreover, the traffic mixture can often be prob-lematic as such experiments can not be run in real networks. In our sce-narios, we attempt to recreate conditions of real-life traffics and attacks byselecting appropriate tools and setting appropriate parameters to generaterealistic traffic. For example, in the DDoS mitigation as a service scenario,video streaming is used as it represents the major application traffic on the

www.necoma-project.eu 33 March 31, 2016

Page 34: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Internet. On the other hand, the MPLS-based defense has benefited fromthe expertise of 6CURE which proposes a similar tool to defend its cus-tomers against DDoS attacks. Therefore, the network environments andsettings are in line with what their customers usually display. Additionally,the technologies used to address the threats are, when implemented, at thestate-of-the-art of what is available off-the-shelf.

www.necoma-project.eu 34 March 31, 2016

Page 35: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

Victim Network

VictimHost

The Internet(Transit ASs, IXs )

Vulnerable Hosts

send request packets with spoofed address (victim address)

Attackers

reply amplified messages to the victim hosts

Figure 3.1: The amplification attack concept

3.1.2 SDN-based DDoS Mitigation

This scenario demonstrates an SDN-based DDoS mitigation mechanism de-veloped in NECOMA and introduced and evaluated in Deliverables D3.3 andD3.5, respectively. We describe the scenario background, and the results ofthe evaluation, according to two metrics we defined in Deliverable D4.1.

3.1.2.1 Overview

UDP amplification-based attacks are an emerging threat on the Internet.Figure 3.1 shows an example of such types of attacks. In general, the attacktraffic is coming from multiple ingress points of the ISP towards victim hosts.The traffic, which is amplified by resolvers, is aggregated on multiple pointsto victim hosts. The aggregated traffic saturates the links between vulnera-ble hosts and victim hosts. Even if the attack could be defended against infront of the victim hosts by Access Control Lists (ACL), a firewall or an IPS,we cannot eliminate the traffic from links upstream on the Internet. Theattack imposes congestion on link bandwidths of inter-domain networks be-tween attackers and victims. Accordingly, the attacks must be mitigated notonly at the victim network but also on the network links from the attackersto the victims. We need mitigation mechanism on every networks againstthe DDoS attacks.

In this scenario, we demonstrate a simple SDN-based mitigation mecha-nism against Domain Name System (DNS) amplification attacks, which rep-resent one common kind of DDoS attack. The mitigation mechanism can be

www.necoma-project.eu 35 March 31, 2016

Page 36: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Probe Node

VictimOpenFlow Switch

OpenFlow Controller DetectorNECOMATter

Open Resolver(amplifier)

Attacker

L2 Switch

L2 Switch

Figure 3.2: Experimental topology of the scenario

deployed at the ISP or at enterprise networks. This scenario demonstratesa pipeline from data acquisition and threat detection to information sharingand and mitigation.

The mechanism is comprised of an OpenFlow controller and a switch.An sFlow monitoring system is used to detect the amplification attacks.When the monitoring system detects traffic of which volume is over a certainthreshold, the system posts the information to the NECOMAtter streams.NECOMAtter is a knowledge sharing platform, spreading cyber threat in-formation among bots (automated agents) and humans in a way similar toTwitter. When information about a threat is posted, a network operatorwho monitors the feed may initiate a mitigation process against the postedattack through the SDN controller. Accordingly, we can block the attack ondeployed OpenFlow switches, and not only at the edge of the network.

3.1.2.2 Topology

Figure 3.2 shows the topology of this scenario. The topology is comprised of3 areas (attacker area, victim area and control area). The topology consistsof the following entities.

3.1.2.2.1 Network This scenario requires a set of IP-capable networks.The networks should be comprised of three segments. One segment will host

www.necoma-project.eu 36 March 31, 2016

Page 37: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

emulated attackers, another one the victim node, and a third segment theresilience mechanisms. The segments have to be connected with switches,and in particular, an OpenFlow-capable switch should bridge the attackerand the victim segments.

3.1.2.2.2 Attacker hosts The hosts generate malicious DNS queries, withthe source IP address of the packets spoofed as the victim node. Also, am-plification nodes are hosted on the same segment as the attacker hosts Theywork as open resolvers that respond to the spoofed DNS queries.

3.1.2.2.3 Detector The node used to detect emulated DDoS attacks. Thedetector should be able to output information on the attacks. When a DDoSattack is detected, the detector shares the attack information with otherhosts in the same segment. Shared information contains source IP addressesof the attack.

3.1.2.2.4 Mitigator This mitigator node is working on the controller inthe topology. The node used to initiate a mitigation against an attack on thenetwork. It is triggered by the reception of attack information shared viathe information sharing platform.

3.1.2.2.5 Probe Node This node is used to measure latency between thisnode and the victim host. We use ICMP packets for measuring the latencyon the platform.

3.1.2.2.6 Victim host The node hosting the service targeted by the am-plification attacks. The node hosts a web service that displays a simple webpage to the users. This node is required to reply to ICMP requests in orderto check for reachability.

3.1.2.2.7 Information sharing platform The platform is used to shareattack source IP addresses in text format between the detector and the miti-gator. At the very least, the platform must be able to share the IP address ofan attack source in text format.

3.1.2.3 Experiments

The scenario topology was deployed on the cloud platform, as describedin Section 2.1, for conducting experiments. We use 3 physical servers asthe virtual environment and deploy our SDN-based DDoS mitigation mech-anism. The topology entities are deployed on the platform.

Figure 3.3 shows the workflow of this scenario. It can be described asfollows:

www.necoma-project.eu 37 March 31, 2016

Page 38: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

DetectorMitigator/Controller

SDN switch

NECOMATter

Attacker

Open Resolvers

⑥⑤

Figure 3.3: Scenario workflow

1. Attacker hosts send spoofed DNS queries to Open Resolvers.

2. Open Resolvers reply with DNS answers to the spoofed target.

3. Detector Switch sends flow information to the Detector.

4. Detector identifies malicious flows by threshold-based detection algo-rithm.

5. Detector posts the information of malicious flows to NECOMAtter.

6. Mitigator monitors NECOMAtter timeline and picks up related infor-mation regarding amplification attacks, and decides to install mitiga-tion rules into the Mitigator Switch.

7. The attack (malicious) traffic is the only traffic impacted by the miti-gation rules, and legitimate traffic is allowed to pass through.

This scenario focuses on demonstrating a fully automated DDoS mitiga-tion, leveraging SDN as well as connections among the entities. It featuresa simple network topology, but the automated mechanism can be applied toother, more complex networks, provided they are based on SDN switches.Aside from SDN technology, automated defense is made possible thanks toother key components. NECOMAtter is a general message sharing platformand feature APIs that can inter-connect analysis modules and defense mech-anisms through the timeline function. That way, network operators or engi-neers can simply connect the sharing platform to their networks. The users

www.necoma-project.eu 38 March 31, 2016

Page 39: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

can replace the SDN switches by other security devices. At the moment,many security devices such as firewalls can be configured the rules from ex-ternal nodes through RESTful APIs or SDN protocols. We can easily makean automation tool that remotely control the security devices.

We measured the following two metrics to evaluate the mitigation mech-anism on the platform.

Time to recover measures the delay between the detection time of a DDoSattack and the time the attack traffic is effectively mitigated by themitigator node. DDoS attacks have to be quickly filtered on the net-works to minimize downtime of the victim services. The measurementresults show how quickly the mechanism responds to attacks.

Latency is induced to legitimate traffic when the SDN-based mitigationmechanism is enabled as the mitigation mechanism may intercept le-gitimate traffic as well as attack traffic. This metric measures how themitigation itself affects legitimate traffic. The measurement revealsadditional latency for the legitimate traffic caused by the mechanism.

3.1.2.4 Results

We observed latency between the victim host and the probe host with andwithout the mitigation mechanism. When activated, the mitigation mech-anism induced, on average, a 0.397-millisecond delay between the victimand the probe host. The mitigation mechanism does not induce a longerdelay than when it is inactive.

Figure 3.4 shows a box plot of the time to recover measured in thisscenario. The experiment was repeated 50 times on the platform. In thisgraph, the y-axis is the time between attack detection and mitigation. Here,the measured time includes collecting sFlow data from the switch, analyz-ing the flow data, sharing detection results, getting information from theNECOMAtter by the mitigator. From this graph, the mitigation mechanismrequires up to 10.8 seconds in average to be started. The reaction time isquite short compared to the duration of common DDoS attacks. Accordingto a report from Arbor Networks[1], 77% of all DDoS attacks are less thanone hour in duration.

3.1.2.5 Conclusion

In this section, we reported on the demonstration of an SDN-based DDoSmitigation mechanism. We demonstrated a complete security pipeline fromdata acquisition to defense. While a simple threshold algorithm was ap-plied for DDoS detection, this scenario does not make assumptions on aspecific algorithm to be used, but we are aware of the time delay inducedby using more complex algorithms. Furthermore, we can change the virtual

www.necoma-project.eu 39 March 31, 2016

Page 40: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

10.0

10.5

11.0

11.5

time

[sec

ond]

Figure 3.4: Time to Recover

SDN switch by other OpenFlow switches. Thus, you can easily expand thepipeline model to the other use case such as enterprise networks.

The demonstrated mitigation mechanism can be applied to various net-works. In this scenario, we adopted the mechanism as a security function onan Internet eXchange (IX). The developed SDN-based IX (SDN IX) is com-prised of OpenFlow switches and a controller. The SDN IX allows ASes tobe inter-connected through the OpenFlow switches and dynamically controlconnections and traffic across ASes. Legacy IXs do not usually provide anymitigation services for their customer ASes. However, AS operators requiredefense mechanisms on inter-domain networks to effectively mitigate largescale DDoS attacks. More information on the SDN IX can be found in Section3.3 of Deliverable D3.51.

1Deliverable D3.5: Countermeasure Application - Results

www.necoma-project.eu 40 March 31, 2016

Page 41: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

3.1.3 DDoS Mitigation as a Service

In this scenario, we evaluate a framework to integrate and automate thedifferent stages of DDoS mitigation, in order to complete the whole securitypipeline, from prevention and detection to characterization and response.Assuming an SDN infrastructure, the framework does not require additionalsoftware modules or dedicated hardware devices for mitigating malicioustraffic, reducing the non-trivial overhead associated with deployment, op-eration and management of DDoS mitigation mechanisms. The frameworkallows for a fast and coordinated detection and response by facilitating thecollaboration between ISPs and their customers. In particular, this frame-work allows (1) the customers to detect DDoS attacks, and share alert in-formation with ISPs in an automated way; (2) the ISPs to provide a specificinterface or function that enables its customers to request for DDoS miti-gation services, and enforce corresponding security policies without heavyhuman intervention.

In this particular scenario, we attempt to preserve the quality of experi-ence of users streaming videos. Our evaluation relies on well-know qualityof experience (QoE) metrics, in order to assess the efficiency and timelinessof our framework in maintaining the service.

3.1.3.1 Overview

As shown in Figure 3.5, our proposed framework is distributed across theISP and customer networks, both of which are logically separated as controlplane and data plane. As such, four essential functional components, i.e.,monitoring, detection, policy, and mitigation actions, can be systematicallyand automatically integrated, despite the fact that they are managed by ISPsand customers independently. In particular, monitoring and detection runin customer network, while policy and mitigation are managed by ISPs. Inorder to close the gap between these four functional components and buildthe pipeline between monitoring and mitigation, which is one of the majorobjectives of our NECOMA project, a set of technical challenges need to betackled.

In this scenario, users are located in the customer network and are re-questing streaming services. Attackers compete to flood their network links,degrading the quality of their video streams. Once detected at the customerside, the ISP can take actions to restore the quality of service. The opera-tional workflow (labeled with step numbers), of how flows traverse the ISPnetwork and are processed, is described as follows,

1. As network traffic enters the ISP network, the SDN controller assignsa unique flow identifier, FlowID, for each new flow by leveraging thepacket-in message. FlowID will be used by the customer controller

www.necoma-project.eu 41 March 31, 2016

Page 42: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Figure 3.5: The proposed framework enabling DDoS mitigation as a service

to request the ISP for mitigation service. Also, the flows are assignedlabels for fast forwarding in the ISP network.

2. Flow statistics are periodically collected via OpenFlow or sFlow agentsin the customer network.

3. Anomaly detection engine runs with an attack database, which caninteract with external databases via certain threat exchange methodslike n6.

4. Detection engine generates security alerts in the presence of maliciousand suspicious traffic flows, and triggers reaction via local policy en-gine, which is in charge of generation and installation of policy rulesat the ingress switch for traffic processing.

5. The FlowID of suspicious and malicious flows are encapsulated into se-curity requests to the Mitigation engine residing in the SDN controller,at the ISP side, via a security API.

6. The Mitigation engine communicates with the policy engine, and

7. The Mitigation engine further checks with the path lookup module(step 7) to infer and distribute the effective mitigation countermea-sures across the switches.

8. Mitigation actions, e.g., traffic redirection to the security middlebox,are finally taken by simply updating the labels of the flows of interest.

www.necoma-project.eu 42 March 31, 2016

Page 43: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

Figure 3.6: The toplogy used in our experiments: the applications of La-bels and FlowID are also illustrated, where action FWD means packet/flowforwarding.

3.1.3.2 Topology

The topology of our experimental platform is shown in Figure 3.6, wherethe ISP and customer networks are managed by their respective SDN con-trollers. For simplicity, two routing paths are configured for the ISP net-work: one is QoS-guaranteed and exclusively used for legitimate traffic (go-ing through switches S1, S2, and S4), while the other is for suspicious ormalicious traffic (going through S1, S3 and S4). In the customer network,the OpenFlow switch S5 is attached to the customer controller. Also, threehost machines are virtualized in the platform, respectively serving as legiti-mate host (e.g., serving video contents), attack host and customer machine.

Table 3.1: Platform specifications

Component Quantity Specification or Configu-ration

Role

IBM RackSwitch G8052 5 48 Gigabit ports OF enabled switchDELL server r620 2 8-core 2.9 GHz CPU,

16GB RAM, 10 Gb/s Eth-ernet NICs

SDN controller host (ISP, cus-tomer)

Hosts 3 Ubuntu server 12.04.2,6-core 3 GHz CPU, 4GBRAM,20GB HDD

Playing as video streamingserver L, attack host A andvideo streaming client C

SDN Controller 2 Ryu 3.20 Controlling switches in ISP (S1

to S4) and customer networks(S5)

www.necoma-project.eu 43 March 31, 2016

Page 44: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Hardware and software specifications of the SDN-based testbed, as de-scribed in Sect. 2.3, in particular the components and specifications of theplatform, are summarized in Table 3.1. On the legitimate host, the servicewe have chosen to simulate for this use case is video streaming as men-tioned previously. Indeed, video traffic accounts for more than 70 percentof all consumer Internet traffic [3] today. Specifically, TAPAS [2], a videostreaming tool is used to generate video traffic between the customer ma-chine C and the legitimate host L, and in total 10 TAPAS video clients areemployed to generate the legitimate video traffic. On the attack host, Bot-Net Simulator(BoNeSi) is used to generate volumetric DDoS attacks with ahigh traffic rate, i.e., nearly 250, 000 packets per second.

3.1.3.3 Experiments

The purpose of our experiments is to validate the effectiveness and timeli-ness of our proposed DDoS mitigation prototype in terms of several perfor-mance metrics of concern.

Threat model. We are particularly concerned with DDoS attacks againstthe customer of an ISP, and we focus on a single customer network. Also,we restrict our focus on DDoS flooding attacks, two of which are explainedbelow.

• Target flooding attacks: an adversary sends a huge amount of trafficto the customer network to rattle up the customer’s communication.UDP flood, TCP SYN flood and ICMP flood can be used to achieve thatpurpose.

• Bandwidth flooding attacks: an adversary aims to congest the commu-nication link between ISP and customer networks. This type of attackusually causes collateral damage to other customers which share thesame ISP as the target customer network, because it floods the up-stream links of the ISP. UDP flood and ICMP flood are widely used tolaunch such attacks.

We are confident that this scenario can be extended to a multiple-customerscenario, as each customer network only needs to connect through the Se-curity API exposed by the ISP controller.

Evaluation metrics. In our scenario, the protected target is the videostreaming service provided to the customer. The experimental purpose is tomeasure the extent of quality deterioration in the face of aforementionedDDoS attacks, in terms of those metrics specified in Table 3.2, which areessentially related to QoE. For example, DDoS flooding attack can cause thetime to rebuffer to increase, average buffer length to decrease, and durationof player in the paused mode to increase, eventually leading to serious dete-rioration of overall QoE. These QoE metrics are measured thanks to TAPAS

www.necoma-project.eu 44 March 31, 2016

Page 45: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

Table 3.2: Defined metrics to measure the impact of DDoS attacksMetric Definition Unit Impact of attack

Time to rebuffer the time taken to rebuffer thevideo when the streaming buffergets empty due to network con-gestion

seconds increases

Time to start the time taken to complete theinitial buffering to start the video

second increases

Duration of pausedmode

the time player is in the pausedmode for the buffer to fill up

seconds Paused for moretime

Average buffer length the buffer length of the player seconds increasesAverage goodput rate of data transfer KB/sec decreasesPlayer status indicates whether the video

streaming is paused or notBoolean values: true(Playing) or false(Paused)

becomes false

Figure 3.7: Time to rebuffer

clients. We took the average of all clients’ collected values to compute ourmetrics.

3.1.3.4 Results

We conducted several rounds of tests to comparatively study the given met-rics in three cases:

• Video streaming services in normal conditions (without attack traffic);

• Video streaming services under different attacks without mitigation;

• Video streaming services under attack with our mitigation module ac-tivated.

www.necoma-project.eu 45 March 31, 2016

Page 46: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Figure 3.8: Status of the player

Time to rebuffer. The client was not able to play the video as the stream-ing buffer got depleted due to the overwhelming attack traffic. If this condi-tion holds for a long time, the quality of user experience (QoE) is degraded.Figure 3.7 well illustrates this fact: the TAPAS video client was able to com-plete the video transmission without any interruptions when there was noDDoS traffic, and the buffer length was maintained above 15 seconds. Incontrast, under DDoS traffic, the video client was unable to buffer the video,leading to the depletion of the playout buffer down to zero. When the mit-igation scheme was activated by the ISP upon the request of the customercontroller, the playout buffer returned to a normal level, allowing the videoclient to play the video again. In particular, when the buffer started, it took10 to 15 seconds for the playout buffer to return to the normal level.

Player status. It is not surprising to observe that during the experi-ments without DDoS traffic, the status of the video player was true, andit was always playing except during the initial buffering phase, as shownin Figure 3.8. However, when DDoS attack occurred, the player status re-mained to true for some time thanks to the initial buffering, then the playerwas not able to fill the buffer, eventually causing the player status to be-come false. When the mitigation module was activated, the player was ableto buffer after approximately 500 seconds in the case of ICMP and TCP SYNflood attacks, so the status became true again. However, in case of a UDPflood attack, the player was able to buffer close to 300 seconds before theplayer status becomes true. Without mitigation, the player status remainedto false once the attack traffic starts.

Average buffer length. We also measured the average buffer length,and observed that in the case of UDP, TCP SYN and ICMP flood attacks,

www.necoma-project.eu 46 March 31, 2016

Page 47: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

Figure 3.9: Average buffer length

Figure 3.10: Duration of the player in paused mode

we were able to maintain the buffer length close to 12 seconds with ourmitigation module activated. In the case of video traffic without attack, thebuffer length is maintained close to 18 seconds. However, the average bufferlength was close to 6 seconds when the mitigation module is not activatedduring attack.

Duration of paused mode. This metric was evaluated to compare theduration of the player in paused mode during initial buffering and in themidst of attack. The results of this metric are depicted in Figure 3.10, whichhas the following implications:

www.necoma-project.eu 47 March 31, 2016

Page 48: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Figure 3.11: Average Goodput

• In normal conditions, the total duration that the player is in the pausedmode is approximately 2 seconds during the initial buffering phase.

• The video player was paused for 12 seconds, 15 seconds and morethan 30 seconds in the presence of UDP flood, TCP SYN flood, andICMP flood attacks, respectively.

• The video buffer was populated again and the player resumed playingwhen the mitigation module was activated.

• Without the mitigation module activated, the player was not able tobuffer the video to play.

We also examined the average goodput, and observed that it was main-tained close to 700 KB/sec in the absence of attacks. Then the average good-put dropped down below 200 KB/sec when DDoS attack was launched, andit could not return to a normal level. However, when our mitigation mod-ule was activated, the average goodput could be maintained at 600 KB/secin the presence of UDP flood attack, and could be maintained close to 500KB/sec in the presence of ICMP and TCP SYN flood attacks, as shown inFigure 3.11.

3.1.3.5 Conclusion

We leveraged the programmability of SDN to develop an autonomic DDoSmitigation framework, with the objective to facilitate the collaboration be-tween ISPs and their customers and create a pipeline from traffic monitor-ing to attack mitigation. In particular, we demonstrated that the centralized

www.necoma-project.eu 48 March 31, 2016

Page 49: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

SDN controller can significantly simplify the deployment of security func-tions (such as blocking or redirecting) and the enforcement of mitigationpolicies without human intervention. Also, the threat intelligence sharingbetween the ISPs and customers can be established thanks to the communi-cation between SDN controllers. The experimental results clearly indicatedthat our proposed framework can ensure that the protected asset, a videostreaming service, was able to maintain satisfactory performance in the pres-ence of flooding attacks, in terms of several major QoE metrics.

www.necoma-project.eu 49 March 31, 2016

Page 50: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.1.4 Pushing Defenses Upstream

In this scenario, we place ourselves in a situation where the DDoS attacktraffic saturates resources close to the target as the attack traffic convergestowards the victim. The objective is to be able to distribute the defensesupstream from the choking point in order to gain headroom at target level.

By saturating resources, we mean, for example, network links and equip-ment, but also potential on-premise defense mechanisms, such as firewalls,DDoS mitigation systems, etc.

By distributing the defenses upstream, we mean being able to filter outat least part of the attack. We consider that there is no need to be able tofilter the whole attack with a distributed mechanism, for two reasons:

• Equipment like routers provide only coarse-grained filtering capabili-ties, based on layer 3, and eventually layer 4 criteria. Such filteringwill capture suspicious traffic, which in addition to attack traffic cancontain legitimate traffic as well.

• We suppose that we have on-premise defense mechanisms, capable offiner-grained filtering, at our disposal.

Thus, as long as the attack volume remains below the congestion level,we want it to reach the on-premise defense mechanisms – this allows us tolimit the collateral damage caused by the coarse-grained filtering when it isnot absolutely needed.

By headroom, we mean gaining sufficient effect so that on-premise mech-anisms can function. Depending on the situation, this might mean justavoiding saturation of the link or the router (i.e., the bottleneck was thenetwork) or avoiding the saturation of the on-premise defense mechanismsin case their capacity is lower than the network’s.

The saturation at target level is a reality many organizations having lessnetworking capacity than an ISP face regurarly.

3.1.4.1 Overview

The mechanism uses two MPLS tunnels between the ingress and egresspoints of the traffic, one used for legitimate (all traffic not considered explic-itly as suspicious) and one for suspicious (not necessarily malicious, giventhe coarse granularity of classification we have at this level).

The tunnel for suspicious traffic has less resources available than thetunnel for legitimate traffic, thus in case of congestion the traffic travellingin the suspicious tunnel suffers first, allowing the traffic in the legitimatetunnel to continue to flow.

The mechanism uses router Access Control Lists (ACL) to control whichtraffic is sent to which tunnel, and in the scenario, the ACLs are defined

www.necoma-project.eu 50 March 31, 2016

Page 51: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

using source and destination IP addresses (other choices could be made,e.g., using UDP/53 as source and large packet size to define suspicioustraffic). The information to define the ACLs is obtained via local detectionmechanisms and through external threat analysis information provided byNECOMA partners via n6 API.

The attacks most likely to cause resource saturation, before the on-premise defenses, are amplification attacks. In this scenario, the focus ison DNS amplification attack.

3.1.4.2 Topology

The scenario is executed on the MPLS testbed, described in Sect. 2.2, onwhich we deploy additional elements for the scenario, shown in Fig. 3.12.The MPLS tunnels for malicious traffic (red, filled lines) have a rate limitat 5 Mbps each, thus allowing a maximum of 10 Mbps (i.e., 10% of theavailable bandwidth) of traffic considered as suspicious to continue to flowas long as the legitimate traffic is not impacted. There is no rate limiting forthe legitimate tunnels (shown in green, filled lines).

Figure 3.12: The demonstrator on the MPLS testbed. Black, filled linesshow IP connectivity, green and red filled lines show MPLS tunnels, hashedlines depict information flow - green for attack and threat related, red forconfiguration orders, and blue for measurement collection

www.necoma-project.eu 51 March 31, 2016

Page 52: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.1.4.3 Experiments

The scenario workflow is as follows:

1. Monitoring traffic (ping, httperf) is launched.

2. Replay of legitimate traffic starts towards both routers R1 and R2 andfor a few minutes, allowing measures to be taken in order to build aprofile of the state before attack.

3. Replay of attack traffic then starts at the lowest attack volume and runfor a few minutes.

4. Iteratively, we step up the attack volume, awaiting a few minutes be-tween each increase.

5. The migitigation mechanism is triggered only at this moment, in orderto observe the network performance metrics and volumes at differentattack rates without mitigation in place.

6. We use the following two variants for emulating the detection of theattack and the identification of attack sources:

(a) Using a 6cure TP as on-premise detection mechanism, providinginformation about source IP addresses generating the traffic thatis being dropped.

(b) Fetching a list of known DNS amplifiers available in NASK’s datasetsvia n6 API.

These sources will be considered as suspicious and traffic sent by themis redirected to the MPLS tunnel reserved for suspicious traffic. Thistraffic will suffer first when the resource usage on the links exceedsthe allocated thresholds (i.e., in case of congestion).

The legitimate traffic is replayed from two different packet captures, onetowards R1 and another towards R2. The two replays use the same constantbandwidth (and thus at different packet rates due to differences in the sizeof packets contained in the two captures for each /24 destination extractedfrom the MAWI capture used for legitimate traffic, see Sect. 2.2.2.2.2 fordetails) during each attack step.

3.1.4.3.1 Experiment timeline We have reproduced the chronology ofone test run below:

• Start – 09:17: the platform runs empty with only monitoring traffic

• 09:17 – 09:25: only legitimate traffic is being replayed

www.necoma-project.eu 52 March 31, 2016

Page 53: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

• 09:25 – 09:28: attack traffic is replayed at the lowest intensity,

• 09:28 – 09:32: attack intensity increases

• 09:32 – 09:34: attack intensity increases

• 09:34 – 09:40: attack reaches its maximal intensity

• 09:40 – 09:55: mitigation is activated and starts iteratively reconfig-uring routers R1 and R2, which start discarding more and more attacktraffic

3.1.4.3.2 Evaluation Metrics The mitigation mechanism is assessed interms of

• overhead with latency caused to legitimate traffic;

• efficiency with precision, recall and specifity; and

• time to recover with the ACL deployment delay.

It is important to mention the following definitions to better interpretthe results: in this scenario, a “true positive” is a malicious packet thathas been dropped, a “false positive” is a legitimate packet that has beendropped, a “true negative” is a legitimate packet that is able to reach thetarget, and a “false negative” is a malicious packet that reaches the target.On the platform, packets can be dropped only due to congestion, regard-less of whether they are considered legitimate or suspicious, since there areno explicit packet dropping decisions made by the mitigation mechanism.However, considering a source IP suspicious reduces its likelihood of pack-ets from that source to survive in case of congestion.

With these definitions, the precision, recall and specificity measure thefollowing characteristics:

Precision or positive predictive value gives the likelihood of a packet froma source we have considered as suspicious, based on information pro-vided by the 6cure TP or obtained via the n6 API, being dropped bythe routers due to congestion.

Recall or true positive rate gives the likelihood of a packet sent by a sourceIP we have flagged as suspicious being actually dropped due to con-gestion.

Specificity or true negative rate gives the probability of a packet from asource considered as legitimate being forwarded to the destination,despite eventual congestion.

www.necoma-project.eu 53 March 31, 2016

Page 54: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.1.4.4 Results

This section discusses the results of our experiments, describing the through-puts observed during the test run in Sect. 3.1.4.4.1; the mechanism over-head in terms of latency during the attack, with and without mitigation, inSect. 3.1.4.4.2; the efficiency in terms of precision, recall, and specificityin Sect. 3.1.4.4.3; and the time to recover in terms of ACL load times inSect. 3.1.4.4.4.

3.1.4.4.1 Throughput Figures 3.13, 3.14, 3.16, and 3.16 show the bits-and packets-per-second rates measured at R1’s and R2’s ingress and egressports, as well as, at R3’s egress port for one test run with five different attackintensities.

Figure 3.13: Traffic volumetry in bps between R1 and R3

Figure 3.14: Traffic volumetry in pps between R1 and R3

Figure 3.15: Traffic volumetry in bps between R2 and R3

www.necoma-project.eu 54 March 31, 2016

Page 55: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

Figure 3.16: Traffic volumetry in pps between R2 and R3

The impact of mitigation can be seen at routers R1 and R2, causing thetraffic to be dropped at R1’s and R2’s egress ports. The resource exhaustiontakes place at R3 (router processing capacity) and/or on R3 egress (egresslink capacity).

In the figures, R1 inbound corresponds to attack and legitimate traffic,R1 outbound corresponds to attack and legitimate traffic with the effect ofmitigation visible. R3 outbound shows the total amount of traffic (includingthe traffic from R2 as well), i.e., both attack and legitimate traffic, with thecongestion effects.

The platform egress (through R3) is a 100 Mbps link and we can observe,from Figs. 3.13 and 3.15, how the light blue line reaches its maximal level of95 Mpbs from 09:25 onwards, as the attack begins and the overall volumeexceeds the egress link capacity. Later, when the attack intensity furtherincreases, the R3 egress volume drops lower than the egress link capacityat 09:32, indicating equipment overload. This phenomenon is repeated at09:34 when the attack intensifies. The congestion also disturbs the linkbetween R1 and R3 as can be seen in the short drops in the R2 egress volume(orange line) between 09:30 and 09:40 in Fig. 3.13.

The mitigation starts at 09:40 and its effect is clearly visible e.g., inFig. 3.15 in the R2 eggress (red line) as the ACLs are being pushed graduallyonto the routers and start routing suspicious traffic through the tunnel withless resources.

3.1.4.4.2 Overhead Caused by the Mechanism To understand the over-head caused by the mitigation, we measured the latency on the platform.When looking at the latency in Fig. 3.17, we can see that as soon as the firstbatch of attack traffic hits in, around 09:25, the round trip time observedjumps from the millisecond range to between 30 and 50 milliseconds, ac-cording to the path taken. In addition, we already start losing some ICMPecho messages as shown by Fig. 3.18.

After 09:40, the impact of the mitigation becomes gradually visible in thelatency measures and ping droprates. As soon as the congestion disappears(droprate drops down to 0%), the ping latency comes back to the levels

www.necoma-project.eu 55 March 31, 2016

Page 56: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Figure 3.17: Round trip time for ping messages on the platform before theattack, during the attack (before and after mitigation)

Figure 3.18: Ping drop rate on the platform before attack, during attackbefore and after mitigation

observed before the attack. In fact, the mitigation mechanism does not addany notable latency per se, as the mitigation is achieved via normal routingoperations. On the contrary, during the attack and without mitigation, thelatency is at its shortest, over 50 ms for a network with just a few hops.

We would like to point out the situation from 09:47 onwards. At thattime, the traffic from all the amplifiers participating in the attack is consid-ered suspicious, but the volume coming through R1 and R2 towards R3 isslightly higher than before the attack (this is somewhat difficult to see withthe level of detail in figures). Indeed, a fraction of the suspicious trafficis still being routed towards the destination, as intended. Here, the ACLsdo not provide a sufficiently fine-grained detection as to discriminate legiti-mate and suspicious traffic. As long as the legitimate traffic does not suffer,we want to let the suspicious traffic reach the local, more precise mitigationmechanism (a 6cure TP in this case, as shown in Fig. 3.12).

3.1.4.4.3 Efficiency: Precision, Recall and Specificity Figures 3.19,3.20, 3.21, 3.22 show the computed precision, recall and specificity forpacket based statistics obtained by analyzing traffic captured at the MPLSnetwork ingress and egress points. The metrics have been obtained from atest run where, instead of holding back the mitigation (as described in 3.1.4.3,step 5), the mechanism was fully operational right from the beginning. Thetest run described above held back the mitigation in order to demonstrate

www.necoma-project.eu 56 March 31, 2016

Page 57: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

the adverse effects of the attack on throughput and latency, whereas to mea-sure the effectiveness of the solution this neither necessary nor desirable.

0

0.2

0.4

0.6

0.8

1

0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000

Attack volume (pps)

PrecisionRecall

Specificity

Figure 3.19: Precision, recall, and specificity; based on packet drops for R1and 6000 attack flows as the function of attack volume, without mitigation

0

0.2

0.4

0.6

0.8

1

0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000

Attack volume (pps)

PrecisionRecall

Specificity

Figure 3.20: Precision, recall, and specificity; based on packet drops for R1and 6000 attack flows as the function of attack volume, with mitigation

As can be seen, the basic behavior of these metrics is similar on thetwo routers (i.e., the packet size does no affect the macroscopic behavior ofthe mechanism). In the cases where no mitigation was applied (Figs. 3.19

www.necoma-project.eu 57 March 31, 2016

Page 58: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

0

0.2

0.4

0.6

0.8

1

0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000

Attack volume (pps)

PrecisionRecall

Specificity

Figure 3.21: Precision, recall, and specificity; based on packet drops for R2and 11000 attack flows as the function of attack volume, without mitigation

0

0.2

0.4

0.6

0.8

1

0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000

Attack volume (pps)

PrecisionRecall

Specificity

Figure 3.22: Precision, recall, and specificity; based on packet drops for R2and 11000 attack flows as the function of attack volume, with mitigation

and 3.21), we can see some impact when the links start to saturate with anattack volume around 7500 pps. Without any mitigation, both legitimateand suspicious packets are being forwarded approximately at their propor-tional volumes. This causes a rather heavy drop rate for suspicious traffic(i.e., true positives), increasing the precision and recall value, while at the

www.necoma-project.eu 58 March 31, 2016

Page 59: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

same time the legitimate traffic suffers as well, causing the specificity todrop heavily.

When mitigation is applied, much more legitimate traffic reaches thetarget (i.e., true negatives) and thus the specificity remains close to one.Precision and recall increase at lower attack volumes compared to the casewithout mitigation, precision reaching values close to one indicating thatmost dropped packets are actually from suspicious sources. At the sametime, recall remains at lower values since part of the suspicious traffic con-tinues to flow to the target.

3.1.4.4.4 Deployment Latency and ACL Load Times It can be seen inthoughput and latency figures (Figs. 3.15 and 3.17), that the impact ofmitigation appears gradually and after a delay following the activation ofmitigation at 09:40.

This delay has two elements:

Detection The identification of suspicious IP addresses by 6cure TP takes awhile, causing a detection delay.

Deployment Once a set of suspicious IP addresses has been identified, in-sertion of corresponding ACL entries to the routers creates a deploy-ment delay.

The detection delay is caused by either

• the way 6cure TP reports the mitigation that takes place – as the equip-ment is oriented to DDoS mitigation and often handles malicious ac-tivity concerning several thousands (or more) source IPs, thus it re-ports only a set of n most significant offending IPs (even if it might bedropping traffic from all of them) ; or

• the delay we have querying and processing the threat analysis infor-mation via n6 API for known amplifiers.

The detection delay visible in the above mentioned figures is due to6cure TP behavior: the mitigation mechanism uses the n most significantoffending IPs provided by 6cure TP, flags those n IPs as suspicious; awaitsfor a new set of the n most siginificant offending IPs from 6cure TP, and soon.

As the detection delay is not inherent to the mitigation mechanism, weare interested in the deployment delay, the time it takes to push new ACLson the routers. To asses this delay, we have measured both the insertiontime of different number of ACL rules into a router with no ACLs, and thecumulative insertion of ACLs in batches of 10k rules.

Figure 3.23 shows measurements for the former case, the time it takesto insert an ACL on the Idle router R1 as the function of the number of rules

www.necoma-project.eu 59 March 31, 2016

Page 60: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

defined by this ACL. Figure 3.24 depicts the insertion time of 10k ACL ruleson R1, both idling and under load, as the function of the number of rulesalready present on the router R1.

We can see that the delays are sufficiently long to be taken into accountwhen evaluating the suitability of the approach in a given environment, thatthe cumulative insertion of ACL rules is not a linear function of the numberof rules already loaded, and that the router load has an impact. It should benoted that the actual values for these measures are likely highly dependenton the underlying router hardware.

0

100

200

300

400

500

600

700

800

0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000

AC

L in

sert

ion t

ime (

s)

Number of inserted ACL rules

Figure 3.23: ACL loading time as the function of the number of insertedrules on a router with no rules loaded

3.1.4.5 Conclusion

We have described a DDoS mitigation scenario exploring the possibilities ofusing existing MPLS functionality in upstream routers to limit the amount ofsuspicious traffic reaching the target. The main objective was to be able tolimit the amount of traffic reaching the target to a level below the saturationthreshold, even while under a volumetric DDoS attack. The secondary ob-jective was to be able to let a proportion of the suspicious traffic to reach thetarget for more fine grained analysis than can be performed by the routers,as long as the local saturation threshold is not reached.

We deployed the developed defense mechanism on a demonstration plat-form consisting of Cisco routers. The defense mechanism was coupled witha detection mechanism provided by the 6cure’s commercial DDoS mitigationsolution deployed in front of the target and used also external threat analy-

www.necoma-project.eu 60 March 31, 2016

Page 61: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

0

200

400

600

800

1000

1200

1400

1600

1800

0 50000 100000 150000 200000 250000

AC

L in

sert

ion t

ime (

s)

Number of ACL rules loaded into the router

Idle routerRouter under load

Figure 3.24: ACL loading time; based on the cumulative insertion of abatches of 10000 rules

sis information provided by NASK through the n6 API to have a full pipelinefrom external data collection and analysis to countermeasure application.To experiment this setup and our mechanism, we replayed both legitimatetraffic consisting of real traffic captured on Internet backbone and syntheticattack traffic and gathered different metrics on the platform in various testruns.

In the light of these experiments, the mechanism seems to reach theobjectives we had set, allowing us to improve the resilience of a protectedsystem against volumetric DDoS attacks. In situations where the platformegress router and/or link was overwhelmed by the volume of the attacktraffic up to the point where the packet drop rate for legitimate traffic wasabove 95%, the defense mechanism was able to bring the drop rates oflegitime traffic down to 0%. These results apply for the test conditions andtraffic we used, but care should be taken for any generalization. Overall, theprinciple of our mechanism should be applicable to attack traffic which canbe characterized using criteria expressible with the router ACL granularity(varying between providers and maybe models). The exact numbers areheavily dependent on at least two factors: how precisely the attack trafficand only the attack traffic is captured by these ACLs, and on the resourceallocations made for the tunnels for legitimate and suspicious traffic.

Thus we consider that these experiments demonstrate the technical fea-sibility, but future work would be required to better understand how to de-cide and/or optimize the resource allocation for the tunnels. Furthermore,the current mechanism works under one administrative domain where the

www.necoma-project.eu 61 March 31, 2016

Page 62: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

upstream routers can be controlled by the entity wishing to push the de-fenses upstream - a situation where the target cannot itself reconfigure theupstream routers remains to be addressed. Current work on threat sig-nalling, such as DDoS Open Threat Signaling (DOTS)2 could be used forthe target to request a mitigation.

2https://datatracker.ietf.org/wg/dots/documents/

www.necoma-project.eu 62 March 31, 2016

Page 63: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

3.1.5 Conclusion and Lessons Learned

In this use case, we may separate SDN-based scenarios from the MPLS-basedone. Indeed the usage of either of the technologies involves different im-pacts on the target network, to begin with. If MPLS is a widespread pro-tocol in legacy networks, SDN is an emerging paradigm that still strive togarner adhesion. In particular, the community has started to divide itselfaround a couple of technologies, with OpenFlow losing speed to the bene-fit of other solutions, such as VMWare’s Network Virtualization Platform 3,Cisco’s Open Network Environment 4, or the OpenCompute’s Open NetworkInstall Environment 5.

In the SDN scenarios, our designs take advantage of the salient fea-tures of SDN, i.e., the network programmability and the centralized networkview to implement two functions in the security pipeline. These built-incapabilities were not available before in legacy networks, and imply ad-ditional development and deployment in the MPLS scenario. Where ad-ditional network equipment and extended features are needed, the SDNparadigm oppose controller applications that allow to dynamically recon-figure the switches, as well as deploying other functions, which are eitherbuilt-in or added on to the application layer. The application layer has ac-cess to the whole topology view through the controller which centralizesevents reported by the switches. SDN has therefore the potential to com-plete the loop of the security pipeline, i.e., exporting the data from theswitches; processing these data at the controller or at the application layer;selecting a countermeasure at the application layer; and, eventually havingthe switches enforce the defense pushed by the controller. What SDN tech-nologies do not provide inherently are the applications running on top ofthe controller, although the controller implementations often expose APIs toenable them. Our scenarios often focus on these two components includinganalysis and resilience mechanisms designed and developed in workpack-ages 2 and 3, and perform well with respect to the metrics considered. Inparticular, the time to recover from an attack does not exceed 30 seconds ineither scenario.

While deploying our scenarios on SDN testbeds is relatively easy, the dis-crepancies between the implementations prevented us from using heteroge-neous networks and focus on a single-vendor topology. It also preventedus from using multiple controllers, especially different software implemen-tations, since the implementation of OpenFlow versions differ from one tothe other, not to mention that their ability to handle flows from heteroge-neous switches is not guaranteed either. We have encountered difficulties

3https://www.vmware.com/products/nsx/4https://www.cisco.com/web/solutions/trends/open_network_

environment/index.html5http://www.opencompute.org/wiki/Networking/ONIE

www.necoma-project.eu 63 March 31, 2016

Page 64: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

to stabilize the platforms at times, which illustrates a lack of maturity fromboth the vendors and the operators. Although the potential indicated aboveoffers interesting perspectives (described below), these obstacles are not infavor of a wider SDN, in particular, OpenFlow deployment. Another obsta-cle, we have not necessarily dealt with, was the security of control chan-nels. In our scenarios, we often assumed out-of-band channels to transportOpenFlow messages, but it was rarely the case during our experiments. Itis another inherent limitation of SDN implementations, although securingcontrol channels with TLS is part of the roadmap, and often recommendedto be implemented.

Ultimately, a first roadmap to autonomic security can be drawn, regard-less of the technology we used (MPLS, OpenFlow, etc.), while it is particu-larly facilitated by the SDN technologies. As indicated previously, and ev-idenced by some of our contributions, as well as related works, data col-lection can be embedded in SDN networks. The SDN-based IXP scenariouses sFlow, while the DDoS mitigation as a service scenario leverages na-tive OpenFlow statistics exports. On the other hand, we have been able toreconfigure the network by pushing new rules to the switches that wouldspecifically match and process flows detected as suspicious. In between,the controller often hosts a simple application that processes alerts (eitherby processing feeds from NECOMAtter or alerts sent by the customer-sidecontroller) and modifies the controller flow tables. Policies implemented inthese scenarios may be nuclear and limited, but the workflow and interfacesare often already there. For example, we have proposed to implement othersecurity functions in both scenarios, as it is made possible by the imple-mented mechanisms. An alternative approach would be Network FunctionVirtualization, a related paradigm that offers to virtualize legacy networkfunctions, eliminating the need for dedicated network equipment, and hav-ing all-purpose SDN switches offloading flows to software-based networkfunctions located at the controller. It allows for more flexible and customiz-able security function chaining.

Another perspective, much more related to DDoS mitigation, is the usageof tunnels to preserve legitimate flows, and increase resilience of the net-work. Similar to past proposals where legitimate flows would bear tokens,termed capabilities, to allow network equipements to prioritize them overprobably suspicious traffic, we took inspiration from MPLS to implementlabel-switching in our DDoS mitigation as a service scenario. Although, itis expected to be self-protective by mitigating flow table explosion thanksto its per-label approach, it also suffers from some limitations, namely thelength of the label field (12-bit VLAN field), or the consistency of the flowidentifier (the Flow ID may be preserved thanks to label switching, but itsstorage and exchange imply additional communication overhead). Studyingother tunneling protocols for label switching in SDN networks should point

www.necoma-project.eu 64 March 31, 2016

Page 65: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.1. DDOS MITIGATION

towards possible solutions. But compliance with the SDN paradigm is yet tobe confirmed.

Finally, the demonstrators gave us some other insights with respect tobuilding realistic experiments. As much as we took great care in selectingtools and approaches to achieve more realistic scenarios, not all demonstra-tors exhibit a level of real-lifeness that would allow to completely validateour results, if not their ability to sustain a high level of stress, as it is theoriginal goal of this use case. The SDN-based DDoS mitigation scenario hasactually been deployed in a real deployment use case during the INTEROPTokyo 2015 event. But it was not completed at that time. The MPLS-basedscenario does include legitimate traffic replay, but attacks are generated bysome tools, which carefully tune up the parameters to mimic real attackclasses. Such parameters were reused in the DDoS Mitigation as a Servicescenario, in which both legitimate and attack traffic are generated syntheti-cally. The scopes of the latter two scenarios are intended to be constrainedenough (although it may span several networks each) as to minimize theabsence of other networks. We believe that focusing on specific attacks andenvironment settings, and providing accurate rendition of these character-istics helped us attain a certain degree of realism, that we will strive toreproduce in future works.

www.necoma-project.eu 65 March 31, 2016

Page 66: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.2 Botnet Introspection

Please use d42 scenario TEMPLATE.tex to describe the implementation ofyour scenario on a given platform and how you run the experiments, the met-rics you measured and interpreted the results.

Botnets may serve as a powerful arsenal to launch various massive at-tacks, such as DDoS and spamming. However, completely eradicating bot-nets is mission impossible in practice due to both technical and social chal-lenges. This use case is dedicated to leveraging and correlating partial ob-servations at distinct layers for constructing a holistic framework to analyzebotnet activities with more certainties, ultimately reducing response latencyand yielding more effective mitigation mechanisms.

In particular, we focus on the investigation of initial rendezvous of com-munication between bot programs, which run on compromised hosts andthe C&C server, as well as the traffic leveraging threat analysis from pas-sive measurement. The intelligence obtained by workpackage 2’s outcomescontributes to identifying the malicious traffic generated by controlled botprograms in a sandbox environment or a synthetic tool.

3.2.1 Evaluation Framework

3.2.1.1 Datasets

Observing botnets in the wild is not an easy task as it requires a distributednetwork of probes, both at the infrastructure and endpoint layers. The lattermay actually be considered too intrusive and is not really deployed. Moni-toring the network for bot traffic is usually best described by the needle-in-a-haystack problem. Often, we may not even know how the needle looks like.Therefore, blindly looking for anomalies may result unsuccesful, providedwe have actually a good knowledge of our own traffic.

The approach considered in this botnet introspection use case is rathersimple and assumes that compromised machines, the bots, will eventuallycontact their C&C server. One privileged channel for remote command andcontrol is HTTP(S) traffic, but other less common protocols may be used, aslong as the bots are able to identify their C&C server. Attackers use URLsas identifiers for their contact point, instead of IP addresses as the machinehosting the C&C server may change often. Introspection is hence the moni-toring of alleged bots’ communication with remote hosts identified by someURLs. URL filtering is often used to prevent bots from establishing connec-tions with their C&C server. But threat reports and research works haveevidenced the popular use of Domain Generation Algorithm (DGA) to gen-erate URLs deterministically, so that botnets could evade takedowns, at thedomain name (DNS) level. Breaking the algorithm could allow to anticipatethe generation of URLs and hence proactively consitute a blacklist of soon-

www.necoma-project.eu 66 March 31, 2016

Page 67: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.2. BOTNET INTROSPECTION

to-be-issued malicious URLs. But it is not that trivial, and URL informationexchange remains a solid way of spreading the word, once a new maliciousURL has been detected. Detecting these URLs is another challenge whendealing botnets, as it is not always easy to spot the C&C communicationsamong URLs, when the presence of bots is unknown. This is why honeypotand sandboxing approaches strive to collect such important information.

Our introspection scenario on botnet behaviors relies on various datasetscollected or generated from outside of the experimental system. As men-tioned above, the necessary datasets to carry out introspection experimentsare as follows:

• malicious domain name list;

• bot traffic.

Malicious URLs lists are now a common occurrence, although the fresh-ness is not guaranteed. Indeed, DGA has the downside to make URLs short-lived. Still, even old instances of C&C identifiers will allow to detect botslying within the network. So maintaining such URL list is quintessential fordetecting and monitoring botnets.

The second requirement is to have actual bot malware running in theexperimental environment, in order to carry out introspection. As a matterof fact, observing synthetic bot behaviors does not validate as botnet intro-spection. Compromised machines running in a contained environment isthe closest we can get to real-life demonstration, but we believe that at leastcaptured traffic may be sufficient, as we are only interested in monitoringthe network. What ensues is lack of control on the communications, if theC&C server is not hosted locally as well.

3.2.1.2 Metrics

Table 3.3: Metrics to be collected for botnet introspection scenariosMetric Description Units RemarkDetectionaccuracy

accuracy of detection %

Detectionprecision

how many detected items are“actual bad”?

%

Detectionrecall

how many “actual bad” itemswere detected?

%

Collected in-formation atdecoy server

the amount of collected com-mands

[n]

www.necoma-project.eu 67 March 31, 2016

Page 68: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Table 3.3 summarizes a subset of the metrics proposed and defined inDeliverable D4.1 to assess this use case. The statistical variability metrics(i.e., accuracy, precision, and recall) are used to interpret how the analysismodules used in the introspection system works while the numeric metric(i.e., collected information) intends to describe how the information collec-tion method in the system performs.

Additional metrics include the delay imposed on legitimate traffic. Whileanalyzing traffic may actually slow down throughput and impact the deliv-ery of legitimate traffic, our approach to botnet introspection does not havesuch dire consequences, unless the blacklist, against which users’ traffic istested, grows unmanageably long. Analysis is only directed to labeled mali-cious traffic, and this delay is managed, so that it does not get suspicious toattackers themselves.

Detection time was also proposed, and although it is a good indicatorwhen observing suspicious traffic, our approach does not allow to figure outan accurate time to get compromised. An approximation could be given, butthe time a bot can be dormant remains to be confirmed. As we only seek toobserve and gather intelligence from botnet behaviors, it is not an essentialcharacteristic in our scenario.

3.2.1.3 Experiments

Botnet introspection is a fairly new and sensitive use case. While it has thepotential to help analysts extract a fair amount of actionable information, itneeds to be performed with caution and scrutiny. Analysts need to be ableto collect information from live bots while avoiding infection or evasion.As mentioned above, an important asset to performing botnet introspectionis to have a contained environment available. However, modern malwaresamples have often evolved anti-detection or anti-analysis traits that makethem difficult or impossible to process in a virtualized environment. Given,we are able to provide such stealthy analysis environment, we still need toensure that bots are able to communicate with an external entity. Whileultimately, we wish to detect C&C servers as well, it is not the focus of thisuse case.

In the decoy server-based scenario, malware samples are run in a sand-boxed environment with several simulated services, such as DNS and Webto answer domain lookup and C&C transaction requests, respectively. Thedecoy server acts as a collector and the outcomes of its interception are di-rectly actionable: filters in real user networks can be immediately updatedto include new signatures (URLs mainly) or behaviors.

www.necoma-project.eu 68 March 31, 2016

Page 69: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.2. BOTNET INTROSPECTION

3.2.2 C&C Server Introspection with DNS Queries

This scenario investigates traffic between live bot clients and their C&Cserver using a decoy server, and validates/deploys its actionable informa-tion (i.e., malicious domain names list) to be used onto live traffic usingpacket filtering software. This scenario uses the Request Policy Zone (RPZ)mechanism to redirect traffic generated by bot-infected clients and directedto their C&C server. RPZ is a mechanism implemented in the BIND9 name-server software. RPZ allows the operator to modify responses to queries: ituses a simple rewrite ruleset. An operator could replace IP addresses linkedto botnet’s hostnames with an IP address controlled by him (i.e., the decoyserver).

In this scenario, we use a malicious domain names list created by ourC&C domain name detection method developed in workpackage 2. Thismethod detects malicious domain names by applying a machine learning al-gorithm to observed DNS data – dynamic analysis of live bot(s), DNS trafficon a cache server, and DNS traffic on a root nameserver. The nameserverrewrites IP addresses of listed hostnames in order to redirect malicious traf-fic to the decoy server that inspects connections to malicious hosts.

3.2.2.1 Overview

The scenario consists of the following two phases, run in parallel, in itshigh-level view to introspect the bot traffic towards C&C servers throughDNS queries:

1. monitoring in a sandbox environment (qname blacklisting)

2. applying the blacklist to live traffic (in-field deployment)

Figure 3.25 depicts the whole network topology of this botnet intro-spection scenario. Each step of the introspection and filtering process aredescribed as follows (item numbers correspond to the ones in Fig. 3.25):

1 Data collection and threat analysisData of interest collected by our probes is continuously stored in ouranalysis platform (MATATABI, see Deliverable D2.2). The stored DNStraffic is processed by analysis modules and outputs a list of maliciousdomain names.

2 Domains list installationThe domain names list of C&C servers, generated from the previousstep, is: 1) used to configure DNS cache servers (i.e., BIND9); 2)shared via NECOMAtter (see Deliverable D3.2) to provide this action-able information to different entities.

www.necoma-project.eu 69 March 31, 2016

Page 70: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Switch Decoy C&C Server

DNS Cache Server

Internet

Bot

dynamic_analysis m_root dns_

pcaps

MATATABI 1. data probes

3.malicious DNS query

4.decoy response

6. intercept and record

domain list for RPZ

DNS Analysis Module�

5.connect to decoy

Firewall

Internet

User Device

DPI (d4c)

NECOMAtter

domain list for d4c

2. domain list install

Sandbox Network User Network

7. filtering malicious domains

Figure 3.25: Topology and sequence of the scenario: the sandboxed net-work (right side) learns malicious domain names based from the sandboxedtraffic, the user network (left side) employs this malicious domain lists tofilter out the traffic in the wild.

3 Malicious queries generationIn the sandboxed network, bot programs send DNS queries, which arecomputed by domain generated algorithm (DGA).

4 Decoy responseOur hosted DNS server answers a fake response if the queried domainname is in the domains list, created at step 2. The IP address in theDNS answer record is rewritten to the one of the decoy server, insteadof the C&C server.

5 Connection to decoy serverThen, the bot program attempts to connect to the decoy server accord-ing to the fake response.

6 Interception and recordingThe decoy C&C server acts as an HTTP transparent proxy server. Thedecoy server accepts client requests, relays the HTTP traffic, thenrecords the whole transaction to packet traces (i.e., pcap files).

7 Filtering malicious domainsOn the user network side, a DNS packet filtering software, called d4c,filters out malicious DNS queries based on the generated DNS black-list. The blacklist is periodically updated from information spread byNECOMAtter.

www.necoma-project.eu 70 March 31, 2016

Page 71: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.2. BOTNET INTROSPECTION

In this demonstration of a botnet introspection system, the differentmechanisms we designed integrate well into the NECOMA pipeline, as il-lustrated in Fig. 3.26.

dns_pcaps

dynamic_analysis

MATATABI NECOMAtter

bind9

d4c

data collection threat analysis information sharing policy enforcement

lynx

Figure 3.26: A pipeline from data processing to filtering

3.2.2.2 Experiments

3.2.2.2.1 Collecting bot traffic by sandbox and decoy server We in-stalled two types of bot programs in the sandboxed network, Shinobot6 andZeuS gameover[8], on virtual machines running on a Linux-based hypervi-sor machine.

We used the Cuckoo7 sandbox as our analysis environment. Cuckoosandbox is a malware analysis platform which uses virtualization technol-ogy. Cuckoo can trace and record OS-level API calls, network traffic, mem-ory dump on the virtual environment.

Since DGA type malware uses the date of time or web contents on theInternet as a random seed, it is crucial to disguise the sandbox as a real-world environment. The analysis is carried out for 8 hours every day. At thebeginning of the analysis, we restore a plain OS environment from the storedVM image. Then, we deploy malware binary images into the sandbox, andrecord all behaviors of the target system.

During the whole experiment, packet traces are recorded at the decoyserver and the sandbox software on the hypervisor machine. In order to easethe analysis of bot transactions, the packet capture is limited to recordingsome specific packets, thanks to packet filtering.

Two specific packet traces are collected that aim to analyze: 1) fromthe DNS traffic, how many DNS query attempts are sent to get in touch

6http://shinobot.com/7https://www.cuckoosandbox.org/

www.necoma-project.eu 71 March 31, 2016

Page 72: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

with C&C servers and how many of those are successfully resolved to IPaddresses, and 2) from HTTP traffic, what kind of messages are exchangedbetween bot programs and C&C servers.

3.2.2.2.2 Collecting malicious domain name list Our introspection mech-anism needs malicious domain names to redirect botnet traffic. We collectmalicious domain names using two analysis methods: 1) a dynamic mal-ware analysis, and 2) a DGA-based botnet detection system developed inWP2 (see Deliverable D2.1, ”2.2.3 DGA-based botnet detection system”).

We also ran some other malware samples in the sandbox network tocollect DNS query traffic from the malware.

3.2.2.2.3 Implementation An academic network is used to witness theimpact of reconfiguration through the filtering of actual DNS queries usingour collected malicious domain names list. To achieve mitigation, the mech-anism performs the following steps. First, the domain names list collectedin the sandbox is being shared to NECOMAtter every hour. Second, oncethe mitigation software acquires the domain list from NECOMAtter throughits streaming API, it initiates domain filtering on OpenFlow switches thatinterconnect the border of the academic network and the Internet. Thecontroller sets forwarding entries for UDP 52 packets (DNS) towards d4c,a domain-based packet filtering software developed in NECOMA. If a DNSpacket contains a qname that matches an entry of the malicious domain list,the packet will be discarded by d4c.

3.2.2.3 Results

Our evaluation on this scenario relies on a set of metrics measured in ourexperimental network:

• the amount of collected information at the decoy server indicates howsuccessful our introspection is;

• the number of detected domain names represents how the system getsreconfigured to detect the behavior of bot programs;

• filtered domain names on an actual network which has deployed ourmechanism.

3.2.2.3.1 Metric 1: collected information at decoy server Figure 3.27and the numbers detailed in Table 3.4 represent the amount of DNS trafficobserved in the sandbox network and distinguish (using colors in Fig. 3.27)successful and failed queries to the domain names. Most DNS queries faildue to unreachability to the C&C servers: only 0.34% of queries can be

www.necoma-project.eu 72 March 31, 2016

Page 73: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.2. BOTNET INTROSPECTION

10

100

1000

10000

100000

20150930

20151005

20151010

20151015

20151020

20151025

20151030

20151105

20151110

20151205

20151210

20151215

20151220

20151225

20151230

20160105

20160110

20160115

20160120

# o

f D

NS

queries

Date

SuccessFailure

Figure 3.27: Successful IP address resolutions observed in DNS traffic

Table 3.4: DNS statistics for malicious queries.Time # of responses with A record # of error responses2015/09 51 15,7832015/10 2,363 610,7952015/11 818 180,2992015/12 1,660 556,3612016/01 1,049 366,521

resolved to actual IP addresses of C&C servers, out of about 1.7M queries in4 months.

Within those successful DNS queries, Fig. 3.28 depicts a distribution ofHTTP request messages and replied status code. Most of the query trafficissued by bot programs to contact C&C servers returns with a reply error sta-tus (403, 404), which are triggered by POST request. During our 4-monthexperiment, we observed quite a few real contacts from the bot programsto C&C servers, i.e., returning HTTP 200 status code to the POST requestmessages. This response was first detected on 2nd October 2015. Table 3.5lists the partial results of C&C servers that our sandboxed network was ableto contact. Those are actionable information which operators or resiliencemechanisms are able to utilize to mitigate security threat based on botnetactivities.

www.necoma-project.eu 73 March 31, 2016

Page 74: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

0

50

100

150

200

20151005

20151015

20151020

20151025

20151030

20151105

20151110

20151205

20151210

20151215

20151220

20151225

20160105

# o

f H

TT

P m

essages

Date

GETPOST

403404200302

Figure 3.28: HTTP traffic

Table 3.5: Partial list of the IP addresses of C&C serversquery (C&C) IP address FQDN9tr3cq1xedhdqkn2bt51ma3w4.com 5.2.189.251 atlantis.extremeart.ro.xohlzx15hg1l4ef8vyh1h5s614.com 95.211.230.75 sinkhole.ru.shinobot.com 54.245.114.184 ec2-54-245-114-

184.us-west-2.compute.amazonaws.com.

3.2.2.3.2 Metric 2: Detected Domain Names Figure 3.29 shows theobserved domain names that were flagged as malicious. detected nxdomainrepresents the number of domain names which were queried by DGA botnetclients but were not resolved successfully. Because a C&C server can onlyuse a part of the generated domain names, there are a lot of unregisteredmalicious domain names. However, these domain names are still useful todetect other botnet clients, because of the possibility that they may requestthese same domain names.

detected noerror represents the number of domain names which werequeried by DGA botnet clients and were succesfully resolved. These domainnames denote potential botnet C&C servers. They are especially useful fortaking down currently existing botnets.

www.necoma-project.eu 74 March 31, 2016

Page 75: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.2. BOTNET INTROSPECTION

0

100

200

1 04 1 11 1 18 1 25 2 01 2 08 2 15 2 22Date

coun

t

codedetected_noerrordetected_nxdomain

Figure 3.29: Observed malicious domain names

3.2.2.3.3 Metric 3: Filtered Domain Names We applied filters to droprequests to suspicious URLs based on the DNS analysis module. This wascarried out on a network of a NECOMA project member’s organization us-ing d4c. The suspicious URLs were fed from the analysis module throughNECOMAtter. Finally, we injected 480 suspicious domains in the entire net-work traffic for 2 months from February 2 to March 31. The list have beenmaintained every hour based on NECOMAtter’s information during this ex-periment. Table 3.6 shows results of the number of filtered DNS queries dur-ing the experiment. During the experiment, d4c blocked 3647 DNS querieswhich contained suspicious domains. Additionally, we did not have any crit-ical failures related to the d4c mechanism. It means that d4c can deployeddurably on actual environments such as enterprise or academic networks.

3.2.2.4 Conclusion

We proposed a botnet introspection system using the DNS RPZ mechanismand an HTTP proxy. As a result, our system discovered some malicious ses-sions between bot clients and C&C servers. The proposed mechanism useda malicious domain names list obtained through the dynamic analysis of botclients and a DGA detection system which was developed in WP2. We alsodeveloped a traffic mitigation system which uses results from our introspec-tion system, indicating applicability to an actual network environment. Wewere able to demonstrate the usefulness of the threat information pipelinein this scenario.

www.necoma-project.eu 75 March 31, 2016

Page 76: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Table 3.6: Example of Blocked Domain on the d4c

1002mvb1v223gw13l9dx5rg02ay.biz16n86acyn6d2x1gwovoa1xhi94y.biz4ah1co10ma6ix1qd5k9z1d5sdoy.biz31q72l1vvmu0x14be2ewwqfsxy.biz1055dy110wokabcnjsod18xhkez.biz1855dfl1hdd10ssladtnhwxsrz.biz1e5i2klp3ts3o1048jcs15zfxuz.biz1gofv3r1q94e141lnoue71jb80z.biz27v3jd1p8d1yn12fefnb1jid71z.biz1yvhiwh1cq7kaj1hv3rpyrlqfz.biz108jb431k7qfnc1rlt0w115x6mq2.biz1bj6okk18v1igves06pa14dl2f2.bizc1ff2smmp6hvflx2n1bs79v2.biz19dtad28ha5di10o1fgzp5fe22.biz

www.necoma-project.eu 76 March 31, 2016

Page 77: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.2. BOTNET INTROSPECTION

3.2.3 Conclusion and Lessons Learned

Botnets are fundamental evolution over traditional malware in that theyleverage distributed computing nodes and decentralized protocols to im-plement and replace their functions as necessary. In this regard, collectiveefforts to develop introspection capabilities within operating systems as wellas in decentralized protocols are essential.

In this demonstrator, we targeted particular strain of botnet that contactC&C server through DNS using domain generation algorithms. We success-fully demonstrated that the MATATABI platform and NECOMAtter can col-lectively facilitate the analysis and mitigation against DGA botnet throughthe custom-built threat analysis module. Here, introspection techniques forDNS and HTTP protocols are applied to C&C communications that are re-produced in our testbed, which in turn makes use of commonly availableVM introspection technique, such as the Cuckoo sandbox.

According to our experiments, most communications of individual bot-net nodes are meaningless, leading to nonexistent destinations; only a smallfraction of the entire traffic is used to perform malicious activities. Thus it isimperative to conduct long-term data acquisition and analysis, for which themechanisms developed in NECOMA have demonstrated their effectiveness.

The botnet introspection technique can be improved further to deal withdiverse strains of botnets in the real world, beyond the DGA strain. Con-tinued development of analysis modules to deal with ongoing diversifica-tion of strains are left to business entities and research communities. It isworth noting, however, that the development of such analysis modules aresignificantly facilitated by the NECOMA platforms, such as MATATABI andNECOMAtter. We also demonstrated that limited access to diverse strains ofbotnet are not going to be a major issue for research communities, as we canemploy synthetic botnets in testbed environments to evaluate and improvethe introspection capabilities.

This demonstrator also served as a useful example of pipeline for dataacquisition, cross-layer analysis and cyberdefense. The malicious domainlist as used in this demonstrator has been derived by combining the out-put of the malware analysis sandbox with the output of the DGA detectorwhich was developed as part of workpackage 2. They are integrated inMATATABI, thus threat information from both endpoint and infrastructureare made available to analysis modules. The demonstrator also employedtwo cyberdefense mechanisms, in order to filter or introspect malicious DNSqueries.

www.necoma-project.eu 77 March 31, 2016

Page 78: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.3 Smartphone User Protection

This use case focuses on how to prevent smartphone users from unsuspect-ingly participate into malware campaigns. As evidenced by the state of theart, malware authors leverage the constrained environment of smartphones(in particular, with respect to the user interface which has been adapted tofit in a small screen) to deceive their users. Each scenario, presented below,addresses a threat, be it a common browser vulnerability or a lesser-knownsmartphone-specific attack. Other aspects covered in these scenarios includepersonalization of defense, resilience, reconfiguration, and education. Twoof these scenarios are concerned with common web threats, where the at-tacker purposedly takes advantages of the lack of appropriate indicator inthe smartphone user interface:

• a browser extension prevents users from accessing malicious web pagesbased on a domain-based filter. It assumes that URL redirections burieddeep under layers of JavaScript obfuscation are more than suspicious.

• a browser augmented with an eye-tracking device is able to infer thelevel of awareness of a user and reconfigure analysis and defense ac-cordingly to be adapted, based on a machine-learning method. Secu-rity is difficult to balance with usability, especially in web interfaces.By assessing the level of security expertise of a user, we were able to of-fer an appropriate phishing response. Additionally, the user interactionwith the web page input forms are subjected to the acknowledgementof the application that the user has understood the risk, educating thelatter in the process.

• a mobile access point is reconfigured to stand as a network-edge fire-wall to protect and isolate smartphones connecting to it. It replacessmartphone-based firewalls which could be energy-consuming, espe-cially when signatures get pushed regularly, as it assumes the firewallrunning as a daemon, and polling for updates periodically. This glob-ally prevents smartphones from joining attack campaigns in a mobilearea network.

• a lightweight smartphone monitor opposes CAPTCHAs to prevent un-desired calls and SMS from malicious applications. Its operation actu-ally improves the awareness of the smartphone user by putting him atthe center of the security decision-making process.

These four scenarios contribute to the overall improvement of the userawareness of security risks when using a smartphone, as well as, to thelimitation of the ever-growing botnets.

www.necoma-project.eu 78 March 31, 2016

Page 79: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

3.3.1 Evaluation Framework

3.3.1.1 Datasets

In the set of smartphone user protection scenarios, our developed cyberdefense systems should be evaluated with various types of data. For exam-ple, in the case of the phishing prevention scenario, we construct severalphishing sites in the test experiments by using Web Datasets. The defensesystems themselves were also developed based on User Behavior Datasetsas well as Web Datasets, but due to the nature of this dataset, i.e., recordsof past users’ activities, we could not use it for demonstration purposes.

Aside from the phishing prevention scenario, a set of scenarios for driveby download prevention, offloading smartphone’s FW, and SMS fraud pro-tection are developed based on the knowledge extracted from our collecteddatasets, but these datasets were not used in the evaluation.

Instead, we employed a realistic set of data. We carefully analyzed thecharacteristics of threats, namely visiting drive-by download web pages,downloading malicious applications, joining DDoS campaigns and mak-ing unwanted phone calls by receiving SMS fraud messages . We thenextracted the characteristics of these cyber threats, and finally generatedrealistic events causing these cyber threats for evaluation purpose.

3.3.1.2 Metrics

Table 3.7: Confusion matrix for effectiveness metricsLabel as Threat Label as Benign

Actual Threat True Positive (TP) False Negative (FN)Actual Benign False Positive (FP) True Negative (TN)

Referring to what has been described in Section 4.2 of Deliverable D4.1,our scenarios deal with the following metrics. Note that Table 3.7 summa-rizes the factors for calculating each metric.

Effectiveness The defense mechanisms must achieve high detection accu-racy. Apparently, user safety would be compromised if the defensesystem labeled malicious entities as benign. Conversely, users wouldalso complain if the defense system labeled benign entities as mali-cious because of the interruption caused by the system.

It should be noted that in the case of phishing prevention scenario, wealready evaluated the detection accuracy with Detection Precision andDetection Recall metrics, in Deliverable D2.1.

Detection precision was the number of true positive detections out ofthe total number of positive detections for cyber threats. It can becalculated by TP

(TP+FP ) (the higher the better).

www.necoma-project.eu 79 March 31, 2016

Page 80: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Detection recall was the number of true positive detections that wereactual positive events. It can be calculated by TP

(TP+FN) (the higher thebetter).

Overhead While the primary design objective of our defense mechanismsis to effectively prevent, detect and mitigate cyber threats and recoverany damages, we also expect them to be lightweight and work in re-source efficient ways. Since resilient cyberdefense aims at recover-ing damages resulting from attacks, a restoration procedure shouldbe performed in a short time period. With respect to that, Delay is aresonable metric for presenting overhead.

Usability This is an optional metric for representing the loss of users’ con-venience due to the security mechanisms. Since there are trade-offsbetween security and convenience, our cyber defense systems will im-prove users’ safety but penalize users’ convenience. If the loss of con-venience is not marginal, it means that the system is far from beingpractical.

Battery consumption We also regarded battery consumption as an optionalmetric. The battery of the smartphone will be drained if the system re-quires a lot of computational resource. From the aspect of smartphoneavailability, the cyber defense would work with fewer resources.

3.3.1.3 Experiments

Generally, typical cyber threats aimed to endpoint systems are Denial ofServices (DoS), malware, and phishing websites and email. We thereforedeveloped cyber defense systems against these threats to protect endpointdevices, and then attempted to utilize the systems for smartphone devices.

One of the most popular sources of malware infection is drive-by-download.Our developed system described in Section 3.3.2 prevent this attack by ter-minating an access to a malware distribution server. The experiments needsa landing server, a distribution server, a victim host runs a cyber defensesystem which interacts to NECOMAtter threat information sharing systems.

In the case of preventing phishing websites described in Section 3.3.3, itsevaluation should address the fact that the defense system can work regard-less of the size of the screen. The smartphones and tablet computers havelimited screen size in comparison to personal computers. The experimentswere designed to assess the protection ability using testbeds described inSection 2.4.

Besides, messages encountered in smartphones, that induce victims tovisit phishing sites and/or execute malicious applications, are usually SMSmessages rather than emails. With respects to this peculiarity, We used

www.necoma-project.eu 80 March 31, 2016

Page 81: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

an Android emulator instance as our testing platform as described in Sec-tion 2.5.

We also considered the fact that smartphones are not often victims ofDDoS attacks. Rather, they sometimes join DDoS attacks controlled by re-mote attackers. To get a better view of this situation, we developed a systemfor preventing smartphones to join DDoS attack. For its evaluation, the ex-perimental testbed involves a (pseudo) victim node of DoS attack as wellas smartphones and their defense systems. In this case, we used the Tablet-based Testbed described in Section 2.4.

www.necoma-project.eu 81 March 31, 2016

Page 82: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.3.2 Drive by Download Prevention

3.3.2.1 Overview

The scenario aims at demonstrating how to prevent drive-by-download at-tacks on a user’s browser. Drive-by-download attacks use obfuscated mali-cious JavaScript code on a web server to redirect users to malware distri-bution servers. A user who accesses a landing server downloads maliciouscode and executes it in the context of the browser. The malicious code ex-ploits a vulnerability that will allow to redirect the browser’s execution flow.Hijacking the browser’s execution usually leads to downloading a malwarehosted at a malware distribution server. Finally, the malware is injected inthe user’s host, where it runs with the privileges of the browser, waiting forsome commands from the attacker.

In this scenario, we demonstrate a blocking mechanism which performson top of a browser and terminates an access to a distribution server. Thedefense mechanism works as a browser extension on the Chrome browser.When the browser downloads an HTML file with obfuscated JavaScriptcode, the extension de-obfuscates the code. If the de-obfuscated code in-cludes access requests to domains different from the domain of the landingsite, the extension blocks the access and raises a warning to the user.

Figure 3.30 shows the sequence diagram of the scenario. We explain thesequence as follows:

1. The victim’s device opens the URL of the malicious server.

2. The browser downloads web contents which contain obfuscated mali-cious JavaScript code.

3. The browser executes the code.

4. The browser extension analyzes the code and blocks access to thedownload server.

5. The extension displays an alert to the user.

6. The extension posts the URL of the malicious site and the downloadsite on the NECOMAtter information sharing platform.

7. In case the extension is disabled or absent, the victim’s browser ac-cesses the download server.

8. The malware is downloaded to the victim host and the host executesthe malware.

The prevention mechanism can be involved in a security pipeline throughthe NECOMAtter. Suspicious URLs that are detected by the mechanism are

www.necoma-project.eu 82 March 31, 2016

Page 83: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

LandingServer

Victim host(Chrome + extension)

Malware Distribution

Server

NECOMAtter

③④⑤

Figure 3.30: Scenario workflow for drive-by-download prevention

shared on the NECOMAtter. After the sharing, other clients can get the de-tected URL lists from the NECOMAtter and update the URL filtering lists bythemselves.. Accordingly, the clients drops Web accesses to the URLs with-out detection on itself. Furthermore, the detection results also use in URL-based firewalls such as D4C3.2.2. The pipeline improves cyber resilience onendpoint users.

3.3.2.2 Experiments

We implemented the proposed mechanism as a Chrome extension. Fig-ure 3.31 shows the operational sequence of the extension. First of all,the extension loads a web page that is accessed by a user. Then, this ex-tension parses and analyzes the source code of the web page. During thepre-analysis, the extension extracts domains (URLs) listed in internal or ex-ternal resources such as CSS files, JavaScript files and more. The extracteddomain names are only written in human or machine readable format andencoded in the source code. Some of the domains are obfuscated to hidemalicious activity from endpoint security software. The proposed mecha-nism recognizes fraudulous acccess to external resources that are not listedon pre-analysis results The mechanism is sensitive to false positives, possi-bly classifying legitimate accesses into malicious accesses. The implementedChrome extension works on both PCs and tablets.

www.necoma-project.eu 83 March 31, 2016

Page 84: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Content Script Background Pages Web Page

beforeRequest Event

main_frameDomainCheck()

connection accept

ScriptInject()

script data

send script data save script data

DomainCheck()

DomainCheck()

beforeRequest Event

connection accept

beforeRequest Event

connection accept

LegitimateContents

MaliciousContents

Figure 3.31: Sequence diagram of the proposed mechanism as a Chromeextension

3.3.2.3 Results

We evaluated the proposed prevention mechanism on an actual browsingenvironment. In this evaluation, we measured the page rendering time withand without the proposed mechanism on Chrome.

We accessed the top 70 web pages listed on the Alexa Top Website list.We collected IP addresses accessed by the listed web sites to minimize theaddress resolution time. As a matter of fact, the address resolution processcould introduce an additional and unstable delay. After the IP address col-lection, we stored the pairs of FQDN and IP addresses on the local hostsfile of the experimental PC environment. This allowed to minimize the DNSprocessing delay at rendering time.

For each of the 70 web sites, we repeated the rendering time measure-ment 10 times. The rendering time is automatically measured by the ex-tension. In Figure 3.32, the x-axis represented the FQDN of accessed sites,while the y-axis is the average rendering time for each site. In this graph,the blue bars denote the rendering time without the extension. On the otherhand, orange bars are rendering time with the extension activated. Thisgraph shows that rendering time with the extension is longer than withoutone in most web sites. Rarely, the latency with the extension enabled is

www.necoma-project.eu 84 March 31, 2016

Page 85: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5http://google.com

/http://facebo

ok.com

/http://youtub

e.com/

http://yaho

o.com/

http://am

azon

.com

/http://wikipedia.org/

http://google.co.in/

http://qq

.com

/http://twitter.com

/http://live.com/

http://taob

ao.com

/http://msn.com

/http://yaho

o.co.jp/

http://linkedin.com/

http://sina.com

.cn/

http://google.co.jp/

http://weibo

.com

/http://bing.com

/http://yand

ex.ru

/http://vk.com

/http://hao1

23.com

/http://instagram.com

/http://ebay.com

/http://google.de/

http://am

azon

.co.jp/

http://mail.ru/

http://google.co.uk/

http://google.ru

/http://36

0.cn/

http://pinterest.com

/http://t.co/

http://google.com

.br/

http://redd

it.com/

http://netflix.com

/http://google.fr/

http://tm

all.com

/http://paypal.com

/http://sohu

.com

/http://microsoft.com

/http://wordp

ress.com

/http://blogspot.com

/http://google.it/

http://google.es/

http://tumblr.com

/http://on

clickads.net/

http://imgur.com

/http://ok.ru

/http://aliexpress.com

/http://xvideos.com/

http://apple.com/

http://imdb

.com

/http://stackoverflow.com

/http://fc2.com/

http://ask.com/

http://google.com

.mx/

http://am

azon

.de/

http://google.com

.hk/

http://office.com/

http://alibaba.com/

http://google.ca/

http://google.com

.tr/

http://rakuten.co.jp/

http://po

rnhu

b.com/

http://xinh

uanet.com

/http://google.co.id/

http://github

.com

/http://tianya.cn/

http://craigslist.o

rg/

http://soso.com

/http://blogger.com

/http://am

azon

.in/

http://am

azon

.co.uk/

http://nicovideo.jp/

http://kat.cr/

http://ou

tbrain.com

/http://go.com

/

withoutextensionwithexetension

Figure 3.32: Rendering time with/without the extension

shorter than with no extension. Accesses to external (possibly, malicious)web sites are terminated by the extension.

Table 3.8 gives the average rendering time in each case. The averagerendering time for the drive-by protection scenario is one-third longer thanwhen the extension is disabled. Such latency is believed to be acceptable tousers, especially since it is just a little longer than a second.

3.3.2.4 Conclusion

The scenario proposes to protect web users against drive-by-download at-tacks with minimal disruption. We evaluated the rendering time of thebrowser extension with popular web sites that are listed by Alexa. As aresult, we revealed the additional latency caused by the extension is notcritical for users. The mechanism can be involved in a security pipeline as adetector and a policy enforcer. We can make the following pipeline by usingthe mechanism. When browsers which has been installed the mechanismextension, detected suspicious URLs will be shared on the NECOMAtter. Af-ter the sharing, the shared URLs will be distributed to other clients that arewatching the NECOMAtter timeline. Therefore, they can apply the suspi-

Table 3.8: Average rendering timeaverage rendering time

without extension 0.81 [seconds]with extension 1.2 [seconds]

www.necoma-project.eu 85 March 31, 2016

Page 86: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

cious URLs on themselves. The clients can also feedback their detectionresults on the timeline.

We have already published the source code of the extension on ourGitHub repository8 with a user manual.

8https://github.com/necoma/drive_by_download_protector

www.necoma-project.eu 86 March 31, 2016

Page 87: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

3.3.3 Phishing Protection

The goal of this work is to provide a suitable phishing prevention based onthe user’s awareness level. Basically, there are trade-offs between improvingprecision and improving recall. If a user is a novice, who is easily deceivedby phishing attacks, our defense should improve recall rate in spite of de-creasing precision. On the contrary, an expert who can identify phishingcorrectly would require high precision for phishing prevention systems. Wetherefore decide to provide a different defense for users.

Our phishing prevention system monitors users’ eye movements. Ourprior analysis [5] evidenced that the eye movements of users could allowto identify the type of users (e.g.,, expert or novice) when captured whileassessing the websites’ credibility. We then perform difference types of diff-ence and observe their performance.

3.3.3.1 Overview

This scenario demonstrates a personalized phishing protection mechanism.Here, personalization means that the mechanism can be customized perthe needs of the end user. The mechanism involves the assessment of theuser’s awareness when visiting trustworthy or phishing websites, and thedifferentiation of adopted countermeasures according to the inferred skill ofthe user. In particular, the awareness assessment is a fundamental conceptfor determining the skills of a user. For example, based on our analysisresults [5], novices usually fail to make correct decisions, since they tendto be visually attracted by web contents (which always appear to be similarbetween phishing sites and legitimate sites), rather than the URL or SSLindicators. By contrast, an expert tends to examine the site’s URL and/orthe browser’s SSL indicator rather than the contents of the web page todetermine the credibility of the sites.

The system operation is summarized in Fig. 3.33. The defense system,named EyeBit+, runs on Google’s Chrome web browser. The system checksthe level of awareness with respects to the address bar for each participant.By interacting with the eye-tracking device, the system will determine ifthe participant is a novice, an expert or another type of participant. Foran expert participant, the defense system employs ATOS’s high precisionanalysis modules. For a novice participant, the defense system uses UTokyo’smachine learning-based analysis modules. These modules were describedin Deliverable D2.1. Further details on EyeBit+ system can be found inDeliverable D3.5.

It should be noted that the system was designed to provide resilient cy-ber defense and hence, it just deactivates web input forms instead of filter-ing/blocking the whole Web content. Even if a legitimate website is misla-beled as phishing, the users’ loss of convenience might be limited since most

www.necoma-project.eu 87 March 31, 2016

Page 88: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

BrowserExtensionModule

PDPModule

Eye-TrackingModule

(1) (6)

Analysis Platform

Defense System

ATOS’sModule

UT’sModule

(3)A (4)A (3)B (4)B

(2) (5)

(1)

Figure 3.33: System Overview

of the web contents would be shown to the users. This loss of inconvenienceis partly addressed through reconfiguration: the defense can reactivate theforms when the user checks the address bar when the site deemed as deter-minately phishing.

3.3.3.2 Experiments

In this scenario, we employed three metrics, namely, precision/recall, delay,and user experience. The precision and recall were already measured inNECOMA Deliverable 2.1. Two detection modules for phishing sites wereused:

• “High Precision Phishing Detection Module”, developed by ATOS. Themodule checks the number of dots in the FQDN, and label the websiteas phishing or not. The module can achieve high precision despite oflow recall, and it works faster than the other module.

www.necoma-project.eu 88 March 31, 2016

Page 89: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

• “Machine Learning-based Phishing Detection Module”, developed byUT. This module checks websites’ contents as well as the FQDN, ex-tracts features with heuristics described in [4], and calculates the like-lihood of the website being a phish using the NECOMA Threat AnalysisPlatform. It can achieve higher recall in comparison to ATOS’s module,although it needs much time to output a detection result.

In order to analyze performance, we chose to measure delay to assess themechanism’s overhead. If our system cannot achieve a near real-time per-formance, the users would complain when using this defense. The test en-vironment is comprised of an emulated phishing site, made from web pagesof legitimate enterprises. We also implemented ATOS’s and UT’s phishingdetection modules in our test environment, and then performed evaluationwith ApacheBench, a stress test tool.

Finally, we considered the user experience. Since there are tradeoffs be-tween security and usability, our phishing prevention systems will improveusers’ safety but penalize users’ convenience. If the loss of convenience isnot marginal, it means our phishing prevention system is far from practicaluse. This is a qualitative variable, collected through a questionnaire. For theevaluation, we performed a participant-based experiment. It must be notedthat our experiments must not collect and/or analyze any personally iden-tifiable information. The experimental design, concept and methodologiesfor recruiting participants are also explained below.

1. Recruitment of participants through a poster advertisement at a col-lege campus.

2. Explanation of our experiment to the participant.

• Our purpose is to observe the user’s activity, in particular withrespects to assessing the credibility of websites.

• Our goal is to develop security mechanisms for protecting usersfrom phishing.

• Before the experiments, each participant will be asked his/herage and sex.

• During the experiments, each participant will be monitored byan eye-tracking device, and be shown several websites. Theiractivity will be monitored, and they will be asked if each websiteseems to be phishing or not. They will also be asked the reasonof their decision.

• Collected data consists of the participants’ age, sex, decision re-sult, decision criteria, and eye-tracking data.

www.necoma-project.eu 89 March 31, 2016

Page 90: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Table 3.9: Precision and Recall of the Analysis ModulesATOS’s module UT’s module

Precision 100.00% 86.1%Recall N/A 75.8%

0

500

1000

1500

2000

2500

3000

1 10 20 30 40 50

Mea

n T

ime

Per

Req

uest

(m

s)

Concurrency Level

UT ATOS

Figure 3.34: Stress Test Analysis of the Analysis Modules using ApacheBench

• Collected data is shared with both European and Japanese re-search members.

3. Display of 4 websites, including legitimate websites and pseudo phish-ing sites.In the experiment, the phishing sites are not real phishing sites toavoid information leakage. We refer to this procedure as “first step.”

4. Showing participants how our system works.

5. Display of 4 other websites, including legitimate websites and pseudophishing sites. We refer to this procedure as “second step.”

6. Asking participants to assess our defense according to a five-gradeevaluation system, e.g., (1) Strongly disagree, (2) Weakly disagree,(3) Neither, (4) Weakly agree, and (5) Strongly agree.

3.3.3.3 Results

Below, we provide the results of our performance and usability experiments.According to our independent evaluation described in Deliverable D2.1, theresults were briefly summarized in Table 3.9.

www.necoma-project.eu 90 March 31, 2016

Page 91: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

Table 3.10: Conditions of each site used for evaluating EyeBit# website Phish Lang Description1 Yahoo yes JP dmiurdrgs.cher-ish.net, once reported as a phishing site2 PayPal no EN EV-SSL3 eBay yes EN signin-ebay.com, similar to legitimate URL

signin.ebay.com

4 DMM no JP SSL5 Amazon yes EN www.importen.se, once reported as a phishing site6 Bank of America no EN EV-SSL7 Facebook no JP SSL8 Square Enix yes JP hiroba.dqx.jp..., similar to legitimate URL

hiroba.dqx.jp

9 Twitter yes JP twittelr.com

10 Google no JP SSL11 Battle.net no EN EV-SSL12 Sumitomo Mitsui Card yes JP www.smcb.card.com...., similar to legitimate URL

www.smbc-card.com

Figure 3.34 shows the benchmark results of using UT’s analysis moduleand ATOS’s analysis module on several concurrency levels, i.e., the num-ber of clients which concurrently send requests to the modules. We selectedwebsites for this benchmark from the websites listed in Table 3.10. However,since all phishing sites were already taken down at the time of experiment.Instead, we used legitimate ones to measure the performance and moni-tored the average delay. We queried these analysis modules via HTTPS 100times, and observed the mean time per request. Since UT’s analysis moduleneeds to analyze the content with natural language processing, we foundthat the overhead of the UT’s module was greater than for the ATOS’s mod-ule. It can be naturally assumed that there might be a trade-off betweensecurity and convenience for users in choosing one or the other module.However, the performance overhead for UT’s module was just a few seconds,therefore, we speculated that the loss of convenience might be acceptablefor web users who are more likely to be victims of phishing sites.

For evaluating user experience, we used phishing sites shown in Ta-ble 3.10, all of listed sites were used in our past analysis [5]. We usedwebsites #1-4 for the first step, and then showed randomly either websites#5-8 or #9-12 during the second step.

The result is summarized in Fig. 3.35. We asked our 15 participants torate their perceived level of inconvenience, (represented by blue bars). Thepredominant answer was that users weakly disagreed to a perceived loss ofconvenience. Users who already have a habit of checking the address bardo not feel more stressed while using this system. In the case of novices,the detection takes much more time in comparison to experts, however, theperceived loss of convenience was limited. We must highlight the fact thatour defense deactivated web input forms, but did not block the whole con-tent of the websites. So we assumed that users did not feel inconveniencesince most of the website contents were available.

www.necoma-project.eu 91 March 31, 2016

Page 92: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Inconvenience Safety

12

34

5

Fiv

e−gr

ade

Deg

ree

Figure 3.35: Analysis of Usability Experiment

We also asked the partcipants to rate their perceived level of safety (rep-resented by yellow bars in Fig. 3.35). The predominant answer was thatusers weakly agreed to perceiving a better level of safety. The participantswho labeled their experience with “disagree” tended to state the importanceof education: they told us that seeing the address bar without the knowl-edge of its security indicators was not helpful to identify phishing.

3.3.3.4 Conclusion

Our phishing prevention system is designed to provide suitable defensebased on users’ awareness. Before detecting phishing, the system tries toidentify the user’s awareness by analyzing their eye movements.

We consider that both information pipelining and resilience are key com-ponents of our defense system. Regarding threat information pipelining, the

www.necoma-project.eu 92 March 31, 2016

Page 93: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

system call on different detection modules via the n6 message format. Thesystem can therefore utilize the threat analysis platform and respond witha suitable defense. Our defense also addresses the matter of resilience. Thesystem runs on a web browser that can deactivate web contents and re-activate them if necessary. We therefore believe that the proposed systemwas demonstrated as not heavily penalizing users’ convenience, making oursolution usable.

www.necoma-project.eu 93 March 31, 2016

Page 94: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.3.4 Offloading Smartphone’s FW to OpenFlow-capable AP

This scenario demonstrates a smartphone firewall which aims at offloadingthe smartphone firewalling function to network switching devices.

Our research group explored the suitable protection schemes, and foundthat the OpenFlow-capable Access Points (APs) are able to facilitate config-uring filtering rules, and also perform defense at the closest point to theissued device, as well as saving energy consumption. Indeed, firewall appli-cations running on top of smartphones heavily drain their battery life.

Since OpenFlow provides powerful traffic control schemes, it facilitatesthe implementation of firewall functions which can filter traffic based onthe header information of the network and transport protocols such as IPaddress, and TCP/UDP port numbers. The key idea is that our implemen-tation runs on an access point for filtering packets. It contributes towardsaving network bandwidth resources between the access point and otherfiltering devices. Additionally, our implementation is developed based onopen source applications and standard protocols. It may therefore provideflexibility and operability while being vendor-agnostic.

It should be noted that once a rule is created by the NECOMA threatanalysis, it is available to enforce cyberdefense for both infrastructure andend-point layers. We expected that SDN-based policy enforcement pointswould enable rapid deployment of cyber defense.

3.3.4.1 Overview

As we mentioned, the defense at the smartphone itself is difficult. Instead,we decided to offload smartphone firewalling function to network switchingdevices.

One type of defense mechanism is URL filtering, intended to block spe-cific web sites for all smartphones. The objective of filtering is to preventmalware infection, botnet C&C, and phishing. In such attacks, the HTTPprotocol is respectively used for malicious applications download, commu-nications with the bot master, and fraudulent websites access.

Packet filtering is another possible defense to prevent smartphone de-vices from joining DDoS campaigns. Once a smartphone device is infectedwith a trojan horse, it starts to send data packets to a specified host (the vic-tim) whenever it receives a DDoS attack command. It should be noted thatthe scope of DDoS mitigation is usually at the infrastructure layer ratherthan the endpoints, even when protecting smartphones. However, a ma-jor principle of DDoS protection stipulates that the mechanism should filterDDoS traffic as close to the attacker as possible. We therefore address DDoSmitigation at the AP, which may be the closest to the source as possible whenwe consider the case of a smartphone joining a DDoS campaign.

www.necoma-project.eu 94 March 31, 2016

Page 95: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

We also consider that the smartphone firewall has the ability to pre-vent infected smartphones from attacking neighboring smartphones in sometypes of network. For example, carrier networks tend to prohibit that asmartphone connects to other smartphones in the same network. How-ever, in future networks, there is a possibility that carrier networks allowa smartphone to connect to other smartphones (e.g., virtual access point asproposed in IETF NVO3 WG9). In such cases, an AP-based firewall will behelpful to thwart the attacks.

SmartphoneMalware

C&CAttacker

Victim nodeof DDoS

Packet Filtering

OpenFlow Switch/ Wireless AP Smartphone

OpenFlow controller

Figure 3.36: Operational sequence for the smartphone protection scenariousing OpenFlow-capable APs

Fig. 3.36 illustrates a high level overview of the architecture we propose.As we can see, APs are placed at the closest point to smartphone devices, andprovide URL filtering to thwart smartphone malware, C&C, and phishing, aswell as packet filtering to prevent smartphones from participating in DDoScampaigns.

From these considerations, we have two requirements for the APs. Thefirst one is Filtering ability: i.e., the APs must be able to protect smartphonesfrom malicious entities, therefore they should be able to filter maliciousURLs and block suspicious IP addresses. Another requirement is Operability:i.e., the APs should be able to scale with the load of configurations.

3.3.4.2 Experiments

If the delay for configuring the AP-based firewall was large, it might be dif-ficult to perform cyber defense flexibly. Thus we considered the delay as thepredominant metric in this scenario. We generated IPv4 address randomly,

9https://datatracker.ietf.org/wg/nvo3/documents/

www.necoma-project.eu 95 March 31, 2016

Page 96: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

0.00.51.01.52.02.53.03.54.04.5

1 2 3 4 5 6 7 8 9 10 20 30 40 50 60 70 80 90 100

200

300

400

500

Adding Deleting

Figure 3.37: Stress Test Analysis of the OpenFlow APIs using Siegeh

calling OpenFlow application to add/delete the IPv4 address into its flowentry. We ran our implementation on the Tablet-based testbed, and thenevaluated its performance with Siege 10

We queried these APIs via HTTP 1000 times, and observed the averagenumber of transactions per second.

We also considered the battery consumption of a smartphone as an op-tional metric. We configured the smartphone to send web requests to avictim server at the speed of 50 requests per second (pps), and monitoredthe battery consumption in several different cases as follows.

• AntiVirus:We ran several AntiVirus applications in a smartphone, and observedthe average energy consumption. Our tested AntiVirus applicationsare namely Trendmicro’s VirusBuster, Symantec’s Mobile Security, andKaspersky’s Internet Security.

• Filtering:In this case, a smartphone sends packets and our AP-based firewallfilters the packets.

• Disassociating:In this case, a smartphone sends packets and our AP-based firewalldisassociates the smartphone.

3.3.4.3 Results

This section provides the results of our preliminary evaluation. Figure 3.37shows the benchmark results for adding and deleting blacklisted IP addresson several concurrency levels, i.e., the number of clients which concurrently

10https://www.joedog.org/siege-home/

www.necoma-project.eu 96 March 31, 2016

Page 97: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

0%

2%

4%

6%

8%

10%

12%

14%

0 30 60 90 120 150 180 210 240 270 300

Disassociate Associate AntiVirus

Figure 3.38: Effects on Battery Consumption with and without an AP-basedFirewall

send requests. For each concurrency level, 1000 of randomly generated IPaddresses were inputted. The x-axis denotes the mean time per request andy-axis the concurrency level. The blue and orange bars denote the perfor-mance for adding and deleting the IP addresses, respectively. When thenumber of concurrent clients reach 500, we observe the worst performance.The delay is still shorter than 4.0 milliseconds though.

We also measured the effectiveness on battery consumption by offload-ing firewalling functions. Before this experiment, we charged the battery ofour smartphone to full tank in order to compare the battery consumption inthe several cases. Figure 3.38 shows the effect on offloading the firewallingfunctions. The x-axis denotes the time (in minutes), and the y-axis denotesthe battery consumption (percentile of battery). The red, yellow, and bluebars respectively denote the battery consumption in the cases of AntiVirus(firewall runs on the smartphone), Filtering with AP-based firewall, and Dis-associating with AP-based firewall.

3.3.4.4 Conclusion

In this scenario, we presented new firewall work on OpenFlow-capable wire-les access points (APs). Our proto-type was based on OpenWrt, and wasconfigured to deal with an OpenFlow protocol in order to facilitate addingnew filtering rules.

We then implemented an OpenFlow controller to provide RESTful APIsthat are available for blocking or unblocking the specified IP address, as wellas disconnecting a smartphone device to which was assigned a specified IPaddress.

www.necoma-project.eu 97 March 31, 2016

Page 98: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

It is a fact that OpenFlow has a powerful configuration capability to addand delete flow rules, and hence, it is easy to reconfigure packet filteringschemes on cyber defense systems powered by OpenFlow. We thus believethat our developed firewall is resilient against an attack in which smart-phones are involved in.

www.necoma-project.eu 98 March 31, 2016

Page 99: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

3.3.5 SMS Fraud Protection

On Android, when an application is being installed, users explicitly acceptall the permissions requested by it and cannot alter or revoke any of themlater. The only option is to uninstall the application. Thus, an applicationthat is allowed to send SMS messages may send any number of messages toany number, even in the background without user consent.

The system we present adds an extra layer of security to counter that.Any application attempting to send a SMS message or call a phone numberis intercepted and ultimately control is given to the user. The destinationnumber of the action is presented and the ability to blacklist the numberor allow the action is given. As a result, users become aware of every ac-tion that would cost money, performed by any application installed on thesystem.

3.3.5.1 Overview

The SMS Fraud Protection mechanism consists of two parts. First, the An-droid core framework is modified to enable the interception of costly ac-tions, such as sending SMS messages or making phone calls. Those modifi-cations are then combined with a new system application that allows for apop-up dialog to show up whenever an action is intercepted and user inputis needed.

Currently, the implementation is done on Android 4.4.3 but can be portedto any Android version in the future. The SmsManager framework class ismodified, specifically the functions that allow an application to send a SMSmessage, namely sendTextMessage() and sendMultipartTextMessage(). Dueto the inability to display a dialog from the SmsManager class, a new An-droid system application is also introduced.

This application enables the communication between the SmsManagerclass and the end user. When any application attempts to send a message,a dialog is shown by the system application allowing the user to blacklistthe destination number or allow the action. In order for either option tobe performed, the user has to solve a CAPTCHA or a simple math opera-tion. Furthermore, the selected action is stored in the system applicationdatabase, which cannot be accessed by any other installed applications.

In addition, we can add the ability to synchronize with malware sourcesand sources of known malicious premium-rate numbers that use the n6 SDK,in order to automatically update the blacklist database, as well as be awareof new threats that our mechanism should be able to handle. As a result,users will rarely have to decide whether a number and an action is mali-cious, as the blacklist is automatically kept up-to-date.

www.necoma-project.eu 99 March 31, 2016

Page 100: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.3.5.2 Experiments

In order to confirm the effectiveness of the SMS Fraud Protection mecha-nism, we utilised an instance of the Android emulator. That specific instanceis running a modified Android 4.4.3 build that enables the interception ofall applications.

Figure 3.39: Flow of the real-world scenario.

Typically, a user has to go through the following scenario when usingour system:

• The attacker publishes a malicious app in one of the multiple applica-tion stores, some of which do not have strict security checks.

• The victim user browses those application stores and eventually in-stalls the malware on his device. Note that malicious code can beinjected in popular applications, which are repackaged and then pub-lished.

• The malicious application is executed.

• When the application attempts to send a SMS message, a pop-up dia-log is populated.

• The dialog informs the user of the message being sent, along with thedestination number and offers three options.

• It is possible to either reject the message and blacklist the number, al-low the message and whitelist the number or decide later. A CAPTCHAor a trivial math problem has to be solved in order to be able to pickan action.

www.necoma-project.eu 100 March 31, 2016

Page 101: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

• In the end, if the SMS is sent to the premium-rate number, the mal-ware author gains revenue.

On the other hand, during the evaluation stage selected malware sam-ples are directly installed on the virtual device. Therefore, there is no needto browse application stores looking for malicious apps. Malware sampleshave been collected from repositories such as Malgenome and Contagio.

Using the debug tool ’Android Debug Bridge’ (adb), installing any appli-cation package is straightforward. The installed application is then manu-ally launched and browsed, in order to trigger the malicious actions which inturn will bring up the pop-up dialog of the system. At this point, the systemhas been proven effective against the application that is being tested.

This process can be automated using debug and instrumentation tools,for example Android ”UI/Application Exerciser Monkey”, which would thor-oughly explore the application that is being scanned, thus triggering the ma-licious action with a higher probability and in less time. Even then, it is notcertain that the application would proceed to malicious actions during theevaluation process, as it may be waiting for specific events.

3.3.5.3 Results

As Android applications can act on system or other application events (In-tents, Broadcasts) it is not trivial to ensure that the malicious code will beexecuted during testing. However, the implementation presented ensuresthat no application is able to send a message without being intercepted asthe interception is performed at system framework level.

Current malware is unaware of this approach and thus unable to by-pass it. A possible workaround by malware authors would need to utilize asignificant platform exploit to gain elevated privileges (”root”). Such mal-ware would be able to either alter the contents of the blacklist and whitelistdatabases of our system application or even disable the interception mecha-nism entirely.

www.necoma-project.eu 101 March 31, 2016

Page 102: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Family Month ofdiscovery

Description

FakeInst 12/2011 Sends SMS messages to pre-mium numbers

FakeMart 08/2012 Sends SMS messages to pre-mium numbers

FakeNotify 12/2011 Sends SMS messages to pre-mium numbers

FakePlayer 08/2010 Sends SMS messages to a pre-mium number

HippoSMS 07/2011 Sends SMS to a premiumnumber, deletes incomingSMS from a certain number

Jifake 10/2011 Sends SMS to a premiumnumber

OpFake 02/2012 Sends SMS messages to pre-mium numbers

Zsone 05/2011 Intercepts and sends SMSmessages

Only malware of interest has been tested against our implementation,such as the families listed in Table 3.3.5.3. This is only a small selection ofthe malware circulating that attempts to charge the user by using premium-rate numbers.

3.3.5.4 Conclusion

We present an innovative mechanism that restricts the ability of applicationsto send SMS messages and make phone calls, allowing the user to selec-tively accept or reject such actions and maintain a whitelist and a blacklistof numbers. Currently, the system is effective and immune against malware,but this may change especially with malware that acquire elevated privi-leges (”root”), which would enable them to tamper with the whitelist andblacklist databases or even disable this security mechanism entirely.

Our solution clearly achieves its target, as users become aware of ap-plications that attempt to perform actions that may cost money and de-cide whether those actions are legitimate or not. The system is easy touse and straightforward, although it could potentially be improved in or-der to reduce the need of user input, by syncing with premium-rate numberproviders.

Future work includes fine-tuning the interaction of our system with theuser, for example by allowing the user to view and manage the blacklist andwhitelist databases from within the system application. It is also vital to

www.necoma-project.eu 102 March 31, 2016

Page 103: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

explore ways of hardening the implementation against privileged malware,as well as integrate it with the latest Android release.

www.necoma-project.eu 103 March 31, 2016

Page 104: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.3.6 Conclusion and Lessons Learned

In the set of scenarios, we demonstrated the developed system aimed atmitigating typical cyber threats targeting endpoint devices. Through thedemonstration, we addressed mainly the issue of their resilience. Each de-fense might be reconfigureable even if the status of cyber threats changesfrom time to time. It is therefore important to reduce the impact of the delaywhile performing defense. In the Drive-By-Download Prevention scenario,the list of the malware distribution servers was configurable to provide bet-ter cyber defense. The NECOMAtter system, as used in this scenario, playedan important role to share threat information and to provide an interface tocontrol the list. We also considered metrics such as the average renderingtime for measuring the induced delay, and confirmed that the impact wasmarginal since it only required an additional 0.36 second.

The Phishing Protection scenario was designed to check the usability ofthe cyber defense system. In order to better defend end users, we attemptto check whether a user is a novice or an expert. The two types of anal-ysis developed by ATOS and UT had distinct characteristics: ATOS’s waslightweight and focused on providing the highest precision despite lowerrecall. We utilized the defense based on the ATOS’s analysis module forexperts. Conversely, UT’s analysis modules required much more time incomparison to ATOS’s. However, it was designed to provide better recallalthough its precision was not so as good as ATOS’s. The system provides acountermeasure based on “deactivation”, i.e., web input forms were madeunavailable to users. Aside from this explicit exception, the user can browsethe entire web page content as usual. Besides, the system had functions forreactivating input forms if the user did adopt the right behavior, and if thewebsite was not a determined to be a phishing site. Even when the delaywas 2.6 seconds in the worst case, the loss of usability felt by users waslimited.

In the SMS fraud protection scenario, we demonstrated a system whichprevents malicious applications from contracting premium rate numbers bysending SMS messages and making phone calls. The detection was baesd onthe whitelist and blacklist of the numbers all of which users can maintain.We constructed test environment for receiving SMS fraud and demonstratedthat the defense stopped the known malware.

Finally, we studied a DoS defense system for smartphones in which com-promised devices are prevented from joining a DDoS campaign. We definedthe requirements along with smartphone’s facilities and designed and imple-mented the defense at OpenFlow-capable wireless access points. It shouldbe noted that the list of suspicious IP addresses would be generated forDDoS mitigation methods usually running at the infrastructure layer, andthe same list would be reused for the AP-based firewall. Especially, in thecase of SDN-based DDoS prevention, if an operator or a system inject flow

www.necoma-project.eu 104 March 31, 2016

Page 105: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.3. SMARTPHONE USER PROTECTION

rules for mitigating DoS attacks, they will be made available for endpointlayer defense as well as infrastructure layer defense. As we mentioned inSection 3.1.5, the SDN paradigm opposes controller applications that allowto dynamically reconfigure the APs. Our defense works as an applicationthat accepts requests for adding and deleting filtering rules. From the viewpoint of battery consumption, the application also had a function for disas-sociating the compromised device to save smartphone’s battery.

Through the set of scenarios, we addressed the defense systems’ re-silience runs on multiple policy enforcement points. When actionable threatinformation was provided for defense systems, we confirmed that the sys-tems would work as we expected. In particular, we developed cyber defensesystems that were reconfigurable to cope with the targeted cyber threats.Although it was beyond the scope of the project, we also considered to ex-change various types of information obtained at endpoint layers. For ex-ample, the website’s URL that a user is browsing will give hints to analyzesuspicious websites. However, the exchange of these information amonganalysis modules might violate privacy concerns. The social acceptance forsharing cyber threat information is a key for facilitating multilayer analysisin our future work. If we could enrich analysis modules on the platform,it would be possible to match other types of defense, in addition to thedemonstrated defenses against typical cyber threats.

www.necoma-project.eu 105 March 31, 2016

Page 106: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.4 Malware Campaign Mitigation

The malware campaign mitigation scenario verifies the usability of NECOMAtools in a real-world setting. The scenario (only one in this case study) isdeliberately implemented without a testbed, although such approaches werealso considered earlier.

The scenario assumes a typical CERT/CSIRT setting (such as CERT Pol-ska, involved in the implementation of the scenario). The scenario docu-ments the work of an analyst, having access to large sets of data and taskedwith handling of security incidents in a large network or collection of net-works. This analyst is equipped with NECOMA tools, including the analysismodules developed in WP2 and datasets made available as part of NECOMAWP1/WP3 through the n6 interface.

The goal of the analyst is to identify a malicious campaign in the largesets of seemingly non-related incidents and extract as much actionable in-formation [6] as possible.

The experiments described in this section were conducted entirely onreal data. The results shown document the detection of actual malwarecampaigns, real C&C servers and infected machines. The manual steps wereperformed by experts who routinely deal with similar problems. The sce-nario is built as a real-world demonstration, using real data both as inputfor the analysis and as means of verification of mitigation results.

www.necoma-project.eu 106 March 31, 2016

Page 107: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.4. MALWARE CAMPAIGN MITIGATION

3.4.1 Evaluation Framework – Datasets

The analysis performed in the scenario generally is based on the third-partymalicious URL datasets (see deliverable D1.4 for details), containing mali-cious URLs from multiple different sources. The correlation phase leveragedthe production instance of the n6 platform with all datasets that it contains,including ones containing information on command & control servers, mal-ware infections and phishing.

During verification, several datasets were used. The MAWI NetFlow datais used to verify the presence of suspicious IP addresses in Japanese traffic.The verification was also performed on two datasets which cannot be sharedin full. The first was the NetFlow data from the network operated by NASKin its ISP role. This provides the possibility of assessing the remediationeffectiveness in a real, large-scale setting. The second closed dataset collectsNetFlow data from an academic segment of the NASK network. This datasetis richer (higher sampling frequency), but yielded no useful results, sincethe data were collected in a different time period – presumably the detectedcampaigns were not active at the moment.

www.necoma-project.eu 107 March 31, 2016

Page 108: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.4.1.1 Experiments

The experiment involves multiple steps, some of which have been auto-mated as part of the NECOMA project. Some of the steps cannot howeverbe automated, as discussed below.

1. Starting with the malurl datasets, group related malicious URLs, tok-enize them and identify common ones (automated).

2. Using the FP-growth algorithm, identify frequent itemsets – sets oftokens that appear in many unrelated URL groups. (automated)

3. Selection of promising patterns in the results of the previous step. Thisstep has to be performed manually – the step involves a significantamount of expert knowledge, which so far cannot be automated effec-tively.

4. Cross-correlation with external datasets. This step cannot be fully au-tomated, as the selection of relevant information requires expert input.However, the collection of the information for expert assessment canbe and has been simplified and automated. For NECOMA datasets thisgoal is achieved using the n6 APIs. For external sources, searchableusing commercial search engines, the tokenseeker developed in WP1is used.

5. Non-automated additional research. This step involves the expert’sexperience and may involve different sources depending on the par-ticular case. In practice, during experiments in WP4 VirusTotal Intelli-gence proved to be a valuable resource, most often used by the experts– this may allow future automation.

6. PCAP extraction. This step depends on the malware sample involved.In most cases the campaign involves malware already seen in the wildand information can be gathered from known sources such as VirusTo-tal. If no data is available, the samples can be collected and analyzedin sandbox environments.

7. C&C analysis, comparing the new PCAP data with previously knownsamples. The process lists the IP addresses contacted by the samples,filters out benign ones and selects the ones most likely to be C&Cservers based on similarity of behavior and known benign/C&C ad-dresses. (automated)

8. Manual verification of IP addresses – this step is basically a repetitionof step 4 using the IP addresses as starting tokens.

www.necoma-project.eu 108 March 31, 2016

Page 109: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.4. MALWARE CAMPAIGN MITIGATION

9. Generation of blocking rules for the selected IP addresses. In realitythe rules would be implemented in the network. In the demonstrationscenario their usefulness is assessed using NetFlow datasets.

3.4.1.2 Results

The following section describes the different metrics used to assess the re-sults of the scenario. The metrics, unless otherwise stated, are defined as inDeliverable D4.1.

During the experiment, multiple campaigns were identified (see 3.4.1.2.2).After a multi-step filtering process performed by an analyst assisted by NECOMAtools, it was possible to identify two campaigns of modern malware familiesand collect enough actionable information for mitigation. The first cam-paign used Dridex malware and its activity peaked in January 2015. Thesecond campaign used the Pony bot, started in January 2016 and is stillactive at the time of writing of this report.

3.4.1.2.1 Detection precision The definition of this metric presented indeliverable D4.1 proved to be too strict to be applicable in this scenario. Thereason for this is the fact that the terms “true positive” and “false positive”are fuzzy in case of malicious campaign identification – even with expertknowledge it is not always clear whether certain patterns should be con-sidered “true positives”. Therefore we define the following three classes ofpatterns instead:

C campaign – the pattern clearly identifies a single malware campaign;

G generic – the pattern is strongly correlated with malicious activity but isnot indicative of any particular campaign – it appears in URLs involvedin several unrelated campaigns or might be common in URLs (bothmalicious and legitimate) in general;

L legitimate – the pattern is not indicative of any malicious campaigns, itappears regularly in benign URLs, most of the malurl entries exhibitingthis pattern are not malware-related; identification of such pattern isusually an effect of presence of false positives in the source data.

Using these classes we can define the following metrics, more suitablefor this use-case:

campaign detection ratio C/(C +G+ L)

false positive ratio L/(C +G+ L)

Evaluation was performed on 100 maximal frequent itemsets with biggestsupport, with the following results: campaign detection ratio 38%, false

www.necoma-project.eu 109 March 31, 2016

Page 110: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

positive ratio 3%, and the remaining 59% fell into the generic category.Such result is encouraging, since it shows that the chosen approach can pro-vide patterns that can be used to pinpoint specific malware campaigns. Thepresence of false positives confirms the assumption that patterns obtainedthrough text mining can expose incorrect input data and must be furtherverified before beining used for blocking or alerting. Finally, the relativelyhigh number of generic patterns is expected and does not pose a major prob-lem, since an analyst should be able to quickly reject such results and focuson the more specific ones instead.

3.4.1.2.2 Detection rate The test dataset spans 14 months, from 2015-01-01 to 2016-02-24. In that time 2216896 unique URLs pointing to bina-ries were collected and included in the analysis. A campaign is detected ifa maximal frequent itemset identified in the source data has sufficient sup-port to suggest that a significant number of machines might be involved inthe campaign. The support is defined as the number of distinct IP addresseshosting malicious URLs containing the given pattern.

During tests, the minimal support for tokens was set to 20. As a conse-quence, no itemset with support of less than 20 will appear in the results.Most of maximal frequent itemsets have support close to the minimal value,detailed distribution is presented in table 3.11.

Table 3.11: Quantiles of support values for maximal frequent itemsetsmined from the malicious URL database.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%20 20 20 20 20 21 22 23 25 31 802

While every frequent itemset could be considered an indicator, they haveto be validated by a human analyst, so it is worth focusing on the largestones, i.e. ones affecting a large number of IPs (high support). A thresh-old to distinguish between large and insignificant campaigns is arbitrary, sothe decision will most likely depend on resources available for the task ofinvestigating them.

Based on the distribution of the test data, a first natural cutoff valueis support 21 (inclusive), which is satisfied by 3866 itemsets (55% of allmaximal itemsets). This rate translates into approximately 9 reports per dayon average, which should be feasible for manual review. Since the supportvalues provide an easy way to prioritize campaigns, it is possible to reducethe number of reports as needed without missing the most common ones.

Note that the detection rate is quite arbitrary in this use case, as it de-pends heavily on the support threshold and expert tolerance in pattern se-lection phase. The FP-tree algorithm will generally provide large numbersof itemsets, most of them with very small support and not interesting for

www.necoma-project.eu 110 March 31, 2016

Page 111: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.4. MALWARE CAMPAIGN MITIGATION

further analysis. The approach taken during the experiment results fromearlier experience and balances the false positive ratio, the detection ratioand the processing capability of the experts involved in manual steps. Themeasured detection rate is a consequence of this choice.

3.4.1.2.3 Detection time The metric measures the time between thestart of analysis of the data and the availability of actionable information.Note that this is not limited to the campaign detection process itself – itextends to the entire verification and analysis process, ending only whenactual actionable information is present, enabling mitigation.

Since the process involves many phases (as described in detail in section3.4.1.1) it is instructive to discuss the time consumed by each phase.

1. tokenization (automated): 6 minutes 11 seconds as measured for3,189,050 URLs on a single Intel Core2 Quad processor at 2.66 GHz

2. maximal frequent itemsets generation (automated): 112 seconds asmeasured for 331043 items (tokens) in 102729 transactions (URLs)on a single Intel Core2 Duo processor at 1.8 GHz

3. pattern selection (manual): ≈ 10 seconds per pattern

4. correlation with source datasets (manual): 1-5 minutes per pattern

5. obtaining information from external sources, including VTI (manual):1-5 minutes per pattern

6. obtaining PCAPs (semi-automated): 5-20 seconds11

7. PCAP C&C analysis time (automated): 0.57 seconds12

8. IP address verification time (manual): from 10 seconds to 5 minutes

The total detection time varies from 1-2 hours in routine cases to 1-2days for complex campaigns, involving previously unknown malware andC&C servers and using many connections to uncommon benign servers toobfuscate the C&C traffic.

11This measurement applies if PCAPs for the sample are already available. If not, sandboxanalysis must be performed, which may take upwards from 5 minutes – the upper limit isarbitrary and depends on the time the sample is allowed to run.

12This assumes using pre-learned knowledge. The learning process is performed periodi-cally on updated sets of samples and takes several minutes for 1000 samples.

www.necoma-project.eu 111 March 31, 2016

Page 112: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.4.1.2.4 Mitigation time This metric measures the time from the mo-ment the actionable information about a campaign has been obtained to themoment the activity is actually mitigated.

Assuming that the start of the period is marked by submission of thecollected information to the n6 system, the total mitigation time dm is com-posed of four components:

n6 processing delay dn representing the time until the submitted informa-tion becomes available to clients;

polling delay dp, representing the time it takes the clients to become awareof the new data; this delay is negligible (packet RTT) for users of then6 stream API as they are automatically notified (push approach); forusers of the n6 REST API this delay is entirely dependent on the client’susage pattern and may theoretically range from negligible to infinite;

deployment delay dd, representing the time it takes to deploy appropriaterules in all desired policy enforcement points in the client’s network.

Therefore, the total mitigation time is given as

dm =

{dn + dp + dd for REST API usersdn + dd for stream API users

• dn: It can be observed that the delay in event processing introduced bythe production deployment of the n6 platform at CERT Polska variessignificantly thorough the day. In the first quarter of 2016, messagedelay ranged from less than a minute to almost two hours. Such vari-ance can be explained by the n6 architecture: each event goes throughmany processing steps — in particular multiple enrichment stages –which are implemented by specialized components. Since the volumeof incoming data can vary greatly during the day (especially whendata is fetched in large batches from some sources), some of thesecomponents become saturated, which results in queuing of events.

As of March 2016, the slowest enrichment component has the maxi-mal throughput of 56 events per second, while the incoming data itmust handle can reach approximately 600 events per second (dailypeak). While the average overall throughput is currently sufficient,NASK plans to improve performance of individual components. Therealistic goal for the 3rd quarter of 2016 is to achieve median eventdelay of less than 10 minutes.

• dp: An attempt to estimate the propagation time of information fromthe n6 platform to recipients was made. Analysis of access patternsin the first quarter of 2016 by clients regularly fetching data through

www.necoma-project.eu 112 March 31, 2016

Page 113: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.4. MALWARE CAMPAIGN MITIGATION

the REST API allowed to establish their polling periods. Clients withpolling period of less than an hour constitute only 41% of the activepopulation, 38% downloaded data once per day only, and 19% did itexactly once per hour. The median polling period for all active clientsis one hour.

Since on average, the expected value of delay of the information flowis equal to the half of the polling period, currently 60% of recipientswill experience delay of either 30 minutes or 12 hours.

• dd: The deployment time is dependent on the policies of the clientand may vary greatly. In case of manual deployment with no strictpolicies the delay may even reach days (until the administrator de-cides to update his rules). We will ignore this case, as the delay isnot only unpredictable, but unrelated to the operation of the systemitself. In the more common case of automation or expedited manualdeployment, the time is much shorter.

In small networks, automatically deploying new rules directly to a sin-gle policy enforcement point the operation is nearly instantaneous. Inlarge networks with complex automated systems the propagation ofrules to all enforcement points may often take up to an hour. Manualverification of the new rules, performed by some clients, may extendthat time but can be ignored in our analysis as an extra precaution.

The deployment delay can therefore be expected to range from afew seconds to approximately one hour depending on the size of theclient’s network and the mechanisms used to deploy rules.

In total, the median mitigation time dm for stream API users will gener-ally range from several minutes to 2-3 hours, mostly depending on deploy-ment method and n6 platform’s load. For REST API users the mitigationtime is typically longer and may range from less than an hour to about 15hours, mostly depending on the polling period of the client.

3.4.1.2.5 Actionable information count The average number of poten-tial C&C IP addresses collected per campaign in the conducted experimentsis 12.4. However, it should be noted that this metric is nonlinearly depen-dent on the amount of PCAP samples collected. The changes would affectall other metrics as well, so this metric should be discussed in context of allothers.

3.4.1.2.6 Mitigation effectiveness The scenario produces two kinds ofindicators which can be used to mitigate the threat.

www.necoma-project.eu 113 March 31, 2016

Page 114: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

Common patterns identified in the FP-tree analysis and verified later canbe used to detect new URLs belonging to the same campaign in the produc-tion traffic. Such campaign indicators can be deployed on HTTP proxies,mail filters, or even end-user machines, to block or alert whenever maliciouscontent is detected and protect users from getting infected. Since email andHTTP are two biggest infection vectors, such approach could reach 100%effectiveness in blocking specific campaigns.

The main problem with such approach is potentially high ratio of falsepositives. It could be addressed by checking if detected patterns do notappear in a whitelist consisting of legitimate URLs (ones that have highprobability of never being used for malicious purposes). This enhancementis left as further work.

The main focus of the scenario is the second type of the actionable infor-mation obtained from the analysis: IP addresses of C&C servers, which arethe main output of the presented processing pipeline. The C&C addressesidentified in later stages of the analysis can be used to block C&C traffic anddetect infected machines in the monitored network. Such indicators canbe used by IDS/IPS systems, firewalls, or matched against traffic records todetect suspicious behavior in the past.

The main goal of the mitigation effectiveness metric is to estimate theeffect of using such indicators on a large scale. For this purpose, the C&Caddresses were matched against MATATABI and NASK datasets containingnetwork activity (NetFlow and sFlow). The correlation period was Septem-ber 2014 – March 2016 for MATATABI datasets and December 2014 – March2016 for the NASK dataset. All datasets were heavily sampled, with typicalratio of 1 : 104.

Results of the correlation are presented in the figure 3.40. It was as-sumed that all IP addresses communicating with confirmed C&C servers arepotential victims of malicious activity. Thanks to this approach it was possi-ble to identify actual malicious network activity and a number of suspicioustraffic. If indicators generated by the NECOMA analysis modules were usedfor network defense, it would be possible to protect these users by prevent-ing the infection or blocking C&C channels.

It can be observed that the number of affected users is not large. Themost probable explanation for this fact is that the monitored networks didnot contain significant populations of infected machines and perhaps thecampaigns were targeted at different countries or networks. The high sam-pling ration is also a contributing factor, as many connections were simplynot registered in the collected datasets.

According to observations, while URL patterns can be specific for a spe-cific campaign, these campaigns often share certain parts of their infrastruc-ture, in particular C&C servers, with other malicious activities. Thereforeblocking a single IP address associated with the campaign might result inpreventing other attacks, beyond the original campaign that was identified.

www.necoma-project.eu 114 March 31, 2016

Page 115: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.4. MALWARE CAMPAIGN MITIGATION

0

100

200

300

0

10

20

30

MA

TATA

BI

NA

SK

2014

-09

2014

-10

2014

-11

2014

-12

2015

-01

2015

-02

2015

-03

2015

-04

2015

-05

2015

-06

2015

-07

2015

-08

2015

-09

2015

-10

2015

-11

2015

-12

2016

-01

2016

-02

2016

-03

2016

-04

month

num

ber o

f pot

entia

l vic

tims

Figure 3.40: Number of potential victims identified in network trafficdatasets, based on indicators obtained through analysis of two malwarecampaigns.

A consequence for this metric is that it is impossible to determine exactlywhat type of malicious activity was prevented by using just IP addresses.Therefore the results presented here most likely include some behavior as-sociated with other malicious campaigns. It can be confirmed by comparingthe timing of campaigns with the number of potentially affected users (mea-sured by unique IP addresses) in the figure 3.40. While the Pony campaignthat peaked in January and was active afterwards clearly corresponds to theactivity detected in NASK network, there is no obvious link between peaksof activity in MATATABI datasets (August 2015 and September 2014) andthe two campaigns that were associated with the C&C addresses used formatching traffic.

www.necoma-project.eu 115 March 31, 2016

Page 116: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

3.4.2 Conclusion and Lessons Learned

The scenario has been successful, not only in extracting actionable informa-tion, but – perhaps more importantly – in showing how the NECOMA toolsincrease the efficiency and effectiveness of the work of a security analyst.New automated analyses of large datasets create good starting points foran expert and provide valuable data. The common interfaces enable fastand effective research to be performed on multiple rich datasets, withoutthe need for switching between various access methods and repeating thesame searches in multiple locations.

From the point of view of a security expert performing the scenario,several elements constitute significant progress from the state of the art:

• The frequent pattern analysis provides an effective way to extract com-mon patterns from the rich, but relatively low-quality and hard to an-alyze malicious URL datasets. To the best of our knowledge no com-parable tool has been developed and deployed.

• The PCAP C&C analysis is a useful tool, automating what is often man-ual work. Existing automated analyses tend to focus on a single, iden-tified malware type. The analysis module enables mixed, unfilteredPCAP data to be processed, providing a useful clustering (with theability to match samples to known malware groups if present in learn-ing data or at least group related samples), extracting potential C&Caddresses and filtering out benign ones. Such tools, implemented asready-to-use analysis modules are also currently missing, although asignificant amount of research has been performed worldwide on sim-ilar approaches and R&D implementations obviously exist.

• The n6 API as a common interface for many different datasets is prov-ing valuable for data fusion. Correlation between different datasetsis easy and efficient. Scalability is achieved through common seman-tics and search mechanisms [7]. Also, a common user interface makesthe mechanism easy to use – while a steep learning curve may notbe a significant hindrance for a highly skilled expert, avoiding it isalways worthwhile and saves an expert’s time. While a lot of workhas been performed on developing common interfaces, tools and stan-dards, the n6 REST API has proved to be very simple, practical, easy touse, easy to integrate with new datasets and client tools. The large setof datasets already accessible through this API and the availability ofearly, but operational versions of the new stream API make n6 a veryuseful platform for a security analyst.

The tools developed during the NECOMA process will therefore be sup-ported, developed further and put to operational use by CERT Polska, asspecified in the exploitation plan (Deliverable D5.6).

www.necoma-project.eu 116 March 31, 2016

Page 117: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

3.4. MALWARE CAMPAIGN MITIGATION

3.4.2.1 Lessons Learned

The experiments have shown that there are many adware/spyware cam-paigns represented in the collected data, the more dangerous threats aremuch less common. This fact makes the manual verification of results by anexpert even more crucial, as the automated analyses cannot realistically dis-cern serious malware campaigns from relatively benign ones and prioritizeappropriately.

Focus on executable files proved to be valuable. The patterns involvingURLs of executable files have the greatest probability of actually being mali-cious. Even if a pattern not involving an executable file is indeed connectedto a malicious campaign, a sample of the malware is difficult to obtain. Thiskind of filtering is also an effective method of dealing with the relativelyhigh number of false positives in the malicious URLs datasets.

An interesting and perhaps worrying observation is that there are morecharacteristic templates present in data from the first half of 2015 thanin later data. It is likely that this is the effect of more effective obfusca-tion strategies being employed by malware authors, making URL-parsingapproaches to campaign detection less effective.

While the automated analysis successfully identifies common patterns,the work of an analyst is still crucial – the results need to be reviewed toweed out the benign examples corresponding to common frameworks, pop-ular binaries duplicated on many servers, etc. A whitelist approach may beattempted to reduce (but not eliminate) this need. Note that the analyst’sexperience with the new tools influences significantly the processing time inall phases of the process. The tools can be very efficient if used correctly butlack of experience may extend the time of analysis several times.

A rather disappointing conclusion from the performed experiments is thelow quality of the datasets used as input. This problem was already indi-cated by the dataset ratings, as presented in D2.2 (table 4.2). The maliciousURL datasets have the highest false positive rate of all datasets available inNECOMA.

Some datasets actually tend to repeat old, outdated data, increasingnoise. One such example found during the experiments was a dataset,which repeated reports of another, public one, after 6 months. Data qualityis therefore a significant limiting factor in the analysis, increasing the needfor expert input. False positives are impossible to avoid and any results ob-tained during the analysis require manual verification before being used forremediation purposes.

Another characteristic of the input dataset is the disproportionate re-porting of some data items. The initial analyses attempted processing of theraw data resulting in many false positive errors. Multiple instances of sim-ilar URLs pointing to the same target were interpreted as campaigns, eventhough the underlying reason for the existence of common patterns was e.g.

www.necoma-project.eu 117 March 31, 2016

Page 118: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 3. CASE STUDIES

the structure of a web site. Meanwhile, actual malicious campaigns can beexpected to appear in many unrelated places. This problem was solved bygrouping the input data. Domain-based grouping, counting multiple entriesfrom the same domain as one entry increased the precision significantly.Even better results have been obtained by grouping entries resolving to thesame IP address.

An important weakness of the input datasets is the lack of meaningfullabels indicating the cause of marking the URL as malicious by the source,even the classification methods are usually not identified. The previouslymentioned problems may be related to that fact. There is no way to separatethe URLs used for drive-by-downloads, hosting malware binaries, or used forC&C infrastructure.

From all information sources used during the analysis, a surprisinglylarge amount of actionable information was provided by the automated ex-ternal knowledge collection mechanism – tokenseeker – which uses externalsearch engines to collect unstructured data. The n6 datasets available atNASK also provided useful information, although they could not be used toclassify all the investigated IP addresses. MATATABI sources could be used toconfirm infections and activity of C&C servers in some cases but in generaldid not provide much information relevant to the investigations. The lowutility of MATATABI information in Europe-based experiments is not surpris-ing, as earlier work has shown relatively low correlation between Europeanand Japanese datasets. In the cases where data is available in MATATABI,that information itself is meaningful, as it indicates a global phenomenon,observed in both regions.

3.4.2.2 Future work

The ability to find common patterns in URL datasets proved to be usefulfor analysts. Currently, there are no directly comparable tools for this task,therefore research into more effective algorithms and extensions such aswhitelists is planned in the near future.

Effective ways of dealing with the lack of labels in malicious URL datasetsare also clearly needed. Methods of separating these different kinds of ma-licious URLs using correlations with other datasets are a possibility and willbe researched in the near future. Experiments with C&C datasets (well la-beled) show that such correlation, when possible, increases markedly thequality of results. Depending on the quality of the grouping provided bythe frequent pattern analysis, a successful correlation of even a small set ofURLs may allow effective classifiers to be created.

Further work on the n6 platform and associated APIs is also worthwhile,as the common interface to multiple datasets is very useful in practice, low-ering the costs of multi-layer analysis.

www.necoma-project.eu 118 March 31, 2016

Page 119: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

4Conclusion

The main matter of this deliverable has focused on reporting the ultimateoutcomes of this project, putting into perspective the results obtained in for-mer workpackages. However, as evidenced by the lessons learned at theend of each chapter, developing and integrating a full demonstration of theNECOMA threat information pipeline does not come easily. In this deliver-able, we have considered four different use cases to demonstrate how theNECOMA threat information pipeline unfolds in different settings, scopesand contexts. The wealth of scenarios attempts to illustrate as much as pos-sible the NECOMA approach, integrating outcomes of the technical work-packages. In particular, with each use case addressing different concernsand goals, we show that the integrated approach from data collection toanalysis to reaction was feasible in diverse settings: enabling autonomicDDoS mitigation, correlating diverse observations in botnet introspection,adapting defense mechanisms for smartphone users, and supporting an an-alyst in fighting against malware campaigns. Ultimately, we were able toprovide an improved resilience for systems at the infrastructure level, andusers at the endpoints. This translates to the following major results, aspresented in this deliverable:

1. we have implemented autonomic DDoS mitigation to some extent,thanks to the SDN paradigm, leveraging (1) the global view of thenetwork through the centralized controller, and (2) the programma-bility of the data plane by reconfiguring the switches, both of whichenforce our mitigation policy. We have demonstrated the ability tocomplete the autonomic loop, and to recover from incidents in a rea-sonable time, in spite of volumetric attacks;

2. we have enabled a way to gather intelligence on botnets via a dualapproach where the prior collection of malware domains allows theinterception of interesting traffic, while live bot monitoring in a sand-

119

Page 120: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

CHAPTER 4. CONCLUSION

boxed environment helps in diverting their traffic to a decoy C&Cserver, allowing the observation of their behaviors;

3. we have improved the awareness of smartphone users through a setof tools able to counter common but critical threats, not only com-mon web threats like drive-by downloads and phishing, but also thosethreats which are more specific to smartphones like energy-efficientfirewalling and SMS fraud. The tools leveraged each an importantaspect of the NECOMA philosophy: personalization of defense mech-anisms, cross-layer analysis, automated reconfiguration, and raisingawareness;

4. we have provided a method to support the work of a security ana-lyst, by extracting actionable information from a malware campaign,blending automated analyses of large datasets with a common inter-face for fast and effective searches. This use case demonstrates thatthe NECOMA approach is repeatable and obtains results that were notpossible in the past without labor-intensive work.

Additionally, each use case provides an evaluation framework to (1) re-produce our approach, leveraging the disseminated datasets and tools, and(2) put our results in perspective. We have detailed our choice of metricsto capture relevant aspects of our designs, which later help to interpret theresults of our experiments. We have put particular stress on providing test-ing environments that are more realistic. This particular requirement was agreat challenge, and could not always be completely fulfilled, although wetried to reproduce attack traffic characteristics as much as possible. Finally,we gained much experience in designing and developing testing platformsto host our scenarios, and we expect to use and/or share them in futurecollaborative projects.

www.necoma-project.eu 120 March 31, 2016

Page 121: SEVENTH FRAMEWORK PROGRAMME · SEVENTH FRAMEWORK PROGRAMME Information & Communication Technologies ICT Cooperation Programme Nippon-European Cyberdefense-Oriented Multilayer threat

Bibliography

[1] D. Anstee, D. Bussiere, and G. Sockrider. Worldwide Infrastructure Security Report 2012Volume VIII. Technical report, Jan. 2013.

[2] L. De Cicco, V. Caldaralo, V. Palmisano, and S. Mascolo. Tapas: A tool for rapid prototyp-ing of adaptive streaming algorithms. In Proceedings of the 2014 Workshop on Design,Quality and Deployment of Adaptive Video Streaming, VideoNext ’14, pages 1–6, NewYork, NY, USA, 2014. ACM.

[3] F. Dobrian, V. Sekar, A. Awan, I. Stoica, D. Joseph, A. Ganjam, J. Zhan, and H. Zhang.Understanding the impact of video quality on user engagement. SIGCOMM Comput.Commun. Rev., 41(4):362–373, Aug. 2011.

[4] D. Miyamoto, H. Hazeyama, and Y. Kadobayashi. An Evaluation of Machine Learning-based Methods for Detection of Phishing Sites. Australian Journal of Intelligent Informa-tion Processing Systems, 10(2):54–63, 2008.

[5] D. Miyamoto, T. Iimura, G. Blanc, H. Tazaki, and Y. Kadobayashi. EyeBit: Eye-TrackingApproach for Enforcing Phishing Prevention Habits. In Proceedings of the 3rd Inter-national Workshop on Building Analysis Datasets and Gathering Experience Returns forSecurity (BADGERS), Sept. 2014.

[6] P. Pawlinski, P. Jaroszewski, P. Kijewski, Ł. Siewierski, P. Jacewicz, and P. Zielony. Ac-tionable information for security incident response. https://www.enisa.europa.eu/activities/cert/support/actionable-information, Nov 2014.

[7] P. Pawlinski and A. Kozakiewicz. Lowering cost of data exchange for analysis and de-fence. In Proceedings of the Coordinating Attack Response at Internet Scale (CARIS) Work-shop, June 2015.

[8] Symantec Security Response. International Takedown Wounds Gameover Zeus Cy-bercrime Network. Available at: http://www.symantec.com/connect/blogs/international-takedown-wounds-gameover-zeus-cybercrime-network,2014.

121