Product Security Incident Response Team (PSIRT) – Изнутри Cisco PSIRT

Inside Cisco's Product Security Incident Response Team (PSIRT)

Alexey Lukatsky

Business Security Consultant, Cisco GSSO

[email protected]

© 2014 Cisco and/or its affiliates. All rights reserved.BRKSEC-2012 Cisco Public

На этом можно было бы и закончить

2

«Их практика работы направлена на то, чтобы шантажировать заказчика и покупателя. Они вывешивают в открытом доступе систему своих уязвимостей и говорят — коллеги, если вы хотите, чтобы эти уязвимости не были использованы, заплатите нам за поддержку и мы их устраним»Помощник президента России Игорь Щеголев


Agenda• Introduction

• PSIRT’s Mission Process and Engagement

• Vulnerability Management Process

• Customer Expectations

• PSIRT Publications and Triage

• Cisco Security Development Lifecycle (CSDL)

• New Trends in Vulnerability Management

• Case Studies

• Security Automation & Cisco’s Machine Readable Content

• Conclusion

3


Introduction

4

The Cisco PSIRT is a dedicated, global team that manages the receipt, investigation, and public reporting of security vulnerability information that is related to Cisco products and networks.


Security Research & Operations (SR&O)

5

PSIRT

IntelliShieldApplied Security Research

IPS Signature Team

Applied Security Intelligence

SIO PortalSecurity Technology Assessment Team

(STAT)

Security Blog

ASIG

Talos


PSIRT’s Mission• Global team assisting customers with the ongoing security of their networks through identification, resolution and prevention of vulnerabilities in Cisco products and industry-wide vulnerabilities.

PROTECT CUSTOMERS AND PROTECT CISCO

6


PSIRT’s Mission (continued).

Single point of contact for receiving and resolving internal and external reports of vulnerabilities in all Cisco products since 1995.

7


When Does PSIRT Engage?

• Cisco products likely to be affected, but not always

• Maintenance contract not necessary• Customer requests PSIRT involvement

• Support engineer feels attack is new or unknown or escalation is required

• Caller is a member of external incident response team

• Law enforcement is already involved

8

PSIRT’s PUBLICATIONS

9


PSIRT’s Publications

PSIRT creates and publishes:Cisco Security Advisories, Notices, and Responses

Fair public disclosure: everyone notified at the same time.

www.cisco.com/go/psirtwww.cisco.com/security

10

http://www.cisco.com/web/about/security/psirt/security_vulnerability_policy.html

The following table summarizes the methods used by Cisco to notify customers about the security vulnerabilities and other security information.

Email SIO Portal RSS CNS Bug Search Tool

Security Advisories Yes Yes Yes Yes Yes

Security Notices No Yes Yes No Yes

Security Response Yes Yes Yes Yes Yes

Cisco Event Responses No Yes Yes No No

Threat Outbreak Alerts / IntelliShield Alerts No Yes Yes No No

Release Note Enclosures No No No No Yes

11


Cisco uses the following CVSS guidelines when determining which security publication will include a particular vulnerability:

Publication CVSS Score

Cisco Security Advisory 7.0 – 10.0

Cisco Security Notice 4.0 – 6.9

Bug Release Note Enclosure 0.1 – 3.9

Cisco Security Responses address issues that require a response to information discussed in a public forum, such as a blog or discussion list. The responses are normally published if a third party makes a public statement about a Cisco product vulnerability.

12

INDUSTRY LEADERSHIP & COLLABORATION

13


PSIRT’s Security Community Engagement

Coordination as required with external agencies (CERT/CC, CPNI, etc.)

14


PSIRT’s Security Community EngagementRepresents Cisco in the incident response and security communities.

Cisco is a founding member of the Industry Consortium for Advancement of Security on the Internet (ICASI) enhances the global security landscape by driving excellence and innovation in security response practices, and by enabling its members to proactively collaborate to analyze, mitigate, and resolve multi-vendor, global security challenges.

15

PSIRT’s PROCESS

16

PSIRT Collaborates With Experts Across Cisco

Many other teams

Technology Groupsproduct experts

Technical Assistance Centersupport experts

Legal & Public Relations

Advanced Serviceshigh touch support experts

Security Research & Operations

Security Experts


PSIRT Scope: More Than Vulnerability Handling • Provide security expertise to Cisco’s product development and testing organizations

• Deliver security training and education, internally and externally

• Share best practices in industry through customer forums, executive briefings, security conferences, and cisco.com content

• Mentor others building vulnerability handling capabilities to strengthen collective response

19

Cisco Security Development Lifecycle (CSDL)

20


Cisco Secure Development Lifecycle (CSDL)

Why Security is Good Business Sense:

• Reduced cost of fixing bugs• Remove expense and pain of changing security architecture

• Reduces TTM (time to market) over time• Day-one advantage over our less security savvy competitors

• Improve customer satisfaction• Lower PSIRT and customer cases

Perform GAPAnalysis

PreventSecurityAttacks

DetectSecurityDefects

ValidateRequirementsand Resiliency

Identify andAddressSecurityThreats

Register and Update 3rd PartySoftware

21

CASE STUDY 1 - HEARTBLEED

22


What is Heartbleed?

• If the specified heartbeat request length is larger than its actual length, this memcpy() will read memory past the request buffer and store it in the response buffer which is sent to the attacker

• OpenSSL1.0.1 – 1.0.1f are vulnerable

• Bug was introduced in December 2011 but not found/disclosed until April 2014– OpenSSL is used by 2/3 of Internet web servers and many products

• Approximate 534,156 services are vulnerable

• Cisco was one of the first security companies to provide IPS coverage


Background

• Exploitation Allows Access to Device Memory Contents• Attackers could potentially extract sensitive information• Cryptographic keys and certificates are of particular concern

• Impact of Exploitation Depends on Multiple Factors• Role of affected device in the network• How OpenSSL is used on the device


Cisco’s Response

• Announced Publicly on April 7th 2014• No industry coordination;; vulnerability was disclosed before vendors were informed

• Cisco PSIRT Coordinating Response and Investigation• Cisco Security Advisory published April 9th• Cisco among the first vendors to respond• Initial focus on accurate listing of Cisco products and services• Updated daily as new information is discovered

• Detection and Mitigation Strategies Include:• Cisco Sourcefire and Cisco IPS signatures are available• Technology-specific guidance and best practices


Security Impact

• Bigger than 443• Any SSL service is being targeted • Most prominent sites have already patched• Many, many, smaller sites are not patched…

• Worst case: Private keys, credentials and more leaked• Hijacked accounts -> more exploit kits• Embedded devices are unlikely to patch• May enable lateral movement• Without security monitoring there is no real way to know if you were exploited

• The client side attack is also concerning


Services Being Targeted

Destination Port/ICMP Code465 (smtps)/tcp995 (pop3s)/tcp993 (imaps)/tcp443 (https)/tcp


Cisco Product Impact

• Cisco Impact Varies per Product/Service• PSIRT assumes worst-case in product assessment• Deployment architecture may significantly reduce “real” risk

• Potential Exposure of Critical Data• Remediation Steps• Upgrade to a fixed version of software• Reissue cryptographic keys and certificates• Force password resets• Detection• IPS can detect and block attack attempts


High-Level Assessment of Potential Exposure


Step 1: Identify the vulnerable SSL/TLS product or software • One method of determining vulnerable devices is through vendor security advisories. For example, Cisco’s OpenSSL Heartbeat Extension Vulnerability in Multiple Cisco Products security advisory

• An alternate method of identifying vulnerabilities is through the utilization of specifically designed tools. Examples:

• Clients: pacemaker https://github.com/Lekensteyn/pacemaker• Web-based tools: https://filippo.io/Heartbleed/


Step 2: Identify the affected features• It is important to know which product feature is impacted.

• If a product were only vulnerable when using feature X, it would mean that it is not vulnerable when the feature is not in use.

• Note: Administrators should note that devices that have SSH (not a TLS feature) enabled are not affected by this vulnerability.

BUG

FEATURE


Additional Steps

• Is the client connecting to pre-determined/trusted or unpredictable/untrusted servers?• Pre-determined/trusted servers• Unpredictable/untrusted servers(i.e., a browser which is connecting to any random website)

• Can you verify with certainty that the vulnerable product is using process memory separation?

• Is the client authenticated by the server?

• Is the vulnerable server feature accessible from untrusted networks?


Vulnerable Server Remediation Options

• Apply patch from software vendor

• To protect against the Heartbleed vulnerability, the vulnerable server would need to be upgraded or recompiled

• The latest OpenSSL fixed version 1.0.1g or newer should be used

• If it is not possible to upgrade to the fixed release of OpenSSL, vulnerable software can be recompiled linking to OpenSSL with the handshake removed from the code by compile time option -DOPENSSL_NO_HEARTBEATS


Coverage

• Sourcefire IPS• 30510 - 30513 inbound connection attempts beyond a normal threshold• 30514 - 30517 large outbound heartbeat responses (successful exploitation)• 30520 - 30525 outbound vulnerable client traffic

• Cisco Legacy IPS• 4187-3 - inbound connection attempts beyond a normal threshold• 4187-4 - large outbound heartbeat responses (successful exploitation)/outbound vulnerable client traffic


Online ResourcesCisco Security Portal:

• Security Advisoryhttp://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20140409-heartbleed

• Event Response Pagehttp://www.cisco.com/web/about/security/intelligence/ERP-Heartbleed.html

• IntelliShield Alerthttp://tools.cisco.com/security/center/viewThreatOutbreakAlert.x?alertId=33695

Blog Posts including Mitigation, Detection and Best Practices:• http://blogs.cisco.com/security/openssl-heartbleed-vulnerability-cve-2014-0160-cisco-products-and-mitigations

• http://vrt-blog.snort.org/2014/04/heartbleed-memory-disclosure-upgrade.html• http://blogs.cisco.com/security/heartbleed-transparency-for-our-customers/

Cisco Security and Services:• http://www.cisco.com/go/security• http://www.cisco.com/c/en/us/products/security/service-listing.html

CASE STUDY 2 – SPECIALIZED & CUSTOM MALWARE IN INFRASTRUCTURE DEVICES

37


New Threat Landscape

• Targeted attacks and custom malware against infrastructure devices (routers, switches, etc.)

• These attacks go undetected for a longer time than traditional attacks

Infrastructure Devices


History

• Theoretical Research in 2005-2006 (FX & Mike Lynn)

• Recent incidents (2013 & 2014)– Custom malware to change infrastructure device configurations– Remote code execution– Persistent attacks


Custom Malware

• Malware is software created to modify a device's behavior for the benefit of a malicious third party (attacker).

• One of the characteristics of effective malware is that it can run on a device stealthily in privileged mode.

• Malware is usually designed to monitor and exfiltrate information from the operating system on which it is running without being detected.

• Potentially sophisticated Cisco IOS malware would attempt to hide its presence by modifying Cisco IOS command output that would reveal information about it.


Context: Malware seen targeting IOS Classic

http://blogs.cisco.com/security/evolution-of-attacks-on-cisco-ios-devices


Infrastructure Device Infection

On Cisco devices running Cisco IOS Software, a limited number of infection methods are available to malware. Malicious software in Cisco IOS Software may be introduced in the following ways:

• By altering the software image stored on the onboard device file system. These types of malware would be persistent and would remain after a reboot.

• By tampering with Cisco IOS memory during run time. In this case, the malware is not persistent and a reload will restore the Cisco IOS device to a clean state booted from the image stored in the flash.

• By modifying the ROM monitor on systems with flash-based ROM monitor storage.

• By a combination of some or all of the preceding mechanisms


Attack Methods

• Some Cisco IOS devices offer a limited set of commands that are intended to be used by Cisco Technical Assistance Center (TAC) engineers during the process of troubleshooting a technical problem. Such advanced troubleshooting and diagnostic commands require privileged EXEC level and require valid credentials to execute. Thus, these commands could be an area that attackers can focus on to identify ways to run malicious software in Cisco IOS.

• It is important to note that not all Cisco IOS platforms offer advanced diagnostic commands. Of the platforms that do, only a very limited set of such commands is usually available. Additionally, to run these commands, a user needs administrative access to the device. Thus, following common authentication and command authorization security best practices will help prevent a malicious user from even attempting to install malicious software in Cisco IOS Software.

Commands

43


Attack Methods (cont.)

• It is possible that an attacker could insert malicious code into a Cisco IOS Software image and load it onto a Cisco device that supports the image.

• This attack scenario applies to any computing device that loads its operating system from an external, writable device.

• Even though such a scenario is not impossible, there are image verification techniques, discussed in the Cisco IOS Image File Verification section of this document that could prevent the router from loading such an image.

Manipulating Cisco IOS Images

44


Attack Methods (cont.)

• As with every operating system, there is a possibility that a vulnerability could exist in Cisco IOS Software that, under certain conditions, could allow malicious code execution.

• An attacker who exploited the vulnerability would install or run malicious code in Cisco IOS Software, which could then be used to take malicious action, such as modifying device behaviors or exfiltrating information.

• PSIRT identifies, manages, and releases all vulnerabilities in and fixes for Cisco products.

• Any vulnerability that Cisco is made aware of is investigated and released in accordance with the Cisco vulnerability disclosure policy.


Vulnerabilities

45


Identification Techniques

MD5 hash calculation and verification using the MD5 File Validation feature can be accomplished using the following command:

verify /md5 filesystem:filename [md5-hash]

Network administrators can use the verify /md5 privileged EXEC command to verify the integrity of image files that are stored on the Cisco IOS file system of a device. The following example shows how to use the verify /md5 command on a Cisco IOS device:

R1# verify /md5 sup-bootdisk:c7600rsp72043-advipservicesk9-mz.151-3.S3

.....<output truncated>.....Done!

verify /md5 (sup-bootdisk:c7600rsp72043-advipservicesk9-mz.151-3.S3) = e383bf779e137367839593efa8f0f725

Using the Message Digest 5 File Validation Feature

46

Network administrators can also provide an MD5 hash to the verify command. If the hash is provided, the verify command will compare the calculated and provided MD5 hashes as illustrated in the following example:

R1# verify /md5 sup-bootdisk:c7600rsp72043-advipservicesk9-mz.151-3.S3 e383bf779e137367839593efa8f0f725

.....<output truncated>.....Done!

Verified (sup-bootdisk:c7600rsp72043-advipservicesk9-mz.151-3.S3) = e383bf779e137367839593efa8f0f725router#



Cisco IOS Software image file verification using this feature can be accomplished using the following commands:

file verify auto

copy [/erase] [/verify | /noverify] source-url destination-url

reload [warm] [/verify | /noverify] [text | in time [text] | at time [text] | cancel

The following example shows how to configure the file verify auto Cisco IOS feature:router# configure terminalrouter(config)# file verify autorouter(config)# exitrouter#

Using the Image Verification Feature

47



Network administrators can also verify the integrity of the run-time memory of Cisco IOS.

The best way to verify the integrity of run-time memory for IOS is to analyze the region of memory called “main:text.”

The main:text section contains the actual executable code for Cisco IOS Software after it is loaded in memory. As such, verifying its integrity is particularly relevant for detecting in-memory tampering. This region of memory should not change during normal Cisco IOS Software operation, and should be the same across reloads.

Because this region of memory holds the actual operating system code, it should not change between devices as long as they are the same model and running the same release number and feature set. However, if the Cisco IOS release in use is ASLR enabled, these assumptions become invalid. A side effect of ASLR is changing some parts of the operating system code. This means the memory contents will be different across devices, even if they are running the same operating system release and feature set.

http://www.cisco.com/web/about/security/intelligence/integrity-assurance.html

Cisco IOS Run-Time Memory Integrity Verification

48


Additional Indicators of Compromise

The presence of the following commands should trigger further investigation. The asterisk symbol * indicates any text that follows the command itself.

gdb *test *tlcsh *service internalattach *remote *ipc-con *if-con *execute-on *show regionshow memory *show platform *do-exec version of any of the above

Check logs for the presence of “unusual” commands

49


Additional Indicators of Compromise (cont.)

Cisco IOS devices support exporting the contents of the running memory. After the export, comparisons between the running memory dump, also called core dump, and the associated sections in the Cisco IOS image file can be performed to detect modification of the run-time memory contents.

Most Cisco IOS releases support a memory dump via the write core command.

The following example shows how to search suspicious commands captured in a core dump file by using the Linux utility string:

$ strings <CORE> |grep ^CMD:CMD: 'verify /md5 system:memory/text' 06:59:50 UTC Wed Jan 15 2014CMD: 'service internal | i exce' 07:02:41 UTC Wed Jan 15 2014CMD: 'conf t' 07:02:45 UTC Wed Jan 15 2014CMD: 'exception flash procmem bootflash:' 07:02:54 UTC Wed Jan 15 2014CMD: 'exception core-file CORE compress ' 07:03:31 UTC Wed Jan 15 2014

Checking Command History in the Cisco IOS Core Dump

50


Resources

• This document analyzes injection of malicious software in Cisco IOS Software and describes ways to verify that the software on a Cisco router, both in device storage and in running memory, has not been modified.

• Additionally, the document presents common best practices that can aid in protecting against attempts to inject malicious software (also referred to as malware) in a Cisco IOS device.

http://www.cisco.com/web/about/security/intelligence/integrity-assurance.html

SECURITY AUTOMATION & CISCO MACHINE READABLE CONTENT

52


Robust support for relevant standards to ensure multi-‐layer

interoperability

EMERGING TECHNOLOGIES

Completely closed solutions

EVOLVING MATURITY MATURE IMPLEMENTATIONS

Adoption of basic interoperability standards

Security Automation Evolution

Industry’s perception of the security automation evolution

WE ARE HERE

PAST FUTURE

53


OVAL: Cisco IOS Vulnerability Assessment• Cisco PSIRT is including Open Vulnerability and

Assessment Language (OVAL) definitions in Cisco IOS security advisories.

• OVAL provides a structured and standard machine-‐readable content that allows customers to quickly consume security vulnerability information and identify affected devices.

• OVAL can also be used to verify that the patches or fixes that resolve such vulnerabilities were successfully installed.

• OVAL content can be downloaded from each Cisco IOS security advisories

Common Vulnerability Reporting Framework (CVRF)• In addition to OVAL definitions, PSIRT is

also publishing CVRF content for all Cisco security advisories.

• CVRF allows vendors to publish security advisories in an XML (machine-‐readable) format.

• CVRF has been designed by the Industry Consortium for Advancement of Security on the Internet (ICASI), of which Cisco is a member and took a major role in its development.

Vulnerability Machine Readable ContentCisco is committed to protect customers by sharing critical security-related information in different formats.

54


CISCO PSIRT - PROTECTING CISCO CUSTOMERS

Cisco is committed to protect customers by sharing critical security-related information in different formats.


More information at: http://oval.mitre.orgCisco OVAL White Paper: http://cs.co/90035hJ3

Introduction to OVALOpen Vulnerability and Assessment Language (OVAL) -‐ an international community standard to promote open and publicly available security content and to standardize the transfer of this information in security tools and services.

OVAL provides a structured and standard machine-‐readable content that allows customers to quickly consume security vulnerability information and identify affected devices.

OVAL can also be used to verify that the patches or fixes that resolve such vulnerabilities were successfully installed.


CISCO PSIRT - PROTECTING CISCO CUSTOMERS

Security Automation: Cisco IOS OVAL Content• Cisco PSIRT is creating OVAL content

(“definitions”) for Cisco IOS security advisories.

• OVAL content can be downloaded from each Cisco IOS security advisory; Cisco Security Event Response Pages and from the following link/repository:

http://tools.cisco.com/security/center/ovalListing.x

57


What is OVAL?What are the use cases?

There are four main use cases, also called “classes,” of OVAL definitions:

• Vulnerability: Determine if the device is affected by a given vulnerability• Compliance: Validate a device configuration against a known or approved valid configurations (i.e., best practices)

• Inventory: Check for a specific version of software installed on the system• Patches: Find a specific patch on the system

58

59

OVAL ComponentsOVAL Definitions

•XML files that are used to check the presence of a vulnerability or a configuration best practice.

OVAL Schemas•OVAL definitions are XML documents;; thus they need schemas.

•The purpose of an XML Schema is to define the building blocks of an XML document

•OVAL XML Schemas define elements, attributes, and data types that are part of an OVAL definition

•Example: how OVAL checks for affected versions;; different configurations (i.e., ACLs, Interfaces, Routing Protocols, etc.)

Authoring Tool

•Cisco created internal tools to support the creation of IOS vulnerability definitions

System Characteristics Producer

•Generates and keeps details of the system being evaluated

•Examples: jOVAL Definition Interpreter, McAfee Policy Auditor, etc.

Definition Repository

•A repository of OVAL Definitions made available to the community (free or pay).

•Cisco publishes OVAL definitions that can be downloaded from each IOS security advisories.

Definition Evaluator

•A product that uses an OVAL Definition to guide evaluation and produces OVAL Results (full results) as output.

•Examples: jOVAL


Authoring ToolDefinition

Evaluator/Scanner(openscap/jOVAL)

Definition Repository

OVAL Definition

OVAL Definition

OVAL System Characteristics

OVAL Definition

ResultsConsumer

OVAL Results

How Everything Works Together…High-level

60


Additional OVAL SchemasEnhancements and New Schemas

61

Cisco Security Research and Operations (SR&O) recently numerous enhancements to the Cisco IOS OVAL Schemata and created new schemas for:

• IOS-XE

• Cisco ASA

Example: Assessing an IOS device using OVAL

62


Topology

63

Machine with jOVAL (OVAL Scanner)

R1:172.18.122.246


Technical DetailsVulnerability details and router configuration

64

• CVE-2012-0381 addresses a vulnerability that affects the Cisco IOS Software Internet Key Exchange (IKE) implementation.

• R1 is configured for IPsec and it is running an affected version. • The following is an excerpt of the IPsec/IKE configuration of R1:

crypto isakmp policy 10encr aes 256authentication pre-share!crypto map test 10 ipsec-isakmpset peer 10.10.10.10match address 101!interface FastEthernet0/1ip address 14.4.1.126 255.255.255.0crypto map test


jOVAL ExamplejOVAL Configuration and OVAL Definition Information

65

The OVAL definition filename is cisco-sa-20120328-ike-CVE-2012-0381_oval.xml and it resides in a directory called DEFINITIONS.

To scan R1, the jovaldi.bat utility was used, as shown in the following example:

D:\joval>jovaldi.bat -plugin remote -m -o DEFINITIONS\cisco-sa-20120328-ike-CVE-2012-0381.xml


jOVAL ExamplejOVAL Output

66

D:\joval>jovaldi.bat -plugin remote -m -o DEFINITIONS\cisco-sa-20120328-ike-CVE-2012-0381.xml… <output omitted for brevity> ** parsing D:\joval\DEFINITIONS\cisco-sa-20120328-ike-CVE-2012-0381.xml

- validating xml schema.** checking schema version

- Schema version - 5.10** skipping Schematron validation** creating a new OVAL System Characteristics file.** gathering data for the OVAL definitions.

Collecting object: FINISHED** saving data model to system-characteristics.xml.** skipping Schematron validation** running the OVAL Definition analysis.

Analyzing definition: FINISHED** OVAL definition results.

OVAL Id Result-------------------------------------------------------oval:cisco.oval:def:13 true-------------------------------------------------------

** finished evaluating OVAL definitions. ** saving OVAL results to results.xml.** skipping Schematron validation** running OVAL Results xsl: xml\results_to_html.xsl.

True = device is vulnerable


HTML ReportjOVAL Report Example

67

True = device is vulnerable


HTML ReportjOVAL Report Example

68

In the following example, IPsec was disabled on R1. After this change, the device was not vulnerable.

False = device is not vulnerable


ResourcesCisco’s OVAL and CVRF Resources

69

Resource Description LinkWhite Paper Details of the SCAP components, as well as step-by-step

instructions on how to use OVAL content with available open source tools.

http://cs.co/9001V4vP

FAQ Published to help answer common questions related to Cisco’s OVAL adoption.

http://cs.co/9004V4vr

Cisco SIOPortal

Early-warning intelligence, threat and vulnerability information, and proven Cisco mitigation solutions to help customers protect their networks.

http://cisco.com/security

Security Blog Cisco Security Blog posts providing information about OVAL, CVRF and security automation.

http://cs.co/9000V4vE

http://cs.co/9009V4vD

Cisco’s SecurityVulnerability Policy

Cisco’s public security vulnerability policy including information about OVAL and CVRF content.

http://cs.co/9008V4vM

Product Security Incident Response Team (PSIRT) – Изнутри Cisco PSIRT

Internet

Transcript of Product Security Incident Response Team (PSIRT) – Изнутри Cisco PSIRT