SECURITY OF THE SECURITY TOOLS

76
SECURITY OF THE SECURITY TOOLS Fu-Hau Hsu National Central University 1

description

SECURITY OF THE SECURITY TOOLS. Fu-Hau Hsu National Central University. Background. Plenty of critical personal information is stored in computers Many commercial activities are proceeded through computers and networks. Results. - PowerPoint PPT Presentation

Transcript of SECURITY OF THE SECURITY TOOLS

Page 1: SECURITY OF THE SECURITY TOOLS

SECURITY OF THE SECURITY TOOLSFu-Hau HsuNational Central University

1

Page 2: SECURITY OF THE SECURITY TOOLS

Background2

Plenty of critical personal information is stored in computers

Many commercial activities are proceeded through computers and networks.

Page 3: SECURITY OF THE SECURITY TOOLS

Results3

Computers become a major battlefield between attackers and computer users.

Page 4: SECURITY OF THE SECURITY TOOLS

Solutions4

Diverse security tools,have been developed to handle this critical work. anti-virus software auto-patch mechanisms firewall intrusion detection systems and so on

Page 5: SECURITY OF THE SECURITY TOOLS

New Problem5

The complexity of these security tools increases.

The security tools become attack targets.

The security of the security tools also becomes a crucial issue.

Page 6: SECURITY OF THE SECURITY TOOLS

Disable Anti-virus Software

Research has shown that diverse approaches have been developed to disable different anti-virus software.

6

Page 7: SECURITY OF THE SECURITY TOOLS

Problems (1)7

How to guarantee the normal operation of anti-virus software.

Page 8: SECURITY OF THE SECURITY TOOLS

Problems (2)8

If attackers can disable anti-virus software, is it possible that they can also change the behavior of the anti-virus software through replacing, injecting, or hooking code?

Page 9: SECURITY OF THE SECURITY TOOLS

Impacts of Anti-virus Parasites Given the fact that most, if not all,

computers allow only one anti-virus application to be installed on them at one time, an attacker that can control the anti-virus application of a computer can fully command the computer without being detected.

9

Page 10: SECURITY OF THE SECURITY TOOLS

Auto-Patch

How could a computer guarantee that the auto-patch code it executes is not malicious code?

10

Page 11: SECURITY OF THE SECURITY TOOLS

Impacts

If the correctness of auto-patch code cannot be protected, instead of being a tool to seal the security breaches of a computer, auto-patch may become an efficient channel for attackers to intrude a computer or spread malicious code.

11

Page 12: SECURITY OF THE SECURITY TOOLS

12

Antivirus Software Shield against the Antivirus Terminator

Page 13: SECURITY OF THE SECURITY TOOLS

Purpose13

We proposed ANtivirus Software Shield (ANSS) to protect antivirus software from being terminated by malware.

Page 14: SECURITY OF THE SECURITY TOOLS

Arms Race between Malware Writers and Antivirus Programmers14

The simplest way for a computer user to secure his computer is to install antivirus software on his computer. 81% of computer users utilize antivirus software to protect

their computers, which means over a hundred million computer users use antivirus software in the USA alone.

Attackers attempt to break any defense mechanism by masquerading social engineering hindering antivirus software detection

Page 15: SECURITY OF THE SECURITY TOOLS

Malware Self-Defense Techniques (1)

15

Disorder signature-based virus detection methods by Transforming Encoding Compressing Packing malware code.

Avoid executing themselves when they are monitored by debuggers hypervisors.

Page 16: SECURITY OF THE SECURITY TOOLS

Malware Self-Defense Techniques (2)

16

Unmeaning bytes may also be inserted into malware to hinder static malware analysis.

More aggressive and dangerous malware may even directly disrupt the functionality of security solutions, especially antivirus software

Page 17: SECURITY OF THE SECURITY TOOLS

Antivirus Terminator17

Malware that terminates the execution of antivirus software without the consciousness of the antivirus software users is called an antivirus terminator.

Page 18: SECURITY OF THE SECURITY TOOLS

Who Protect Antivirus?18

The most effective solution to antivirus terminators is antivirus software itself.

Page 19: SECURITY OF THE SECURITY TOOLS

The Problem19

Even though antivirus companies claim that they have implemented protection mechanism to protect their products against antivirus terminators, our experimental results shown that techniques utilized by antivirus terminators to terminate antivirus software still could terminate antivirus software.

Page 20: SECURITY OF THE SECURITY TOOLS

20

Approaches to Terminate Antivirus Software

Page 21: SECURITY OF THE SECURITY TOOLS

Null Debugger Method21

An antivirus terminator can use API DebugActiveProcess to attach itself to an antivirus process as a debugger to control it.

Then right after invoking DebugActiveProcess, the antivirus terminator exits.

Because the controller of the antivirus process does not exist, the process will crash immediately.

Page 22: SECURITY OF THE SECURITY TOOLS

Dll Unloading Method22

An antivirus terminator can use API ZwUnmapViewOfSection to unload some important dll files, such as ntdll.dll, from an antivirus process to erase some portions of the virtual address space of the antivirus process.

ntdll.dll defines many common used windows Native APIs, hence, once the dll file is unloaded from the antivirus process and an API in the dll file is invoked, the process will crash.

Page 23: SECURITY OF THE SECURITY TOOLS

Process Termination Method

23

An antivirus terminator gets the handle of an antivirus process by calling API OpenProcess with the process ID of the antivirus process.

Then the antivirus terminator can use APIs NtTerminateProcess or ZwTerminateProcess to terminate the antivirus process in the kernel level.

Page 24: SECURITY OF THE SECURITY TOOLS

Close Message Method24

An antivirus terminator can use API FindWindow to search all windows running on the system to find the window matching the name of an antivirus software window (e.g. avguard of antivir or kavsvc of kaspersky).

Then the antivirus terminator continues sending messages, such as WM_CLOSE or WM_QUIT, to the related antivirus process by APIs SendMessage or PostMessage until the process is terminated.

Page 25: SECURITY OF THE SECURITY TOOLS

Mouse Simulator Method25

An antivirus terminator can use API SendInput to counterfeit a series of mouse events which lead to the suspension of an antivirus process.

E.g.: An antivirus terminator may find the icon of an

antivirus process first. Then it forges a series of mouse events to move

the cursor to the right lower corner of the screen, press the right button to expand a work menu, and then chose to suspend an antivirus process.

This approach was designed by us. Hence, we have not found an antivirus terminator in the wild utilizing this approach to terminate antivirus software.

Page 26: SECURITY OF THE SECURITY TOOLS

Registry Modification Method (1)

26

An antivirus terminator can modify the registry so that a NULL debugger will be attached to an antivirus process when the antivirus process begins its execution.

The above steps will stop the execution of the antivirus process.

Page 27: SECURITY OF THE SECURITY TOOLS

Registry Modification Method (2)

27

Besides, an antivirus terminator can also modify the registry to delete antivirus related processes from the startup process list; hence, the system will be booted without the protection of an antivirus process.

API ZwOpenKey can be used to open a registry key.

And API ZwSetValueKey can be used to modify a registry key value.

Page 28: SECURITY OF THE SECURITY TOOLS

Thread Termination Method28

An antivirus terminator can use API TerminateThread to terminate the threads of an antivirus process one by one till the antivirus process stops.

Page 29: SECURITY OF THE SECURITY TOOLS

Table I: Methods Utilized by Various Malware to Terminate Antivirus Software

29

Page 30: SECURITY OF THE SECURITY TOOLS

Combine Two or More Terminating Approaches

30

As Table I shown, attackers may combine two or three techniques together to strengthen their malware.

Page 31: SECURITY OF THE SECURITY TOOLS

Market Share Survey31

Page 32: SECURITY OF THE SECURITY TOOLS

Most Popular Antivirus Software 5 famous brands of antivirus software

were used in our tests. Avira Antivir Personal Kaspersky Anti-virus 2012 Norton Antivirus 2011 ESET NOD32 Antivirus 4 AVAST Free Antivirus 6.

32

Page 33: SECURITY OF THE SECURITY TOOLS

Terminator Program33

For each of the above methods we wrote a program, called terminator program, to see whether the approach alone can terminate antivirus.

None of them deems the above 7 programs as malware.

Page 34: SECURITY OF THE SECURITY TOOLS

Impact of Terminator Programs

34

The results show that if a piece of malware bundles itself with one or more of the first 6 terminator programs; it becomes an effective antivirus terminator.

As a result, we believe antivirus terminators will become a severe threat to many computer systems which are usually protected by antivirus.

Page 35: SECURITY OF THE SECURITY TOOLS

Frangible APIs35

The Windows native APIs that are executed when the above 7 techniques are used to terminate antivirus software are called frangible APIs .

The native APIs may not be called directly by antivirus terminators, but are called by the APIs utilized by the antivirus terminators.

Page 36: SECURITY OF THE SECURITY TOOLS

List of Frangible APIs36

Page 37: SECURITY OF THE SECURITY TOOLS

Intercept Windows API Calls

37

ANSS monitors invocations to frangible APIs.

Frangible APIs calls with parameters that will stop or suspend antivirus software are blocked by ANSS directly.

ANSS uses SSDT (System Service Descriptor Table) hooking to intercept and examine invocations of frangible APIs.

Page 38: SECURITY OF THE SECURITY TOOLS

System Service Dispatch Table

38

The Windows kernel provides various system services for application programs to use.

Inside the kernel the addresses of these system services are stored in the SSDT, KiServiceTable.

Page 39: SECURITY OF THE SECURITY TOOLS

Usage of SSDT39

When a program issues a system call to execute a system service, kiSystemService looks up the SSDT to find the address of the corresponding system service.

Then kiSystemService uses the address to invoke the system service.

Page 40: SECURITY OF THE SECURITY TOOLS

SSDT Hooking40

Through SSDT hooking, ANSS modifies some function addresses stored in the KiServiceTable service descriptor table and replaces them with ANSS API handlers.

After an ANSS API completes its task, it invokes the original API.

Page 41: SECURITY OF THE SECURITY TOOLS

Execution Flow of a System Call with and without SDDT Hooking

41

Page 42: SECURITY OF THE SECURITY TOOLS

ANSS Structure42

ANSS consists of three components: interceptor, filter, and blocker.

Page 43: SECURITY OF THE SECURITY TOOLS

Interceptor43

When a program is going to invoke a frangible API, the interceptor intercepts the execution flow of the program and transfers the execution flow to the filter before the code of a frangible API is executed.

ANSS utilizes SSDT hooking to accomplish the above task.

Page 44: SECURITY OF THE SECURITY TOOLS

Frangible APIs and ANSS APIs

44

For each frangible API, there is a corresponding ANSS API in the ANSS filter.

In other words, an invocation to a frangible API is transferred to its corresponding ANSS API first through SSDT hooking.

Page 45: SECURITY OF THE SECURITY TOOLS

Filter45

Based on some build-in rules, the filter examines the parameters of the API call to filter out the intention to abuse the API that will terminate or suspend antivirus software.

For a normal API call, the filter transfers the execution flow back the original frangible API code.

Page 46: SECURITY OF THE SECURITY TOOLS

Blocker46

An API call with a malicious intention is further passed to the blocker which returns an error code, ACCESS_DENY, as the output of the API call.

Besides, the blocker prints out an error message to notify users that it just blocked an attack on the antivirus software.

Page 47: SECURITY OF THE SECURITY TOOLS

Filter Rules47

The ANSS filter utilizes various rules to prevent frangible APIs from being executed with malicious parameters.

Page 48: SECURITY OF THE SECURITY TOOLS

Antivirus Process Handle Rule

48

The ANSS filter restrains an application from obtaining the process handle of an antivirus program, because once an attacker obtains the handles of an antivirus program, the attacker can terminate the program easily. To get the process handle of an application

program, attackers can use API OpenProcess which utilizes frangible API NtOpenProcess inside the kernel land to finish its work.

Page 49: SECURITY OF THE SECURITY TOOLS

NtTerminateProcess Rule49

The filter does not allow an application to use NtTerminateProcess to terminate antivirus software.

Page 50: SECURITY OF THE SECURITY TOOLS

DebugActiveProcess Rule50

The filter does not allow an application to utilize DebugActiveProcess to attach itself as a debugger of antivirus software.

DebugActiveProcess uses frangible API NtDebugActiveProcess in the kernel address space to accomplish its task.

Page 51: SECURITY OF THE SECURITY TOOLS

PostMessage Rule51

The ANSS filter prohibits a program from sending messages, such as WM_CLOSE or WM_QUIT, to antivirus software, because an antivirus terminator can use API PostMessage to send the above messages to the antivirus software to stop it.

API PostMessage sends messages through the kernel land frangible API NtUserPostMessage.

Page 52: SECURITY OF THE SECURITY TOOLS

Registry Key Rule52

The filter disallows an application to modify or delete Windows Registry keys to protect antivirus software, because attackers can use registry APIs, like NtDeleteKey, NtSetValueKey, and NtCreateKey to disable antivirus software.

Page 53: SECURITY OF THE SECURITY TOOLS

NtUnmapViewOfSection Rule

53

The ANSS filter forbids an application to use NtUnmapViewOfSection to unload a .dll file from antivirus, because an antivirus terminator can use the frangible API to stop antivirus software.

Page 54: SECURITY OF THE SECURITY TOOLS

SendInput Rule54

When the foreground process is antivirus software, the ANSS filter inhibits an application to use API SendInput to deliver mouse events to the process, because an antivirus terminator can use API SendInput to imitate a series of action leading to the termination of antivirus software.

Page 55: SECURITY OF THE SECURITY TOOLS

Limitation55

ANSS uses SSDT hooking to intercept frangible APIs calls.

If after ANSS is installed in a computer, an antivirus terminator can bypass the detection of the antivirus software in the computer and hook the system too, then ANSS may no longer work as expected.

Page 56: SECURITY OF THE SECURITY TOOLS

Solution56

To provide a thorough protection, ANSS needs to work with the anti-hooking mechanism which is a well studied issue in the literature.

Page 57: SECURITY OF THE SECURITY TOOLS

Evaluation - Compatibility57

We made diverse experiments to test whether ANSS influences the operations of normal applications.

Page 58: SECURITY OF THE SECURITY TOOLS

Compatibility Evaluation Samples

58

We use false positive tests to evaluate the influence caused by ANSS upon normal applications.

There are two types of applications used in our tests. The first type of application is antivirus

software. Any non-antivirus application is classified

as the second type.

Page 59: SECURITY OF THE SECURITY TOOLS

Antivirus Compatibility Evaluation

59

For antivirus software, we installed ANSS first. Then we manually disabled the antivirus

software, and manually enabled the antivirus software to check whether these operations proceeded as usual.

Besides, we also updated the virus definition files made regular system scan shut down the operating system.

All operations proceeded as usual.

Page 60: SECURITY OF THE SECURITY TOOLS

Result60

Page 61: SECURITY OF THE SECURITY TOOLS

Non-Antivirus Compatibility Evaluation

61

We installed ANSS in 8 our daily used computers to test if ANSS influences their normal operations.

Then we used these computers as usual.

Page 62: SECURITY OF THE SECURITY TOOLS

Hardware Specification of Test Hosts

62

These computers had installed Microsoft Windows XP Professional Service

Pack 3 with Intel 2.66 GHz Core 2 Quad CPU Q9400 1.96 GB RAM.

Page 63: SECURITY OF THE SECURITY TOOLS

Software Used in the Test Hosts

63

Besides, lots of popular applications, such as Firefox Windows LiveMessenger iTunes Oracle VMVirtual Machine Avira AntiVir Personal Microsoft Office Adobe Reader.

Page 64: SECURITY OF THE SECURITY TOOLS

Result64

After 6 months’ tests, none of the applications or computers was disabled, crashed, or blocked by ANSS.

In other words, ANSS did not influence the operations of applications and computers during our test periods.

Page 65: SECURITY OF THE SECURITY TOOLS

Evaluation - Functionality65

We made diverse experiments to test whether ANSS can protect antivirus software.

Page 66: SECURITY OF THE SECURITY TOOLS

Functionality Evaluation Samples

66

Then we used the 18 antivirus terminators that we abstracted from 2063 malware samples to test the 5 famous brands of antivirus software: Avira Antivir Personal Kaspersky Anti-virus 2012 Norton Antivirus 2011 ESET NOD32 Antivirus 4 AVAST Free Antivirus 6

Page 67: SECURITY OF THE SECURITY TOOLS

Challenge67

However, the virus definition files of some of the above antivirus software have already had the signatures of the antivirus terminators used in our tests.

Hence, before the antivirus terminators execute, they will be detected and blocked by antivirus software.

Page 68: SECURITY OF THE SECURITY TOOLS

Solution68

But our experiments only want to test whether the mechanisms that antivirus terminators use to terminate antivirus software can really close antivirus software.

Hence, when making our experiments, we temporarily did not allow antivirus software to scan files so that we can accurately test the terminating mechanisms without interference.

Page 69: SECURITY OF THE SECURITY TOOLS

Result69

Experimental results show that every antivirus terminator can terminate some of these antivirus programs.

However, if these antivirus programs were protected by ANSS, none of the antivirus terminators can close them.

Page 70: SECURITY OF THE SECURITY TOOLS

Survivability of Various Antivirus Programs with Or without ANSS70

Page 71: SECURITY OF THE SECURITY TOOLS

Performance Evaluation Environment

71

We used Windows benchmark tool called PassMark Performance Test to test the CPU and memory performance overhead of ANSS on a Windows XP system with Intel 2.66 GHz Core 2Quad CPU Q9400 448MB RAM.

Page 72: SECURITY OF THE SECURITY TOOLS

Performance Overhead Measurement

72

Performance overhead measurement was performed under two different system setups. In the first setup, we installed a Windows

XP system which did not execute ANSS. In the second setup, we installed a

Windows XP system that executed ANSS. Under each setup, we executed PassMark a hundred times.

For each benchmark, we use its average value to represent its final value.

Page 73: SECURITY OF THE SECURITY TOOLS

CPU Performance Overhead73

Page 74: SECURITY OF THE SECURITY TOOLS

Memory Performance Overhead

74

Page 75: SECURITY OF THE SECURITY TOOLS

Conclusion75

Experimental results show that many commercial antivirus programs will be closed by antivirus terminators.

However, when being protected by ANSS, antivirus software can function as usual even under the attacks of antivirus terminators.

ANSS introduces at most 0.42% CPU performance overhead and 1.77% memory write performance overhead.

Therefore, ANSS is an effective and efficient solution against antivirus terminators.

Page 76: SECURITY OF THE SECURITY TOOLS

76

Thanks