Exploiting the Testing System
-
Upload
frisksoftware -
Category
Technology
-
view
544 -
download
1
description
Transcript of Exploiting the Testing System
![Page 1: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/1.jpg)
Exploiting the testing system
Viorel Canja,Head of BitDefender Labs
![Page 2: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/2.jpg)
Contents
What does the title mean ? Testing detection on wildcore Testing detection on zoo collections Retrospective detection tests Examples Feedback from the industry Q&A
![Page 3: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/3.jpg)
What does the title mean ?
Purpose of tests: to define metrics and measure the performance of
AV products to find am approximation for the real world
performance of AV products to give feedback to AV researchers about their
products to allow the users to make an informed decision
![Page 4: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/4.jpg)
What does the title mean ?
“Define:exploit” use or manipulate to one's advantage draw from; make good use of overwork: work excessively hard
![Page 5: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/5.jpg)
What does the title mean ?
To use the limitations of the testing procedure to one’s advantage.
The focus is on those actions which have questionable benefits for the user.
![Page 6: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/6.jpg)
Types of tests
detection tests on wildcore detection tests on zoo collections retrospective detection tests
![Page 7: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/7.jpg)
Testing detection on wildcore
• What is wildcore ?“WildCore is a set of replicated virus samples that
represents the real threat to computer users.”“When a virus is reported to us by two or more
Reporters, it's a pretty good indication that the virus is out there, spreading, causing real problems to users. We consider such a virus to be 'In the Wild'.”
![Page 8: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/8.jpg)
Testing detection on wildcore
The Wildcore samples are known to all AV companies as soon as wildcore is published.
Tests are likely to be performed on exactly the same samples. This is always the case with samples of malware which does not replicate.
![Page 9: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/9.jpg)
Testing detection on wildcore
Quick hack: just sign all the samples with dumb ( aka automatic ) signatures.
Disable heuristics to avoid false positives ( if the testbed is already known there is no need for technology that detects previously unknown threats )
![Page 10: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/10.jpg)
Testing detection on zoo collections
Zoo should contain a large number of files so that the statistics are as accurate as possible
Threats should be replicated ( where applicable ) or large numbers of samples should be used for polymorphic malware or malware that is re-generated on the server
The zoo should not contain garbage
![Page 11: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/11.jpg)
Testing detection on zoo collections
Hacks:- use customized settings for the test. Heuristics
should be set to paranoid mode. Automatically sign all previously missed samples and white-list all previously reported false positives.
- automatically sign all samples detected by at least one AV product just to be on the “safe” side
![Page 12: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/12.jpg)
Testing detection on zoo collections
Hacks (2):
- add detection routines for garbage that is usually found in collections. This includes detecting known false positives of other products, detecting damaged executables, detecting files produced by different analysis tools.
![Page 13: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/13.jpg)
Retrospective detection tests
Signature databases are frozen at a certain moment Detection is tested against samples received after
that moment Testing should be done with default settings because
most of the products are marketed as “install and forget” and the majority of users will not change the settings
![Page 14: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/14.jpg)
Retrospective detection tests
Has the disadvantage that it will not take into account proactive detections introduced by generic routines created for malware families that appear after the signatures are frozen
These routines ( or signatures ) will detect proactively subsequent variants of the same family
Favors aggressive heuristics if not correlated with false positive tests
![Page 15: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/15.jpg)
Examples
Automatic signing: Av01 (1st pair) : TR/Zapchast.CP Av02 : Collected.Z Av03: W32/KillAV.3B84!tr Av04: Trojan.Downloader.Asks Av05: Program:Win32/SpySheriff (threat-c) Av06: Trojan.Gen Av07 : Win32:Trojan-gen. {Other} Av08: Win32/Dewnuttin.B Av09: W32/Tofger.CD Av10: Application/KillApp.A Av11: (2nd pair) TROJ_PROCKILL.DJ Av12: Trojan.Xtssksastsm Av13: (1st pair) Trojan.Win32.Zapchast.cp Av14: (2nd pair) application ProcKill-DJ Av15: Win32/ProcKill.1hj!Trojan Av16: Trojan.Zapchast.CT
![Page 16: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/16.jpg)
Examples
Detecting other products’ false positives:
Av01: Backdoor.X
Av02: FalseAlarm.Av01.Backdoor.X
![Page 17: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/17.jpg)
Feedback from the industry
Automatic sample processing …- is a must given the number of samples received
![Page 18: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/18.jpg)
Feedback from the industry
… and adding detection based on the output of other AVs
- illegal, immoral, plain wrong- bad idea- it’s common practice - it probably started as an attempt to have common
names- there is no other way
![Page 19: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/19.jpg)
Feedback from the industry
Reporting packed files- if they are not malicious we should not detect them- some of the packers should be blacklisted while
others are too widely used so must be allowed- an unfortunate necessity- professional companies do not need to use dodgy
packers
![Page 20: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/20.jpg)
Feedback from the industry
White-listing clean apps instead of black-listing malware
- it’s not possible- does not scale- it’s ok in controlled environments- better and better idea as time passes
![Page 21: Exploiting the Testing System](https://reader035.fdocuments.in/reader035/viewer/2022062319/5575c438d8b42a312a8b4c1e/html5/thumbnails/21.jpg)
The end …
Q&A