HotFuzz Developers Guide

8/13/2019 HotFuzz Developers Guide

1/58

HotFuzzDevelopers guide

Authors:Dusan Domany, Stepan Henek, Peter Kmet, Jan Stanek, Martin Zember


2/58

HotFuzz Developers guide

Special Thanks

We would like to thank our pro e!t leader Daniel "oropila #or his in$aluable !omments andleadership skills that lead this pro e!t to a su!!ess#ul #inish% We would also like to e&press ourne$er'endin( (ratitude to Pa$el Kanko$sky #or his ideas, hints and ad$i!es durin( the whole pro e!tde$elopment phase%

)


3/58


Table of Contents* +ntrodu!tion%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

*%* Purpose o# the Pro e!t%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%*%) -omponents%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%*%. Similar Work%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

"he Pea!h 0u11in( Plat#orm%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%0usil the #u11er%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Sulley%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%3unny the 0u11er%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%SA45%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"A60%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%SP+K5%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Web#u11er%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

S!rat!h%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Man(le%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

) 5mployed "e!hnolo(ies%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%)%* Pea!h%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%)%) Wireshark%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%)%. 8t%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

. Ar!hite!ture%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%.%* Pea!h in the Middle%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Pea!h%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Pitm%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

4 + -ommuni!ator%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Pitm S!hema%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Pro&y < "-P =ersion%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Handlin( Data%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%?e!ordin(@0u11in(%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%0inishin( the +teration%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Monitorin( the Appli!ations%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Pro&y < DP =ersion%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Pitm 0iles 6$er$iew%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%.%) Data Analysis%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Data)p!ap Module%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Data)p!ap in Hot0u11%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Data)p!ap sa(e%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%How to -reate a 0ake DP Pa!ket and Sa$e it to a P!ap 0ile%%%%%%%%%%%%%%%%%%%%%%%%%%%%How to -reate a "-P Pa!ket and Sa$e it to a P!ap 0ile%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%-reate a "-P -onne!tion%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Modi#i!ation and 5&tension o# Data)p!ap%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Module tm e&port%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

+ntrodu!tion%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%"e!hni!al +ssues%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

sa(e%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%5&ample sa(e%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

+nput@6utput Stru!tures%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Modi#i!ation and 5&tension o# tm e&port%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

.


4/58


-allin( Wireshark -ode #rom Python%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%-on$ertin( Wireshark Stru!tures into Pea!h Stru!tures%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

.%. Data Mat!hin(%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

.%9 ?e!orded Data A((re(ation %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

.% Stru!ture o# the Hot0u11 -on#i(uration 0ile%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%9 4 +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Sour!e -ode 4eneration%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Main 0un!tions 6$er$iew%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Appli!ation 0ile 6$er$iew%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

9%* Dialo(s%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%3asi! Dialo( Bo(i!%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Standard 8t Dialo(s%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Main Window Dialo(%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Appli!ation Settin(s%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%+ntro Dialo(%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Pre#eren!es Dialo(%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Cew Pro e!t Dialo(%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%?e!ent Pro e!t Dialo(%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Pro e!t +n#o Dialo(%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%A!tion =iew Dialo(%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

9%) Pro e!ts %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Pro e!t 0iles%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

MB Manipulation%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Dump 0iles%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

9%. Wid(ets%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%6wn Wid(ets%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%8t Desi(ner Plu(in%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

ndo A!tions%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%9%9 5&ternal Pro(rams %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Pea!h -ommuni!ator%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Startin( Pitm%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%Stoppin( Pitm%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

De$elopment o# Hot0u11%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%* History o# the Pro e!t%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%) De$elopment o# the Disse!tion Pro!ess in Hot0u11%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

; Strate(i! De!isions%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

;%* Why Pea!h is sed%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%;%) Why the Wireshark Bibraries are sed%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%> 0uture Work%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

>%* 4 +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% Appendi& A: Stru!ture o# the Hot0u11 -on#i(uration 0ile%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

9


5/58


1 Introduction

1.1 urpose of the ro!ect"he aim o# the Hot0u11 pro e!t is to pro$ide a tool #or dis!o$erin( se!urity $ulnerabilities% +t uses

a widely used approa!h !alled E#u11in(F that is based on pro$idin( in$alid data to a pro(ram so thatit !ould lead to an une&pe!ted beha$iour o# the pro(ram% "he Hot0u11 pro e!t implements a pro&y#un!tionality between network appli!ations% +t modi#ies the data be#ore #orwardin( them to the#u11ed pro(ram% "he bene#it o# the Hot0u11 pro e!t is that it !an automati!ally parse the streams o#data without a detailed user'pro$ided proto!ol spe!i#i!ation and apply the #u11in( rules to theresultin( parsed blo!ks%

1." Co#ponents0i(ure *%* Gon the ne&t pa(e displays relations between Hot0u11 !omponents% Short des!riptions

o# the !omponents are listed below%

Peach in the middle: Pea!h in the middle is an adaptation o# Pea!h, whi!h trans#orms Pea!h into anetwork pro&y and inte(rates most o# the important Hot0u11 ideas% +t !ontains the !entral lo(i! o#the appli!ation and !onne!ts the other Hot0u11 !omponents to(ether%

Proxy: Hot0u11 Pro&y repla!es the !lassi!al Pea!h State Model and handles the !ommuni!ation between the !lient and the ser$er appli!ation% "he pro&y !an run in re!ordin( or in #u11in( mode% +tsupports both "-P and DP !ommuni!ation%

Recorded data aggregation: +t mer(es similar messa(es #rom all the re!orded test !ases into asin(le data model% -reates a !ompa!t data representation o# multiple !ases o# a !lient'ser$er!ommuni!ation%

Configuration file generation: -reates a !on#i(uration #ile #or the #u11in( phase based on the datare!orded durin( the re!ordin( phase and the !on#i(uration done by the user $ia the 4raphi!al ser+nter#a!e%

Data matching: +t identi#ies mutable elements durin( the #u11in( pro!ess% "he mat!hin( is based

on the !omparison o# the a!tual data #orwarded throu(h the pro&y with the models re!orded by theuser durin( the re!ordin( phase%

Customized WindowsDebugEngine monitor: -ustomi1ed $ersion o# the Pea!h debu( monitor%"he !lass was modi#ied to pro$ide the #un!tionality reIuired by Hot0u11%

Custom Process monitor: -ustom Hot0u11 monitor #or handlin( basi! pro!ess manipulation% +n!omparison with the ori(inal Pea!h pro!ess monitor it !an handle mu!h more !ompli!ated

pro!esses% "he monitor was written #rom s!rat!h to satis#y Ho#0u11 needs%


6/58


;

Figure 1.1: Relations among the HotFuzz components


7/58


Netstat based port scanning: "he port s!anner retrie$es in#ormation about !urrently opened portson the lo!al ma!hine% +nstead o# blindly tryin( to !onne!t to !ertain ports, it uses Windows #un!tionsto retrie$e the in#ormation% "he !omponent works only on ma!hines runnin( Mi!roso#t Windows%

Custom publisher: "he !omponent stores $alues that are important #or the !ommuni!ation with!lient and ser$er%

Custom Random uzzing strategy: "he Hot0u11 ?andom Mutation strate(y needed to be written#rom s!rat!h to satis#y Hot0u11 needs% 6ne o# its #eatures is that it !an be pro$ided with a randomseed to a!hie$e a pseudo'random beha$iour% "he strate(y meets basi! reIuirements on Pea!hmutation strate(ies%

!"# Communicator: "he !omponent !ontains a lo(i! #or re!ei$in( instru!tions #rom the (raphi!aluser inter#a!e and sendin( ba!k an in#ormation about the !urrent state o# the re!ordin(@#u11in(

pro!ess% "he !ommuni!ation runs in parallel with the re!ordin(@#u11in( pro!ess in a separate thread% Data analysis: Data analysis is used to analyse the data passin( throu(h Hot0u11% "he input to thedata analysis pro!ess is the data !omin( #rom the !lient appli!ation to the ser$er appli!ation and inthe other dire!tion too% "he output o# the data analysis pro!ess is a data model a tree'like stru!turewhere e$ery node represents a part o# data with a spe!i#i! meanin( and a des!ripti$e name Gwhere

possible %

Data recei$e: "he data are re!ei$ed #rom a so!ket and stored to a bytestrin(% "he bytestrin( is then pro$ided to the pa!ket re!onstru!tion pro!ess%

Pac%et reconstruction: ?aw data re!ei$ed #rom a so!ket are wrapped up in a #ake pa!ket% "he #ake pa!ket !onsist o# a p!ap (lobal header, p!ap pa!ket header, 5thernet header, +P header and "-P or

DP header%

Pac%ets dissection: Pa!ket disse!tion is a pro!ess o# detailed pa!ket analysis% "he pa!ket isse(mented to the smallest parts o# data with its own meanin(% "hese parts are also or(ani1ed to

blo!ks with its own meanin( Gwhere possible %

&dditional data analysis: Additional data analysis !onsists o# a #ew pro!esses that are applied tothe results o# the pa!ket disse!tion% "hese are mainly !he!ks o# data types and !onsisten!y%

'ransforming C(structures into Python(structures: "his !omponent is desi(ned to !on$ert allne!essary - stru!tures into Python stru!tures% +t also makes the !allin( o# shared library #un!tionswritten in - possible within Python sour!e !ode%

Peach structures creation: "he !omponent !reates proper Pea!h stru!tures based on the pro$idedPython stru!tures% Due to the di##eren!es between the Wireshark stru!tures and the Pea!h stru!tures,$ariety o# thin(s need to be !orre!ted in!ludin( data types and ali(nment%

)trings to%enization: "he !omponent per#orms additional tokeni1ation o# strin( elements tomaintain (reater (ranularity o# the disse!ted data% "he tokeni1ation uses set o# separators and

bra!kets to split the strin(s into smaller parts% 0u11in( o# spe!i#i! ?B parameters would not be

possible without this !omponent%

>


8/58


inding relations: A relation #indin( pro!ess tries to #ind relations between di##erent parts o# data!ontained within one messa(e% We ha$e implemented si1e relations so #ar Gwhen a data se(mentrepresents the si1e o# another data se(ment, et!% %

!raphical "ser #nterface: "he 4 + !omponent is desi(ned to make use o# the Hot0u11 pro(ramas easy and e##e!ti$e as possible% +t !onsists o# !omponents listed below%

Dialogs: "he 4raphi!al ser +nter#a!e !ontains many dialo(s whi!h handle intera!ti$e!ommuni!ation with the user% "hese dialo(s pro$ide the user with $ariety o# !ontrols #or easymana(ement o# the appli!ation%

*ain window: "his window is the most important !omponent o# our 4 +% +t opens all otherdialo(s and pro!esses si(nals #rom numerous types o# e$ents% Moreo$er, all e&ternal pro!esses arestarted within this dialo(%

Process handling: 6ne o# the main purposes o# the 4 + is to mana(e the Pea!h in the middle pro!ess, whi!h has to !arry out the re!ordin( and the #u11in( phases% "here are also other e&ternal pro(rams, whi!h are started #rom the 4 +: a te&t editor and a debu((er%

)toring application settings: Whene$er the Main Window is startin(, it loads appli!ation settin(sand modi#ies its internal $ariables a!!ordin( to these settin(s% Whene$er the Main Window is!losin(, it prompts whether to store unsa$ed !han(es to the pro e!t and then it stores its internal$ariables to the appli!ation settin(s%

+*, manipulators: "he 4 + appli!ation is !apable o# manipulatin( Gstorin( and loadin( threetypes o# MB #iles < fuzzing.xml, recording.xml and project.xml % "he #u11in( and the re!ordin(

MB are passed to Pea!h in the middle as one o# its parameters% "he pro e!t MB is used to storeo$erall pro e!t in#ormation%

-iewing crash details: When Pea!h in the middle is started in the #u11in( mode a new dire!tory inproject dir/dumps/ appears% "his dire!tory is !reated by Pea!h and the 4 + appli!ation is used todisplay its !ontents%

1.$ Si#ilar %ork"he +" se!urity !ommunity was asked * to su((est a topi! #or the pro e!t% We were re!ommended

to enhan!e some o# the e&istin( tools instead o# buildin( yet another #u11er or a #u11in( #ramework%"here#ore we de!ided to implement e&tend an e&istin( #u11er so that it !ould be used also as a#u11in( pro&y% "here are some !ommer!ial #u11ers whi!h implement this #eature, but none o# theopen'sour!e #u11ers #ully inte(rates this #un!tionality%

Di##erent #u11ers and the #u11in( #rameworks were tested by the de$elopment team at the earlysta(e o# the pro e!t% Here is the list o# the e&amined #u11ers%

* Dailyda$e mailin('list: http:@@lists%immunityse!%!om@mailman@listin#o@dailyda$e0u11in( mailin('list: http:@@www%whitestar%linu&bo&%or(@pipermail@#u11in(@

/
http://lists.immunitysec.com/mailman/listinfo/dailydavehttp://www.whitestar.linuxbox.org/pipermail/fuzzing/http://www.whitestar.linuxbox.org/pipermail/fuzzing/http://lists.immunitysec.com/mailman/listinfo/dailydave


9/58


The each Fuzzing latfor#

Pea!h is a #ramework #or !reatin( #u11ers% +t is written in Python and it is easily e&tendible% Wede!ided to use this pro(ram as the one o# the main parts o# the Hot0u11 pro e!t% Pea!h is !urrently

bein( de$eloped and supported by the main author GMi!hael 5ddin(ton % Mi!hael !ame up withmany ideas that we used when desi(nin( the initial stru!ture o# our pro e!t% A detailed des!ription o#the Pea!h 0u11in( Plat#orm is lo!ated in the other part o# this do!ument%

Fusil the fuzzer

0usil the #u11er is an open'sour!e #u11in( #ramework written in Python% +t was su!!ess#ully used #or#indin( se!urity'related bu(s in so#tware like PHP, (lib!, libe&i#, -lamA=% Many bu(s were #oundin Python itsel# Gsee http:@@bitbu!ket%or(@haypo@#usil@wiki@Python%

Sulle&Sulley is similar to Pea!h in the way that it has the abilities not only to (enerate data, but also tomonitor the network, the tar(et pro(ram, to re$ert the tar(et pro(ram state i# needed and to tra!kand !ate(ori1e dete!ted #aults%

+t has an installation tool that !an install all dependen!ies% We were inspired by this #eaturedurin( the !reation o# our Hot0u11 installer%

'unn& the Fuzzer

3unny the 0u11er is a EsmartF #u11er that uses a loop #rom the tra!ed pro(ram% "his #eature pro$ides the #u11er with run'time #eedba!k on how to alter the inputs to in!rease the !ode !o$era(e%

S()*

SA45 is a proprietary tool% +t mana(ed to !rash an appli!ation shipped as part o# 6##i!e )77> 9.times G#tp:@@#tp%resear!h%mi!roso#t%!om@pub@tr@"?')77>' /%pd# %

T(+F

"A60 G"he art o# #u11in( is a #u11in( #ramework that is easy to use e$en #or people without pro(rammin( skills and it is probably a (ood start #or those that are new in the #ield o# #u11in(% "heo##i!ial webpa(e pro$ides some $ideos on how to dis!o$er $ulnerabilities usin( "A60%

S I,*

SP+K5 is an AP+ that enables a pro(rammer to spe!i#y a network proto!ol% "he #u11ed data are(enerated a!!ordin( to this spe!i#i!ation% +t was !reated in )77) and the lan(ua(e used is -%

2
http://bitbucket.org/haypo/fusil/wiki/Pythonhttp://bitbucket.org/haypo/fusil/wiki/Python


10/58


%ebfuzzer

Web#u11er does not pro$ide any user'#riendly inter#a!e, it is #o!used on ad$an!ed users who !anmodi#y the sour!e !ode a!!ordin( to their reIuirements% +t does not pro$ide almost anydo!umentation and the sour!e !ode itsel# is rather poorly !ommented% +t has not been updated #oryears%

http:@@(un1ip%alter$ista%or(@(%php #Lpro e!ts%

Scratch

S!rat!h is another #u11er that needs to modi#y the Python sour!e !ode in order to !on#i(ure it% +tdoes not pro$ide any do!umentation at all e&!ept a #ew !omments in the !ode% +t is probably notde$eloped any more%

http:@@pa!ketstormse!urity%or(@ C+ @mis!@s!rat!h%rar

-angle

Man(le has less than 77 lines o# !ode% +t !reates a -4+ s!ript #rom a - sour!e !ode whi!h #u11esH"MB synta& and tries to !rash the $iewin( browser%

http:@@l!amtu#%!oredump%!&@so#t@man(leme%t(1

*7
http://gunzip.altervista.org/g.php?f=projectshttp://gunzip.altervista.org/g.php?f=projectshttp://packetstormsecurity.org/UNIX/misc/scratch.rarhttp://lcamtuf.coredump.cx/soft/mangleme.tgzhttp://gunzip.altervista.org/g.php?f=projectshttp://packetstormsecurity.org/UNIX/misc/scratch.rarhttp://lcamtuf.coredump.cx/soft/mangleme.tgz


11/58


" *#plo&ed Technologies

".1 eachPea!h is a #ramework #or buildin( #u11ers% +t has been de$eloped sin!e )779 by Mi!hael

5ddin(ton% Pea!h was released under M+" li!ense, so any person is allowed to obtain a !opy o#Pea!h and deal in the So#tware without restri!tion%

Pea!h is basi!ally bein( !reated by a sin(le author and it is still under a!ti$e de$elopment% Mr%5ddin(ton or(ani1es workshops #or parti!ipants, who represent somethin( like a small Pea!h!ommunity% Do!umentation o# Pea!h !an be #ound on http:@@pea!h#u11er%!om@% Many parts o# thedo!umentation are in!omplete or missin(% Many parts are also out o# date% Howe$er, it is probablythe best pla!e to start (ettin( #amiliar with Pea!h%

Another (reat sour!e o# in#ormation is the Pea!h mailin('list at pea!h#u11 (oo(le(roups%!om%Pea!h users ask many Iuestions there and Mr% 5ddin(ton a!ti$ely mana(es this mailin('list,answers the Iuestions and (i$es ad$i!es on how to make Pea!h runnin(, deals with di##erent issues,et!% Some o# the workshop students started to parti!ipate on the mailin('list mana(ement re!ently,too% More e&perien!ed users send bu( #i&es and proposals #or new #eatures% Some o# them are bein(inte(rated into up!omin( $ersions o# Pea!h% 6ne o# the inte(rated bu( #i&es was also a !orre!tion o#the Pea!h A(ent beha$iour proposed by the Hot0u11 team%

"." %iresharkWireshark is a multiplat#orm pa!ket analyser% +t is the su!!essor o# 5thereal and has Iuite a lon(

de$elopment history be(innin( in the *227s% Althou(h it is de#initely not the only pa!ket analysere&istin( nowadays, we !an say that it is surely the most widely used one% "he bi((est ad$anta(es o#Wireshark are that it is #ree and open sour!e%

Pa!ket analysis is a pro!ess in whi!h the network !ommuni!ation is !aptured and its partsGpa!kets are inspe!ted and se(mented into smaller parts with a de#ined meanin(% Pa!ket analysersare automated tools that pro$ide the !ommuni!ation !apturin( and pa!ket analysis #un!tionality,o#ten o##erin( somethin( e&tra like statisti!s, (raphs et!% More in#ormation !an be #ound athttp:@@en%wikipedia%or(@wiki@Pa!ket analy1er % We !an also point you to the user manual part o# theHot0u11 do!umentation whi!h !ontains the des!ription o# some basi! terms !onne!ted with this

problemati! in the 3asi! !on!epts se!tion%

We do not wish to waste time and spa!e by repeatin( thin(s that are a$ailable in numerous pla!eson the +nternet so i# you are interested in usin( Wireshark, you should de#initely (o to its homepa(eat www%wireshark%or(% +# you only need to know some basi! in#ormation about Wireshark were!ommend you to pay a $isit to its Wikipedia pa(e at http:@@en%wikipedia%or(@wiki@Wireshark %

+# you plan to adapt Wireshark or its libraries #or your own pro e!t we hi(hly re!ommend you to oin the Wireshark de$elopment !ommunity and use one o# the numerous mailin( lists a$ailable athttp:@@www%wireshark%or(@lists@% "his !an sa$e you Iuite some time and ner$es%

**
http://peachfuzzer.com/http://en.wikipedia.org/wiki/Packet_analyzerhttp://en.wikipedia.org/wiki/Wiresharkhttp://www.wireshark.org/lists/http://en.wikipedia.org/wiki/Packet_analyzerhttp://peachfuzzer.com/http://en.wikipedia.org/wiki/Wiresharkhttp://www.wireshark.org/lists/


12/58


".$ t"he 4 + uses the Py8t9 library, whi!h pro$ides bindin(s #or the CokiaNs 8t appli!ation

#ramework% "his makes the 4 + appli!ation $ery portable, be!ause the Python interpreter and

Py8t9 are ported to the $ast ma ority o# nowadays operatin( systems%"o !reate the !ode more e##e!ti$ely, we de!ided to use the 8t Desi(ner to desi(n a skeleton o#dialo(s%

*)


13/58


$ (rchitecture

$.1 each in the -iddle

each

"o understand the each in the middle , it is #irst needed to understand how Pea!h works% "hePea!h e&e!ution pro!ess !an be brie#ly des!ribed as #ollows:

"he #irst step is the initiali1ation o# Pea!h, whi!h !onsists o# parsin( o# !ommand line optionsand !he!ks whether the Pea!h dependen!ies are properly installed%

"he Pea!h 5n(ine is then started% +t uses the Pea!h arser to parse the input !on#i(uration #ileand to !reate the appropriate !omponents% A#ter !reation o# all the ne!essary !omponents and

per#ormin( some additional initiali1ation, the Pea!h 5n(ine enters the main pro!ess loop, in whi!hthe test !ases are e&e!uted%

Durin( ea!h o# the test !ases, Pea!h runs a !tate "achine % "he State Ma!hine is a deterministi!#inite state ma!hine !ontainin( states based on the users !on#i(uration% 6ne o# these States needs to

be #la((ed as initial% 5a!h o# these States !onsists o# one or more A!tions% When the State Ma!hineenters a State, it runs seIuentially all o# its A!tions% 0or ea!h A!tion the user !an also spe!i#y

!ir!umstan!es under whi!h the A!tion should be per#ormed% Pea!h has a #i&ed list o# a$ailable

*.

Figure #.1: each $rchitecture


14/58


A!tion types, whi!h in!lude !onne!tin( to a remote host G!onne!t , a!!eptin( !onne!tion Ga!!ept ,sendin( data Goutput , re!ei$in( data Ginput , !allin( spe!i#i! Python method G!all , !han(in( stateG!han(eState et!% +# all o# the A!tions #rom a sin(le State are e&e!uted without !han(in( a State, thee&e!ution o# the State Ma!hine is ended%

5a!h output A!tion needs to ha$e a template% "his template is !alled %ata "odel and representsthe stru!ture o# the messa(e bein( sent% Whene$er Pea!h runs an output A!tion, it per#orms amutation on the spe!i#ied template, !on!atenates all the $alues #rom the template and uses the resulto# the !on!atenation as an output%

"he mutation is per#ormed by a spe!ial ob e!t !alled "utation !trateg& , whi!h uses its internallo(i! to sele!t elements #rom the Data Model and applies "utators to these elements% Mutators

pro$ide $alues #or the elements and these $alues are then used instead o# the ori(inal ones%Durin( the whole pro!ess Pea!h intera!ts with a Pea!h A(ent to maintain !ontrol o$er the tested

appli!ation and to re!ei$e in#ormation about the appli!ations !urrent state% "he user needs to spe!i#ya Monitor whi!h is used by the A(ent to handle Gstart@stop and monitor the tested appli!ation% A#terea!h iteration Pea!h reIuests the A(ent to dete!t whether any #ault o!!urred% A typi!al e&ample o# a#ault is an appli!ation !rash% +# Pea!h re!ei$es a positi$e answer Ge%(% the appli!ation !rashed , itreIuests the A(ent to send any in#ormation related to the #ault that are a$ailable% "o satis#y thisPea!h reIuest, the Monitor needs to implement a spe!i#i! method, whi!h is used to retrie$e thereIuested data% A typi!al e&ample o# su!h data is a WinDb( dump file % "his #ile is (enerated by theWindows Debu( 5n(ine a#ter an appli!ation !rash, i# the appli!ation ran with Windows Debu((eratta!hed%

"he last ob e!t in the Pea!h Ar!hite!ture s!hema is !alled Pea!h stru!tures% "hese stru!tures arede#ined in the #ile dom.py and in!lude ob e!ts like Pea!h Strin(, Pea!h Cumber et!%, whi!h !omewith number o# use#ul methods and make the Pea!h !ode mu!h more !lear%

it#Pea!h in the middle Gshortly itm !han(es the ori(inal Pea!h e&e!ution pro!ess in an early sta(e

o# the Pea!h State Ma!hine run% +# !ertain options are spe!i#ied on the !ommand line, then thee&e!ution o# the Pea!h State Ma!hine !ode is inter!epted and Hot0u11 !ode is used instead%

As the matter o# #a!t, the !han(es were done $ery !are#ully, so Pitm !an still be used as anori(inal Pea!h and only i# the user spe!i#ies !ertain options, the Hot0u11 #un!tionality is a!ti$ated%We took !are to limit modi#i!ations in the Pea!h !ode and to keep them lo!ali1ed, be!ause Pea!h asa so#tware is still e$ol$in( and we wanted to be able to easily mi(rate the Hot0u11 #un!tionality tothe newer $ersions o# Pea!h i# needed%

)/I Co##unicator

6ne o# the important enhan!ements introdu!ed in Hot0u11 is that the whole pro!ess !an be!ontrolled $ia a 4raphi!al ser +nter#a!e% We de!ided not to inte(rate the 4raphi!al ser +nter#a!edire!tly, be!ause that would reIuire a lot o# modi#i!ations to the ori(inal Pea!h !ode, whi!h wewanted to pre$ent% +nstead, the 4 + runs as an independent pro!ess and intera!ti$ely !ommuni!ateswith the Pitm pro!ess $ia a lo!al port% "he Pitm pro!ess runs a separate thread, whi!h handles theintera!ti$e !ommuni!ation with the 4 +%

"he thread re!ei$es DP pa!kets on the spe!i#ied port% ?e!ei$ed messa(es are e&pe!ted to be.)'bit inte(ers en!oded into he&'#ormat, so the len(th o# the messa(es is e&pe!ted to be e&a!tly /

bytes% "he messa(es are de!oded and the #ollowin( $alues are re!o(ni1ed as instru!tions:

*9


15/58


E)F: Store the address #rom whi!h the pa!ket was sent and periodi!ally send number o# the!urrent iteration to this address% +# a #ault was identi#ied in the last iteration, then send theiteration number as a ne(ati$e inte(er

E.F: "erminate the entire pro!ess GreIuested when the button !top or the button !top $ll is pressed

E9F: Pause the main Pitm thread GreIuested when the button ause is pressed E F: npause the main Pitm thread GreIuested when the button 'npause is pressed %

=alue E*F was reser$ed #or instru!tin( a sin(le A(ent to stop runnin( its appli!ation and is not!urrently used% "he intera!tion with the main Pitm thread is handled usin( the shared memory% Parto# the #un!tionality is implemented in the method PpAction.guiCommunicator() , whi!h themain Pitm thread runs at the be(innin( o# ea!h iteration% "he #ollowin( a!tions are per#ormed whena termination is reIuested:

the main Pitm thread is interrupted to pre$ent ra!e !onditions the A(ents are instru!ted to terminate any appli!ation that they are !urrently runnin( and toswit!h to a passi$e mode

the 4 + is in#ormed that the termination was su!!ess#ul the whole pro!ess e&its

"he A(ents need to be !onta!ted remotely as they !an run on di##erent ma!hines% +# the A(entsare not respondin(, the whole termination pro!ess mi(ht take lon(er%

*


16/58


it# Sche#a

Cow let us take a look at a s!hema o# the main Pitm thread% +n a simple way, the s!hema !an bedes!ribed as #ollows:

"he Hot0u11 initiali1ation is basi!ally an e&tended Pea!h initiali1ation% Additional !ommandlineoptions are parsed here, whi!h !an be used to a!ti$ate the Pitm #un!tionality and pass some related$alues% Some o# the important Hot0u11 stru!tures are also !reated here and the thread that handlesthe !ommuni!ation with the 4raphi!al ser +nter#a!e is also !reated and started at this point%

"he e&e!ution pro!ess then !ontinues by startin( the ordinary Pea!h 5n(ine and per#orms all thestandard Pea!h operations until it starts e&e!utin( the Pea!h State Ma!hine% "he ori(inal Pea!h

parser was written in a $ery #le&ible way and we were there#ore able to desi(n the stru!ture o# theHot0u11 !on#i(uration #ile, so it !ould be pro!essed by only sli(htly modi#ied $ersion o# the parser%

"he HotFuzz rox& (ngine is started at the point, where the Pea!h State Ma!hine wouldnormally e&e!ute% +n !ase that the user spe!i#ies !ertain options, the standard Pea!h !ode isinter!epted and the Hot0u11 !ode is used instead% +# the user spe!i#ies an option --hotrec on the!ommandline, the Hot0u11 Pro&y 5n(ine starts in a re!ordin( mode% +# the user spe!i#ies an option--hotfuzz , the 5n(ine starts in a #u11in( mode% 6therwise the e&e!ution o# the standard Pea!h StateMa!hine is per#ormed%

"he Hot0u11 Pro&y 5n(ine uses so!ket select() to dynami!ally handle the !ommuni!ationwith the !lient and the ser$er appli!ation% +t plays an important role in our appli!ation and wentthrou(h multiple re!onstru!tions to meet all the reIuirements related to the Hot0u11 ideas Gsome o#

them were #i(ured out durin( the de$elopment pro!ess % "he pro&y needs to be able to

*;

Figure #.): HotFuzz architecture


17/58


simultaneously handle the !ommuni!ation with the !lient and the ser$er% At the same time, it needsto ha$e a (ood !ontrol o$er the !ommuni!ation, make sure that some messa(es are not sent tooearly et!% "he pro&y needs to know when it is supposed to read data and when it is allowed to sendthem% +n short the #eatures !an be des!ribed as #ollows:

Storin( re!ei$ed messa(es ?eIuestin( a real'time data analysis and a real'time #u11in( o$er the stored messa(es Handlin( the situations when the data analysis reIuires more data be#ore it !an return the

result Propa(atin( !onne!tion !lose and dealin( with network errors 0inishin( the !ommuni!ation when the end o# the iteration is identi#ied

"here are di##erent instan!es #or "-P and DP, be!ause these two proto!ol #amilies reIuire asli(htly di##erent approa!h% Bet us take a look at the "-P $ersion #irst%

ro0& TC 2ersion

"he pro&y #irst binds a so!ket #or a !ommuni!ation with the !lient and then reIuests !lient A(entto run the $cti*ating +ommand Gor start the !lient appli!ation in !ase o# !tart n+all % So when the!lient tries to !onne!t, the pro&y is already listenin( on the spe!i#ied port% "he pro&y then tries to!onne!t to the ser$er appli!ation% 3e!ause the ser$er appli!ation mi(ht not be ready yet, the pro&ymakes .7 attempts durin( .7 se!onds be#ore it (i$es up%

Ce&t, the pro&y enters a readin( phase% Durin( this phase it tries to read as mu!h data as possiblein a $ery short time% "his is the #irst line o# de#en!e a(ainst an issue that we !alled se(mentation%"he se(mentation means that only part o# the sent data were already re!ei$ed, so the data analysismi(ht not be able to #ully analyse the re!ei$ed messa(e% "here is no (uaranteed way to prote!ta(ainst this issue% "he se(mentation mi(ht ha$e many #orms% "he messa(e Gpa!ket !an be split inthe headers part, in the body part, it !an be missin( only a #ew last !hara!ters, or the re!ei$ed part o#the messa(e mi(ht be so short, that it is impossible to tell what it is% "he disse!tion pro!ess mi(ht bethere#ore !on#used by the data it was supplied with% We brou(ht multiple te!hniIues into ourappli!ation to deal with the issue and the !urrent state is !onsidered to be $ery stable% Ma ority o#the te!hniIues #orms part o# the %ata anal&sis modules%

+# the pro&y does not re!ei$e any data in 7%7* se!ond, it swit!hes to a sendin( phase% Durin( this phase the pro&y #irst reIuests the data analysis o$er the stored messa(es% +# the data analysis doesnot return any result, it means that the data it was supplied with were probably in!omplete% +n that!ase the pro&y !ontinues readin( messa(es and passin( them to the data analysis until the dataanalysis returns a $alid result%

Handling Data

Cow let us take a !loser look at this step, be!ause there is more (oin( on here under the hood%"he data analysis need to be supplied with some in#ormation re(ardin( the data% "his in#ormationin!ludes the proto!ol #amily and the proto!ol standard port and is used to !reate proper #ake pa!ketsin the data2pcap module% "he #ake pa!ket is analysed by the tm_export module usin( Wiresharkdisse!tion libraries and the result is then translated into Python stru!tures% 3ut it is not #ully

prepared to be used in the #u11in( yet%"he #irst thin( is that the $alues o# the elements are en!oded into a he&'#ormat% "he reason #or

that is that otherwise they !ould !ontain a O \0N!hara!ter, whi!h would !ause, that the $alues would

*>


18/58


be!ome shortened in the pro!ess o# the stru!tures translation% "he $alues need to be there#orede!oded at this point% Another thin( is that durin( the disse!tion pro!ess, some not important byteso# the supplied data mi(ht be skipped, whi!h makes it impossible to a!!urately re!onstru!t theori(inal messa(e% "hese missin( data need to be #illed ba!k in at the !orre!t positions%

"he stru!tures need to be translated into proper Pea!h stru!tures ne&t% A deep understandin( o#these stru!tures was ne!essary when implementin( this part, be!ause some o# them ha$e di##erent

beha$iour than the others% n#ortunately, it is not possible to #ully rely on the data types identi#ieddurin( the disse!tion pro!ess% 0or e&ample a $alue o# a E +ontent-length header is a strin( that!ontains also a number, but the whole $alue is identi#ied as a number% "hou(h this mi(ht makesense in some !onte&t Glike in the Wireshark 4 + , it is not desired when !reatin( the Pea!hstru!tures% "he data types need to be !orre!ted at this point%

As the last thin( we apply somethin( that we !all heuristi!s% "wo heuristi!s are !urrentlyimplemented% "he #irst one is a strin( tokeni1ation% "he $alues o# the elements that were identi#iedas strin(s are tokeni1ed based on a de#ined set o# separators% +t works simultaneously in two ways:

Separation by bra!kets: works re!ursi$ely and sear!hes #or spe!i#i! types o# bra!kets in thestrin(

Separation by sin(le tokens: (oes throu(h the strin(, looks #or spe!i#i! !hara!ters andseparates the strin( in pla!es where these !hara!ters are #ound

"he reason #or this heuristi! is that #or e&ample the ?B #rom a H""P reIuest !omes #rom thedisse!tion pro!ess as a sin(le strin( element% "o be able to mutate parts o# this element, like the

?B parameters, it needs to be split into smaller parts%"he se!ond heuristi! sear!hes #or relations between di##erent elements in the stru!ture% -urrently

it sear!hes only #or si1e relations% +t identi#ies that a $alue o# one element is a si1e o# anotherelement% Bater, durin( the pro!ess o# #u11in(, i# the $alue o# the se!ond element is mutated and its

si1e is !han(ed, the $alue o# the #irst element is a!!ordin(ly modi#ied as well to make the data lookauthenti!%As a result we #inally (et a #ully Iuali#ied Pea!h Data Model and we !reate a Pea!h A!tion that

!ontains this Data Model% +n Hot0u11 we use Pea!h A!tions and Pea!h States only as pa!ka(es #orour data%

3ecording4Fuzzing

+# we run in a re!ordin( mode, the A!tion is stored and the data are #orwarded un!han(ed% +# werun in a #u11in( mode, then the %ata "atching module is reIuested to #ind a mat!hin( A!tionamon( the pre$iously re!orded data% +# the mat!hin( A!tion is #ound, then its !on#i(uration is!opied and a mutation is per#ormed on the Data Model o# the !urrent A!tion% 6therwise the data are#orwarded un!han(ed%

"he mutation is per#ormed by the HotFuzz "utation !trateg& % "he strate(y identi#ies mutableelements and randomly sele!ts a subset o# them% +t then identi#ies the Pea!h Mutators that areappli!able to the elements in the subset, randomly !hooses one Mutator #or ea!h element and

per#orms the mutation% +t is possible to pro$ide the strate(y with a seed to make it beha$e as pseudo'random%

A#ter the mutation, the $alues in the Data Model are !on!atenated into a sin(le strin( and sentinstead o# the ori(inally re!ei$ed data%

*/


19/58


Finishing the Iteration

When the pro&y has no more messa(es to send, it swit!hes ba!k to the readin( phase% "he pro&yswit!hes between the readin( and the sendin( phase until one o# the #ollowin( e$ents o!!urs:

"he pro&y has no messa(es to send and no data were re!ei$ed in a number o# se!ondsspe!i#ied by the user as an +teration timeout%

"he %ata matching module identi#ies the end o# the iteration% "his o!!urs only in the#u11in( phase%

"he end o# the iteration !an be identi#ied usin( one o# three methods Gwhi!h one is used isde!ided in the %ata $ggregation pro!ess durin( the re!ordin( phase :

0inish when the Data Model o# the re!orded A!tion with the #la( terminateTestCase ismat!hed

0inish when the ser$er !loses !onne!tion 0inish when a !ount o# the output a!tions rea!hes the !ount o# the re!orded output a!tions

and the !ount o# the input a!tions rea!hes the !ount o# the re!orded input a!tions

When the iteration is ended, the !onne!tions are !losed and a partial !lean up is per#ormed% +# werun in a re!ordin( mode and the number o# the !urrent iteration rea!hes the number o# iterations to

be re!orded, the %ata $ggregation is reIuested to a((re(ate the re!orded data and the result is usedto !reate a !on#i(uration #ile #or the #u11in( phase%

-onitoring the (pplications

Durin( the pro!ess itm intera!ts with two Pea!h A(ents < one #or the !lient appli!ation and one

#or the ser$er appli!ation% "he A(ents are the same as the ori(inal Pea!h A(ents, but the monitorsare di##erent% Hot0u11 implements two !ustom monitors #or !ontrollin( the appli!ations andreportin( #aults%

.ot uzz Process *onitor : -ustom Hot0u11 monitor #or handlin( basi! pro!essmanipulation% -ompared to the ori(inal Pea!h pro!ess monitor it !an handle mu!h more!ompli!ated pro!esses by usin( Windows Jobs% "his monitor does not report anyin#ormation about the dete!ted !rashes% "he moti$ation #or writin( this monitor was Mo1illa0ire#o&, whi!h !ould not be handled by the ori(inal Pea!h pro!ess monitor%

.ot uzz Debug *onitor : -ustomi1ed $ersion o# the Pea!h debu( monitor% +t was modi#iedto pro$ide the #un!tionality reIuired by Hot0u11% "his monitor uses the Windows Debu(5n(ine to report in#ormation about the dete!ted !rashes% When a monitored appli!ation

!rashes, the e$ent is identi#ied and reported by the Hot0u11 Debu( Monitor as a #ault andWinDb( dump is sent to itm , whi!h lo(s the #ile to(ether with messa(es that were sentdurin( the !urrent iteration

3oth monitors implement the #ollowin( #eatures that are typi!al #or Hot0u11: $cti*ation command < +t is possible to run the !lient appli!ation persistently and spe!i#y a

!ommand that !auses the appli!ation to make a reIuest% Running on port < +t is possible to spe!i#y a port that is opened by the appli!ation when it

starts% "his is use#ul when the appli!ation takes lon( to start% "he Pitm pro!ess is de#erreduntil the port is opened Gbut #or the ma&imum o# *7 se!onds % "his is espe!ially use#ul #orthe DP $ersion o# the pro&y%

%etecting CLOSE_WAIT ports < we ha$e obser$ed that the -esar0"P appli!ation tends to

*2


20/58


(et stu!k a#ter it re!ei$es a !ertain number o# reIuests !ontainin( the O 7N !hara!ter% "his!hara!ter is a $ery !ommon part o# the $alues (enerated by the Pea!h Mutators% "heappli!ation stops a!!eptin( any !onne!tions, so it is not possible to e##e!ti$ely !ontinue inthe #u11in( pro!ess% We were not able to #i(ure out the e&a!t !ause, but we #ound out that

the e$ent went hand in hand with an in!reased number o# ports in a CLOS _!"#$ state% We based our heuristi! upon this obser$ation and whene$er the number o# CLOS _!"#$ portsin!reases and the ser$er port is amon( them, it is reported as a #ault and the s!enario isrestarted, so the #u11in( pro!ess !an !ontinue% n#ortunately we are not !urrently able to

pro$ide any in#ormation about the #ault as it is not !onsidered to be a !rash%

"he O 7N !hara!ter seams to be hard to handle in (eneral% 0or e&ample when 3ad3lue appli!ationre!ei$es a reIuest !ontainin( this !hara!ter, it does not reply in any way% "he !ommuni!ation doesnot !ontinue and it is not possible to e##e!ti$ely identi#y the end o# the iteration Gonly by thetimeout %

"he se!ond and the third des!ribed #eature both use somethin( that we !alled etstat ased port scanner % "he !omponent does not try to !onne!t to any ports, but uses Windows AP+ to retrie$e$arious in#ormation about !urrently opened ports on the lo!al ma!hine%

ro0& /D 2ersion

"he DP $ersion o# the pro&y runs the same Hot0u11 A!tions G#or re!ordin( and #or #u11in( asthe "-P $ersion, but the pro&y itsel# naturally works sli(htly di##erent% +t re!ei$es and sends datawithout establishin( any !onne!tions% +t a!ti$ates the !lient only a#ter it binds the so!ket andswit!hes between readin( and sendin( phase ust like the "-P $ersion% 3ut there is no (uaranteedway how to make sure that the ser$er appli!ation is already runnin( and ready to re!ei$e data% "he

Running on port #eature o# the ser$er monitor there#ore plays an important role here%

it# Files +vervie5 peach.p& < -ontains all the ne!essary initiali1ations be#ore startin( the main 5n(ine% "hese

initiali1ations in!lude !he!ks whether the ne!essary dependen!ies are installed, !ommandline options parsin( et!% Hot0u11 adds a !ouple o# new !ommand line options and doessome initiali1ation o# its own%

each/(ngine/state.p& < -ontains Pea!h methods #or mana(in( the Pea!h State Ma!hineand its a!tions% Method !tate(ngine.0run!tate 2 was modi#ied so that when spe!i#i!

$ariables are set, the Hot0u11 !ode is e&e!uted instead o# the standard Pea!h !ode% each/(ngine/ppaction.p& < -ontains the Hot0u11 Pro&y 5n(ine% each/$gent/ppprocess.p& < -ontains the Hot0u11 Pro!ess Monitor% each/$gent/ppmonitor.p& < -ontains the Hot0u11 Debu( Monitor% each/$nal&zers/ppto3enizer.p& < -ontains methods #or strin( tokeni1ation and methods #or

!reation o# Pea!h stru!tures% each/$nal&zers/pprelations.p& < -ontains methods #or #indin( relations amon( di##erent

elements in a Data Model% each/ u lishers/pppu lisher.p& < -ontains a !lass #or storin( $alues that are important #or

the !ommuni!ation with the !lient and the ser$er% each/"utate!trategies/ppstrategies.p& < -ontains Hot0u11 Mutation Strate(ies% each/(ngine/pp4ui+ommunicator.p& < -ontains the lo(i! #or re!ei$in( instru!tions #rom

)7


21/58


the 4raphi!al ser +nter#a!e and sendin( ba!k the in#ormation about the !urrent state o# thePitm pro!ess%

$." Data (nal&sisData analysis is one o# the !ore !omponents o# Hot0u11% +t is used to analyse the data passin(

throu(h Hot0u11% "he input to the data analysis pro!ess are the data !omin( #rom the !lientappli!ation to the ser$er appli!ation and in the other dire!tion too% "he output o# the data analysis

pro!ess is a datamodel a tree'like stru!ture where e$ery node represents a part o# data withspe!i#i! meanin( and a des!ripti$e name Gwhere possible %

"he o$erall ar!hite!ture o# the whole data analysis pro!ess !an be seen in the #ollowin( pi!ture%

As we !an see, the data analysis pro!ess !onsists o# three ti(htly !onne!ted but separable phases%+n the #irst phase, the raw data in a #orm o# a byte stream are !on$erted to #ake pa!kets, wrappin(them up usin( appropriate arti#i!ially (enerated headers% "hese #ake pa!kets are used as an input #orthe pa!ket disse!tion pro!ess in the se!ond phase% +n the third phase, the output #rom the disse!tion

pro!ess is !on$erted #rom - stru!tures into Python ob e!ts%3e!ause these phases are separable, we de!ided to !reate a module #or ea!h o# them, !reatin( a

typi!al modular ar!hite!ture #or the whole data analysis pro!ess% "his de!ision o##ers #uturede$elopers the possibility to use their own modules #or the mentioned separated phases thus makin(the de$elopment pro!ess easier%

+t mi(ht look a bit e&traordinary to !reate #ake pa!kets e$en thou(h we only want to analyse theraw data and to !reate at #irst - stru!tures whi!h then need to be !on$erted to Python stru!tures et!%We know it mi(ht look !on#usin( but trust us that these de!isions were taken a#ter thorou(hin$esti(ation o# the possibilities o# how to implement the data analysis pro!ess% +t will be bettere&plained in the des!ription o# the indi$idual modules%

Bets ha$e a !loser look at ea!h o# the data analysis modules now%

)*

Figure #.#: %ata anal&sis


22/58


Data"pcap -odule

Data2pcap is a Python module that pro$ides the #un!tionality o# !reatin( #ake pa!kets #rom rawdata and sa$in( them in a p!ap #ormat% 0ake pa!kets are #ormed #rom the raw data wrapped up inarti#i!ial headers so that the data !an be used as an input #or pro(rams whi!h e&pe!t pa!kets on theirinputs% "hese pa!kets are well'#ormed, meanin( that they ha$e $alid !he!ksums and are 1ero'

padded when needed% -urrent $ersion o# this module pro$ides support #or DP and "-P on thetransport layer, +P on the internet layer and 5thernet on the link layer, but !an be easily e&tended%

We ha$e de!ided to !reate this module be!ause o# two main reasons one is that our datadisse!tion module is based upon Wireshark libraries Gmore details about the disse!tion module !an

be #ound in the #ollowin( se!tion


23/58


Data"pcap /sage

"he data2pcap module was written with the emphasis on simpli!ity o# use !ombined with the possibility to set all important parameters o# the !reated pa!kets% +# you want to use the data2pcapmodule you need to use Python $ersion )% and to know only a #ew basi! #un!tions% "hese are:

merge_headers_and_data(raw_data, protocol, src_ip, src_port, dst_ip,dst_port, seq=1 , ac!=1 )

sa"e_pcap_pac!et_to_pcap_#ile(data, pcap_#ile)

and i# you want to !reate "-P pa!kets with $alid seIuen!e and a!knowled(ement numbers thenalso

update_tcp_control_length(mlen, old_tcp_ctrl)

With these #un!tions you will be able to !reate #ake DP and "-P pa!kets #rom raw data andsa$e them to a p!ap #ile% 5$en thou(h we pro$ide a #ull Do&y(en do!umentation #or this modulethat !ontains enou(h in#ormation to be able to work with the data)p!ap module, we ha$e written a#ew short e&amples to make the work with the module easier #or you%

Ho5 to Create a Fake /D acket and Save it to a cap File

Suppose we ha$e raw data in a bytestrin( !alled raw_data and want to !reate a #ake DP% "he bytestrin( !an be either read #rom a so!ket or !onstru!ted #rom any Python strin( usin( he&li#y oryou !an use whate$er ma(i! you need to (et it% We want to !reate a DP pa!ket !omin( #rom the +Paddress %&%&%&% port '0 to the +P address )%)%)%) port (0 % We want to sa$e the pa!ket to a lo( #ile!alled lo(%p!ap in the !urrent dire!tory% "he #ollowin( #ew lines are all we need to use:

$ create a %&P pac!et with speci#ied parametersmydata = merge_headers_and_data(raw_data, 'udp', '1.1.1.1', ' ',' . . . ', '* ')$ create a new #ile and write the created pac!et to itout#ile = #ile('log.pcap','w+')sa"e_pcap_pac!et_to_pcap_#ile(mydata, out#ile)out#ile.close()

And that is it% Qou !an try it and open the lo(%p!ap #ile in any pa!ket analyser to see the result

whi!h should be well'#ormed DP pa!ket with all parameters set up a!!ordin( to the spe!i#i!ation%+# you want to simulate an DP !ommuni!ation !ontainin( more pa!kets, you !an do it easily%Qou ust ha$e to use a loop% Mo$e the #ile openin( and #ile !losin( operations outside the loop andthe data readin( operation into the loop !ondition, lea$in( merge_headers_and_data andsa"e_pcap_pac!et_to_pcap_#ile inside the loop and e$erythin( should work like a !harm%

Ho5 to Create a TC acket and Save it to a cap File

"his is $ery similar to !reation o# a DP pa!ket with one notable di##eren!e% "he "-P!ommuni!ation !ontains seIuen!e and a!knowled(ement numbers% +# you want to !reate only one

pa!ket, you do not ha$e to !are about these and you !an use the e&ample presented #or DP withthe simple !han(e o# EudpF to Et!pF in the !all o# the merge_headers_and_data #un!tion% Qou do

).


24/58


not need to update the seIuen!e and the a!knowled(ement numbers, sin!e we de#ined de#ault$alues #or them in the abo$e mentioned #un!tion%

3ut i# you want to simulate a !ommuni!ation #low between two sides !ontainin( more pa!ketse&!han(ed in both ways the situation be!omes a bit more !ompli!ated% 0or more details read the

E+reate a 5+ +onnection F part%

Create a TC Connection

+t is a little tri!ky to simulate a "-P !onne!tion% +t is ne!essary to keep the a!knowled(ement andseIuential numbers #or ea!h pa!ket #rom within the same !onne!tion updatin( !orre!tly% We wereaware o# this beha$iour and so we added support #or !omputin( ri(ht seIuen!e anda!knowled(ement numbers #or "-P pa!kets% Howe$er, to keep thin(s simple the resultin(me!hanism mi(ht seem a bit !on#usin(% We will not (o into details o# S58@A-K arithmeti! in here,i# you are interested in it please !onsult http:@@pa!ketli#e%net@blo(@)7*7@ un@>@understandin('t!p'

seIuen!e'a!knowled(ment'numbers@ whi!h is a $ery (ood des!ription o# the sub e!t%"he #a!t is that to simulate a "-P !ommuni!ation between A and 3 you need two inte(ers% "he#irst will ser$e as a seIuen!e number #or A and also as an a!knowled(ement number #or 3, these!ond will ser$e as a seIuen!e number #or 3 and !onse!uti$ely as an a!knowled(ement number#or A% "o keep the numbers ri(ht you ha$e to !all the update_tcp_control_length() #un!tionon the #irst number e$ery time A sends data to 3 and use the same #un!tion on the se!ond numbere$ery time 3 sends data to A% Also it is re!ommended to initiali1e these numbers randomly at thestart o# the !ommuni!ation% +nitiali1ation to * mi(ht lead to problems when there are any se(menteddata at the be(innin( o# the !ommuni!ation%

We know that the des!ription mi(ht be !on#usin( Ge$en thou(h we tried it to be as !lear as possible so we pro$ide a simple e&ample%

$ sequence num+er o# A, ac!nowledgement num+er o# seqA = $ sequence num+er o# , ac!nowledgement num+er o# Aseq = -$ A sends data to , data are captured and trans#ormed into a #a!e TCPpac!etmydata = merge_headers_and_data(raw_data, 'udp', '1.1.1.1', ' ',' . . . ', '* ', seqA, seq )$ seqA is updatedseqA = update_tcp_control_length(len(raw_data), seqA)$ any processing necessary happens here, new data #rom arri"es

$ sends data to A (reply to the pre"ious data #rom A)mydata = merge_headers_and_data(raw_data, 'udp', ' . . . ', '* ','1.1.1.1', ' ', seq , seqA)$ seq is updatedseq = update_tcp_control_length(len(raw_data), seq )$ any processing necessary happens here, new data #rom A arri"es$ A sends data to (reply to the pre"ious data #rom )mydata = merge_headers_and_data(raw_data, 'udp', '1.1.1.1', ' ',' . . . ', '* ', seqA, seq )$ seqA is updatedseqA = update_tcp_control_length(len(raw_data), seqA)$

)9
http://packetlife.net/blog/2010/jun/7/understanding-tcp-sequence-acknowledgment-numbers/http://packetlife.net/blog/2010/jun/7/understanding-tcp-sequence-acknowledgment-numbers/http://packetlife.net/blog/2010/jun/7/understanding-tcp-sequence-acknowledgment-numbers/http://packetlife.net/blog/2010/jun/7/understanding-tcp-sequence-acknowledgment-numbers/http://packetlife.net/blog/2010/jun/7/understanding-tcp-sequence-acknowledgment-numbers/


25/58


"his way you !an simulate a "-P !ommuni!ation whi!h will be !orre!t% Qou !an !he!k this byusin( a pa!ket analyser, it should not ob e!t a(ainst the seIuen!e or a!knowled(ement numbers andshould line up the pa!kets appropriately%

-odification and *0tension of Data"pcap

Bet us #irst note that data pcap is open and a$ailable #or any type o# modi#i!ation you mi(htneed to do% +n #a!t we will be (lad i# you will reuse it in your own work and i# it !an spare you somede$elopin( time%

"he !urrent $ersion o# data pcap supports only !reation o# #ake DP and "-P pa!kets o$er +Po$er 5thernet% "his is be!ause we needed only these types o# pa!kets #or Hot0u11 and we did notneed to use any other types% Howe$er, we knew that someone mi(ht need to !reate other types o#

pa!kets too and so we ha$e written data pcap module to be easily e&tensible%5$ery #ake pa!ket !reation is !omposed o# !alls to create_///_hdr() #un!tions where

stands #or the desired proto!ol type% A "-P pa!ket is !reated by create_eth_hdr() ,create_ip_hdr() and create_tcp_hdr() in this order% A p!ap wrapper enablin( the pa!ket to be written to a p!ap #ile is !reated by !alls to create_pcap_glo+al_hdr(),create_pcap_pc!t_hdr() and then the #un!tion to !onstru!t the desired pa!ket type% "hisapproa!h lea$es you an easy way o# modi#i!ation i# you #or e&ample need to simulate a token rin(instead o# 5thernet you !an write a new #un!tion create_tr_hdr() and !all it instead o#create_eth_hdr() % And that is it%

We ha$e also prepared some #un!tions #or manipulation with he&ade!imal data in Python% "hese#un!tions !an be #ound in the #irst part o# the data pcap sour!e% "he !he!ksum #un!tion !omputin(a one'!omplement binary number used in many headers is also prepared #or you to use i# ne!essary%

-odule t#6e0port

Introduction

"he tm e&port module was written in - and pro$ides the pa!ket disse!tion #un!tionality% +t is based mainly upon the Wireshark libraries% +n short, we !an say that it is a wrapper around theWireshark !ore disse!tion pro!ess that takes a bytearray !ontainin( a pa!ket on the input and

produ!es a tree'like stru!ture deri$ed #rom the results o# the disse!tion pro!ess on the output%3e!ause we needed to sli(htly !han(e some o# the Wireshark sour!es in order to make tm_e0portwork we re!ommend you to read the 5echnical issues part i# you want to reuse or modi#y this

module% Cote that due to the #a!t that the tm e&port module was !reated in a #orm o# a dynami!ally linkedlibrary there are almost no restri!tions on the pro(rammin( lan(ua(e in whi!h you !reate yourappli!ation that will use this module% "he only thin( is that you must be able to load the library% 0orthis purpose we ha$e used the ct)pes module to (enerate a wrapper #or Python but you areabsolutely #ree in this de!ision%

Technical Issues

"he tm_export module is based upon the de$eloper $ersion o# Wireshark *%.% , re$ision .)99;%3e!ause Wireshark is e$ol$in( Iuite #ast and it is not e&traordinary that a bi( portion o# the !ode isaltered to better suite the new approa!h taken by its de$elopers it is $ery important to use e&a!tly

)


26/58


this re$ision i# you e$er need to re!ompile the tm_export module% +# you !annot, #or any reason, usethis re$ision, we hi(hly re!ommend you to !he!k the $ersions o# indi$idual headers that ha$e been!han(ed in the tm_export sour!e !ode a(ainst the $ersion o# these headers in the re$ision you wantto use% +# these re$isions di##er then the !ompatibility mi(ht be broken and e$en thou(h you mi(ht

be able to re!ompile the module, the result mi(ht not work ri(ht%Also there is a known issue with the tm_export module when another $ersion o# Wireshark is

already installed on the ma!hine where you want to use this module% "his situation mi(ht produ!e atri!ky error when e$erythin( seems to be ust #ine until you try to a!tually !all any #un!tion #romwithin the tm e&port module% "hat leads to a Windows pop'up messa(e sayin( somethin( like 5heli 9sutil.dll li rar& cannot e loaded. ou ma& fix this & reinstalling the conflicting program."his error has nothin( to do with the li*+sutil library at all% "he problem is in the load'!hain o# thelibraries% "he tm_export module depends on the libwireshark%dll library whi!h depends on a do1eno# other libraries% "he problem is that one o# those libraries tries to load Wireshark plu('ins #rom thedire!tory path whi!h it #inds in the re(istry and sin!e tm_export does not write anythin( to there(istry, it takes path to the plu('ins o# the installed $ersion o# Wireshark% Sin!e the plu('ins areusually in!ompatible with the $ersion o# Wireshark libraries used by the tm e&port module, an erroro!!urs% "here is a way how to sol$e this problem EmanuallyF, one ust needs to !han(e the dire!torywhere the plu('ins are stored% 0or more in#ormation about this see se!tion E 5rou leshooting: Ho9to 3eep ;ireshar3 installed in the installation manual% 6ther solution mi(ht be to !han(e the pathin the re(istry but it did not work #or us when we tried it% +# you ha$e time and will, you surely !anlook #urther into this and we will be $ery (lad i# you pro$ide us with a better solution%

/sage

"he tm e&port module was desi(ned to be as easy'to'use as possible% Qou ust pro$ide the data in

a bytearray, the module !he!ks whether it looks like a $alid pa!ket and i# so, it !alls the Wiresharkdisse!tion !ore% "his approa!h was #ine #or us sin!e we ust needed the pa!ket to be disse!ted i# thedisse!tion !ore !an do it and an indi!ation o# #ailure otherwise% +# you need to !ontrol the disse!tion

pro!ess anyhow, you mi(ht try to modi#y the tm_export module on your own Gsee the se!tion "odification and extension of tm0export but we re!ommend you to write to the Wiresharkde$eloper #orum #or some hints at #irst sin!e adaptin( the Wireshark !ode is one o# a hard !akes to

bite%"o be able to use the tm_export module, you basi!ally need the #un!tions a$ailable in the

tm_e0port.h #ile% We will skip their detailed des!ription sin!e you !an #ind it in the (eneratedDo&y(en do!umentation or in the sour!e !ode itsel# and we will ha$e a look at the input and outputstru!tures and at a simple e&ample o# one pa!ket disse!tion%

*0a#ple /sage

As we mentioned earlier, there are ust a #ew #un!tions you need to use when you want to use thetm_export module% Qou ha$e to initiali1e (lobal disse!tion stru!tures and memory #or them at the$ery be(innin(% "hen you ha$e to initiali1e lo!al disse!tion stru!tures #or e$ery indi$idual!ommuni!ation Gwell, you does not ha$e to, you !an use the same lo!al disse!tion stru!tures #ordi##erent !ommuni!ations but it is a bit memory'!onsumin( then % "hen you !all the disse!tion

pro!ess itsel# and use the output as you need% A#ter you do not need the output any more you should#ree the memory that was allo!ated #or it% When you are #inished with disse!tion you !all the!leanup #un!tions #or lo!al and (lobal disse!tion stru!tures and that is all%

And how does it look in the pseudo!ode

);


27/58


$ initiali e glo+al dissection structuresh#_dissect_init()$ initiali e local dissection structuresh#_one_iteration_init()$ do the dissection( stands #or the 23&4 %5 mode)result = h#_dissect_one_pac!et(data, )$ now do some result processing you want to do$ #ree the structures o# the dissected pac!eth#_#ree_datamodel(result)$ #ree the local dissection structuresh#_one_iteration_cleanup()$ #ree the glo+al dissection structuresh#_dissect_cleanup()

Input4+utput Structures"here are only two #un!tions that e&pe!t an input #rom you% "he #irst is the

h#_dissect_one_pac!et() #un!tion that !alls the disse!tion o# the data you pro$ide it with% "hese!ond is the h#_#ree_datamodel() #un!tion that e&pe!ts to be !alled on the input o# the #irst#un!tion a#ter you no lon(er need it% "here#ore you ust need to understand what to pro$ide to theh#_dissect_one_pac!et() #un!tion and what it returns you%

"he input should be a bytearray !ontainin( a $alid pa!ket% Qou !an use the wrapper (eneratedaround raw data usin( merge_headers_and_data() #un!tion #rom the data2pcap module or!onstru!t it usin( other ways% 0or an e&ample o# su!h input you may use the data2pcap module%

sin( - !on$entions, the input is de#ined as

unsigned char 6 in_data7

"he output is a tree stru!ture !onstru!ted #rom nodes with indi$idual meanin(% 5$ery node hasthe same stru!ture whi!h !an be seen in the Do&ymentation to the h# mynode stru!t or in thetm e&port%h sour!e #ile% A $isualisation o# the tree stru!ture resultin( #rom a simple H""P pa!ketdisse!tion is in the #ollowin( pi!ture%

)>


28/58


-odification and *0tension of t#6e0port

+# you would like to e&tend or modi#y the tm_export module on your own, we en!oura(e you todo so and be!ause we ha$e (one throu(h some nasty surprises durin( its !reation we o##er you somead$i!e to the be(innin(%

At #irst, you should use Windows and =isual Studio #or de$elopment% We ha$e !reated thetm_export pro e!t in it and it will sa$e you some time% We are sure that i# you need to port this to

Binu&, it should work sin!e Wireshark is portable and hope#ully we did not made any Windows'spe!i#i! !ode too but we !annot (uarantee it%

Also make sure that you ha$e Wireshark sour!es #or $ersion *%.% and re$ision .)99;% +# youha$e another $ersion o# sour!es and do not want to !han(e, you may try your lu!k but in that !aseyou ha$e to (o throu(h all the header #iles in the tm_export header sour!e dire!tory and !he!k theirre$isions a(ainst the re$isions o# the same headers in the Wireshark sour!e% +# they di##er you willha$e to #ind the di##eren!e and !he!k whether it is $ital #or the #un!tionality you need or not% Wewould not ad$ise you to try this approa!h but i# you really need to, we wish you lu!k%

5ssential #or the tm_export module are the Wireshark libraries% "here#ore you ha$e to learn howto !ompile Wireshark #rom sour!es% We !an re!ommend you a $ery (ood manual how to do thisunder Windows usin( nmake Gand supposedly it will (o similar way, maybe easier, usin( makeunder ni& % "he manual is a$ailable at:

http:@@www%wireshark%or(@do!s@wsd( html !hunked@-hSetupWin.)%html

n#ortunately, this pa(e is Iuite o#ten down and there#ore we de!ided to atta!h it to thisdo!umentation% +t !an is en!losed in the 9ireshar309insetup.html #ile%

0or better understandin( o# the tm_export module sour!e !ode we tried to do!ument it as mu!h as possible usin( Do&y(en and in'sour!e !omments% We hope that these will help you understand the pro!ess and #ind the pla!e where you want to make !han(es to% A short summary o# the pro!ess o#the tm_export module ob:

)/

Figure #.


29/58


read the input bytearray !he!k the p!ap (lobal and lo!al headers #or $alidity and strip them o## prepare Wireshark disse!tion stru!tures and $ariables do the disse!tion pro!ess !he!k the sanity o# the output #rom the disse!tion translate the output o# the disse!tion pro!ess into a tree stru!ture !leanup the disse!tion stru!tures and $ariables return the output in a #orm o# a tree stru!ture

+# you ha$e any troubles durin( your modi#i!ation work, #eel #ree to !onta!t us i# it will be aboutthe tm_export !ode or use Wireshark de$eloper #orum to #ind answers to Iuestions about the

beha$iour o# Wireshark libraries%

Calling %ireshark Code fro# &thon

"he Wireshark analyser is written in - and the Pea!h #u11in( #ramework is written in Python%"his was Iuite a problem, sin!e we wanted to use some parti!ular parts o# Wireshark inside Pea!hand we did not want to slow down the #u11in( pro!ess Ge%(% by startin( a new pro!ess durin( ea!hiteration %

0ortunately, Python !ontains the ct)pes module whi!h allows Python s!ripts to !all - #un!tions#rom shared libraries G ,&so ,&dll% So a#ter we !reated the shared library, we were able to use the

!types module to !all a parti!ular - #un!tion in our Python !ode% n#ortunately, usin( the ct)pesmodule was not so easy% So we de!ided to use a pro(ram !alled ct)pesge. % +t (enerates a !ode,whi!h wraps all ne!essary - #un!tions and stru!tures G#rom a sin(le header #ile into a $alid Pythons!ript #ile usin( the ct)pes module%

"his was Iuite bene#i!ial, be!ause whene$er we made a !han(e in the AP+ o# the shared library,we were able to (enerate a !orrespondin( Python !ode with no e&tra e##ort%

Converting %ireshark Structures into each Structures

"he !ommuni!ation between Pea!h and the Wireshark analyser pro!eeds as #ollows% Pea!h sends

the entire messa(e as a parameter o# a - #un!tion and re!ei$es a pointer to a - tree stru!turerepresentin( one Wireshark datamodel% "his tree is not well suited #or a #urther usa(e, so it isne!essary to per#orm some !on$ersion a!tions% "he !on$ersion !han(es the stru!ture o# the tree#rom the pointer linked stru!tures G- approa!h to array linked stru!tures GPython approa!h % +t alsoremaps the Wireshark types into the Pea!h types% "he in!omin( - stru!tures ha$e more or less thesame layout as the out(oin( Python stru!tures%

)2


30/58


$.$ Data -atching6ne o# the main tasks o# Hot0u11 durin( #u11in( pro!ess is to lo!ate the mutable elements in the

data !omin( #rom the sour!e appli!ation so they !an be then altered be#ore they are #orwarded to thedestination appli!ation% "his means to !ompare the !urrently pro!essed data to the pre$iouslyre!orded data models, to !hoose the model whi!h #its the data and mark mutable elements based onthe model% +# the data would ne$er !han(e #rom the one run o# the test s!enario to another, a simplesolution would be to !ompare the bit or strin( representation o# the data and e$ery known model%3ut sin!e the network !ommuni!ation !an !ontain $ariable items Glike !ounters or timestamps asmarter approa!h is ne!essary% So Hot0u11 sear!hes #or the best mat!hin( data model to the (i$enmessa(e% "he !omparison is per#ormed on the already analysed data, whi!h are split to proto!olelements and or(ani1ed in tree stru!tures% "hen the similarity is measured by !omparin( di##erent

aspe!ts o# these stru!tures%"he in#ormation about the data is stored in a $ariable o# the Pea!h type $ction % 5%(% it indi!atesthe dire!tion in whi!h the data are #orwarded% "hat makes possible to distin(uish the reIuests sent#rom the !lient to the ser$er #rom the responses sent #rom the ser$er to the !lient, so only i# thedire!tion o# the !urrently pro!essed messa(e mat!hes the dire!tion o# the data model, they are!ompared%

"he !ommuni!ation is disse!ted into a tree'like stru!ture% "his stru!ture is then taken intoa!!ount in the ne&t step o# mat!hin(% "hese trees G#or both the messa(e and the model are tra$ersedsimultaneously in the depth #irst sear!h and e$ery node is !he!ked whether the #ollowin( propertiesmat!h: the number o# !hildren Gstru!ture o# the tree , element name assi(ned by the disse!tor #or thenode Gthe type o# the messa(e #ield #rom the $iewpoint o# the proto!ol and the node $alue type Gthe

type o# the $alue !arried in the messa(e #ield % +t is e&pe!ted that #or the similar messa(es, all the properties ha$e the same $alue% "hese properties are !he!ked and i# some o# them does not mat!h,the model is pronoun!ed as not similar to the messa(e and the pro!ess starts a(ain with another!andidate model #rom the set o# the re!orded models%

+# the basi! properties #it, additional attributes o# the same substru!ture are !ompared: the $alueo# the nodes G!ontent o# the messa(e #ield and the $alue len(th G!he!ked only #or the strin( typednodes % "hese attributes do not ha$e to mat!h in all nodes o# the tree% "hey are used to pi!k the mostsuitable model #or the messa(e Gamon( the suitable models % 0or ea!h mat!h with the messa(e, themodel re!ei$es a number o# points% 0ull points #or the $alue mat!h and hal# points #or the $aluelen(th mat!h% At the end o# the !omparison, all the points are summed up, representin( the s!ore o#the model Gthe models that do not #it in the basi! properties ha$e the s!ore o# 1ero % "he s!ore is!omputed #or all the !ompatible models and the pro(ram keeps a re#eren!e to the model with the

.7

Figure #.>: ;ireshar3 + structures con*erted into each &thon structures


31/58


!urrently hi(hest s!ore% At the end o# the pro!ess, the most su!!ess#ul model is used to markmutable #ields o# the messa(e%

Due to many possible ways how the models !an di##er G!aused by a wide ran(e o# proto!ols andtheir messa(es , there were many solutions !onsidered #or this task% "he main !on!ern is to #ind a

reasonable !ompromise between !hoosin( a model whi!h in #a!t does not belon( to the messa(eGtoo loose !omparison and not identi#yin( a model whi!h in #a!t does belon( to the messa(e Gtoostri!t !omparison % We started with a simple test o# the isomorphism between the !ommuni!ationtrees, whi!h pro$ed to work as a basi! reIuirement and later we de$eloped a se$eral ideas #or the#iner sele!tion, in!ludin( the !onstru!tion o# sets o# tree operations needed to trans#orm a model to a(i$en messa(e% Howe$er, this approa!h appeared to be $ery di##i!ult to implement in an early sta(eo# its de$elopment% "here#ore we started to e&periment with the s!orin( #un!tion whi!h !ould beimplemented mu!h #aster and its basi! idea #its the needed sele!tion well% "he testin( on di##erent

proto!ols showed that the #un!tion (i$es satis#yin( results% +n the most !ases it !hooses the modelwhi!h really belon(s to the messa(e, thus we adopted it as the #inal solution% "he disad$anta(e o#the #un!tion is that it reIuires a rather !omple& modular desi(n #or the indi$idual tests so they !an

be used separately and in the !on$enient way% "his howe$er !auses that the same trees ha$e to betra$ersed multiple times, makin( data mat!hin( slower%

$.7 3ecorded Data (ggregation"he purpose o# the re!orded data a((re(ation is to mer(e the data models produ!ed in the

multiple test !ases o# the re!ordin( pro!ess% "he mer(in( is meant to make the resultin( modelssmaller, more e##i!ient and easier to use% 0or e&tensi$e !ommuni!ation like H""P, one test !ase !aneasily produ!e data models o# a si1e o# me(abytes Gwhi!h is needed to be edited and pro!essed , sowith more test !ases the a((re(ation has si(ni#i!ant impa!t% "he main aim o# the a((re(ation is to

eliminate all possible dupli!ities o# the messa(es that are already a part o# the !umulati$e datamodel set% At the be(innin( the set !ontains un!han(ed data models o# the #irst test !ase% "hen themodels #rom the #ollowin( test !ases are in!luded only i# they are distin!t #rom all the models,whi!h are already in%

"he de!ision about the dupli!ities and di##eren!es is based on the $ery same s!orin( #un!tion thatis used #or the data mat!hin( G#or detailed des!ription o# the #un!tion, see the Data mat!hin(!hapter % "his is apparent and natural as multiple runs o# re!ordin( !an be, in terms o# data modelsimilarity, $iewed the same as multiple runs o# a #u11in(% +# a (i$en messa(e would be mat!hedwith a (i$en model durin( the #u11in( anyway, it makes sense to mer(e the messa(e into the modelri(ht away%

Qet, some A!tions ha$e to be treated in a spe!ial way% "hese are the A!tions that do not !ontainany data Gone #orm o# the terminatin( A!tion or ha$e the terminate5est+ase #la( set Ganother #ormo# the terminatin( A!tion % More pre!isely, this is a problem how to a((re(ate an A!tion that will letHot0u11 know that the !urrent test !ase should be #inished% As soon as su!h A!tion is dete!ted, the

pro&y terminates the test !ase and does not #orward data any more% +t is !lear that only oneterminatin( A!tion should be in the resultin( data model, whi!h is not always the !ase, parti!ularlywhen the multiple test !ases are !onsidered% "hus, to a!hie$e this, be#ore sear!hin( #or thesimilarities a((re(ation, the pro(ram looks #or the number and the position o# the !lose A!tions%"he !lose A!tion is tri((ered when the !lient !loses the !onne!tion to the ser$er so there should beone !lose A!tion at the $ery end o# e$ery re!orded test !ase% +# this holds, the !lose A!tion is alsoin!luded in the a((re(ated data model in the same manner% +# not, the !lient does not !lose the!onne!tion itsel# and it should be !he!ked whether a test !ase !an be terminated by a data a!tioninstead% "his is possible i# in the ea!h !ase o# !ommuni!ation is the $ery same last data A!tion,

.*


32/58


a(ain de!ided by data mat!hin(% +n this !ase, the terminate5est+ase #la( is set #or the last a!tion inthe a((re(ated data model%

$.8 Structure of the HotFuzz Configuration File"he MB S!hema in the $ppendix $ des!ribes the stru!ture o# the Hot0u11 !on#i(uration #ile%

"he Hot0u11 !on#i(uration #iles are in many aspe!ts similar to Pea!h !on#i(uration #iles% "hereason #or that is that we wanted to minimi1e !han(es to Pea!h !on#i(uration #ile parser, whi!h is#ortunately written in a $ery #le&ible way, so we were able to insert additional in#ormation into our!on#i(uration #iles without needin( to write a parser o# our own% 6ne o# the results is that it is

possible to in!lude some o# the Pea!h #eatures, like Pea!h monitors, and use them to enhan!e theHot0u11 #un!tionality% Howe$er #ull !ompatibility !an not be (uaranteed and a des!ription o# thePea!h #eatures that !an be used alon( with Hot0u11 is beyond the s!ope o# this MB S!hema%

.)


33/58


7 )/I

"he main purpose o# the 4 + appli!ation is to simpli#y the use o# the Pea!h #u11in( #ramework ba!k'end%

+t does not pro$ide any e&tra #u11in( related #un!tions and all its #un!tionality !an be a!hie$ed byusin( the standard !onsole Pitm appli!ation% Moreo$er the 4 + appli!ation does not !o$er e$ery#u11in( !on#i(uration, whi!h !an be set usin( the Pea!h #u11in( #ramework% +t ust !o$ers areasonable subset o# its #un!tionality related to the Hot0u11 pro e!t% 4 + is distributed under thesame li!ense as the Pea!h #u11in( #ramework, so #eel #ree to e&tend and modi#y its #eatures%

Source Code )eneration

"he 8t Desi(ner does not dire!tly (enerate Python sour!e #iles% +t (enerates ui #iles% G"hese #ilesuse an MB #ormat to represent #orm elements and their !hara!teristi!s% Py8t9 pro$ides with thep)uic/ utility, whi!h is !apable o# !on$ertin( the ui #iles to the Python sour!e #iles% "here is also oneresour!e #ile G4'7/images/resource.=rc !ontainin( in#ormations about the ima(es used, whi!h hasto be !ompiled usin( the p)rcc/ utility%

"o automate these a!tions use the Python s!ript #ile 4'7/ uild.p& or use the make !ommandinside the 4'7/ dire!tory%

-ain Functions +vervie5 Pro$idin( a dialo( lo(i! GDialo(s Startin( #u11in( and re!ordin( e&ternal pro!esses GMain window dialo(, pro(rams Manipulatin( with #u11in( and re!ordin( MBs GMain window dialo(, MB manipulation =iewin( !rashes GMain window dialo(, A!tion $iew dialo(, Dump #iles Keepin( appli!ation settin(s GAppli!ation settin(s, Main window dialo( "he pro e!t mana(ement Gpro e!ts Pro e!t 0iles, +ntro dialo( Pro$idin( undo@redo a!tions G ndo A!tions, Main window dialo(

(pplication File +vervie5 4'7/udpcommunicator.p& < -lasses used #or the !ommuni!ation with Pitm 4'7/xmlmanipulators.p& < -lasses used #or the MB pro!essin( Gloadin( and storin( 4'7/action*ie9.p& < A lo(i! o# the A!tion =iew dialo( 4'7/dumpreader.p& < -lasses used to pro!ess the !rash in#ormations 4'7/glo als.p& < 4lobal de#initions 4'7/hotfuzzplugin.p& < Plu(ins #or 8t desi(ner 4'7/hotfuzz9idget.p& < Spe!ially modi#ied wid(ets G#or 8t desi(ner and main window 4'7/intro.p& < A lo(i! o# the +ntro Dialo( 4'7/mutators.p& < A !lass, whi!h links the mutator wid(et with the element o# the MB tree 4'7/preferences.p& < A lo(i! o# the Pre#eren!es Dialo( 4'7/project.p& < A pro e!t !lass

..


34/58


4'7/projectinfo.p& < A lo(i! o# the Pro e!t +n#o Dialo( 4'7/projectne9.p& < A lo(i! o# the Pro e!t Cew Dialo( 4'7/projectrecent.p& < A lo(i! o# the Pro e!t ?e!ent Dialo( 4'7/settings.p& < Appli!ation settin(s 4'7/shared.p& < A #ile !ontainin( shared non'!lass #un!tions 4'7/testport.p& < A !lass #or testin( whether the (i$en port on the (i$en host is a!!essible 4'7/undoactions.p& < -lasses #or per#ormin( 4 + undo a!tions 4'7/9indo9.p& < A lo(i! o# the Main Window Dialo( 4'7/schema/project.xsd < A s!hema #or $eri#i!ation o# pro e!t MB #iles 4'7/images/ < A dire!tory #or ima(es 4'7/templates/ < A dire!tory with templates #or #u11in( and re!ordin( phases and a de#ault

pro e!t MB 4'7/ui/ < A dire!tory !ontainin( ui #iles !reated in the 8t Desi(ner

7.1 Dialogs

'asic Dialog 9ogic

"he 4 + appli!ation is started by e&e!utin( hotfuzz4'7.p&9 #rom the pro e!t root dire!tory% +tinitiali1es the "ain ;indo9 , shows the splash s!reen and tri((ers the 7ntro %ialog % "he purpose o#the +ntro Dialo( is to open a pro e!t% +# the 7ntro %ialog e&its without openin( a pro e!t, the wholeappli!ation e&its% "he +ntro Dialo( !an be opened only durin( a startup%

All other dialo(s !an be started #rom the "ain ;indo9 %ialog %A#ter a new dialo( is started, the Main window be!omes ina!!essible until the dialo( is !losed%

"he "ain ;indo9 be!omes ina!!essible a#ter a new dialo( is started% +t be!omes a!!essible a(aina#ter the dialo( is !losed%

Standard t Dialogs

We tried to use as many standard 8t dialo(s as possible% Standard 8t dialo(s are o#ten!ustomi1ed a!!ordin( to the !urrent window mana(er o# a parti!ular operatin( system, thus all 8tdialo(s look like nati$e system dialo(s%

We use these dialo(s to Iuery #or the path to e&istin(@non'e&istin( #iles@dire!tories G !tandardopen/sa*e dialogs , #or short error messa(es and #or short !on#irmation messa(es%

-ain %indo5 Dialog

"his dialo( is the most important dialo( o# our appli!ation% +t opens all other dialo(s and pro!esses si(nals #rom numerous types o# e$ents% Moreo$er all e&ternal pro(rams are started withinthis dialo(%

+ts basi! skeleton was !reated in the 8t desi(ner Gsee 4'7/ui/9indo9.ui and 4'7/ui09indo9.p&and the main lo(i! was added in 4'7/9indo9.p& % +t !onsists o# 9 tabs, the Main Menu and theStatus 3ar%

"he #irst tab is responsible #or editin( re!ordin( MBs and startin( Pitm in a re!ordin( mode%

Cote, that the most o# its wid(ets start with a Ere!ordin(F pre#i&%

.9


35/58


"he se!ond tab is responsible #or editin( mutators and the EmutableF part inside the datamodelelement in #u11in( MBs% +t !an also restore last re!orded MB GEpro e!t dirF@re!ordin('out %

"he third tab is responsible #or editin( #u11in( MBs and startin( Pitm in a #u11in( mode% Cotethat the most o# its wid(ets start with a E#u11in(F pre#i&%

"he #ourth tab pro$ides the user with a dump $iewin( #un!tionality% +t opens the A!tion =iewDialo( GA!tion $iew dialo( and starts an e&ternal debu((er pro(ram Gprobably windb( %

"he Main Menu !ontains: standard sa$e@open dialo(s, import@e&port re!odin(@#u11in( MBs dialo(s, the Pre#eren!es Dialo(, About dialo(, the Pro e!t +n#o Dialo(, undo@redo a!tions%

We did not want to bother users with annoyin( messa(e bo&es, so the Status 3ar is used todisplay all important messa(es%"he Main Window also stores in#ormations about the !urrently opened pro e!t%

(pplication Settings

Whene$er the Main Window is startin(, it loads appli!ation settin(s and modi#ies its internal$ariables a!!ordin( to these settin(s% Whene$er the Main Window is !losin(, it prompts whether tostore unsa$ed !han(es to the pro e!t and then it stores its internal $ariables to the appli!ationsettin(s%

"hese settin(s are stored in an ini #ormat to a system dependent lo!ation usin( thereimplementation o# the standard 8t 8Settin(s !lass% +n Windows, it is the #ile +:?%ocuments and!ettings?'!(R $"(?$pplication %ata?HotFuzz?HotFuzz 4'7.ini.

"hese settin(s !an be restored to de#aults or modi#ied in the Pre#eren!es Dialo(%

Intro Dialog

An introdu!tory appli!ation dialo(% "his dialo( is not started #rom the Main Window Dialo(, but#rom an appli!ation start wrapper G hotfuzz4'7.p&9 % +t !an not be started #rom the Main Window%When this dialo( e&its without openin( $alid pro e!t, it !loses the main window thus !ausin(appli!ation to e&it%

+ts basi! skeleton was !reated in the 8t Desi(ner Gsee 4'7/ui/intro.ui and 4'7/ui0intro.p& andthe main lo(i! was added in 4'7/intro.p& % +t !onsist o# three tabs%

"he re!ent pro e!ts tab reads a list o# re!ent pro e!ts pro$ided by the Main Window -lass anddisplays some basi! in#ormations about these pro e!ts% "he ma&imum !ount o# the re!ent pro e!ts!an be set in the references %ialog %

"he new pro e!t tab reads pro e!t template MB #iles G 4'7/templates/fuzzing/,4'7/templates/recording/, 4'7/templates/project/ and displays them in a #u11in(@re!ordin(template list% A#ter hittin( the -reate 3utton, 8t dialo( is opened% A path to a new pro e!t !an be!hosen there% "hen these templates are !opied to the new pro e!t dire!tory% "he pro e!t MB ismodi#ied a!!ordin( to the editable pro e!t in#ormations%

"he open pro e!t tab browses a path to an e&istin( pro e!t% +# the pro e!t MB #ile is $alid, anin#o is shown and the pro e!t !an be opened by the 6pen 3utton%

.


36/58


references Dialog

"his dialo( is responsible #or updatin( Main Window $ariables, whi!h represent !urrentappli!ation settin(s% "he appli!ation settin(s themsel$es is updated when the Main Window e&its%Howe$er, pressin( the ?estore De#aults 3utton !auses that the appli!ation !lears its settin(s, thusrestorin( the state, whi!h was present durin( the #irst start o# the appli!ation%

+ts basi! skeleton was !reated in the 8t Desi(ner Gsee 4'7/ui/preferences.ui and4'7/ui0preferences.p& and the main lo(i! was added in 4'7/preferences.p&.

"his dialo( !an be tri((ered only throu(h the Main Menu or by pressin( a !orrespondin(short!ut%

:e5 ro!ect Dialog

"he e9 roject %ialog is $ery similar to the new pro e!t part o# the 7ntro %ialog % "he !ode ismore or less the same%

+t reads pro e!t template MB #iles G 4'7/templates/fuzzing/, 4'7/templates/recording/,4'7/templates/project/ and pla!es them into a #u11in(@re!ordin( template list% A#ter hittin( the-reate button, a 8t dialo( is opened% A path to a new pro e!t !an be !hosen there% "hen thesetemplates are !opied to the new pro e!t dire!tory% "he pro e!t MB is modi#ied a!!ordin( to theeditable pro e!t in#ormations%

+ts basi! skeleton was !reated in the 8t Desi(ner Gsee 4'7/ui/projectne9.ui and4'7/ui0projectne9.p& and the main lo(i! was added in 4'7/projectne9.p& %


3ecent ro!ect Dialog"he Recent roject %ialog is $ery similar to the re!ent pro e!t part o# the intro dialo(% "he !ode

is more or less the same%+t reads a list o# re!ent pro e!ts pro$ided by the Main Window -lass and displays some basi!

in#ormations about these pro e!ts% "he ma&imum !ount o# the re!ent pro e!ts !an be set in thePre#eren!es Dialo(%

+ts basi! skeleton was !reated in the 8t desi(ner Gsee 4'7/ui/projectrecent.ui and4'7/ui0projectrecent.p& and the main lo(i! was added in 4'7/projectrecent.p& %


ro!ect Info Dialog

"his is a $ery simple dialo(, whi!h ust modi#ies the in#ormations about the !urrently opened pro e!t% "he pro e!t stru!ture itsel# is the one o# a Main Windows $ariables%

+ts basi! skeleton was !reated in the 8t Desi(ner Gsee 4'7/ui/projectinfo.ui and4'7/ui0projectinfo.p& and the main lo(i! was added in 4'7/projectinfo.p& %


.;


37/58


(ction 2ie5 Dialog

"he purpose o# this dialo( is to show users the !ommuni!ation between a ser$er and a !lient be#ore the pro(ram !rashed% +t !an open parts o# the !ommuni!ation in an e&ternal te&t editor%

+ts basi! skeleton was !reated in the 8t Desi(ner Gsee 4'7/ui/action*ie9.ui and4'7/ui0action*ie9.p& and the main lo(i! was added in 4'7/action*ie9.p& %

"his dialo( !an be tri((ered only throu(h the dump tab o# the "ain ;indo9 %

7." ro!ectsAn essential part o# the 4 + appli!ation is a pro e!t%Pro e!t is a !lass used

HotFuzz Developers Guide

Documents

Transcript of HotFuzz Developers Guide