Writing XKS Fingerprints - Electronic Frontier Foundation€¦ · 6/7/2015 · Fingerprints 101...

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Writing XKS Fingerprints

November 2010

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

• Naming Fingerprints • Simple Keywords • Boolean Logic • Variables • Context-Sensitive

TOP SECRET//COMINT

• • • • • • • •

Fingerprints 101

• Whats in a name?

• The XKS Fingerprint naming convention can help organize fingerprints and make searching easier so its important to make sure you name your fingerprint inline with the existing convention

TOP SECRET//COMINT

what's in a name • For example, fingerprint names look like this:

• encryption/archive/rar • encryption/archive/pkzip • encryption/archive/pkzip

• Notice the directory-like structure so that all encryption fingerprints are within the same "folder" and all encryption/archive fingerprints are within the same "folder"

TOP SECRET//COMINT

• • • • • • • •

what's in a name

• This allows for smarter searching because you could look for all encryption fingerprints by searching for encryption/* or search for all encryption/archive fingerprints by searching for encryption/archive/* and etc.

TOP SECRET//COMINT

what's in a name

• When you want to submit a new fingerprint, look to see if it would fit into any existing fingerprint folders.

• Best way to do this is to use either the "Field Builder" or "Tree Field Builder" next to the AppID+Fingeprints field in the search forms

AppID (+Finaerprints) f fu l l text l :

T I rPopulate with Field Builder] I "¿1 rPopulate with Tree Field Builder"!

TOP SECRET//COMINT

CRET STRAPi

fr-r- ^^BL. , , « -« « « » » « i . . . . . . ' . il I

What's in a name

• The field builders allow you to browse existing fingerprint directories to see if one already exists for your new fingerprint

Field Bui lder

J ^ A p p I D (+ Fingerprints)

> j A p p l i c a t i o n s

0 r C T _ M O

!> Q G P S j t u c f c m g

H T M L 5 _ G e o l o c a t i o n

i> C J I C B M

0 L Q U A N T U M B O T

l> a T A O

> L adver t isement

;> a f g h a n _ m o i

> analyt ics

!> ano

i> a n o n y m i z e r

> _ antivirus

i> L I app

i> i app l ica t ion

l> app l ica t ions

associate d_sessi on

> L backdoor

!> L backdoors

> t z

D- L _ botnet

A

TOP SECRET//COMINT

TOP SECRET STRAPi

ingerprint directories Field Builder

Field Builder

AppID (+Fingerprints)

top ic/w md/iran/ir is I

top ic/w md/iran/ir isl/ed il/chat_body

top ic/w md/iran/

top ic/w md/iran/

top ic/w md/iran/

top ic/w md/iran/

top ic/w md/iran/

top ic/w md/iran/ir is l/ed 13


r isl/ed il/docurnent_body

risl/edil/email_body

r isl/ed il/filename

r isl/ed il/ur I _path

r isl/ed ¡2

mojahe V

encryption/mo]aheden2

encryption/mo jaheden2/encodedheader

encryption/mo jaheden2/hidden

encryption/mo jaheden2/hidden2

encryptton/mojaheden2/hidden44

encryption/mo jaheden2/secure_file_encoded

encryption/mo jaheden2/securefile

Field Builder


botnet/black| v

botnet/blackenergybot/command/die

bolnet/blackenergybot/command/flood

bo tne t/b lackenergybo t/co m mand/ic mp

bolnet/blaokenergybot/command/stop

botnet/blackenergybot/command/syn

botriet./blackenergybot/comrnarid/wait

TOP SECRET//COMINT

M

WK———

what's in a name

• If no existing directory makes sense for your fingerprint, you can always create a new one.

TOP SECRET//COMINT

Fingerprints 101: Getting Started

• The first step is to define the name of the fingerprint.

• To do that, follow the syntax below:

fingerprint( encryption/archive/test_new') =

TOP SECRET//COMINT

TOP SECRET STRAPi M

m

Steir-s^S^.vjb U iV VVVW aBB9 ' v . ^ f — ~TFEr

Fingerprints 101: Getting Started

• Note that fingerprint names can not have spaces or any other punctuation other than / which denote directories and _ which can be used in the place of spaces to make fingerprint names easier to read

fingerprint (encryption/archive/test_new?)

TOP SECRET//COMINT

Fingerprints 101 As an example, let s say we want to fingerprint traffic like this:

Met 20D7 : ij fi 1,19?

. J-jiyJ \\iA-s

3M 08:04 ,2003-18-02

g tawab

a

### Begin ASRAR El Mojahedeen v2.0 Encrypted Message ### r/RgTzT/ATRhN2E1 Zjg1 D WQyN WRj M mE2ZTd IN zZmZD h IO D UxZWZh M D Q1 Mj YwMjViZGUO

ZG YwMj d k M m J m NTA4ZD Y2Yj kOM G U2N G N i Yjg5 M z NjZTcS MThjY2 Y1 Zm Y5MTg zZD I kYj hj MTE x0GYzYjc12DdiMDAxNTQzZmVINDVIY2YyMGJjYjU20DkyYjdmYjFjYjAzMWM5ZDQ20WFIMzg

4NThhM2l1 Mjc50DkzZGNhOGRmNWJmNjVIZjQOMjMxNDM4MDIyO Tq1 MmRjM G J iN GN k YTN kYTQ4 Mz MxZj Ri N2 F i Nj 13 Mj E1N GI3 MTA3ZDQ4 N WRm YzMyOTU zZ jZl Mj g3Nj Q10 GQ4 MTA3 N

TU2 N2Zk N2Zj Yz U z Yz Yy Mj FIOD AwN2 VkM2 U6 MTZi N D Y2 M m M2ZTV IYj Q2 YzIO Q G Q2 OD U x W W VkMJI2M WVIND Ay O GlOMThkMTdhNTY1 YzlxMD gy O GZIM2I wZWZJ MDgwM2U4MzNINDg10D U xZTc4 OD c 1MTY2 Fv/t2 IONjU5Zj BhZj Vh Nj kOOTl h N GE x OTh m YWVI N m F IZjly N mM wZD A3 M DM0

NjJkZDhhMm!4ZmRhYjc3NmZINDFk0DkyYjBhY] Y3MD010GVIMj dhYmUv/ZTlyNGlx YmQyZDIz ZjliM2E6ZGQ5NmNhZDQxQTM4NTIQMjc3MzBIOWE\vZWE1 Njk3Yj gxY2ViNTQ1 OWULnoiA/D

U LljTE uD J qn $ OG M RH e s i/8PTnZj Q2y qb mK b F k I Pj wM h $7 FU hF AO w74 S+i+ Po k OREcS X h d P+ y9 /

G ul3ju YTv ri EO x G x2 Os SfN S5 kfR X X H1 DaTn b7 Oyufe9 rfi mM IQ6 e6E0SRUIdU6YVupz0hhgd4Dof

SBbFR30vgOS+pUxDYgmEOr/RA+fYi47tuHQMh+-dynZciQspNdmRUmkiEpFqF03sPHS/1 Oinjqo e 1G sf8+xn52 XE2 q JWd nU+4 XJ W nUI s VN AJ v2n s L+S2TG1 IHbgocmpQoxyOBOSX PcRv J+2 J ek V37 k 1 XyO NZk9YH+• V3 aWYP Xt+y m+wGO X NTqP HI U1J WAZq I2N K/c S Xt9 DMtCtc b8 c z Rj6 G9IX v J9

Eny7t06x P d9B G i o9 M+3 Q ull kZH L E mJ i A v g v B6 R/X/3 wh B q k5zM H QLfo+VJcXS um W5m Rtg Cj z S P W6IZZFCGUB4SK4P XT52ZC0B2 kWD8 VMyNffr lsTG4 X U e s gx47 Hd6 x M L8 wfc pjffZwKN K+EfKI P

==Z1 o w2S A9N3 uL IX B X62 L hOyj/1 i qfJ2 F N R7 Al ON SEj wKo g gVm k x D i uG aQ i+Tu rpxBgatlg

m End ASRAR El Mojahedeen v2.0 Encrypted Message m

Jl iiijioJi^ wU^yi

)c o. I a i iLdblI -•-'•L-JI

I ^•^••J, 1 LklfrJI .

uluJdl ,

^UJI JiU, JI . Itl ..nil cr.fft

.s % i % > c 11 ..,, | .. n [

• IT I . nil

-LlafrJI , -'J'JL1

I4L» JX-OLJL ^JybVpj

_ L-ItxoJI .

JI P- lcfrJI

•¡¿koJj r- lccJI

I - JI.V.II •-oJOJ I

, Lub&JI / „cli3

<U£uJI 0UL0JI

1 teibving 1 Hirlrtei fifths

Fingerprints 101 One thing that could be used to find data like this is the String ASRAR El Mojahdeen V 2 . 0 Encrypted Message

Met 2037 : ij fi 1,19?

. Jsiiyj

3M 03; 04 ,2003-18-02

g t t a w a b

Begii jASRAR El Mojahedeen v2.0 Encrypted MessagfliOT r/RgTzT/ATRhN2E1 Zj g 1 D W Q y N W R j W . - " ^ „ M . v ^ n u m n , tiUtttifilGUO

J«» Jbuo

ZG YwMj d k M m J m NTA4ZD Y2Yj kOM G U2N G N i Yjg5 M z NjZTcS MThjY2 Y1 Zm Y5MTg zZD I kYj hj MTE x OG YzYj c1 ZDd i M D Ax NTQzZmVIN • VIY2 Yy M G Jj Yj U2 OD kyYj dm Y j FjYj Az M WM5ZD Q2 O WFIM z g

4NThhM2l1 Mjc50DkzZGNh0GRmNWJmNjVIZjQQMjMxNDM4MDIy0 Tg1 MmRjM G J iN 6 N k YTN kYTQ4 Mz MxZj Ri N2 F i Nj 13 Mj E1N GI3 MTA3ZDQ4 N YVRm YzMyOTU zZ jZl Mj g3Nj Q10 GQ4 MTA3 N

TU2 N2Zk N2Zj Yz U z Yz Yy Mj FI OD AwN2 VkM2 U6 MTZi N D Y2 M m M2ZTV lYj Q2 YzIO 0 G Q2 OD U x N W VkMJ I2MWVINDAyO G lOMTh k MTd hNTY1 Yzl x M D gyO GZIM2IwZWZJ M D gwM2 U4 Mz NIN D g l O D U xZTc4 OD c 1MTY2 M2 IONjU5^ BhZj Vh Nj kOOTl h N GE x OTh mYWVI N m F IZjlyN mM wZD A3 M DM0

NjJkZDhhMm!4ZmRhYjc3NmZINDFk0DkyYjBhY] Y3MD010GVIMj dhYmUwZTlyNGlx YmQyZDIz Zjl ¡M2E6ZG Q6N m HhZDQ x QTM4 NTIQMjc3 M z BIO WE wZWE 1 Nj k3Yj g x Y2 Vi NTQ10WUL n oi A/D

U LljTE uD J qn e OG M RH e s i/SPTnZj 02y qb mK b F k I Pj wM h e7 FU hF AO w74 S+i+Po k OREo5 X h d P+ y9 /

G ul3ju YTv ri EO x G x2 Os SfN S5 kfR X X H1 DaTn b7 Oyufe9 rS mM IQ6 e6E0SRUIdU6YVupz0hhgd4Dof

S Bb F R3 O v g OS+pU x • Yg m EO r/RA+fY147tu H QM h+dynZq Q sp N d mRUmkj Ep F qF03 s P H S/1 Oi nj q o e 1G stB+xn52 XE2 q/Wd nU+4 XJ WnlJI s VN AJ v2n s L+S2TG11H b go c m p Qo xyOBO SX PcR v J+2 J ek V37 k 1 XyO NZk9YH+• V3 aWYP Xt+y m+wG0 X NTqP HI U1J WAZq I2N K/c S Xt9 DMtCtc b8 c z R]6 G9IX v J9

Eny7t06x P d9B G i o9 M+3 Q ull kZH L E mJ i A v g v B6 R/X/3 wh B q k5zM H QLfo+VJcXS um W5m Rtg Cj z S P W6IZZFCGUB4SK4P XT52ZC0B2 kWD8 VMyNffr lsTG4 X U e s gx47 Hd6 x M L8 wfc pj/fZwKN K+EfKI P

==Z1ow2SA9N3uLIXBX62LhOyj/1iqfJ2FNR7AIONSEjwKoggVmkxDiuGaQi+TurpxBgat1g

m End ASRAR El Mojahedeen v2.0 Encrypted Message m

vil LoJI wLl^yi

)c o. I a i iLdblI -•-'•L-JI

I ^•^••J, I LklfrJI .

uluJdl ,

^UJI JiU, JI . |tl ..nil cr.tfl

.s % i % > c M .. i, I .. n I

• IT I . nil

-LlafrJI , -'J'JL1

I4L» JX-OLJL ILyitbyj

_ L-ItxoJI .

JI P- blccJ I

•¡¿koJj I

I - JI.V.II OJ- LL .oJ'jJ I

, LuL&JI / „cli3

<U£uJI oUioJj

Iteilaving I Hirlrtei fifths

Fingerprints 101: Keywords

• So let s create a fingerprint to tag any data that contains that string

ASRAR El Mojahdeen V2.0 Encrypted Message

TOP SECRET//COMINT


First we'd define the fingerprint with a name:

fingerprint( encryption/mojahdeen2?)

TOP SECRET//COMINT


• Then, simply put the string in single quotes to denote that XKS needs to look for it as a keyword:

fingerprint(encryption/mojahdeen2?) = 'ASRAR El Mojahdeen V2.0 Encrypted Message'

TOP SECRET//COMINT


Finally, all fingerprint definitions need to end with a semi colon to tell XKS that the definition is finished

fingerprint(encryption/mojahdeen2?) = 'ASRAR El Mojahdeen V2.0 Encrypted Message';

TOP SECRET//COMINT

• H * —

Fingerprints 101: Keywords Using the fingerprint GUI on XKS Central, we test to see if this compiles:

Fingerprint Validation / Submittal

Step #1 Step #2 Step #3

y ICompile; 1 Test Against Session Data I d Save tfi Help

Signature

fingerprint (T encryption/moj a h d e e n 2 T ) = T Â S R A R El Mojahdeen v2.0 Encrypted Message7;

S u ccess!

Results

SUCCESS!

Congratulat ions, your f ingerpr int was successfully compiled!

Now use the Test button to run it against the designated session data,

Fingerprints 101 Once checked in, the fingerprint will hit on data like this:

Met 2037 : ij fi 1,19?

. J-jiyJ \\iA-s

3M 08:04 ,2003-18-02

g tawab

### Begin ASRAR El Mojahedeen v2.0 Encrypted Messag r/RgTzT/ATRh N2E1 Zj g 1 O W Q y N W R j t & ^ f l U J t t A t i U t f f i t t

J«» Jbuo uo

ZG YwMj d k M m J m NTA4ZD Y2Yj kOM G U2N G N i Yjg5 M z NjZTcS MThjY2 Y1 Zm Y5MTg zZD I kYj hj MTE x0GYzYjc12DdiMDAxNTQzZmVINDVIY2YyMGJjYjU20DkyYjdmYjFjYjAzMWM5ZDQ20WFIMzg

4NThhM2l1 Mjc50DkzZGNhOGRmNWJmNjVIZjQOMjMxNDM4MDIyO Tq1 MmRjM G J iN GN K YTN kYTQ4 Mz MxZj Ri N2 F i Nj 13 Mj E1N GI3 MTA3ZDQ4 N WRm YzMyOTU zZ jZl Mj g3Nj Q10 GQ4 MTA3 N

TU2 N2Zk N2Zj Yz U z Yz Yy Mj FIOD AwN2 VkM2 U6 MTZi N D Y2 M m M2ZTV iYj Q2 YzIO Q G Q2 OD U x W W VkMJI2M WVIND Ay O GlOMThkMTdhNTY1 YzlxMD gy O GZIM2I wZWZJ MDgwM2U4MzNINDg10D U xZTc4 OD c 1MTY2 M2 IONjU5Zj BhZj Vh Nj kOOTl h N GE x OTh mYWVI N m F IZjly N mM wZD A3 M DM0

NjJkZDhhMm!4ZmRhYjc3NmZINDFk0DkyYjBhY] Y3MD010GVIMj dhYmUv/ZTlyNGlx YmQyZDIz ZjliM2E6ZGQ5NmNhZDQxOTM4NTIQMjc3MzBIOWE\vZWE1 Njk3Yj gxY2ViNTQ1 OWULnoiA/D

U LljTE uD J qn $ OG M RH e s i/8PTnZj 02y qb mK b F k I Pj wM h $7 FU hF AO w74 S+i+ Po k OREcS X h d P+ y9 /

G ul3ju YTv ri EO x G x2 Os SfN S5 kfR X X H1 DaTn b7 Oyufe9 rS mM IQ6 e6E0SRUIdU6YVupz0hhgd4Dof

SBbFR30vgOS+pUxDYgmEOr/RA+fYi47tuHQMh+-dynZciQspNdmRUmkiEpFqF03sPHS/1 Oinjqo e 1G sf8+xn52 XE2 q JWd nU+4 XJ W nUI s VN AJ v2n s L+S2TG1 IHbgocmpQoxyOBOSX PcRv J+2 J ek V37 k 1 XyO NZk9YH+• V3 aWYP Xt+y m+wGO X NTqP HI U1J WAZq I2N K/c S Xt9 DMtCtc b8 c z R]6 G9IX v J9

Eny7t06x P d9B G i o9 M+3 Q ull kZH L E mJ i A v g v B6 R/X/3wh B q k5zM H QLfo+VJcX9 um W5m Rtg Cj z S P W6IZZFCGUB4SK4P XT52ZC0B2 kWD8 VMyNffr lsTG4 X U e s gx47 Hd6 x M L8 wfc pj/fZwKN K+EfKI P

==Z1 o w2S A9N3 uL IX B X62 L hOyj/1 i qfJ2 F N R7 Al ON SEj wKo g gVm k x D i uG aQ i+Tu r p x B g at1 g

m End AS RAR El Mojahedeen v2.0 Encrypted Message m

Jl vii ioji wU^yi

^ o. I i iLdblI .•-'•l-J I

i • ••J, 1 JI . «_»l i ->Jd I ,

^UJI JiU, JI

. |tl ..nil cr.tfl .S % i % > c 11 ..,, I .. n [

. irI. nil

-LlafrJI , -'J'JL1

I4L» JX-OLJL gLy llfV

_ L-ItxoJI . .»¿.¿I

I - JI.V.II •-oJOJ I

, Lub&JI / „cli3

<U£uJI 0UL0JI

1 teibving I t*n< Hirlrtei fifths

TOP SECRET STRAPi

101

As a second example, let s say we want to find data like this:

Us ing T X T f o r m a t t e r

Ref: June 07, 201000803/Q-02135 Islamabad: National Deve 1 o pme nt Co iti pi ex Plot Wo : ^ ^ H Street Mo:

A.

Sector: I s l a m a b a d . Attn:| | AH Purchase SUBJECT : QUOTATION AGAINST YOUR ENQUIRY REF:Purchase of RTV Silicon DATED: 18/05/2010 Dear Sir., With reference to your subject enquiry, we are pleased to enclose our Quotation No: Q-02135-05-567 dated: 07/06/2010/ for your perusal. Please see the 1 Terms of Sale1 attached with our quote for any further details. We hope our offer suits your requirements and we look forward to your valuable purchase order in clue

•

TOP SECRET//COMINT

Fingerprints 101

Look for keywords that could be used to find traffic like th is in the future.

Using T X T f o r m a t t e r

Ref: June 07, 201000803/Q-02135 Islamabad: A.

Deve lo pment C omp 1 e>: Street Mo: Plot Wo:

Sector: Islamabad Attn: AH Purchase SUBJECT : QUOTATION AGAINST YOUR ENQUIRY REF:Purchase of RTV Silicon DATED: 18/05/2010 Dear Sir., With reference to your subject enquiry, we are pleased to enclose our Quotation No: Q-02135-05-567 dated: 07/06/2010/ for your perusal. Please see the 'Terms of Sale1 attached with our quote for any further details. We hope our offer suits your requirements and we look forward to your valuable purchase order in clue

TOP SECRET//COMINT

TOP SECRET STRAPi

101

What if we looked for "National Development Complex" and "Quotation"

T X T f o r m a t t e r

I s l a m a b a d | National D eye 1 o pme n t Co tti pi ex

Sector: Islamabad Attn: AH Pure ha; SUBJECT :^QUOTATION JGAINST YOUR ENQUIRY REF:Purchase of RTV Silicon DATED: 18/05/201* Dear Sit., With reference to your subject enquiry, we are pleased to enclose our Quotation No: Q-02135-05-567 dated: 07/06/2010/ for your perusal. Please see the 'Terms of Sale1 attached with our quote for any further details. Lie hope our offer suits your requirements and we look forward to your valuable purchase order in due

A.

M

WK———

Fingerprints 101: Boolean Logic

• Starting with these two keywords, we'd like to use Boolean Logic to create our new fingerprint

• national development complex • quotation

TOP SECRET//COMINT

TOP SECRET STRAPi

"


Again, step one think of a name:

fingerprint( cp/pakistan/agencies/ndc')

TOP SECRET//COMINT

M

WK———


• Step two, put single quotes around all keywords:

fingerprint( cp/pakistan/agencies/ndc') = 'National Development Complex' quotation

TOP SECRET//COMINT

TOP SECRET STRAPi

"


Use the Boolean operator and

fingerprint( cp/paldstan/agencies/ndc') 'National Development Complex' and quotation

TOP SECRET//COMINT

• Finish the expression with the semi-colon.

fingerprint( cp/paldstan/agencies/ndc') = 'National Development Complex' and quotation ;

TOP SECRET//COMINT

Fingerprints 101: Boolean Logic Use the fingerprint GUI to confirm the fingerprint definition compiles

Fingerprint Validation / Submittal Fingerprint Validation / Submittal

S t e p r f M S t e p # 2 S t e p # 3

j C o m p i l e l 0 T&si A g a i n s t S e s s i o n D a t a [ y j S a v e f l H e l p

Signature

fingerprint(T cp/pakistan/agencies/ndcT) 'national development complex7 and fquotation';

II

ÍU(ü(E

S u c o e s s !

Results

SUCCESS! Congratulations, your f ingerprint was successfully compiled!

Now use the Test button to run it against the designated session data. 1 y j r 1 / / 1

TOP SECRET STRAPi

101

This fingerprint will now successfully find all sessions like this in the future!

Using T X T formatter

Pef: June 07, 201000803/Q-02135 Islamabad: National Development- Complex Plot Mo: Street. Ho: Sector Is lame Attn: AM Purchase SUBJECT : QUOTATION AGAINST YOUR ENQUIRY REF:Purchase of RTV Silicon DATED: 13/0.5/2 010 Dear Sir, With reference to your subject enquiry, we are pleased to enclose our Quotation No: Q-02135-05-567 dated: 07/06/2010, for your perusal. Please see the 'Terms of Sale1 attached with our quote for any further details. tte hope our offer suits your requirements and we look forward to your valuable purchase order in due

TOP SECRET//COMINT

Fingerprints 101 However, how can we account for variations of how the traffic might be seen? Maybe "National Development Complex" will be listed as "NDC". Or maybe instead of a "Quotation" it will be a "Invoice" and etc.

Using T X T formatter

Ref: June 07, 201000803/Q-02135 Islamabad: National Development- Complex Plot Mo Mo : | | Sector: Q | Is 1 arriab ad. Attn:| | AM Purchase SUBJECT : QUOTATION AGAINST YOUR ENQUIRY REF:Purchase of RTV Silicon DATED: 19/ 0.5/ 2 010 Dear Sir, With reference to your subject enquiry, we are pleased to enclose our Quotation Wo: Q-0213 5-05-5 67 dated: 07/06/2010, for your perusal. Please see the 'Terms of Sale1 attached with our quote for any further details. tte hope our offer suits your requirements and we look forward to your valuable purchase order in due

TOP SECRET//COMINT

M

WK———


• Keywords can also be grouped together by parentheses to form more complex Boolean logic:

TOP SECRET//COMINT

M

WK———


• For example, we can expand on our previous fingerprint like so

fingerprint( cp/pakistan/agencies/ndc') = ('National Development Complex or 'NDC')

and (quotation or 'invoice') ;

TOP SECRET//COMINT

TOP SECRET STRAPi

"

Quick Aside 1: Context Sensitivity

All keywords in X-KEYSCORE are case-insensitive by default.

So in the previous fingerprint 'NDC will match on ride, NdC, nDC etc.

TOP SECRET//COMINT

Quick Aside 1: Context Sensitivity

• If you want to force a keyword to be case sensitive, simply append a c after the single quotes.

• Ex: 'NDCc will only hit when NDC is found in all caps, or 'ndc c will hit only when ndc is found in all lower case and etc.

TOP SECRET//COMINT

Quick Aside 2: Keyword Scanning

• By default keywords in fingerprints can hit in substrings since for example 'ndc' is found within grandchildren.

• So this fingerprint fingerprint( cp/pakistan/agencies/ndc') =

'NDC; Will hit on terms like:

• grandchildren • handcard • handcuffs • etc.

TOP SECRET//COMINT

• In specific cases to avoid false hits you can use the word7

context. • Or force there to be a space on either or both ends of the term

by including them inside the single quotes • So this fingerprint becomes:

' NDC

OR:

fingerprint( cp/pakistan/agencies/ndc) = woid('NDC');

TOP SECRET//COMINT

Let s say that this fingerprint is producing good hits, but it also hitting on spam E~ mails.

fingerprint(cp/pakistan/agencies/ndc?) = ('National Development Complex' or 'NDC')

and (quotation or 'invoice') ;

TOP SECRET//COMINT

•We can use the Boolean and not to defeat unwanted traffic like below:

fingerprint( cp/pakistan/agencies/ndc') = (('National Development Complex' or 'NDC')

and (quotation or 'invoice')) and not (Viagra or 'herbal supplement7);

TOP SECRET//COMINT

TOP SECRET STRAPi M

B-.v

Steir-sS.vjb U iVVVVW aBB9 ' v . ^ f — -T ggF

_ I | , — P

Fingerprints 101: Variables

• Variables allow you to link to a list of keywords. For example, working with this fingerprint, we could create variables to each grouping of terms.

fingerprint( cp/pakistan/agencies/ndc?) = (('National Development Complex or 'NDC) and

(quotation or 'invoice')) and not (Viagra or 'herbal supplement');

TOP SECRET//COMINT

TOP SECRET STRAPi M

m

Steir-sS.vjb U iVVVVW aBB9 ' v . ^ f — ~TFEr

_ — , • — - f f ™ ^ -


Variables use the same syntax as fingerprints

$NDC_terms = 'National Development Complex' or 'NDC;

$procurement_terms = quotation' or 'invoice'; $spam_defeats = Viagra' or 'herbal supplement';

fingerprint( cp/pakistan/agencies/ndc') = ($NDC_terms and $procurement_terms) and not

$spam_defeats; TOP SECRET//COMINT

TOP SECRET STRAPi M

m

Steir-sS.vjb U • .'VWVVVV aBB9 ' v . ^ f — -T ggF

_ — , • — - f f ™ ^ -


• Variables can be re-used in multiple fingerprints For example, we could have:

fingerprint (cp/pakistan/agencies/ndc?) = ($NDC_terms and $procurement_terms) and not

$spam_defeats; fingerprint( cp/pakistan/angencies/ndc/testingO = $NDC_terms and (missile launch' or 'tactical radio');

TOP SECRET//COMINT

• • • • • • • •


• In the future, you can modify the variable $NDC_terms and it will automatically affect both fingerprints since they use that variable in their definition.

TOP SECRET//COMINT

• For example, take the first scenario: "I want to look for documents from Iran that mention a banned item"

• Just using keywords with Boolean equations, how could we restrict the term to only a document body and only coming from Iran?

TOP SECRET//COMINT

Context Sensitive Scanning • X-KEYSCORE s context sensitive scanning engine

allows you to explicitly say where you want a term to hit.

• As an early example, the Tech Strings in Documents capability allowed analysts to restrict terms to only Email, Chat or Documents Bodies

• The full XKS Context Sensitive Scanning engine allows for over 70 unique contexts to be used as part of an fingerprint

TOP SECRET//COMINT

Context Sensitive Scanning • For example, take the first scenario: "I want to look for documents from Iran that mention a banned item"

• Using the XKS context for Country Code (based on NKB information) and the XKS context for Document Bodies, this easily becomes:

fingerprintCdemo/scenarioi') = ccCir') and doc_body('banned item')

TOP SECRET//COMINT

Context Sensitive Scanning As another example, let s say we want to tag all Iphone usage

Using the XKS context for User Agent this easily becomes

fingerprint(<demo/scenario2') = user_agent('iphone');

TOP SECRET//COMINT

TOP SECRET STRAPi M

Sfcir-swCS.vjb AV iVVVVW »»»9 ^ f — i i —

USSID18/HRA Considerations

XKS Fingerprints may not be USSID18 or HRA compliant if they are queried on by themselves

For example, we may want to fingerprint the use of mobile web devices like the I Phone, so that attribute could be used as part of a more complex query.

But querying for the I Phone fingerprint itself would be a USSID18 and HRA violation.

TOP SECRET//COMINT

TOP SECRET STRAPi

• —-— m B i n r ^

USSID18/HRA Considerations

• But if you want to look for an IPhone user from an Iranian Proxy accessing his MaiLru account:

IP Address :

AppID (+Fingerprints) [fülltest] :

Field Builder

AppID (-1-Fingerprints)

browser/ce I Iphorie/iphoriel V

Add to Field C l o s e

Field Builder


ma i I/web mai I/ma i Ir u ni)))))))))))))))))))))))))))))))))) ni))))))))))))))))))))))))))))))))))

ma i I/web mai I/ma i Ir u

ma i I/web mai I/ma i Ir u/attach ment

ma i l/web ma i I/ma i Ir u/post

V

TOP SECRET//COMINT

Context Sensitive Scanning

What contexts are available for use in XKS Fingerprints?

TOP SECRET//COMINT

HTTP Activity Contexts (1 of 2) html_title(expr) The normalized extracted text web page titles

html titlefhow to' and 'bomb')

http_host(expr) The "Host:" name given in the http header. http_host( yahoo.com')

http_url(expr) Every URL from HTTP GET and POST commands. http_url(7mail/inbox?action=delete')

http_url_a rgs ( expr) All arguments given as part of a URL (ie. all text following the T i n a URL string) http u r 1 ('acti on=de 1 ete')

http_referer(expr) The "Referer:" URL given in the HTTP header http referer(1http://badwebsite/cp?action=show')

http_language(expr) The normalized two letter iso-6393 language code as inferred from any http and or html header info http language('fa or 'de')

TOP SECRET//COMINT

http://badwebsite/cp?action=show'

HTTP Activity Contexts (2 of 2)

http_cookie(expr) The "Cookie:" field given in the http header. httpcooki e(/ P R E F=\d\d [a-z] /)

http_server(expr) The "Server:" type name in the http header.

http_server('GWS/2.r or 'Apache')

http_user_agent(expr) The "User-Agent:" field given in the http header.

http_user_agent(/MozillaV[45]/ or 'Chrome')

web_search(expr) The normalized extracted text from web searches

web_search(<ricin' or 'plague')

x_fo rwa rd e d_for (exp r ) The X-Forwarded For IP address from the HTTP Header

x_forwarded_for('i.2.3.4')

TOP SECRET//COMINT

Protocol Contexts 1 of 2 ip(expr) The source or destination IP address of the session

ipf 127.0.0.1')

from_ip(expr) The source IP address of the session fromipC 127.0.0.1')

to_ip(expr) Every URL from HTTP GET and POST commands. to_ip(£ 127.0.0.1')

ip_subnet(expr) IP subnet in CI DR notation. ip_subnet('7.211.143.148/24')

port(expr) The source or destination TCP or UDP port number. port('22')

from_port(expr) The source TCP or UDP port number. from_port('22')

to_port(expr) The destination TCP or UDP port number. to_port('22')

TOP SECRET//COMINT

Protocol Contexts 1 of 2 cc(expr) The country (either to OR from) based on IP address

cc('ir' or 'pk')

from_cc(expr) The source country based on TP address from_cc('ir' or 'pk')

to_cc(expr) The destination country based on IP address to_cc('ir' or 'pk')

protocol(expr) The textual form of the IP next protocol, protocol ('TCP')

next_protocol(expr) The textual form of the IP next protocol. ip_nextjprotocol(' 17')

mac_address(expr) The MAC address of the target network device. macad dress ('00:16:3E:3F:BD:EF')

TOP SECRET//COMINT

TOP SECRET STRAPi

ommunication Based Contexts email_body(expr) The UTF-8 normalized text of all email bodies.

email bodyfhow to' and 'build' and ('bomb' or 'weapon')) chat_body(expr) The UTF-8 normalized text of all chat bodies.

chat_body('how to' and 'build' and ('bomb' or 'weapon'))

document_body(expr) The UTF-8 normalized text of the Office document. -Office documents include (but are not limited to) Microsoft Office, Open Office, Google Docs and Spreadsheets.

document_body('how to' and 'build' and ('bomb' or 'weapon'))

calendar_body(expr) The UTF-8 normalized text of all calendars. An example is Google Calendar. calendar_body(' wedding')

archive_files(expr) Matches a list of files from within an archive. For example is a ZIP file is transmitted, all names of files within are passed to this context. archive_files('bad.dir or 'virus.doc')

http_post_body(expr) The UTF-8 normalized text HTTP url-encoded POSTs. http post body('action=send' and 'badguy@yahoo')

Communication Based Contexts Aliases

doc_email_body(expr) This covers the email body and document_body contexts doc_email_body('how to' and 'build' and ('bomb' or 'weapon'))

communication_body(expr) This covers the email_body, documentbody arid chat_body contexts chat_body('how to' and 'build' and ('bomb' or 'weapon'))

A guide to XKS contexts can be found

TOP SECRET//COMINT

Context sensitivity Why use context-sensitive scanning?

• More intuitive - you can say what you mean • More accurate - if rmaps.google.com' is mentioned in a

blog post, you don't want to try processing it as a Google Maps session

• Better performance for XKEYSCORE

TOP SECRET//COMINT

Examples • "I want to look for people doing web searches on Jihad from

Kabul"

• Using the from_city() and web_search() context this becomes

fingerprint( clemo/scenar^') = from_city(ckabur) and web_search('jihad');

TOP SECRET//COMINT

Examples

• "I want to look for people using Mojahedeen Secrets encryption from an I Phone"

You can even use existing fingerprints in a fingerprint definition! So this becomes:

fingerprint('demo/scenario4') = fingerprint( encryption/mojahdeen2' and fingerprmt('browser/cellphone/iphone')

TOP SECRET//COMINT

4 • "I want to look for E-mails that mention words from various

categories of interest to CP"

• You can use multiple variables in an equation like this:

email_body($acwitems and $acwpositions and ($acwcountries or $acwbrokers or $aewports));

TOP SECRET//COMINT

• $acwitems = 'machine gun' or grenade or 'AK 47' • $acwpositions = 'minister of defence' or 'defense minister' • $acwcountries = 'somalia or liberia or sudan • $acwbrokers = south africa' or Serbia' or Bulgaria' • $acwports = 'rangood' or albasra' or 'dar es salam'

email_body($acwitems and $acwpositions and ($acwcountries or $acwbrokers or $acwports));

TOP SECRET//COMINT

New Fingerprint GUI New XKS Fingerprint GUI allows analysts to directly test, submit and manage fingerprints through the web

Navigation Menu

d F ingerpr in ts

£] Validate /Submit Approved

¡5] Pending f ^ l My Signatures

Uç Fingerprint Validation J Submittal

Step #1 Step #2 Step # 3 I

s o ? Compile Test Against Sess ion Data t d S a v e « Help

* M

Global Variable Declarations

Type or paste any global VARIABLE DECLARATIONS here

Signature

Type or paste a FINGERPRINT definition here.

P r e s s C o m p i l e w h e n d o n e ed i t i ng

J. ) t . 1 A

New Fingerprint GUI New XKS Fingerprint GUI allows analysts to directly test, submit and manage fingerprints through the web

unqerprint validation / submittal

SlfcfJ #1

jCcmpilej

Step ¿2

> Test Against Cession Ca:a

Step #3

Inl Step ¿2

> Test Against Cession Ca:a

Step #3

Inl uouai vanauie ueciaidiioiis

i r e s t = ' b o m b 1 or ' m i s s l e o r T ±ed';

Signature

fingerprint(Ttest/te3tlT) = email body(Stest)r

© Siucbss!

Results

SUCCESS! congratulat ions, your finge-prin: was successfully compilad!

Now use the Test but ton ta run it against the cesignated sassicn deta.

Questions?

TOP SECRET//COMINT

Syntax Rules

• The definition of the fingerprint will look like this:

fingerprintCtest/blah/something', owner =

Note the single quotes needed for the fingerprint name and owner

TOP SECRET//COMINT

Syntax Rules

Secondly every fingerprint definition must be completed by a semi-colon.

fingerprintCtest/blah/something?, owner = 'badguy ;

TOP SECRET//COMINT

Syntax Rules

Variables also must be completed by a semi-

$badguy = 'bomb' or gun or weapon' ;

fingerprintOtest/blah/something', owner = $badguy;

TOP SECRET//COMINT

Syntax Rules

• Definitions and Variables can span multiple lines

$badguy = 'bomb' or gun or weapon ;

fingerprintOtest/blah/something', owner = $badguy;

TOP SECRET//COMINT

Writing XKS Fingerprints - Electronic Frontier Foundation€¦ · 6/7/2015 · Fingerprints 101...

Documents

Transcript of Writing XKS Fingerprints - Electronic Frontier Foundation€¦ · 6/7/2015 · Fingerprints 101...