Defcon-Moscow, 23/11/13

research

Hashcracking server on generic hardware

Ivan Novikov (@d0znpp)

Cracking hashes? For what?• Web application security audits: whitebox

and blackbox• Passwords -> hashes• Crackings hashes to gain access

2009-2011 - we used CPU on our laptops2011 - first hashcracker, named “Parovoz” based on Bulldozer + HD6990

2013 – parovoz-ng – HD6990 + 2xHD7990 –up to 7xPCIEx16 cards + water cooling

History

GPU for hashcrackingNvidia or AMD ?

Win7 + gtx580 Win7+HD7970 Ubuntu+HD6990 Ubuntu+gtx560ti

www.hashcat.net/oclhashcat-plus

Just INTEGER for AMD ;)Float calculation – Nvidia

Why HD6990 faster than HD7970?

Why AMD faster?

Depends from software…

JohnTheRipper – opensource, bothoclHashcat-plus/lite – free, bin, UNIXExtreme GPU bruteforcer – bin, Win, Nvidiaonly (CUDA)

Win or Linux ?

Debian linuxoclHashcat-plus for every day usingoclHashcat-lite for single hash crackingJohnTheRipper for rules attacks, rare hashes, when hashcat not working ;)

Where we left of

Configure X11 to use a few cards at the same time (thx @defconua)

Install Catalyst driver (.run file with black magic)

Find combination of oclHashcat and Catalyst versions which can works together

Monitoring/reports/etc on unstable hardware

Common software problems

• Cheaper than server platform (starts from $5000 / platform)

• Easier to reproduce• Occupies less space• Funniest ;)

Generic hardware – for what?

First! You need a big tower!

HD6990 - $700 – not in retail nowHD7990 - $1000 – easy to buyMotherboard –from $100 (1xPCIEx16)to $500 (7xPCIEx16)

Smth else? :)

What about costs

CoolingCOOOOOLING !!!

Power supplyPOOOOOWER SUUUPLYY !!!

Common hardware problems

One motherboard – one ATX connectorEach power supply need ATX connector to work

We want to use more that one power supply but only one motherboard!

How to build multi power supply system on general hardware?

How to build multi power supply system on general hardware?

easy hack ;)

~ 500W per card (HD[6|7]990)Seller: 1300W power supply2 x HD7990 on 1300W PS and… F*ck off!!!

Why?

How to choose power supply?

1200W on power supply but…How many channels? (1/2)How many power per channel? (less than 500W ;-( )

How to choose power supply?

Default air cooling on retail cardsHD6990 – good! Works well on 99% load, stock clocks, about 87 C temp

HD7990 – bad ;( Works on 99% load, stock clocks, about 20 seconds (than 99 C temp)

Cooooooling

Water pump - $100-200 1-2 for systemWater block - $150-200 for each cardRadiator - $100-200Other supply - $150 (hoses, water, others)

Water cooling – only one way ;(

Water pump - $100-200 1-2 for systemWater block - $150-200 for each cardRadiator - $100-200Other supply - $150 (hoses, water, others)

Water cooling – only one way ;(

Aircooling• 20 seconds to 97C on HD7990• 87C on HD6990 stableWater cooling:• 60 C stable on HD6990 and HD7990 both

Water cooling – results

<- Waterblock

Radiator ->

Current:• $5000 for all – 42G MD5/s (42*10^10

hashes per second)• $119/1G MD5Target:• $9000 for all – 112G MD5/s• $80/1G MD5

Our results

Double Hawaii core card +40%Overclock of Hawaii +30%

Total about:$9000 for 216.5G MD5/s$41/1G MD5

Optimistic target

The end

Contacts:

@wallarm, @d0znpp

research