DEXCALIBURAUTOMATE YOUR ANDROID APP REVERSE

Or hooking for dummies

https://github.com/FrenchYeti/dexcalibur.git

WHO AM I ?

GEORGES-B. MICHEL

▸ @FrenchYeti

▸ yeti@0xff.ninja

▸ Software Security Evaluator at Thales

▸ Day : Reverse engineering (Android + TEE) apps

▸ HCE Payment applications, Trusted Applications, ARM binaries

▸ Night : Develop reverse / pentest / appsec tools

▸ Frida addict

Aka @FrenchYeti

EXAMPLE OF AN OBFUSCATED ANDROID APPLICATION

MOTIVATION

LET’S IMAGINE AN OBFUSCATED MULTI-DEX APPLICATIONPACKER

CLASS LOADER DEX LOADER

APP CLASSES & METHODS

Clear .dex file & JNI libs

Ciphered secondary .dex file

DECIPHER & LOAD

NATIVEFUNCTIONS

MOTIVATION

LET’S IMAGINE AN OBFUSCATED MULTI-DEX APPLICATIONPACKER

CLASS LOADER DEX LOADER

APP CLASSES & METHODS

Clear .dex file & JNI libs

Ciphered secondary .dex file

DECIPHER & LOAD

INVOKE BY REFLECTION

NATIVEFUNCTIONS

MOTIVATION

LET’S IMAGINE AN OBFUSCATED MULTI-DEX APPLICATIONPACKER

CLASS LOADER DEX LOADER

APP CLASSES & METHODS

Clear .dex file & JNI libs

Ciphered secondary .dex file

DECIPHER & LOAD

INVOKE BY REFLECTION

Ciphered JNI lib

DECIPHER & LOAD

NATIVEFUNCTIONS

WHITE BOXCRYPTO

MOTIVATION

LET’S IMAGINE AN OBFUSCATED MULTI-DEX APPLICATIONPACKER

CLASS LOADER DEX LOADER

APP CLASSES & METHODS

Clear .dex file & JNI libs

Ciphered secondary .dex file

DECIPHER & LOAD

INVOKE BY REFLECTION

Ciphered JNI lib

DECIPHER & LOAD

Class loaded from the network(NetworkClassLoader)

DOWNLOAD,DECIPHER & LOAD

JNI FUNCTIONS

NATIVEFUNCTIONS

WHITE BOXCRYPTO

MOTIVATION

LET’S IMAGINE AN OBFUSCATED MULTI-DEX APPLICATIONPACKER

CLASS LOADER DEX LOADER

APP CLASSES & METHODS

Clear .dex file & JNI libs

Ciphered secondary .dex file

DECIPHER & LOAD

INVOKE BY REFLECTION

Ciphered JNI lib

DECIPHER & LOAD

Class loaded from the network(NetworkClassLoader)

DOWNLOAD,DECIPHER & LOAD

JNI FUNCTIONS

NATIVEFUNCTIONS

WHITE BOXCRYPTO

MOTIVATION

PACKER CLASS LOADER

DEX LOADER

APP CLASSES & METHODS

Ciphered secondary .dex file

DECIPHER

INVOKE BY REFLECTION

Ciphered JNI lib

DECIPHER & LOAD

Class loaded from the network(NetworkClassLoader)

DOWNLOAD,DECIPHER & LOAD

JNI FUNCTIONS

YOU CAN HOOKONLY WHAT YOU SEE

WHAT CAN I HOOK ?

Clear .dex file & JNI libs

NATIVEFUNCTIONS

MOTIVATION

PACKER CLASS LOADER

DEX LOADER

APP CLASSES & METHODS

Clear .dex file

Ciphered secondary .dex file

DECIPHER

INVOKE BY REFLECTION

Ciphered JNI lib

DECIPHER & LOAD

Class loaded from the network(NetworkClassLoader)

DOWNLOAD,DECIPHER & LOAD

JNI FUNCTIONS

IT REQUIRES SEVERAL HOOKING SESSIONS

Clear .dex file & JNI libs

NATIVEFUNCTIONS

WHAT IS INTERESTING TO HOOK ?

MOTIVATION

THE IDEA

▸ Deobfuscate waste of time

MOTIVATION

THE IDEA

▸ Deobfuscate waste of time

▸ Manage hooks not so easy

MOTIVATION

THE IDEA

▸ Deobfuscate waste of time

▸ Manage hooks not so easy

▸ Manual tasks can be automated (start App, …)

MOTIVATION

THE IDEA

▸ Deobfuscate waste of time

▸ Manage hooks not so easy

▸ Manual tasks can be automated (start App, …)

▸ Several devices hooked simultaneously

MOTIVATION

THE IDEA

▸ Deobfuscate waste of time

▸ Manage hooks not so easy

▸ Manual tasks can be automated (start App, …)

▸ Several devices hooked simultaneously

▸ Application size explore bytecode/libs is boring

MOTIVATION

THE IDEA

▸ Show functions invoked dynamically as « xrefs »

▸ Discover automatically classes & bytecode loaded dynamically (DexFile ..)

▸ Generate hook with a single click on the function

▸ Debug a single hook while others are active

▸ Enable/disable hook without lose or pollute the source code

CHRISTMAS WISH LIST 1/2 :

THE IDEA

▸ Multi-user : share the same instrumentation with my friends

▸ Instrumente several devices and merge hook logs (Workflow / IoT)

▸ Be able to run with rooted & non-rooted devices

▸ Offer user-friendly GUI and API,

▸ Free & open-source ! ( license APACHE 2 )

CHRISTMAS WISH LIST 2/2 :

WHAT IS DEXCALIBUR ?

NOT JUST A TOOLBOX

DEX DISASSEMBLER Baksmali

WHAT IS DEXCALIBUR ?

NOT JUST A TOOLBOX

FILE IDENTIFIERS & PARSERS

DEX DISASSEMBLER Baksmali

WHAT IS DEXCALIBUR ?

NOT JUST A TOOLBOX

STATIC BYTECODE ANALYZER

DYNAMIC BYTECODE ANALYZER

FILE IDENTIFIERS & PARSERS

DEX DISASSEMBLER Baksmali

WHAT IS DEXCALIBUR ?

NOT JUST A TOOLBOX

STATIC BYTECODE ANALYZER

DYNAMIC BYTECODE ANALYZER

FILE IDENTIFIERS & PARSERS

DEX DISASSEMBLER Baksmali

INSTRUMENTATION TOOL

WHAT IS DEXCALIBUR ?

NOT JUST A TOOLBOX

STATIC BYTECODE ANALYZER

DYNAMIC BYTECODE ANALYZER

FILE IDENTIFIERS & PARSERS

MODULAR HEURISTIC & SEARCH ENGINE

DEX DISASSEMBLER Baksmali

INSTRUMENTATION TOOL

WHAT IS DEXCALIBUR ?

NOT JUST A TOOLBOX

STATIC BYTECODE ANALYZER

DYNAMIC BYTECODE ANALYZER

FILE IDENTIFIERS & PARSERS

MODULAR HEURISTIC & SEARCH ENGINE

DEVICE MANAGER & FRIDA UTILS

DEX DISASSEMBLER Baksmali

INSTRUMENTATION TOOL

WHAT IS DEXCALIBUR ?

NOT JUST A TOOLBOX

STATIC BYTECODE ANALYZER

DYNAMIC BYTECODE ANALYZER

FILE IDENTIFIERS & PARSERS

MODULAR HEURISTIC & SEARCH ENGINE

WEB SERVER & UI

DEVICE MANAGER & FRIDA UTILS

CONTROLS & CUSTOMIZE

DEX DISASSEMBLER Baksmali

INSTRUMENTATION TOOL

WHAT IS DEXCALIBUR ?

NOT JUST A TOOLBOX

STATIC BYTECODE ANALYZER

DYNAMIC BYTECODE ANALYZER

FILE IDENTIFIERS & PARSERS

MODULAR HEURISTIC & SEARCH ENGINE

WEB SERVER & UI

DEVICE MANAGER & FRIDA UTILS

IMPROVES ATRUNTIME CONTROLS &

CUSTOMIZE

DEX DISASSEMBLER Baksmali

INSTRUMENTATION TOOL

WHAT IS DEXCALIBUR ?

NOT JUST A TOOLBOX

STATIC BYTECODE ANALYZER

DYNAMIC BYTECODE ANALYZER

FILE IDENTIFIERS & PARSERS

MODULAR HEURISTIC & SEARCH ENGINE

WEB SERVER & UI

DEVICE MANAGER & FRIDA UTILS

IMPROVES ATRUNTIME CONTROLS &

CUSTOMIZE

DEXCALIBUR

DEX DISASSEMBLER Baksmali

INSTRUMENTATION TOOL

WHAT IS DEXCALIBUR ?

WHAT IS DEXCALIBUR ?

POWERED BY …

ANDROID SDK

APKTOOL +

BAKSMALI

Today

NATIVE HOOK CANNOT BE GENERATED NO BYTECODE SYMBOLIC EXEC

Functions contained into JNI/native libscan be hooked, but decompilers/analyzersdont support it. So, native hook cannot be generated.

NICE TOOLS :-)

WHAT IS DEXCALIBUR ?

POWERED BY …

ANDROID SDK

APKTOOL +

BAKSMALI

Today

NATIVE HOOK CANNOT BE GENERATED NO BYTECODE SYMBOLIC EXEC

Functions contained into JNI/native libscan be hooked, but decompilers/analyzersdont support it. So, native hook cannot be generated.

ADD NATIVE LIBRARIES SUPPORT SMALI SYMBOLIC EXEC

NICE TOOLS :-)

LIEFR2 LIEF

Tomorrow

RetDec

SMALI VM Z3 SOLVERAND MORE !

DEMO #1

HOW IT WORKS ?

HOW IT WORKS ?

1) START PHASE - FILE ANALYSIS

UNCOMPRESS APKAPK FILE

DEVICE

FILE ANALYZER

Files identified & categorized:key stores, libs, properties, xml,shared pref, cache, …

Pull Application data/data/data/xxx …

Undetected / high entropy files are tagged

notify1

3

4

Parse APK content2

HOW IT WORKS ?

1) START PHASE - ANDROID API ANALYSIS

UNCOMPRESS APKAPK FILE

ANDROID API/STUB

ApplicationGraph

Statically builtDEX

DISASSEMBLER SAST

FILE ANALYZER

3

1 2

Create appgraph

DEVICE

HOW IT WORKS ?

1) START PHASE - APPLICATION BYTE CODE ANALYSIS

UNCOMPRESS APKAPK FILE

ANDROID API/STUB

DEX DISASSEMBLER

notify

SAST

ApplicationGraph

Statically builtDEX

DISASSEMBLER SAST

FILE ANALYZER

12

4

3Update appgraph

DEVICE

HOW IT WORKS ?

2) INSTRUMENTATION PHASE - BEFORE RUN

notify

Categorized Files

Application+Android APIGraph

Statically built

DYNAMIC LOADER

BYTE ARRAY CLASSIFIER FILE ACCESS KEY STORES

…

1 MODULAR HEURISTIC ENGINE

HOW IT WORKS ?

notify

Categorized Files

Application+Android APIGraph

Statically built

DYNAMIC LOADER

NATIVE LIB / JNI

FILE ACCESS DESCRIPTORS

STREAMS

Search pattern &method

Correlate static filesBind a file to a method

KEY STORE

…

1

2

2’

MODULAR HEURISTIC ENGINE

2) INSTRUMENTATION PHASE - BEFORE RUN

HOW IT WORKS ?

Categorized Files

Application+Android APIGraph

Statically built

DYNAMIC LOADER

NATIVE LIB / JNI

FILE ACCESS DESCRIPTORS

STREAMSKEY STORE

…

HOOK MANAGER

Get methodsignature

ASK FOR INSTRUMENTATION

Generatefrida code

HOOKS

3

45

MODULAR HEURISTIC ENGINE

2) INSTRUMENTATION PHASE - BEFORE RUN

HOW IT WORKS ?

2) INSTRUMENTATION PHASE - RUNTIME

Application+Android APIGraph

Statically built

DYNAMIC LOADER

NATIVE LIB / JNI

FILE ACCESS DESCRIPTORS

STREAMSKEY STORE

…

HOOK MANAGER DEVICE

HOOKS

Starts app &deploys

Hook data : args, return, this, …

6

7Correlate graph &intercepted data8

MODULAR HEURISTIC ENGINE

HOW IT WORKS ?

Application+Android APIGraph

Statically built

DYNAMIC LOADER

NATIVE LIB / JNI

FILE ACCESS DESCRIPTORS

STREAMSKEY STORE

…

HOOK MANAGER DEVICE

HOOKS

Starts app &deploys

Hook data : args, return, this, …

6

7Correlate intercepted data8

Push discovered elements & tag node9

MODULAR HEURISTIC ENGINE

2) INSTRUMENTATION PHASE - RUNTIME

« HEY ! GIVE ME THE MOST COMPLETE PICTURE OF THE APPLICATION »

DRAW A COMPLETE PICTURE OF THE APPLICATION

MIX * ANALYSIS WITH INSTRUMENTATION RESULTS

GRAPHSSTATIC

ANALYSIS

ANDROIDINTERNALS

CALLSSTATIC VALUES

DRAW A COMPLETE PICTURE OF THE APPLICATION

GRAPHSSYMBOLIC

VALUESSTATIC

ANALYSIS

DYNAMIC ANALYSIS

ANDROIDINTERNALS

CALLSSTATIC VALUES

SOLVE CONSTRAINT

…

MIX * ANALYSIS WITH INSTRUMENTATION RESULTS

DRAW A COMPLETE PICTURE OF THE APPLICATION

GRAPHSSYMBOLIC

VALUESSTATIC

ANALYSIS

DYNAMIC ANALYSIS

ANDROIDINTERNALS

CALLSSTATIC VALUES

FILE ANALYSIS

KEYSTORES

PROPERTIES

LIBS & DEX

…

STRUCTURESSOLVE CONSTRAINT

…

MIX * ANALYSIS WITH INSTRUMENTATION RESULTS

DRAW A COMPLETE PICTURE OF THE APPLICATION

GRAPHSSYMBOLIC

VALUES

PARAMS & RETURNS

VALUES

STATIC ANALYSIS

DYNAMIC ANALYSIS

DYNAMIC INSTRUMENTATION

ANDROIDINTERNALS

CALLSSTATIC VALUES

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

FILE ANALYSIS

KEYSTORES

PROPERTIES

LIBS & DEX

…

STRUCTURESSOLVE CONSTRAINT

…

MIX * ANALYSIS WITH INSTRUMENTATION RESULTS

CASE #1 DYNAMIC UPDATE OF XREF WITH INVOKED METHODS

METHOD INVOKED DYNAMICALLY

‣ Method.invoke()

‣ Class.getMethod()

From a static point-of-view only two methods are called :

Smali code

DYNAMIC UPDATE OF « XREF FROM »WITH INVOKED METHODS

DYNAMIC UPDATE OF « XREF FROM »WITH INVOKED METHODS

DYNAMIC UPDATE OF XREF WITH INVOKED METHODS

GRAPHS

PARAMS & RETURNS

VALUES

STATIC ANALYSIS

DYNAMIC INSTRUMENTATION

ANDROIDINTERNALS

CALLSSTATIC VALUES

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

DYNAMIC UPDATE OF « XREF FROM »WITH INVOKED METHODS

DYNAMIC UPDATE OF XREF WITH INVOKED METHODS

GRAPHS

PARAMS & RETURNS

VALUES

STATIC ANALYSIS

DYNAMIC INSTRUMENTATION

ANDROIDINTERNALS

CALLSSTATIC VALUES

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

REFLECTION API INSTRUMENTED

DYNAMIC UPDATE OF « XREF FROM »WITH INVOKED METHODS

DYNAMIC UPDATE OF XREF WITH INVOKED METHODS

GRAPHS

PARAMS & RETURNS

VALUES

STATIC ANALYSIS

DYNAMIC INSTRUMENTATION

ANDROIDINTERNALS

CALLSSTATIC VALUES

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

REFLECTION API INSTRUMENTEDSTART APP

DYNAMIC UPDATE OF « XREF FROM »WITH INVOKED METHODS

DYNAMIC UPDATE OF XREF WITH INVOKED METHODS

GRAPHS

PARAMS & RETURNS

VALUES

STATIC ANALYSIS

DYNAMIC INSTRUMENTATION

ANDROIDINTERNALS

CALLSSTATIC VALUES

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

REFLECTION API INSTRUMENTEDSTART APP

HOOK TRIGGED

DYNAMIC UPDATE OF « XREF FROM »WITH INVOKED METHODS

DYNAMIC UPDATE OF XREF WITH INVOKED METHODS

GRAPHS

PARAMS & RETURNS

VALUES

STATIC ANALYSIS

DYNAMIC INSTRUMENTATION

ANDROIDINTERNALS

CALLSSTATIC VALUES

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

REFLECTION API INSTRUMENTEDSTART APP

HOOK TRIGGED

HOOK GATHERS METHOD INFO

DYNAMIC UPDATE OF « XREF FROM »WITH INVOKED METHODS

DYNAMIC UPDATE OF XREF WITH INVOKED METHODS

GRAPHS

PARAMS & RETURNS

VALUES

STATIC ANALYSIS

DYNAMIC INSTRUMENTATION

ANDROIDINTERNALS

CALLSSTATIC VALUES

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

REFLECTION API INSTRUMENTEDSTART APP

HOOK TRIGGED

HOOK SHOWS STACK TRACE

HOOK GATHERS METHOD INFO

DYNAMIC UPDATE OF « XREF FROM »WITH INVOKED METHODS

DYNAMIC UPDATE OF XREF WITH INVOKED METHODS

GRAPHS

PARAMS & RETURNS

VALUES

STATIC ANALYSIS

DYNAMIC INSTRUMENTATION

ANDROIDINTERNALS

CALLSSTATIC VALUES

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

REFLECTION API INSTRUMENTEDSTART APP

HOOK TRIGGED

HOOK SHOWS STACK TRACE

HOOK GATHERS METHOD INFO

HEURISTIC ENGINE UPDATE DB

BEFORERUNTIME

DYNAMIC UPDATE OF THE CALL GRAPH

METHOD INVOKED DYNAMICALLY

BEFORERUNTIME

AFTER RUNTIME

METHOD INVOKED DYNAMICALLY

DYNAMIC UPDATE OF THE CALL GRAPH

UPDATE OF THE CALL GRAPH

Green nodes are internal Android or Java methods

Pink node are invoked dynamically and not discovered statically

Gray nodes have been discovered statically

DYNAMIC UPDATE OF THE CALL GRAPH

DEMO #2

DYNAMIC UPDATE OF XREFS WITH INVOKED METHODS

DYNAMIC UPDATE OF THE CALL GRAPH

CASE #2 ANALYZE DEX FILE LOADED DYNAMICALLY

ANALYZE DEX FILE LOADED DYNAMICALLY

ANALYZE DEX FILE LOADED DYNAMICALLY

PARAMS & RETURNS

VALUES

DYNAMIC INSTRUMENTATION

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

FILE ANALYSIS

LIBS & DEX

CLASS GRAPH

STATIC ANALYSIS

ANDROIDINTERNALS

CALLS

ANALYZE DEX FILE LOADED DYNAMICALLY

ANALYZE DEX FILE LOADED DYNAMICALLY

PARAMS & RETURNS

VALUES

DYNAMIC INSTRUMENTATION

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

FILE ANALYSIS

LIBS & DEX

CLASS GRAPH

STATIC ANALYSIS

ANDROIDINTERNALS

CALLS

DEX LOADING API INSTRUMENTEDSTART APP

ANALYZE DEX FILE LOADED DYNAMICALLY

ANALYZE DEX FILE LOADED DYNAMICALLY

PARAMS & RETURNS

VALUES

DYNAMIC INSTRUMENTATION

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

FILE ANALYSIS

LIBS & DEX

CLASS GRAPH

STATIC ANALYSIS

ANDROIDINTERNALS

CALLS

DEX LOADING API INSTRUMENTED

DEXFILE CONSTRUCTORS TRIGGED

START APP

ANALYZE DEX FILE LOADED DYNAMICALLY

ANALYZE DEX FILE LOADED DYNAMICALLY

PARAMS & RETURNS

VALUES

DYNAMIC INSTRUMENTATION

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

FILE ANALYSIS

LIBS & DEX

CLASS GRAPH

STATIC ANALYSIS

ANDROIDINTERNALS

CALLS

DEX LOADING API INSTRUMENTED

DEXFILE CONSTRUCTORS TRIGGED

START APP

HOOKS ASK IF DEX FILES ARE ALREADY KNOWN

Dex File already analyzed ?

ANALYZE DEX FILE LOADED DYNAMICALLY

ANALYZE DEX FILE LOADED DYNAMICALLY

PARAMS & RETURNS

VALUES

DYNAMIC INSTRUMENTATION

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

FILE ANALYSIS

LIBS & DEX

CLASS GRAPH

STATIC ANALYSIS

ANDROIDINTERNALS

CALLS

DEX LOADING API INSTRUMENTED

DEXFILE CONSTRUCTORS TRIGGED

START APP

HOOKS ASK IF DEX FILES ARE ALREADY KNOWN

Dex File already analyzed ?

COPY OR GET DEX FILE

ANALYZE DEX FILE LOADED DYNAMICALLY

ANALYZE DEX FILE LOADED DYNAMICALLY

PARAMS & RETURNS

VALUES

DYNAMIC INSTRUMENTATION

DATA READ/WRITE SECONDARY

DEX & LIBS

STACK TRACERUNTIME CONTEXT

FILE ANALYSIS

LIBS & DEX

CLASS GRAPH

STATIC ANALYSIS

ANDROIDINTERNALS

CALLS

DEX LOADING API INSTRUMENTED

DEXFILE CONSTRUCTORS TRIGGED

START APP

HOOKS ASK IF DEX FILES ARE ALREADY KNOWN

Dex File already analyzed ?

COPY OR GET DEX FILEDECOMPILE DEX & UPDATE DB

ANALYZE DEX FILE LOADED DYNAMICALLY

CASE #3 BYTECODE CLEANER

BYTECODE CLEANER

BYTE CODE CLEANER : REMOVE NOP

BEFORE

BYTECODE CLEANER

BYTE CODE CLEANER : REMOVE NOP

BEFORE AFTER

REMOVE USELESS GOTO

BEFORE

BYTECODE CLEANER

REMOVE USELESS GOTO

BEFORE AFTER

BYTECODE CLEANER

DEXCALIBUR - NEXT STEPS

IMPROVEMENTS

‣ Use my own customizable Dex Decompiler (or use LIEF)?

‣ Add r2 binding and native hooks

‣ HTTP communications & Intent grabbing

‣ Bytecode & native symbolic exec (Z3) ?

‣ Bytecode emulation (SmaliVM @CalebFenton)?

‣ Offers native instruction hooking (QBDI)?

‣ And fuzz (afl-fuzz params + feedback given by hooking)?

DEXCALIBUR

Thanks

Q&A

ANNEXES

HOW TO INSTALL ?

HOW TO INSTALL ?

‣ Ensure you have the requirements (Frida, NodeJS, apktool)

‣ Or install from DockerHub

git clone https://github.com/FrenchYeti/dexcalibur.git

cd dexcalibur

npm install

docker pull frenchyeti/dexcalibur

docker run -it \

-v <workspace>:/home/dexcalibur/workspace \

-p 8080:8000 —dev=<device> \

frenchyeti/dexcalibur

DEXCALIBUR - WHAT IS IT ?

SEARCH BYTE ARRAY