Post on 15-Jul-2015
THE CURIOUS CASE OF PROTOBUFS…
De-mystifying Google’s hottest binary protocol
Prasanna KanagasabaiJovin Lobo
Agenda
Introduction.
Anatomy of Protobufs Defining Message formats in .Proto files.
Protobuf compiler
Python API to read write messages.
Encoding Scheme
Problem Statement.
Decoding like-a-pro with IronWasp ‘ProtobufDecoder’.
About us
Prasanna Kanagasabai :
Information Security Pentester @ ThoughtWorks Pune.
Speaker @ Nullcon , Clubhack, IIT Guwahati, Etc ..
Moderator of null Pune.
Author of IronSAP @ Automated SAP Testing tool
Passionately love pen-testing web applications.
Jovin Lobo:
Associate Consultant @ Aujas (www.aujas.com)
Proud member of null- The Open Security Community (www.null.co.in)
Author of GameOver – Linux distro for learning web security.
Spoken at nullCon, GNUnify before.
Introduction:
Protocol Buffers a.k.a Protobufs :
Protobufs are Google's own way of serializing structured data .
Extensible, language-neutral and platform-neutral .
Smaller, faster and simpler to implement.
Java, C++ and Python
Anatomy:
Over view :
Defining a .Proto file.
#> less Example.protomessage Conference {
required string conf_name = 1 ;
required int32 no_of_days = 2 ;
optional string email = 3 ;
}
// * 1,2,3 are unique tags. These are used by the fields in binary encoding.* For optimization use tags from 1-15 as higher nos. will use one more byte to encode.
Compiling
Syntax:
protoc –I=$_input_Dir --python_out=$_out_Dir$_Path_ProtoFile
Eg:
protoc –I=. --python_out=. Example.proto
This will generate a Example_pb2.py file in the specified destination directory.
$ProtoFile_pb2.py
The Protobuf compiler generates special descriptors for all your messages, enums, and fields.
It also generates empty classes, one for each message type:
Eg:
Reading and writing messages using the Protobuf binary format :
SerializeToString() serializes the message and returns it as a string.
ParseFromString(data)
parses a message from the given string.
Why Does one Reverse Protobuf It is needed to see the content of payload in
transit.
Any Modifications to payload by changing the stream could break the payload.
Demo: Protobuf… how it wrks
Encoding.
example2.proto
message Ex1 {
required int32 num = 1; // field tag
}
Code snippet:
obj = example2_pb2.Ex1();
obj.num = 290; // field value
obj.SerializeToString();
Output :
08 A2 02 #hex
000010001010001000000010 #binary
Problem statement.
This is what freaked him out
08 A2 02
Lets Decode it ..
Step 1 :
Find the wire type .
Step 2:
Find the field number.
Step 3: Find the field tag.
Step1: finding wire type.
Convert the payload to binary
0000 1000 1010 0010 0000 0010
To find wire type take the first byte: 0000 1000 1010 0010 0000 0010
[0]000 1000 Drop MSB from First byte.
0001 000 The last 3 bits give wire type.
Wire type is 000
type = 0 is Varint.
Wire types
Source : https://developers.google.com/protocol-buffers/docs/encoding
Step 2: Field tag.
What we already have is 0001000
Now we right shift value by 3 bits and the remaining bits will give us the field tag.
0001000
0001 000
‘0001 ‘ i.e. ‘ 1’
So we get the field tag = 1
Step 3: Find the field value
0000 1000 1010 0010 0000 0010
We drop the 1st byte
1010 0010 0000 0010
Drop the MSB’s from each of these bytes
1010 0010 0000 0010
010 0010 000 0010
Reverse these bytes to obtain the field value. 000 0010 010 0010
000 0010 010 0010 i.e 256 + 32 + 2 = 290
So we finally get the value of the field = 290.
So we successfully decoded
example2.proto
message Ex1 {
required int32 num = 1;
}
Code snippet:
obj = example2_pb2.Ex1();
obj.num = 290;
obj.SerializeToString();
Output :
08 A2 02 #hex
000010001010001000000010 #binary
We successfully Decoded Value : “290”
Demo : Lets do this live
Automating all this with IronWasp Protobuf Decoder:
About IronWasp : IronWasp is an open-source web security scanner.
It is designed to be customizable to the extent where users can create their own custom security scanners using it.
Website : www.ironwasp.org
Plugin : https://bitbucket.org/RandomSecurity/protobuf-decoder
Demo
RoadMap
Add additional Wire types
pk@null.co.in