Ruby application based on http

Ruby application based on http

www.ekohe.comWeb Development & Graphic DesignChina Ruby on Rails Development - Rails Consulting - Rails Services - Merb - Offshore Web Development

Who am I?


Richard Huang a.k.a flyerhzm

Work at Ekohe

http://huangzhimin.com

@flyerhzm

http://github.com/flyerhzm

Http Server


Oauth

JSONXML

Params Signature

Web Service

RestfulCloud Computing

HTML XHTML

Http protocol


Request - Response

Header - Body

Client(Browser)

RubyConfChinaServer

Get /session/new

Return session/new page

Post /session

Login success, redirect to /

Get /

Return home page

Ruby API? Official? Community?


Get data from third-party website


if the website provides an api if the website provides an official/community ruby api use the ruby api else write the ruby api by your self endelse write the ruby codes act as a http client (browser)end

What if fetch contacts from gmail?

What if fetch timelines from twitter?

What if fetch friends from facebook?

What if fetch contacts from msn?

What if fetch repositories from github?

Lucky, there are ruby api to do theses!


What if fetch contacts from 163 mail?

What if fetch weather from sina?

What if fetch friends from kaixin001?

What if fetch contacts from fetion?

There are no ruby api to do theses!


Act as a http client (browser)

Analyze the packets sent by http client (browser) in sniffer tools.

Resend same packets by Ruby.


Sniffer


WiresharkDetect any protocol (ssh, ftp, tcp, http, …)

Cross platform (Windows, Linux, Mac OS)

Both raw data and printable text

Not easy to view https encrypted data


Wireshark


filter http one http request

TCP/IPHTTP

HTTP Header/Body

Raw Data andPrintable text

Wireshark


one http response

HTTP AnalyzerOnly detect http/https protocol

Concentrated on http protocol (request timing, cookie, …)

Easy to view https encrypted data

Windows version only


HTTP Analyzer


http request header

http response header

HTTP Analyzer


http response

body

HTTP Analyzer


http request timing

Get session/new


Get /session/new HTTP/1.1\r\nHost: rubyconfchina.org\r\nConnect: keep-alive\r\nUser-Agent: Mozilla/5.0 AppleWebKit/553.4\r\nAccept: text/html,application/xhtml+xml,text/html;q=0.9\r\nAccept-Encoding: gzip,deflate\r\nAccept-Language: en-US,en;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n\r\n

Get session/new response


HTTP/1.1 200 OK\r\nServer: nginx/0.5.32\r\nDate: Sun, 20 Jun 2010 09:25:08 GMT\r\nContent-Type: text/html; charset=utf-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nCache-Control: private, max-age=0, must-revalidate\r\nContent-Encoding: gzip\r\n\r\n<!DOCTYPE html PUBLIC …>\n<html>\n…

Post /session


Post /session HTTP/1.1\r\nHost: rubyconfchina.org\r\nConnect: keep-alive\r\nUser-Agent: Mozilla/5.0 AppleWebKit/553.4\r\nReferer: http://rubyconfchina.org/session/new\r\nContent-Length: 106\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept: text/html,application/xhtml+xml,text/html;q=0.9\r\n\r\nauthenticity_token=…&login=flyerhzm&password=…&commit=Log+in

Post /session response


HTTP/1.1 302 Moved Temporarily\r\nServer: nginx/0.5.32\r\nDate: Sun, 20 Jun 2010 09:25:31 GMT\r\nContent-Type: text/html; charset=utf-8\r\nConnect: keep-alive\r\nLocation: http://rubyconfchina.org/\r\nCache-Control: no-cache\r\nSet-Cookie: auth_token=; _euruko_session=BAh7CTo\r\nContent-Length: 91\r\n\r\n<html><body>You are being <a href=http://rubyconfchina.org/>redirected</a>.</body></html>

Get /


Get / HTTP/1.1\r\nHost: rubyconfchina.org\r\nConnect: keep-alive\r\nUser-Agent: Mozilla/5.0 AppleWebKit/553.4\r\nAccept: text/html,application/xhtml+xml,text/html;q=0.9\r\nAccept-Encoding: gzip,deflate\r\nAccept-Language: en-US,en;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nCookie: auth_token=; _euruko_session=BAh7CTo\r\n\r\n

Http protocol


HTTP/1.1 200 OK\r\nServer: nginx/0.5.32\r\nDate: Sun, 20 Jun 2010 09:25:08 GMT\r\nContent-Type: text/html; charset=utf-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nCache-Control: private, max-age=0, must-revalidate\r\nContent-Encoding: gzip\r\n\r\n<!DOCTYPE html PUBLIC …>\n<html>\n…

Resend same packets by Ruby



Send a GET request


uri = URI.parse("http://rubyconfchina.org/session/new")http = Net::HTTP.new(uri.host, uri.port)headers = { "User-Agent" => "Mozilla/5.0 AppleWebKit/553.4", "Accept" => "…"}response = http.request_get(uri.request_uri, headers) puts response.body =~ /Login/ # 1942


Parse GET response


raise Exception, "get response should success" unless Net::HTTPSuccess === repsonse

if response.body =~ %r(<input name="authenticity_token".*?value="(.*>)" />)m authenticity_token = $1end


Send a POST request


uri = URI.parse("http://rubyconfchina.org/session")http = Net::HTTP.new(uri.host, uri.port)body = "authenticity_token=#{authenticity_token}&login=flyerhzm&password=…"headers = { "User-Agent" => "Mozilla/5.0 AppleWebKit/553.4", "Accept" => "…"}response = http.request_post(uri.request_uri, body, headers)


Parse POST response


raise Exception, "get response should be redirect" unless Net::HTTPRedirection === repsonse

cookie = response["Set-Cookie"]cookie = parse_cookie(cookie)Location = response["Location"]


Send a GET request


uri = URI.parse(Location)http = Net::HTTP.new(uri.host, uri.port)headers = { "User-Agent" => "Mozilla/5.0 AppleWebKit/553.4", "Accept" => "…", "Cookie" => cookie}response = http.request_get(uri.request_uri, headers) raise Exception, "get repsonse should success" unless Net::HTTPSuccess === responseputs response.body =~ /Login/ # nil

Facility


MechanizeA ruby library that makes automated web interaction

easy.

Automatically location redirection

Automatically cookies management


Mechanize


a = Mechanize.new do |agent| agent.user_agent_alias = 'Linux Firefox'enda.get('http://rubyconfchina.org/session/new') do |page| puts page.body =~ /Login/ # 1942 home_page = page.form_with(:action => '/session') do |f| f.login = 'flyerhzm' f.password = '…' end.click_button puts home_page.body =~ /Login/ # nilend

Test


TestTest http requests

Test http responses


Test request


Client Server

GET /ht/sd.aspx?i=1 HTTP/1.1\r\nHost: 221.176.31.39\r\n…\r\n\r\nR fetion.com.cn SIP-C/4.0F: 730020377I: 1Q: 1 RCN: 19D28D4978125CAA4F6E54277BA7D9EFCL: type="pc" ,version="3.6.2020"

SIPP

sid = 730020377i = 0

Test request


@fetion.stubs(:sid).returns(730020377)@fetion.stubs(:i).returns(0)@fetion.stubs(:guid).returns("19D28D4978125CAA4F6E54277BA7D9EF")

expected_sipc_message =<<-EOFR fetion.com.cn SIP-C/4.0F: 730020377I: 1Q: 1 RCN: 19D28D4978125CAA4F6E54277BA7D9EFCL: type="pc" ,version="3.6.2020"

SIPPEOFexpected_sipc_message.gsub!("\n", "\r\n").chomp!SipcMessage.register_first(@fetion).should == expected_sipc_message

Copied from Sniffer tool

Test response


Client Server

HTTP/1.1 200 OK\r\nServer: nginx/0.5.32\r\n…\r\nSet-Cookie: ssic=CBIOAAAm+FiuQgpcnFi; path=/\r\n\r\n<?xml version="1.0" encoding="utf-8" ?><results status-code="200"> <user uri="sip:[email protected];p=6907" /></results>

FakewebA test helper for faking responses to web requests.

No need to modify existing code

No need to write extensive stubs


Test response


FakeWeb.register_uri( :get, 'https://uid.fetion.com.cn/ssiportal/SSIAppSignInV4.aspx? mobileno=15800681509&digest=79cd56b93f21298dc8ae9d26de1258e3d', :body => %Q|<?xml version="1.0" encoding="utf-8" ?><results status-code="200"><user uri="sip:[email protected];p=6907" /></results>|, :set_cookie => Q|ssic=CBIOAAAm+FiuQgpcnFi; path=/|)@[email protected]_code.should == "200"@fetion.sid.should == "730020377"


Test response


FakeWeb.register_uri( :get, 'https://uid.fetion.com.cn/ssiportal/SSIAppSignInV4.aspx? mobileno=15800681509&digest=79cd56b93f21298dc8ae9d26de1258e3d', :body => %Q|<?xml version="1.0" encoding="utf-8" ?><results status-code="401" desc="password error" />|, :status => ['401', 'password error'])lambda { @fetion.login }.should raise_exception(Fetion::PasswordError)


Test ProcessDo some stubs

Copy the http request/response as expected message from sniffer tools

Execute the method to generate http request/respones

Test with expected message

TDD (optional)


Performance


net-http-persistenta thread-safe wrapper for Net::HTTP that performs

persistent connections for you


user system total real

Net::HTTP 8.410000 2.400000

10.810000

( 17.333671)

Net::HTTP::Persistent

8,110000 0.880000

8.990000 ( 12.190094)

typhoeusParallel http requests

based on libcurl and libcurl-multi

500ms response, call 20 times


user system total real

Net::HTTP 0.030000 0.010000 0.040000

( 10.054327)

typhoeus 0.020000 0.070000 0.090000

( 0.508817)

Q&A

Thank you


Ruby application based on http

Technology

Transcript of Ruby application based on http