Continuous Delivery for Linux/Windows/Hadoop...Beta Cluster Hadoop JobTracker Jenkins Slave Hadoop...

Continuous Delivery for Linux/Windows/Hadoop

Wisely Chen

Wednesday, October 23, 13

http://www.cloudbees.com/jenkins/juc2013/juc2013-palo-alto-abstracts.cb#WiselyChen




Agenda

• Background

• Problem

• Solution

• Demo

• Q & A


Who I am

• Wisely Chen ( [email protected] )

• Release manager of Yahoo! Taiwan shopping and data team

• Enjoys promoting open source tech

• Hadoop Summit 2013 San Jose

• Ruby and Rails (All in Taiwan): Coscup 2006, Ubisunrise 2007, OSDC 2007

• Release Practice (All in Taiwan): Webconf 2013, Coscup 2012,2013, PHPConf 2012 , RubyConf 2012


mailto:[email protected]

mailto:[email protected]

Team

Sanching YuWindows

Neal LeeHadoop


Yahoo! Taiwan Shopping

• Top B2C site in Taiwan

• Monday.com acquired in 2008

• 8 engineering teams

• 15K files changed monthly

• 100K lines of code changed monthly


PlatformBackend

(Window)Frontend(Linux)

Data(Hadoop)

Catalog

Mobile

Member

Payment

Cash Flow

CMS

Logistic

Supplier

Machine Learning

CRM

Business Intelligence

Data Mart


Problem

FrontendTeam

BackendTeam

DB

DataTeam

HadoopWebService API


We need a multi-platform build and release solution


Solution

• Create B&R process

• Build CI infrastructure

• Automate everything


DeployCheckPoint

PerfTest

IntegrationTest

UnitTest

Build and Release Process

CommitUATTest


DeployCheckPoint

PerfTest

IntegrationTest

UnitTest

Hadoop Build and Release Process

CommitUATTest


Original DifferenceFrontend Team Backend Team Data Team

Process Scrum Waterfall Waterfall

OS Linux Windows Linux

Language PHP/JAVA .NET Pig/MapReduce

Issue Tracking Bugzilla Custom System N/A

Repos Subversion VSS N/A

CI N/A N/A N/A

Int Test N/A N/A N/A

Release Management

Routine RE Another RE SDE


Multiplatform Complexity Reduction

Frontend Team Backend Team Data Team

Process Scrum(not finish yet)Scrum(not finish yet)Scrum(not finish yet)

OS Linux Windows Linux

Language PHP/JAVA .NET Pig/MapReduce

Issue Tracking BugzillaBugzillaBugzilla

Repos Subversion(migrate to git)Subversion(migrate to git)Subversion(migrate to git)

CI JenkinsJenkinsJenkins

Int Test SeleniumSeleniumSelenium

Release Management

Delicated Release ManagerDelicated Release ManagerDelicated Release Manager


Communication Cost

巨大Huge Large


Solution





WebSite CI InfrastructureJenkins Master

WindowsJenkins Slave

Farm

SeleniumServer Farm

LinuxJenkins Slave

Farm

Beta Server Farm

WindowsFarm

LinuxFarm

Staging Server Farm

WindowsFarm

LinuxFarm


Hadoop CI Infrastructure

Jenkins Master

Alpha

Jenkins Slave

Beta Cluster

HadoopJobTracker

JenkinsSlave Hadoop

nodeHadoop

nodeHadoop

nodeHadoop

nodeSlave Node

Prod. ClusterGateway

PigServer


DeployCheckPoint

PerfTest

IntegrationTest

UnitTest


CommitUATTest


PigUnit

• A simple xUnit framework

• No cluster set up is required in local mode

• We change default PigUnit behavior

• All PigUnit tests run on single pig server

• Speed up PigUnit


DeployCheckPoint

PerfTest

IntegrationTest

UnitTest


CommitUATTest


Vaidya

• Rule based performance diagnosis of M/R jobs

• Extendable framework

• You can add your own rules

• Write complex rules using existing rules


Vaidya Performance Test

Pig Job

Pig Job Conf

Sampling Data



Pig JobPig Job History

Pig Job Conf

Sampling Data




Vaidya

VaidyaRulePig Job

Conf

Sampling Data




Vaidya

VaidyaRulePig Job

Conf

NotifyUser

Sampling Data




Vaidya

VaidyaRulePig Job

Conf

NotifyUser

Performance Result

Next CI Stage

Sampling Data


<Diagnos)cTest>

<Title><![CDATA[Balanaced Reduce Partitioning]]></Title>

//////<ClassName>//////////<[CDATA[org.apache......BalancedReducePar))oning]]>///////</ClassName>///////<Descrip)on>///////////<![CDATA[This/rule/tests/........]]>///////</Descrip)on>///////<Importance><![CDATA[High]]></Importance>///////<SuccessThreshold><![CDATA[0.20]]></SuccessThreshold>///////<Prescrip)on><![CDATA[advice]]></Prescrip)on></Diagnos)cTest>

See/if/the/reduce/job/is/balanced/or/not/

Vaidya Rule Example

Rule/Importance

Diagnosis/success/threshold


Solution





DeployCheckPoint

PerfTest

IntegrationTest

UnitTest


CommitUATTest


Before Deploy

• Check bug status

• Documention

• Create a git tag

• Auto doc generating on bugzilla

• Deploy to production cluster

• Change bug status


Auto comment in bugzilla

Repos urlRelease Note

Issue statuschange


Auto create git tag

Release Note[Bug xxx] log....

Git Tag


Application Logic Monitor

0

25

50

75

100

2-4 4-6 6-8 8-10 10-12 12-14 14-16 16-18 18-20 20-22 22-24 24-2

Today Same day in last week Same day in last month

Monitor today’s sales revenue with baseline

Notify user of something wrong

SalesNum

Time


Result


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Time

11Q4: Original

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


11Q4: Original

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

High Bug Rate(72%)

Low Throughput


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


11Q4: Original

0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


12Q1 : Create B&R Process...


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3



巨大 bug rate drop


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3




0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


12Q2 : Build CI Infrastructure...


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3




0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3



Dramatic throughput increase

Bug rate down to 22%


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3




0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


13Q1 : Automate Everything...


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


13Q2 : Automate Everything...


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


13Q3: Automate Everything...


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3



Deploy time decrease


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3




Hadoop CI Demo

• Demo1 : Unit test

• Demo2 : Performance test

• Demo3 : Deploy


Conclusion

• This process can reduce bug rate

• This process can increase productivity

• Jenkins can fit in any platform


謝謝大家, 祝你有美好的一天


Appendix


0

22.5

45

67.5

90

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

ReleaseBugRate ReleaseThroughput DeployCostTime

On Demand Bi-weekly Weekly


Web Site CI Process

Backend3 Team

(Window)

Cash Flow

Frontend2 Team(Linux)

Catalog

Mobile

Member

Payment

Logistic

Supplier

CMS

Data2 Team

(Hadoop)MachineLearning

BusinessIntelligence

Data Mart

CRM


Build and release processPHP CodeCommit

PHP Unit Test

DeployLinux Beta

BetaAuto Test

Deploy PHP Staging

StagingAuto Test

StagingUAT

Check Test Status

Release Meeting

Code Freeze

Release

.NET CodeCommit

.NET Unit Test

DeployWin Beta

Deploy .NET Staging

Deploy PHP Prod

Deploy .NET Prod

ProdAuto/UAT

DONE


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

Release Bug Rate Release Throughput Deploy Cost Time

Throughput increased dramaticly


Build CI infrastructure


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


Significant bug rate drop

12Q1 : Create B&R Process


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


Throughput increased dramaticly


Build CI infrastructure


0

25

50

75

100

11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3


Deploy cost time drop

Automate Everything


Two B&R ProcessesFrontend

Team

BackendTeam

DataTeam

Website Process

Data Process


Automate Everything

Check PointScript

Bugzilla

Source ReposLog

DeployScript

Notify UserScript

Bug List Yes

No

Check Bug Status

Auto Manual

JenkinsStyle/Static/Similarity Analysis


Continuous Delivery for Linux/Windows/Hadoop...Beta Cluster Hadoop JobTracker Jenkins Slave Hadoop...

Documents

Transcript of Continuous Delivery for Linux/Windows/Hadoop...Beta Cluster Hadoop JobTracker Jenkins Slave Hadoop...