Improving Engineering Processes using Hudson - Spark IT 2010
-
Upload
arun-gupta -
Category
Technology
-
view
4.031 -
download
1
description
Transcript of Improving Engineering Processes using Hudson - Spark IT 2010
![Page 1: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/1.jpg)
<Insert Picture Here>
Improving Engineering ProcessUsing Hudson
Arun Gupta, Java EE & GlassFish GuyOracle Corphttp://blogs.sun.com/arungupta, @arungupta
![Page 2: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/2.jpg)
Never send a human to do a machine's job.
![Page 3: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/3.jpg)
Throughout human history, we have been dependent on machines to survive.
![Page 4: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/4.jpg)
Rise of Continuous Integration
Offload from people, push to computers
$
Time
computers
people
![Page 5: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/5.jpg)
Spend more CPU power to help you
●First on your laptops and workstations● IDEs are at the forefront
●And then to the servers● a.k.a. “Continuous Integration”● More frequent build/test executions● Static code analysis tools● And more to come
![Page 6: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/6.jpg)
Hudson (http://hudson-ci.org)
● Open-source CI server● Emphasis on ease of installation and use
● “java -jar hudson.war” execution● Or your OS-specific package● Configure everything from browser
● Extensibility● 230+ community-developed public plugins● By 230+ contributors
● Estimated 13,000 installations
![Page 7: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/7.jpg)
"Das
hboard [H
udson]"
![Page 8: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/8.jpg)
Basic Feature Set
● Hudson notices that there is a code change● Checks out the source code
●CVS, Subversion, ClearCase, Mercurial, Accurev, Perforce, Git, StarTeam, …
● Builds Ant, Maven, shell script, NAnt, …● Record and publish results● Tool Integration
● Findbugs, Corbertura, ...
![Page 9: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/9.jpg)
Provides Feedback
● RSS feeds
● Email/IM notification● Tray application to notify developers
● Linux/Windows (Toolbar), Mac (Growl)
● . . .
![Page 10: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/10.jpg)
Demo
![Page 11: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/11.jpg)
Why do I care?●Life before Hudson
● Dev forgets to commit a file● Nightly build breaks, Testing blocked● Resources not utilized efficiently
●Life after Hudson● Hudson will tell you in 5 mins if a build broke● No one else notices that you forgot to commit a
file ;-)
![Page 12: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/12.jpg)
●Life before Hudson● Dev makes a change Monday AM● Test nightly finds a bug Monday night● Dev fixes it Tuesday● Again, under utilization of resources
Why do I care?
●Life after Hudson● Dev makes a change Monday AM● Hudson finds a regression few mins later● Dev can fix it before lunch
![Page 13: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/13.jpg)
●Life before Hudson● Test runs every night, results in e-mail● Excitement fades after 1 week● Regressions go unnoticed until it’s too late
Why do I care?
●Life after Hudson● Tests run Hudson after every commit● E-mail sent out only when tests start failing
● So it manages to keep people’s attention
![Page 14: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/14.jpg)
Matrix projects●Run the same thing on
different environments● JDKs● Databases● OSes● App servers● XML parsers● ...
●Results aggregated
![Page 15: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/15.jpg)
FindBugs integration
![Page 16: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/16.jpg)
Tracking changes
16
![Page 17: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/17.jpg)
… and more Browse workspaces Build time trend report
![Page 18: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/18.jpg)
Inter-Project Support
●Host many projects on 1 Hudson● 1+1 > 2
●You can…● Execute tests separately and correlate them
with builds● Mark builds as “promoted” based on test
outcome● Track which build of libraries are used where● . . .
![Page 19: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/19.jpg)
Test/Build Separation●Tests often dominate the build time●Multiple tests are run in parallel
● Improves turn-around time
JAX-WS #10
Unit test with GF
Unit test with Tomcat
SQE test
JAX-WS #11 JAX-WS #12 …
Test w/o container 3OS x 2JDK = 6 tests
time
![Page 20: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/20.jpg)
Build Promotion●CI produces a lot of “successful” builds
● Often overwhelming to downstream consumers
●Run tests as fast as you can●If a build passes tests, promote it
20
![Page 21: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/21.jpg)
Got the idea?
Automation Reduce turn-around time Make things transparent Remove people from the loop Save people’s time Push jobs to servers, keep workstations
idle for you
![Page 22: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/22.jpg)
Doing Distributed BuildsDoing Distributed Builds
22
![Page 23: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/23.jpg)
Distributed builds with Hudson
●Master● Serves HTTP requests● Stores all important info
●Slaves● 170KB single JAR● Assumed to be
unreliable● Scale to at least 100
●Link● Single bi-di byte stream● No other requirements
Master
slave1
slave2
slave3
slave4
slave5
slave6
slave7
slave8
![Page 24: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/24.jpg)
Automated System Installations
●Slaves● Power on, hit F12● PC boots from network (PXE)● Choose OS from menu● Installs non-interactively
●Hudson + PXE plugin● ISO images of OS
Your corporate IT guy & his DHCP server
![Page 25: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/25.jpg)
Automated Tool Installation - JDK
●JDK from http://java.sun.com/● Automatically chooses the right bundle● Always up to date with new releases
25
![Page 26: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/26.jpg)
Automated Tool Installation - Apache
Ant and Maven from Apache
26
![Page 27: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/27.jpg)
Heterogeneous Cluster Challenge Builds/tests need to run in specific environment Dependency on individual nodes hurts utilization
WombatWindows test
Hudson Windows test
Windows #1
jobs slaves
GlassFishWindows test
Windows #2
Solaris#1
Hudson Solaris test
![Page 28: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/28.jpg)
Labels to rescue Label is a group of slaves Tie jobs to labels
WombatWindows test
Hudson Windows test
Windows #1
jobs slaves
GlassFishWindows test
Windows #2
Solaris#1
Hudson Solaris test
Windows
SolarisWindows
#3
![Page 29: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/29.jpg)
Forecasting failures
●Hudson monitors key health metrics of slaves
● Low disk space, insufficient swap● Clock out of sync● Extensible
●Slaves go offline automatically
![Page 30: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/30.jpg)
Load Statistics Monitoring
![Page 31: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/31.jpg)
When it’s time to add more slaves
31
![Page 32: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/32.jpg)
Hudson EC2 plugin
●Automatically provisions slaves on EC2 on demand
● Based on the current load●Picks the right AMI depending on demand●Starts slave agent●Shuts down unused instances
●Can run Hudson master in the cloud too
![Page 33: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/33.jpg)
And a lot more . . .
IDE plugins iPhone/Android apps REST API and CLI Deployment Automation Hadoop, Selenium …
![Page 34: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/34.jpg)
HUMANS: ALL YOUR COMPUTER ARE BELONGTO US
![Page 35: Improving Engineering Processes using Hudson - Spark IT 2010](https://reader033.fdocuments.in/reader033/viewer/2022052900/5563a7aed8b42a2d538b57c0/html5/thumbnails/35.jpg)
Conclusions
●CI is here to stay● Using lots of PCs is a key
●Hudson is very easy to get started● Once started, there is a lot you can do
●hudson-ci.org