Andrey Adamovich and Luciano Fiandesio - Groovy dev ops in the cloud
104
01
description
The talk is focused on open-source Groovy/Java libraries that we have developed in order to solve our daily DevOps problems including SSH connectivity, cloud instance management, OS image management and integration testing of Puppet modules with the help of Groovy and Junit.
Transcript of Andrey Adamovich and Luciano Fiandesio - Groovy dev ops in the cloud
01
Groovy DevOps in theCloud
02
About us03
Luciano Fiandesio
Bio: Developer, speaker, author, kitchen hacker
Company: Aestas/IT (http://aestasit.com)
E-mail: [email protected]
Linkedin: http://www.linkedin.com/in/lucianofiandesio
••••
04
Andrey Adamovich
Bio: Developer, coach, speaker, author
Company: Aestas/IT (http://aestasit.com)
E-mail: [email protected]
Linkedin: http://www.linkedin.com/in/andreyadamovich
••••
05
What's this presentation about?
Our take on:
DevOps
Software Provisioning
Continuous Integration
Continuous Delivery
••••
06
Technologies
AWS - http://aws.amazon.com
Groovy - http://groovy.codehaus.org
Gradle - http://gradle.org
Jenkins - http://jenkins-ci.org
Puppet - http://puppetlabs.com
•••••
07
Developers +Operations =
?08
Silos
09
Conflicts10
Risk
11
Agile
12
What is DevOps?
13
C.A.M.S.
Culture: People over processes and tools. Software is made by and
for people.
Automation: Automation is essential for DevOps to gain quick
feedback.
Measurement: DevOps finds a specific path to measurement. Quality
and shared (or at least aligned) incentives are critical.
Sharing: Creates a culture where people share ideas, processes, and
tools.
•
•
•
•
14
Though...15
It's not abouttools!
16
It's aboutculture and
process!17
But withouttools...
18
...it'sdefinitelyharder!
19
DevOps implyautomation!
20
DevOps implystructure!
21
Infrastructure as Code
Automate the provisioning and maintenance of servers:
Build from source control
Utilize Open Source tools
Ensure testability
•••
22
Configuration propagation
23
Changes
No Manual Changes!
24
Building an automation toolkit
Automation is key
We are JVM hackers
Fragmented ecosystem
Full-stack approach
••••
25
Tooling
Infrastructure connectivity
Infrastructure provisioning
Infrastructure virtualization
Infrastructure testing
••••
26
Sshoogr
27
Sshoogr Features
Remote command execution
File uploading/downloading
Tunneling
•••
28
Sshoogr Example I
remoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm -rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
29
Sshoogr Example II
remoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/application' }
}
}
01.
02.
03.
04.
05.
06.
30
Sshoogr Example III
def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
result.output.eachLine { line ->
if (line.contains('WARNING')) {
throw new RuntimeException("Warning!!!")
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.31
Why Groovy?
Groovy is perfect choice for scripting
Very mature, concise syntax
Extremely easy to produce DSL
We wrote a book about it!
••••
32
Shameless plug
33
Puppet34
Why Puppet?
More mature than competition
Large community
Readable DSL
No need to learn Ruby ;)
••••
35
Puppet example
36
Puppet provisioning
37
Puppet provisioning
38
Puppet provisioning
39
Puppet provisioning
40
Puppet state management
41
Puppet state management
42
Puppet state management
43
Puppet modules
44
Puppet modules
45
Puppet modules
46
Continuous Integration
47
Why Jenkins?
De-facto standard
Stable
There is a plugin for that!
•••
48
Gradle49
Gradle Example I
task uploadModules << {
remoteSession {
exec 'rm -rf /tmp/repo.zip'
scp {
from { localFile "${buildDir}/repo.zip" }
into { remoteDir "/root" }
}
...
01.
02.
03.
04.
05.
06.
07.
08.
50
Gradle Example I
...
exec 'rm -rf /etc/puppet/modules'
exec 'unzip /tmp/repo.zip -d /etc/puppet/modules'
}
}
01.
02.
03.
04.
05.
51
Gradle Example II
task puppetApply(dependsOn: uploadModules) << {
remoteSession {
scp {
from { localFile "${buildDir}/setup.pp" }
into { remoteDir "/tmp" }
}
exec 'puppet apply /tmp/setup.pp'
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.52
Jenkins example
53
In the meanwhile...
We started developing complex Puppet modules
Modules needs proper testing
...on different platforms
•••
54
Do you test, right?
How to test this stuff?
How to reuse a JUnit approach to testing?
We wanted things to be SIMPLE!
•••
55
PuppetUnit
56
PUnit
Simple testing tool for verifying remote server state
Uses sshoogr and JUnit
Reuse reporting features of JUnit and Jenkins
As simple as ...
••••
57
PUnit Example I
class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include derby")
}
...
}
01.
02.
03.
04.
05.
06.
07.
08.
58
PUnit Example II
@Test
def void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
}
01.
02.
03.
04.
05.
06.
07.
08.
59
PUnit Example III
@Test
def void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
60
PUnit Example III
...
// Check if the log of the insert
// operation contains the word ERROR.
assertFalse(
"The script should return at least one error",
fileText("/root/derbytest.log")
.contains('ERROR')
)
...
01.
02.
03.
04.
05.
06.
07.
08.
09.61
PUnit Example III
...
// Check on data that was inserted into a table.
assertTrue(
"The log should contain a SELECT result",
fileText("/root/derbytest.log")
.contains('Grand Ave.')
)
}
01.
02.
03.
04.
05.
06.
07.
08.
62
PUnit
63
More examples
session {
tunnel ('127.0.0.1', 8080) { int localPort ->
// Login to Jenkins.
def driver = new HtmlUnitDriver(false)
driver.manage().timeouts().pageLoadTimeout(300, TimeUnit.SECONDS).implicitlyWait(30, TimeUnit.SECONDS)
driver.get("http://127.0.0.1:${localPort}/login")
...
01.
02.
03.04.
05.
06.
07.
08.
09.
64
More examples
...
def input = driver.findElement(By.name('j_username'))
input.sendKeys('john')
input = driver.findElement(By.name('j_password'))
input.sendKeys('123456')
input.submit()
...
01.
02.
03.
04.
05.
06.
07.
65
More examples
...
// Verify login.
def wait = new WebDriverWait(driver, 30)
wait.until ExpectedConditions.presenceOfElementLocated (By.linkText('John Doe'))
...
}
}
01.
02.
03.
04.
05.
06.
07.
66
More examples
session {
tunnel ('127.0.0.1', 80) { int localPort ->
// Initilize repository connection data.
DAVRepositoryFactory.setup()
def url = SVNURL.create('http', null, '127.0.0.1', localPort, 'repos/cafebabe', true)
def repository = SVNRepositoryFactory.create(url)
println "Verifying SVN repository at ${url}"
...
01.
02.
03.04.
05.
06.
07.
08.
09.
67
More examples
...
// Setup credentials.
def authManager = SVNWCUtil.createDefaultAuthenticationManager('joe', '123456')
repository.setAuthenticationManager(authManager)
// Verify repository is at revision 0.
assertEquals 0, repository.getLatestRevision()
...
01.
02.
03.
04.
05.06.
07.
08.
68
More examples
...
// Commit first revision.
ISVNEditor editor = repository.getCommitEditor("Initial commit.", null)
editor.with {
openRoot(-1)
addFile('dummy.txt', null, -1)
applyTextDelta('dummy.txt', null)
def deltaGenerator = new SVNDeltaGenerator()
...
01.
02.
03.
04.
05.
06.
07.
08.
09.69
More examples
...
def checksum = deltaGenerator.sendDelta('dummy.txt', new ByteArrayInputStream("data".getBytes()), editor, true)
closeFile('dummy.txt', checksum)
def commitInfo = closeEdit()
println commitInfo
}
...
01.
02.
03.
04.
05.
06.
07.
70
More examples
...
// Verify repository is at revision 1 now.
assertEquals 1, repository.getLatestRevision()
}
}
01.
02.
03.
04.05.
06.
71
Nextproblem?
72
Scalability
How do we test on different OS?
How do we run parallel tests on multiple architectures?
How do we avoid selling our houses?
•••
73
Amazon WebServices
74
Elastic Compute Cloud
Mature
Great API
Virtual hardware variety
OS variety
••••
75
Gramazon
76
Gramazon
Groovy based API for interacting with EC2
Integrates with the rest of the stack
••
77
Gramazon Example I
task startInstance(type: StartInstance) {
keyName 'cloud-do'
securityGroup 'cloud-do'
instanceName 'gramazon/cloud-do'
stateFileName 'cloud-do.json'
ami 'ami-6f07e418'
instanceType 't1.micro'
waitForStart true
}
01.
02.
03.
04.
05.
06.
07.
08.
09.78
Gramazon Example II
task terminateInstance(type: TerminateInstance) {
stateFileName 'cloud-do.json'
}
01.
02.
03.
79
Imgr
80
Imgr
A tool for building images
Inspired by Packer
••
81
Images?
82
Supports
Shell
Puppet
••
83
Configuration Example
84
The flow
Start instance(s)
Upload manifests
Run tests
Generate report
Terminate instance(s)
1.
2.
3.
4.
5.
85
Images, manifests, tasks
86
The big picture
87
Aetomation
88
Conclusions
Reuse your existing Java knowledge
...to build a bridge between DEVs and OPs
Reuse development best practices for OPs
Don't be afraid to try new technologies
Automate!
•••••
89
Readingmaterial
90
Groovy 2 Cookbook
91
The Phoenix Project
92
Continuous Delivery
93
Release It
94
Programming Amazon EC2
95
Gradle in Action
96
Technologies to follow
Vagrant - http://www.vagrantup.com/
Docker - https://www.docker.io/
Packer - http://www.packer.io/
Qemu - http://wiki.qemu.org/Main_Page
jclouds - http://jclouds.apache.org/
Cloudbees - http://www.cloudbees.com/
••••••
97
One morething...
98
It's all OpenSource!
99
Source code
Sshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle
PUnit: https://github.com/aestasit/puppet-unit
Gramazon: https://github.com/aestasit/gramazon
Imgr: https://github.com/aestasit/imgr
•••••
100
Seekingcontributors!
101
Questions?102
Thank you!
103
104
