Groovy DevOps in the Cloud for DevOpsDays in Ljubljana 2014
79
Groovy DevOps in the Cloud 01
description
The talk is focused on open-source Groovy/Java libraries that we have developed in order to solve our daily DevOps problems including SSH connectivity, cloud instance management, OS image management and integration testing of Puppet modules with the help of Groovy and Junit.
Transcript of Groovy DevOps in the Cloud for DevOpsDays in Ljubljana 2014
Groovy DevOps in theCloud
01
About me02
Andrey Adamovich
Bio: Developer, coach, speaker, author
Company: Aestas/IT (http://aestasit.com)
E-mail: [email protected]
Linkedin: http://www.linkedin.com/in/andreyadamovich
Twitter: @aestasit
•••••
03
What's this presentation about?
Our take on:
DevOps
Server Provisioning
Continuous Integration
Continuous Delivery
••••
04
Technologies
AWS - http://aws.amazon.com
Groovy - http://groovy.codehaus.org
Gradle - http://gradle.org
Jenkins - http://jenkins-ci.org
Puppet - http://puppetlabs.com
•••••
05
Developers +Operations =
?06
Silos
07
Conflicts08
Risk
09
Agile
10
What is DevOps?
11
It's not abouttools!
12
It's aboutculture and
process!13
But withouttools...
14
...it'sdefinitelyharder!
15
DevOps implyautomation!
16
DevOps implystructure!
17
Infrastructure as Code
Automate the provisioning and maintenance of servers:
Build from source control
Utilize Open Source tools
Ensure testability
•••
18
Configuration propagation
19
Changes
No Manual Changes!
20
Building an automation toolkit
Automation is key
We are JVM hackers
Fragmented ecosystem
Full-stack approach
••••
21
Sshoogr
22
Sshoogr Features
Remote command execution
File uploading/downloading
Tunneling
•••
23
Sshoogr Example I
remoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm -rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
24
Sshoogr Example II
remoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/application' }
}
}
01.
02.
03.
04.
05.
06.
25
Sshoogr Example III
def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
result.output.eachLine { line ->
if (line.contains('WARNING')) {
throw new RuntimeException("Warning!!!")
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.26
Why Groovy?
Groovy is perfect choice for scripting
Very mature, concise syntax
Extremely easy to produce DSL
We wrote a book about it!
••••
27
Shameless plug
28
Puppet29
Why Puppet?
More mature than competition
Large community
Readable DSL
No need to learn Ruby ;)
••••
30
Continuous Integration
31
Why Jenkins?
De-facto standard
Stable
There is a plugin for that!
•••
32
Gradle33
Gradle Example I
task uploadModules << {
remoteSession {
exec 'rm -rf /tmp/repo.zip'
scp {
from { localFile "${buildDir}/repo.zip" }
into { remoteDir "/root" }
}
...
01.
02.
03.
04.
05.
06.
07.
08.
34
Gradle Example I
...
exec 'rm -rf /etc/puppet/modules'
exec 'unzip /tmp/repo.zip -d /etc/puppet/modules'
}
}
01.
02.
03.
04.
05.
35
Gradle Example II
task puppetApply(dependsOn: uploadModules) << {
remoteSession {
scp {
from { localFile "${buildDir}/setup.pp" }
into { remoteDir "/tmp" }
}
exec 'puppet apply /tmp/setup.pp'
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.36
In the meanwhile...
We started developing complex Puppet modules
Modules needs proper testing
...on different platforms
•••
37
Do you test, right?
How to test this stuff?
How to reuse a JUnit approach to testing?
We wanted things to be SIMPLE!
•••
38
PuppetUnit
39
PUnit
Simple testing tool for verifying remote server state
Uses sshoogr and JUnit
Reuse reporting features of JUnit and Jenkins
As simple as ...
••••
40
PUnit Example I
class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include derby")
}
...
}
01.
02.
03.
04.
05.
06.
07.
08.
41
PUnit Example II
@Test
def void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
}
01.
02.
03.
04.
05.
06.
07.
08.
42
PUnit Example III
@Test
def void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
43
PUnit Example III
...
// Check if the log of the insert
// operation contains the word ERROR.
assertFalse(
"The script should return at least one error",
fileText("/root/derbytest.log")
.contains('ERROR')
)
...
01.
02.
03.
04.
05.
06.
07.
08.
09.44
PUnit Example III
...
// Check on data that was inserted into a table.
assertTrue(
"The log should contain a SELECT result",
fileText("/root/derbytest.log")
.contains('Grand Ave.')
)
}
01.
02.
03.
04.
05.
06.
07.
08.
45
PUnit
46
More examples
session {
tunnel ('127.0.0.1', 8080) { int localPort ->
// Login to Jenkins.
def driver = new HtmlUnitDriver(false)
driver.manage().timeouts().pageLoadTimeout(300, TimeUnit.SECONDS).implicitlyWait(30, TimeUnit.SECONDS)
driver.get("http://127.0.0.1:${localPort}/login")
...
01.
02.
03.04.
05.
06.
07.
08.
09.
47
More examples
...
def input = driver.findElement(By.name('j_username'))
input.sendKeys('john')
input = driver.findElement(By.name('j_password'))
input.sendKeys('123456')
input.submit()
...
01.
02.
03.
04.
05.
06.
07.
48
More examples
...
// Verify login.
def wait = new WebDriverWait(driver, 30)
wait.until ExpectedConditions.presenceOfElementLocated (By.linkText('John Doe'))
...
}
}
01.
02.
03.
04.
05.
06.
07.
49
More examples
session {
tunnel ('127.0.0.1', 80) { int localPort ->
// Initilize repository connection data.
DAVRepositoryFactory.setup()
def url = SVNURL.create('http', null, '127.0.0.1', localPort, 'repos/cafebabe', true)
def repository = SVNRepositoryFactory.create(url)
println "Verifying SVN repository at ${url}"
...
01.
02.
03.04.
05.
06.
07.
08.
09.
50
More examples
...
// Setup credentials.
def authManager = SVNWCUtil.createDefaultAuthenticationManager('joe', '123456')
repository.setAuthenticationManager(authManager)
// Verify repository is at revision 0.
assertEquals 0, repository.getLatestRevision()
...
01.
02.
03.
04.
05.06.
07.
08.
51
More examples
...
// Commit first revision.
ISVNEditor editor = repository.getCommitEditor("Initial commit.", null)
editor.with {
openRoot(-1)
addFile('dummy.txt', null, -1)
applyTextDelta('dummy.txt', null)
def deltaGenerator = new SVNDeltaGenerator()
...
01.
02.
03.
04.
05.
06.
07.
08.
09.52
More examples
...
def checksum = deltaGenerator.sendDelta('dummy.txt', new ByteArrayInputStream("data".getBytes()), editor, true)
closeFile('dummy.txt', checksum)
def commitInfo = closeEdit()
println commitInfo
}
...
01.
02.
03.
04.
05.
06.
07.
53
More examples
...
// Verify repository is at revision 1 now.
assertEquals 1, repository.getLatestRevision()
}
}
01.
02.
03.
04.05.
06.
54
Nextproblem?
55
Scalability
How do we test on different OS?
How do we run parallel tests on multiple architectures?
How do we avoid selling our houses?
•••
56
Amazon WebServices
57
Elastic Compute Cloud
Mature
Great API
Virtual hardware variety
OS variety
••••
58
Gramazon
59
Gramazon
Groovy based API for interacting with EC2
Integrates with the rest of the stack
••
60
Gramazon Example I
task startInstance(type: StartInstance) {
keyName 'cloud-do'
securityGroup 'cloud-do'
instanceName 'gramazon/cloud-do'
stateFileName 'cloud-do.json'
ami 'ami-6f07e418'
instanceType 't1.micro'
waitForStart true
}
01.
02.
03.
04.
05.
06.
07.
08.
09.61
Gramazon Example II
task terminateInstance(type: TerminateInstance) {
stateFileName 'cloud-do.json'
}
01.
02.
03.
62
Imgr
63
Imgr
A tool for building images
Inspired by Packer
••
64
Images?
65
Supports
Shell
Puppet
••
66
Configuration Example
67
The flow
Start instance(s)
Upload manifests
Run tests
Generate report
Terminate instance(s)
1.
2.
3.
4.
5.
68
Images, manifests, tasks
69
The big picture
70
Aetomation
71
Conclusions
Reuse your existing Java knowledge
...to build a bridge between DEVs and OPs
Reuse development best practices for OPs
Don't be afraid to try new technologies
Automate!
•••••
72
One morething...
73
It's all OpenSource!
74
Source code
Sshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle
PUnit: https://github.com/aestasit/puppet-unit
Gramazon: https://github.com/aestasit/gramazon
Imgr: https://github.com/aestasit/imgr
•••••
75
Seekingcontributors!
76
Questions?77
Hvala! Thank you!
78
