Post on 21-Apr-2017
@quidryan @quidryan http://www.slideshare.net/quidryan
Netflix Build Language
Justin Ryan <jryan@netflix.com>
@quidryan @quidryan http://www.slideshare.net/quidryan
Justin Ryan <jryan@netflix.com>
Why Netflix Uses Gradle
• Better Dependency Management story
• Flexible lifecycle
• Groovy
How Netflix Uses Gradle
• JVM Languages
• Resolution
• Code Quality
• Publishing
• Deployment Orchestration
How Netflix Sets up Gradle
• Patched Gradle
• Custom Distribution
• Custom Wrapper
apply plugin: ‘nebula’!apply plugin: ‘java’!!nebula {! readyForJava7 = true!}!!dependencies {! compile ‘netflix:platform:latest.release’!}
build.gradle
@NetflixOSS
github.com/Netflix
netflix.github.io
Unite two builds
• Model a responsible project
• Componentize via plugins
nebula-plugins
• Infrastructure
• Use Github
• Use CloudBees
• Use Bintray
• Mailing List
• nebula-plugin-plugin
Github Repositoriesnebula-* or gradle-*-plugin
Ensuring Github• https://github.com/nebula-plugins/ensure
• Ensure repository has
• Description
• Web Hooks
• Ensure “contrib" team has all repositories in it
• Ensure a contrib team exists for every repository
Continuous Integration
• Job DSL to create jobs
• Per branch
• Snapshot job
• Release job
• Lock job
CloudBees jobsRelease, snapshot, and pull request per branch
Job DSL
Bintray Packages
Ensuring Bintray• https://github.com/nebula-plugins/ensure
• Ensure every repository has a package
• Ensure package has the description as Github
• Ensure license is set to Apache 2.0
• Ensure labels are “gradle" and “nebula”
nebula-plugin-plugin
nebula-test• ProjectSpec
• PluginProjectSpec
• IntegrationSpec
• Thanks to Marcin and Luke
• Runs with Tooling API or GradleLauncher
class PluginExampleSpec extends PluginProjectSpec {! @Override! String getPluginName() { return 'plugin-example' }!! def ‘run task’() {! when:! project.plugins.apply(PluginExample)!! then:! def t = project.tasks.get(‘example’)!! when:! t.run()!! then:! new File(projectDir, ‘build/example.txt’).exists()! }!}
PluginProjectSpec
!def 'setup and run build'() {! buildFile << '''! apply plugin: 'java'! '''.stripIndent()!! when:! writeHelloWorld('nebula.hello')!! then:! fileExists('src/main/java/nebula/hello/HelloWorld.java')!! when:! def result = runTasksSuccessfully(‘build’, ‘-v’)!! then:! fileExists(‘build/classes/main/nebula/hello/HelloWorld.class')! result.wasExecuted(':compileTestJava')! def output = result.standardOutput! output.contains('Skipping task \’:compileTestJava\'')!}
IntegrationSpec
nebula-core• Collection of tasks
• Download
• Untar
• Unzip
• AlternativeArchiveTask
• CopySpecHelper
• GradleHelper
class GradleHelper {!! def beforeEvaluate(Closure beforeEvaluateClosure)!! def getTempDir(String taskBaseName)!! def addDefaultGroup(String defaultGroup)!}!!class CopySpecHelper {!! def visitCopySpec(CopySpecInternal copySpec, Closure closure)!! def findCopySpec(CopySpecInternal delegateCopySpec, closure)!}!!class ClassHelper {!! String findSpecificationVersion(Class clazz)!! Manifest findManifest(Class clazz)!! def findManifestValue(Class clazz, String key, defaultValue)!}
nebula-core helpers
nebula-publishing-plugin• Artifact plugins
• nebula-javadoc-jar
• nebula-source-jar
• nebula-test-jar
• Publishing plugins
• resolved-ivy
• resolved-maven
• nebula-sign
apply plugin: ‘nebula-maven-publishing‘!apply plugin: ‘nebula-source-jar'!apply plugin: ‘nebula-javadoc-jar'!apply plugin: ‘nebula-test-jar'!apply plugin: ‘nebula-sign'!apply plugin: 'java'
nebula-publishing.gradle
my-plugin-1.12.0-javadoc.jar!my-plugin-1.12.0-javadoc.jar.md5!my-plugin-1.12.0-javadoc.jar.sha1!my-plugin-1.12.0-javadoc.jar.asc!my-plugin-1.12.0-sources.jar!my-plugin-1.12.0-sources.jar.md5!my-plugin-1.12.0-sources.jar.sha1!my-plugin-1.12.0-sources.jar.asc!my-plugin-1.12.0-tests.jar!my-plugin-1.12.0-tests.jar.md5!my-plugin-1.12.0-tests.jar.sha1!my-plugin-1.12.0-tests.jar.asc!my-plugin-1.12.0.jar!my-plugin-1.12.0.jar.md5!my-plugin-1.12.0.jar.sha1!my-plugin-1.12.0.jar.asc!my-plugin-1.12.0.pom!my-plugin-1.12.0.pom.md5!my-plugin-1.12.0.pom.sha1!my-plugin-1.12.0.pom.asc
gradle-info-plugin• Collects meta data
• ‘info-java'
• ‘info-ci'
• 'info-scm'
• Reports in key/value pairs
• ‘info-jar'
• ‘info-props'
buildscript {!! repositories { jcenter() }!! dependencies { !! ! classpath 'com.netflix.nebula:gradle-info-plugin:1.12.+'!! ! classpath ‘org.eclipse.jgit:org.eclipse.jgit:3.2.0.201312181205-r'!! }!}!apply plugin: 'info'
info.gradle
Manifest-Version=1.0!Implementation-Title=com.netflix.nebula#my-plugin;1.12.1-SNAPSHOT!Implementation-Version=1.12.1-SNAPSHOT!Built-Status=integration!Built-By=jryan!Build-Date=2014-06-10_13:30:44!Gradle-Version=1.12-20140608201532+0000!Module-Source=!Module-Origin=git@github.com:nebula-plugins/my-plugin.git!Change=976292c!Build-Host=localhost!Build-Job=LOCAL!Build-Number=LOCAL!Build-Id=LOCAL!Created-By=1.7.0_45-b18 (Oracle Corporation)!Build-Java-Version=1.7.0_45!Module-Owner=justin@halfempty.org!Module-Email=justin@halfempty.org!X-Compile-Target-JDK=1.7!X-Compile-Source-JDK=1.7
info.gradle output
gradle-contacts-plugin• Express people involved in the project
• Make people and roles available to other plugins
• contacts-manifest
• contacts-pom
apply plugin: 'contacts'!contacts 'minnie@disney.com', ‘mickey@disney.com'!!contacts {! 'club@disney.com'! 'bobby@company.com' {! roles 'notify', 'owner'! }! 'billy@company.com' {! role 'techwriter'! }! 'downstream@netflix.com'! role 'notify'! }!}!
contacts.gradle
gradle-scm-plugin• Attempt to provide SCM abstraction for other plugins
• E.g. gradle-dependency-lock-plugin and gradle-info-plugin
gradle-dependency-lock-plugin
• Developer declare their ideal situation
• Save resolved version
• If tests pass, commit to SCM
{! "com.github.townsfolk:gradle-release": { !! ! "locked": "1.2", "requested": "1.2" },! "com.jfrog.bintray.gradle:gradle-bintray-plugin": { !! ! "locked": "0.3", "requested": "0.3" },! "com.netflix.nebula:nebula-project-plugin": { !! ! "locked": "1.12.0", "requested": "1.12.+" },! "com.netflix.nebula:nebula-test": { !! ! "locked": "1.12.0", "requested": "1.12.+" },! "org.codehaus.groovy.modules.http-builder:http-builder": { !! ! "locked": "0.7.1", "requested": “latest.release" },! "org.jfrog.buildinfo:build-info-extractor-gradle": { !! ! "locked": "2.2.4", "requested": "2.2.+" }!}
apply plugin: ‘gradle-dependency-lock'!
lock.gradle
./gradlew generateLock
nebula-integtest-plugin• Sets up integTest source set
• Adds integTestCompile and integTestRuntime configurations
• Creates integrationTest task
nebula-project-plugin
• Pull together other plugins
• Responsible projects
nebula-plugin-plugin
• Used by plugins
• Strong opinions on how to publish
• Force nebula-project-plugin on projects
gradle-ospackage-plugin• Merging ubuntu-packager-plugin and gradle-rpm-
plugin
• Uses CopySpec definition
• Via just Java, generates RPMs and DEBs
ospackage {!! os = LINUX!! into '/opt/foo'!! from ('dist') {!! ! user 'builds'!! ! exclude '**/*.md'!! }!! postInstall file('scripts/postInstall.sh')!}!!buildRpm {!! requires('bar', '2.2', GREATER | EQUAL)!! from (‘build/metadata.properties’)!! link(‘/etc/init.d/foo’, '/opt/foo/bin/foo.sysv',)!}!!buildDeb {!! link('/etc/init/foo', '/opt/foo/bin/foo.upstart')!}
ospackage.gradle
gradle-override-plugin• Take command line arguments
• Intelligently apply in afterEvaluate
• E.g. -Nfindbugs.enabled=false
Internal Plugins
• netflix-repos
• nebula-ospackage
• nebula-grails
• nebula-findbugs, etc
• ivyimport
• nebula-fixexcludes
• nebula-intellij
buildscript {!! repositories { jcenter() }!! dependencies { !! ! classpath ‘com.netflix.nebula:nebula-plugin-plugin:1.12.+'!! ! classpath ‘org.eclipse.jgit:org.eclipse.jgit:3.2.0.201312181205-r'!! }!}!!description ‘Example Plugin'!apply plugin: ‘nebula-plugin'!!contacts {! ‘jryan@netflix.com’ {! moniker 'Justin Ryan'! github 'quidryan'! }!}
plugin.gradle
curl -s get.gvmtool.net | bash!gvm install lazybones!cd ~/Projects/github/nebula-plugins!lazybones create nebula-plugin <name-of-project>!cd <name-of-project>!git init!git remote add origin git@github.com:nebula-plugins/<name-of-project>.git!./gradlew clean build!git add -A!git add -f gradle/wrapper/gradle-wrapper.jar!git commit -m "Initial template”!git push --set-upstream origin master
Making a plugin
https://github.com/nebula-plugins/nebula-plugins.github.io/wiki/New-Plugins
Getting it out the door • Let “ensure” run
• Run <name-of-project>-release
• Link package to jcenter
• Link package to gradle-plugins
Unopinionated Plugins
Opinionated Plugins
Gotchas
• NamedDomainObjectSet
• Debugging Tooling
• File as @Output
• afterEvaluate
Outstanding• CloudBees permissions
• Bot to create repositories
Participating
• Use individual plugins
• Get on nebula-plugins Google Group
• Move your plugin to nebula-plugins
• Start a new plugin in nebula-plugins
@quidryan @quidryan http://www.slideshare.net/quidryan
We’re hiring
Justin Ryan <jryan@netflix.com>
HOUSE of GRADLE