1 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. … · 2017-12-06 ·...
Transcript of 1 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. … · 2017-12-06 ·...
1 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
2 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
java –jar hudson-3.0.0.war
3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Plugins • descriptors
Projects (Jobs) • config.xml
Slaves
Labels
config.xml
Hudson
Master
4 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Build archival
Job config
Plugin Config
Master/Slave/Views
config.xml
/jobs/${name}
config.xml config.xml /builds/${number}
build.xml
${descriptor}.xml
5 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
PlaIorm IAAS Launcher Master
config
ssh
LXC
Dumb
Linux
Windows
AIX
cloud
CloudStack
OpenStack
Manual My machine
6 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Early efforts
• Single master scalability • “I need to develop a new plugin for generic uptake across
existing projects” • “I need to develop a new plugin for my projects” • “I have a new project coming online (no consumers) and I
would like a little privacy while I set up the job” • When is it okay to reboot a Hudson master?
7 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
1 butler? Butler of butlers?
8 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Carson’s Resource Model
• GET POST
/carson/v1/organizaWons
• GET PUT DELETE
/carson/v1/organizaWons/:id
• GET POST
/carson/v1/projects
• GET PUT
/carson/v1/projects/:id
9 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Carson’s Resource Model
• GET POST
/carson/v1/plugins
• GET PUT DELETE
/carson/v1/plugins/:id
• GET POST
/carson/v1/slaves
• GET PUT DELETE
/carson/v1/slaves/:id
10 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Config Management
(git repo)
• Job config • Plugin config per org • Slave config per org
Shared Resources
Master/Slave nodes
Build Archival (Storage)
JFrog Artifactory for
Artifact Management
Hudson Plugin
Manager
Artifactory For
Internal Plugin promotion
ldap Http
Gateway
IT
11 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Adding new orgs or dev instances
Http Gateway
POST /carson/v1/organization
RouterService
http://host/hudson/crm http://host/hudson/hcm
hcm crm
sync
IAAS
12 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Job Promotion
PUT /carson/v1/project/:id
move
dev
prod
13 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Hudson REST interface /rest/admin/config
/rest/admin/quiet-‐down
/rest/admin/restart
/rest/queue/clear
/rest/queue/nodes
/rest/queue/{nodeName}
14 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Hudson REST interface /rest/projects
• POST GET DELETE
/rest/projects/{projectName}
/rest/projects/{projectName}/builds
/rest/projects/{projectName}/config
• boolean query param
/rest/projects/{projectName}/enable
15 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Plugin Stages of Maturity Dev
Dev masters only
Sprint lifeWme
No bug id
Beta
Available to producWon masters
part of sprint
Prod
ucWo
n bug id assigned Different organizaWons may use different versions
One version per organizaWon
• Maturity not determined by version • Maturity determined by virtual repository (the artifactory way) • Organization decides whether it will use plugins of lower maturity
16 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Hudson plugin containers
Hudson 3.0.0 core
@Extension container
Smoothie Container (jsr-330)
OSGI (jsr-330)
experimental
• Teams did not mind restarting dev masters • Dev Managers care very much about restarting production masters
• Easier if a restart doesn’t cancel any jobs or block any from getting on the queue
• PluginStrategy • ExtensionFinder
17 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Embracing restarts
PUT /carson/v1/plugin/:id
copy
dev
Http Gateway
disable
Update route
enable
prod
18 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Node Management Extension Points
XLCSlave Integrate Linux XLC containers for cheap virtualizaWon
KerberosNodeMonitor (map #(test %) (-‐> (filter #(labels? :ade %) nodes) :Wckets))
NFSNodeMonitor (map #(test %) (-‐> (filter #(labels? :nfs %) nodes) :mountPoints))
CacheManagerLifecycle Implement ivy/gradle/maven cache usage/cleaning policies
19 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
New Extension Points added via Plugins
Hudson Core
abstract ExtensionPoint Impl1
Impl2
Example: “release” versioning
20 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Provisioning Extension Points
@ToolInstallaWon We made the rule that new builders with sofware requirements must provide this ExtensionPoint
@ToolInstaller Plug in chef/puppet scripts that pull installaWons from arWfactory
@ToolProperty Useful to formalize the contract for how different jobs might need to install build tools
21 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
View Extension Points
MatrixView We made the rule that new builders with sofware requirements must provide this ExtensionPoint
ArWfactoryQuotaView Plug in chef/puppet scripts that pull installaWons from arWfactory
CramItBadgePlugin Useful to formalize the contract for how different jobs might need to install build tools
• We wrote a big SCM plugin and quite a few custom @Trigger extensions • Also created a brand new Hudson ExtensionPoint to work with JFrog
22 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
CI hierarchy of needs
Basic Needs: Compile successfully. Assemble testable System. Mostly “negative” feedback.
Secure Foundation: safety net. Reproducibility. Consistency.
Reliance: requests for new reports, new features. Presence is assumed
Self-Actualization: System and CI process evolve together
23 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Dev Org “pre-merge” Spider Sense
Feature Vector SCM
Label Label
Label
Choose feature based on “max information increase”
(Shannon Information)
Update predictive
model
Identify “high-impact”
changes
24 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Application of CI in SOA env
Compile/package/test
Composability - new modules - new dependencies - new endpoints
Refactoring - cohesion, coupling - patchability - idempotency
25 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Components in FusionApps
• ~ 10407 ADF libs – “Model” ADF libraries contain Entity
definitions and Java Service interfaces – “UI” ADF libraries contain the task flows,
page defs, jsff, etc. (resources)
• ~ 470 SCA composites • 1:1 ratio between module/component
and Project
3/4 million versioned things in SCM
10407 patchable arWfacts
56 deployable J2EE applicaWons + 470 SCA composites
26 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Build
Package Provision
Feed Forward: • Package (plan deployments)
• Connection plans • SOA Config plans • Credential plans • Assemble Security Policies • Http Access config • BIP Catalogs • Translation
Continuous Release
27 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Build
Package Provision
Publish
Plan(Abstract) Deploy(Concrete)
Assembling Service Network • Collect all abstract portType information • scac phase runs against contracts in MDS • scac compile phase unordered • Assemble portType to logical provider map
• Enumerate every unbound port • generate SOA config plans • integrate with Deployment Admin
• bind concrete endpoint details to generated plan • Inject environment • Deploy composites
28 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8 © 2006 Oracle Corporation – Proprietary and Confidential
Assembling Service Network
• Build is just scac validation IF we avoid dependency on endpoint urls (runtimes) • Introduction of a “publish” phase and strict separation between “abstract” interface
and concrete endpoints • Endpoints are introduced only at deploy time (post scac compile) using config plans
generated by analysis of the service network (“bottom-up assembly”) • Deployments can be done UN-ordered (ie in parallel) • ConfigPlans are also readable as endpoint requirement specs
• Config Plans are never “versioned” files. • No need to have developers “tokenize” anything.
29 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
CI “events” in SOA lifecycle • new providers • new consumers
– new consumers with new topology requirements – new “wires” within existing topology – How many emulations/tests added?
• call out any schema types not uniquely defined • updates to third-party contracts
30 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Connection Management • Types
– ServiceFactory – WebService – Http – UCM – Forum – WSRP – BI JDBC – BI SOAP – EXTAPP
• Same basic Principle: – Publish (Discover) – Plan (Determine Edges) – Deploy (Bind “logical” to “concrete” and apply pre-ear): topology written into MDS
• Credential Plans – “plan” phase is replaced by an “erase” phase: continuous amnesia?
31 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Resource Types • Permission Owners • Framework-‐driven
App Roles / Privileges • Top-‐Down centrally managed
Grants • bo@om-‐up resource driven
Merged Jazn-‐data policy
Security
32 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Maven Default Lifecycle
validate compile test package integraDon-‐test verify install deploy
maven-ant-plugin weblogic-maven-plugin Custom-hudson-plugin
goal goal goal goal
goal goal
goal goal
goal goal
goal
33 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Extract(“discover”) Create Edges of
Dependency Graph Generate Build
“plan” Send Build “Events” to slave machines
Full Parallelized Builds
34 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
35 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Dep Graph Map “TransacWon Delta” to Nodes
Incremental Rebuild/
Repackage plan Create Patch
Incremental Builds
36 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Impact Analysis
RunWme A
Component A.1
Component A.2 Impacted
RunWme B
Component B.1 impacted
Bug fix
37 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Hudson Maintenance BuildWrapper
Hudson SCM plugin
Impact Analysis
Plan/Execute Build
Build Events
Deployment Manager
Change Events
Policy Pipeline
Translation Module
App Restart Policy
Versioning Policy
Environment Definitions
38 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Change “Events” tracked
39 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Release Pipeline
ArWfact Repo
GDR
Source Repo
Install/Prov Metadata - Dists - Feature Sets - Components
Hudson Jobs
label sources
Publish DOs
Hudson Jobs
Component Archive Component
Archive Component Archive
“populate” archive - use binary repo coords?
Dependency Metadata Dependency
Metadata Dependency Metadata Dependency
Metadata Dependency Metadata
Artifact Management
- populate deps via versioned components - translate between file groups and coordinate sets
- - repos/coords - - pom/ivey metadata generation?
40 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8
Internal Hackfest results
• Build avoidance • Bulk rebuild trigger • LogBeautifiers • Dependency graph views (GraphViz processing.js) • OSN integration
41 Copyright © 2011, Oracle and/or its affiliates. All rights reserved. Insert Information Protection Policy Classification from Slide 8