RULE THEM ALL - JAX Londonliveperson ci / cd: docker / kubernetes dev env build deployment tests /...
Transcript of RULE THEM ALL - JAX Londonliveperson ci / cd: docker / kubernetes dev env build deployment tests /...
RULE THEM ALLCI/CD For Micro-Services Architecture
Nir KorenDevOps CI/CD Team Leader, LivePerson Israel
Nir KorenDevOps CI/CD Team Leader, LivePerson IsraelDoing and implementing DevOps and CI / CD for 10+ years
https://il.linkedin.com/in/nirkoren
https://www.facebook.com/koren.nir
@KorenNir
@nir_koren
AGENDA
▸ About LivePerson▸ About LivePerson Production▸ LivePerson CI/CD in a nutshell▸ Microservices (MSA) in a nutshell▸ Goals for better MSA CI/CD▸ LivePerson CI/CD in a practice
ABOUT LIVEPERSON
“
”
We create a
Digital Contact Center to allow transition
From Voice to Messaging
LIVEPERSON
Founded in 1995, public since 2000 (NASDAQ: LPSN)
More than 20,000 customers worldwide
More than 1000 employees
25B site visits and +300M live engagements Yearly
100% SaaS Cloud Company
Headquartered in NYC with 18 offices in Americas,
EMEA and APAC
SMS
Apps
1990 2000 2010
Phone
Today
People no longer call, they message
Started with SMS in 1992. Now dominant across all age groups and geographies
Messages Facebook Whatsapp
Hangouts Twitter Viber
Snapchat Skype WeChat
LivePerson transforms the world’s largest brands customer care
from voice calls to messaging
Learn more about LivePerson @ www.liveperson.com
ABOUT LIVEPERSON PRODUCTION
LIVEPERSON DATACENTERS MAP11
OaklandVirginia
LondonAmsterdam
Tel Aviv
Sydney
Melbourne
LIVEPERSON PRODUCTION NUMBERS
7Worldwide data centers
3MConcurrent users at peak
1.1MMetrics monitored per minute
10,000Servers (virtual and physical)
7000Production deployments per year
3BVisits per month
ABOUT LIVEPERSON CI/CD
ALIGNMENT: WHAT IS CI?
“A key software development practice where members of a team integrate their work frequently.” Martin Fowler, Continuous Integration: http://martinfowler.com/articles/continuousIntegration.html
LIVEPERSON CI / CD NUMBERS
200Microservices (50% K8S)
7000 Ongoing deployments per year
2500Active GitHub Repositories
15,000Running weekly builds in TeamCity
80TeamCity /Jenkins Build Agents
5CI/CD core Team members
LIVEPERSON CI / CD: PUPPET / RPM SERVICES
BUILD DEPLOYMENT TESTS / SCANS OPERATIONSDEV ENV
REVIEW
CI / CD is orchestrated by TeamCity
LIVEPERSON CI / CD: DOCKER / KUBERNETES
BUILD DEPLOYMENT TESTS / SCANS OPERATIONSDEV ENV
REVIEW
CI / CD is orchestrated by TeamCity
LIVEPERSON MSA RELEASE PROCESS
RELEASE REPO PRODUCTION
All is done by the developer independently
LivePerson developers owns the entire process
from
IDE to PRODUCTION
Continuous Deployment
Continuous Delivery
Auto Auto ManualBuild Deployment Tests Production
Auto Auto AutoBuild Deployment Tests Production
ALIGNMENT: WHAT IS CD?
We at LivePerson runs
CONTINUOUS DELIVERYand on our way to
CONTINUOUS DEPLOYMENT
ALIGNMENT: SUMMARY
END OF SPRINTDEVELOPMENT FIXES DELIVERY
PLAN
FACT
CI
CD
OUR ARCHITECTURE: MICRO-SERVICES
We have a single product but in fact we run
200 products
FROM MONOLITHIC TO MICRO-SERVICES
Monolith Microservices
MONOLITH?
▸ Single Deployment and version.▸ Easy to control the servers in Runtime. ▸ Easy to monitor.▸ Single Codebase / Code repository.▸ Easy for integration.▸ Easy to enforce processes and governance.
MONOLITH?
▸ Spaghetti.▸ Hard to refactor / maintain.▸ Hard to implement Zero-Downtime deployments.▸ Each change ”shakes” the entire system.▸ Complicated and long CI processes.▸ Complicated and long deployments.▸ Hard to implement CD (both deployment and delivery).
MICROSERVICES?
▸ Simplicity.▸ Easy for modeling.▸ Easy to add / remove / refactor services.▸ Increase DevOps Culture & processes.▸ Easy to own a better CI / CD Processes.▸ Both frequent and relevant updates (mainly on cloud).▸ Popular: more and more best practices in the web.
▸ Requires DevOps skills, people and culture. ▸ Hard to control.▸ Hard to perform organizational governance.▸ Complexity of interfaces & integration.▸ Operational overhead.▸ Network and performance overhead.▸ Dependencies management (both build and runtime).
MICROSERVICES?
Microservices or Monolith?
There is no right answer
Microservices or Monolith?
”…don't even consider Microservices unless you have a system that's too complex to manage as a monolith…”
Martin Fowler
Microservices or Monolith?
LIVEPERSON ARCHITECTURE
200 micro-services. Each one has its own:
▸ Code repository (*ies)▸ Code review rules▸ Lifecycle▸ Deployment rules ▸ Server(s) and environment ▸ Expose needed API / Entry points
Doing wrong CI/CD in Microservices
LEADS TO CHAOS
CHAOS IN BUILD
Each team uses whatever makes him feel right
▸ Maven, Gradle, Ant… ▸ NPM? Yarn? Grunt? ▸ JDK / Maven. ▸ Maven dependencies▸ Scripts▸ Releases▸ Artifacts, Versions, Naming conventions... Wild West
▸ Chef, Puppet, Ansible… Each team chooses whatever he likes.
▸ Use docker-compose? Kubernetes? Scripts? Plugins?
CHAOS IN DEPLOYMENT
▸ Different technologies for the same languages.▸ Different code-coverage technologies ▸ Different dependencies versions for the same technology.
CHAOS IN TESTS
CHAOS IN POST BUILD ACTIONS
▸ Team can mark unwanted scans and steps. ▸ Cannot monitor processes and statistics.▸ Unable to enforce security / quality policy
Chaos creates
BAD PRODUCTS
MICROSERVICES CI/CD GOALS
What do we want to achieve in order to
CONTROL THE CI/CD?
What do we want to achieve in order to
SUPPORT OUR DEVELOPERS?
GOAL # 1
All aligned with the right (and relevant) tools
Make all versions (and tools) unified
GOAL # 2
Make (most of the) builds run in the same way
GOAL # 3
Enforce (corporate) governance
GOAL # 4
Transparency to (both) Ops and Dev teams
GOAL # 5
Unified path (and checklist) to Production
GOAL # 6
LIVEPERSON CI/CD IN A PRACTICE
PROJECT ARCHETYPES
IN-HOUSE MAVEN / NPM PLUGINS
IN-HOUSE PARENT POM / NPM
POST-BUILD ACTIONS
DASHBOARD AND REPORTS
UNIFIED PIPELINE TO PRODUCTION
1
2
3
4
5
6
PROJECT ARCHETYPES
PROJECT ARCHETYPES
1-Click self-service that will generate everything you need
Need a new Project? Choose your project technology
Choose your project type
Fill in groupId, artifactId,
Organization…
PROJECT ARCHETYPES
Step 1: Project creation (Maven project example)
Maven directories structure
Deployment Skeleton
Maven POM Skeleton
Dockerfile
TestsSkeleton
PROJECT ARCHETYPES
Step 2: GitHub Repository creation
LivePerson GitHub EnterpriseREST Call
Commit from step 1
PROJECT ARCHETYPES
Step 3: Continuous Integration Job (TeamCity)
LivePerson TeamCity ServerREST Call
PROJECT ARCHETYPES
MASTER Build job (VCS)
BRANCHES Build job
PULL REQUEST Build job
IN-HOUSE MAVEN / NPM PLUGINS
IN-HOUSE PLUGINS – WHY?
▸ You can enforce additional steps by the organizational needs.▸ The plugin does not exists outside. ▸ NPM / Maven Framework will do the job for you. ▸ The user should be able to configure it easily.
IN-HOUSE PLUGINS – WHY?
IN THE SAME WAY TO ALLUnified way to run common actions
IN-HOUSE MAVEN PLUGINS – CODE IT
import org.apache.maven.plugin.AbstractMojo;import org.apache.maven.plugin.MojoExecutionException;import org.apache.maven.plugins.annotations.Mojo;
@Mojo( name = "sayhi")public class GreetingMojo extends AbstractMojo {
public void execute() throws MojoExecutionException {
getLog().info( "Hello, world." );
}}
IMPORT MOJO LIBS
WRITE TO CONSOLE
IN-HOUSE MAVEN PLUGINS – USE IT
<build><plugins>
<plugin><groupId>sample.plugin</groupId><artifactId>hello-maven-plugin</artifactId><version>1.0-SNAPSHOT</version><executions>
<execution><phase>compile</phase><goals>
<goal>sayhi</goal></goals>
</execution></executions>
</plugin></plugins>
</build>
MOJO NAME
WHEN TO RUN IT?
DECLARATION
puppet-deploy-plugin
DEPLOYMENT
ci-provisioning-plugin
OPENSTACK INSTANCE
yum-upload-plugin
YUM REPO UPLOADER
checkmarx-plugin
SECURITY SCAN
k8s-maven-plugin
KUBERNETES DEPLOY
cd-flow-plugin
CONTINUOUS DELIVERY
IN-HOUSE MAVEN PLUGINS @LIVEPERSON
LIVEPERSON PARENTS (POM/NPM)
WHAT IS PARENT POM?
▸ POM (Project Object Model).▸ Parent POM = Super POM. ▸ Specifies parent element and makes your project inherit from it.▸ Enforces projects to use it as a prerequisite for release.
Example of the Maven Parent POM
WHAT IS PARENT POM - EXAMPLE?
Project ReactorLivePerson Parent Module
inherit inherit
Super POM Parent POM Child POM
Main Plugins configurations
maven-compiler-plugin3.7.0
Overwrite / Extend configuration
<configuration><source> 1.8
Declare / use the plugin
<failOnError>false
LIVEPERSON PARENT POM
SERVICE #1
POM
SERVICE #2
POM
SERVICE #3
POM
SERVICE #4
POM
SERVICE #5
POM
LivePersonParent POM
LIVEPERSON PARENT POM
POM
<parent><groupId>com.liveperson.maven</groupId><artifactId>liveperson-parent</artifactId><version><SOME VERSION></version>
</parent>
<plugin><groupId>com.apache.maven.plugins</groupId><artifactId>maven-compiler-plugin</artifactId><version>${maven.compiler.version}</version>
</plugin>
DECLARATION
USAGE
VARIABLE
LivePersonParent POM
LIVEPERSON PARENT POM
No hardcoded versions but Environment Variables Parameters are being injected on build time by Groovy Hooks
$ mvn clean deploy
$ mvn clean deploy -Dmaven.compiler.version=${compilerVersion}
HOOK
def compilerVersion=‘3.7.0’
POSTBUILD ACTIONS
POSTBUILD ACTIONS – WHY?
Enforce Corporate Governance
POSTBUILD ACTIONS – WHY?
▸ Make sure developers cannot comment / remove important tasks.▸ Make sure we are not violating security standards..▸ Make sure we are using legal and permitted software.▸ Enforce corporate quality standards.
POSTBUILD ACTIONS – WHAT DO WE RUN?
Analyze: what do we run on which build type
Build Types:▸ Normal CI Process▸ Schedules (nightly) CI Process▸ Release Build
Identify build type
NORMALRELEASESCHEDULES
Triggers …
POSTBUILD ACTIONS – WHAT DO WE RUN?
[Commercial] Static code scan for security violations
[Open Source] Static code scan for code quality
[Commercial] Dependency scan for security and licenses
POSTBUILD ACTIONS – WHAT DO WE RUN?
Save build data and info into a local database for reports, dashboard, debugging…
Coverage reports (Istanbul / JaCoCo)
POSTBUILD ACTIONS – HOW?
We use Maven Lifecycle Extension. Extending the Maven Lifecycle Class
@Component( role = AbstractMavenLifecycleParticipant.class, hint = ”delegate")public class MyExtension extends AbstractMavenLifecycleParticipant {
@Overridepublic void afterSessionStart(MavenSession session) throws MavenExecutionException {
// Do something…}
@Overridepublic void afterProjectsRead(MavenSession session) throws MavenExecutionException {
// Do something…}
}
POSTBUILD ACTIONS – HOW?
Identify the Executions type (Session: Ended, Succeeded, Failed….)
ExecutionEvent executionEvent = (ExecutionEvent) event;MavenProject project = executionEvent.getProject();
if (executionEvent.getType() == ExecutionEvent.Type.SessionEnded) {// Do something…
}
Place your Maven extension JAR under your Maven installation:
${maven.home}/lib/ext
POSTBUILD ACTIONS – HOW?
Each jar is being replicated to the build agents to ${maven.home}/lib/ext
Build Agent Build Agent Build Agent Build Agent Build Agent Build Agent
DASHBOARDS AND REPORTS
WE COLLECT ANYTHING WE CAN
We write to the DB from any plugin / process we canWe expose any data we need in a REST API Service
REST API
BUILD PUPPET POST BUILD QA DEPLOY PROMOTION PRODUCTION
DASHBOARD FOR CI SERVICES
UNIFIED PIPELINE TO PRODUCTION
LIVEPERSON CD PIPELINE
Can be automated on green CI or manually on demandUnified process for all services
Check if QA environment
status is green
Run the deployment
(Puppet or K8S)
Save Deployment results and info
Update Slack Channel with the
results
Update ServiceNow with the relevant info
Manage and Orchestrate
MAIN IMPACT
Our CI Processes are
ROBUST AND BULLETPROOF
We know the process and can
SUPPORT ALL PROJECTS CYCLES
Onboarding new services and projects
IT’S EASY
Single point of failure
EASY TO FIX, EASY TO MAINTAIN
87
THANK YOU ALL!https://il.linkedin.com/in/nirkoren
https://www.facebook.com/koren.nir
@KorenNir
@nir_koren