ESA UNCLASSIFIED - For Official Use

Continous integration and deployment in SSA Data System

The SSA way towards distributed/cooperative development and operation processes (DevOps)

Contineous Integration in SSA Distributed/Federated Development

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 2

Outline

• Background

• The SSA CI Experience

• Starting point

• Toolset

• Pilot Phase objectives

• CI Principles

• Process flow

• Status & Achievement Summary

• Evolution

• Pilot roadmap

• Vision (CI, CD, DevOps)

• Conclusion

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 3

Background

• Three different segments

• Each with different Operational Concepts

• Heterogeneous procurement

• Workplan

• National Programme funded activity

• Studies

• Needs to support upcoming SW Common Core (SST)

• Combine segment specific and common services development and maintenance

• Functional processing fine tuning required to react to operational events

• 10 development contracts X 3 main delivery X 3 component per delivery 90 processes

• Growing along the SSA Data System assets growth

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 4

• Software controlling is done without the support

of any SW versioning tool

• Use of Software tools to support Data System

Engineering lifecycle processes limited to:

• Ticketing system

• SPR management tools (RC, Jira)

In support of the human driven processes

defined in the applicable ECSS/QA practices

Current Status

Delivery

•FTP/DVD

Build/Compile

•Build from scratch

•RPM

Deployment

•PA -> FA -> Under Production

Unit Testing

•PA -> FA

Starting point

•Adequate support to SW Configuration , Change, Versioning •Increase process efficiency via automate compilation and deployment

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 5

Toolset

SW LifecycleProcesses

Starting point Pilot evolution Notes

Ticketing OTRS OTRSJira

OTRS integrated with JiraOr move to Jira

SW Delivery FTP , DVD and human processes

ESA Gitlab

SW versioning Human processes ESA Gitlab

Compilation&Deployment

Human processes JenkinsMavenDocker

GIT/Gitlab perfectly integrates withjenkins, maven etc

SPR Management RC ESA Jira

Project Management none ESA Jira TBD Variety of tools available on market

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 6

• ESA Gitlab as code repository

• And keeping in focus basic CI principles

• stepwise process automation

• Build

• Deploy

• Test …

Pilot phase objectives

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 7

CI Principles

Principle @ ESA SSA @ Industry

1 Maintain code repository

2 Automate the build

3 Make the build self testing

4 Everyone commit to baseline every day

5 Every commit (to baseline) should be built

6 Keep build fast

7 Test in a clone of the production environment

8 Make it easy to get latest deliverables

9 Everyone can see the result of the latest build

10 Automate deployment

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 8

Process flow

CI pipelinescompilationdeployment

GIT ORIGIN @ESA

Software VersioningInternal Repository

NEO SW App 1

NEO SW App 2

NEO SW App n

SST SW App 1

SWE SW App 1

SST SW App 2

SST SW App n

SWE SW App n

IRE OPE

Consortium 2

GitLab

Consortium 1

GitLab

Consortium 3 GitLab

Study 1GitLab

NEO SSTSWE

git://http://https://

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 9

Process flow: CI/CD @ Industry

CI automated processes

Consortium

Git Repository usedfor delivery to SSA

Application 1

Application 2

Application 3

Application 3

Application 1

Application 1

Application 3

Application 2

Clone of ESA OPE

Subco2

Subco1

Subco3

git://http://https://

ConsortiumWorking CI Repository

CI Repository

Build/Compilation

Deployment

Unit Testing

Integration Test

Authorise Delivery

1. Consortium Working Repository (not visible to ESA)

2. Automate processes• Build• Deploy• Test

3. Fast and frequent code change • Feedback to developers• Repair broken links immediately • Capability to revert back

4. Integrate frequently 1. Feedback to SubCo/ Developers2. Repair broken links immediately3. Capability to revert back

5. Test done onto a clone of the Production Environment

6. Check in validated delivery onto the Consortium Master Repository (visible to ESA)

Feedback

success

1

2

3 & 4

5

6

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 10

Process Flow: CI/CD @ESA SSA

automated processes

IRE &Pre-OPE

git://http://https://

ESA SSA ORIGIN

Code Repository

Build/Compilation

Deployment

PA

FA

Authorise Delivery

1. Pull out from Industry Repository 2. Automate processes

• Build• Deploy• Test

3. PA&FA • Feedback to developers• Capability to revert back• Test include also the test of the

automated processes 4. Failed test result is inputs to the Industry CI 5. Test done onto a clone of the Production

Environment 6. Validated delivery get the go-ahead for

undergo Production (the actual step shall be coordinated)

Feedback

1

2

3

ProductionOPE

Consortium

Git Repository usedfor delivery to SSA

65

4

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 11

Process Flow: CI/CD @Industry

FATDev/maint

Team Integration

Test

A B C D E F

By Grégoire Détrez, original by Jez Humble - This file was derived from: Continuous Delivery process diagram.png, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=43977816

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 12

Process Flow: CI/CD @ESA SSA

FA(Pre-OPE)

SSA DS Team

PA(IRE)

A B C D E F

By Grégoire Détrez, original by Jez Humble - This file was derived from: Continuous Delivery process diagram.png, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=43977816

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 13

FA(Pre-OPE)

SSA DS Team PA (IRE)

GIT Mirror Repository Synchronization

ESA Feedback

FATDev/maint

Team Integration

Test

A B C D E F

A B C D E F

Process flow: Combined processes

By Grégoire Détrez, original by Jez Humble - This file was derived from: Continuous Delivery process diagram.png, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=43977816

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 14

Phase 1

• Scope

• Create a SSA repository reference for all the ongoing development/maintenance

• Support to SW Versioning

• Automate compilation and deployment

• Try out and consolidate configuration set up

• Projects/groups trees

• Repository Structures

• Started with

• ESA GIT GITLAB

• Gitlab supports git, http, https connection

• Challenges

• Selection of tools and initial set-up

• What tool are the most appropriate?

• No industry/partner reference

Output:•Capture key aspect into documents defining guidelines and processes•Initial set-up and tools selection •Familiarization

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 15

Phase 2

• Scope

• Complement pilot phase started in Phase 1

• Fully automated compilation and deployment

• Full Validation of Processes within one of the three segment

selected/identified as pilot

• Prepare for a Common Core Software Engineering lifecycle (required by

one of the three segment)

• Challenges

• Capability to continuously evolve

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 16

Summary

Steps Schedule Planned action Objective

1) Pilot Jan-May2017

Configuration of the selected technology (GitLab) Creation of current repository Integration with plug-in technology (Jenkins, maven , RPM) Open (not imposed) to on going development & maintenance Definition of Guidelines to be applied in SSA development

• Support to SW Versioning • Create 80% of the SSA SW Repository• Automate

• Build 80%• Deployment 50%

• Try out and consolidate configuration set up• Make the approach applicable to all future

SSA development • Consolidate requirements• Tool evaluation

2) Consolidation Jun-Sep2017

Fully deployed set-up Finalization of the applicable Guidelines Formal applicability to all the new activity

• 100% SSA SW repository• Fully automated compilation and deployment• Support to SST Common Core Software

Engineering lifecycle • Security issue (needed by SST)

3) Evolution Q4 2017TBC

Automatic unit testing Documentation management Documentation production Extension of the processes to Portal CMS Full integration with

Ticketing management Requirements management Change management Problem management Project management (desireable) Security Management Asset Management IPR Management (i.e. Blackduck Protex plug-in?)

• Further efficiency of the processes

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 17

Roadmap

Feb-May 2017

Support to SW Versioning

Create 80% of the SSA SW Repository

Automate

• Build 50%

• Deployment 50%

Try out and consolidate configuration set up

Make the approach applicable to all futureSSA development

Consolidate requirements

Tool evaluation and adoption

Jun-Sep 2017

Create and maintain 100% SSA SW repository

Automate

• 90% Build

• 90% deployment

Prepare the ground for SST Common Core Software Engineering lifecycle

Security issue (needed by SST)

From October-2017

Applicable in all new SSA Procurement

Automatic unit/integration testing

Documentation management

Documentation production

Extension of the processes to Portal CMS

Full integration with

•Ticketing management

•Requirements management

•Change management

•Problem management

•Project management (desirable)

•Security Management

•Asset Management

•IPR Management (i.e. Blackduck Protex plug-in?)

•System Monitoring

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 18

Status on 20/06/2017Delivery Repository Compilation Deployment

build binaries IRE OPE Testing

SSA-SWE

swe-services SWE-XI gitlab mirror available jenkins maven manual manual manual

swe-resources SWE-XI gitlab mirror available jenkins maven manual manual manual

swe-bindings SWE-XI gitlab mirror available jenkins maven manual manual manual

swe-common-libraries SWE-XI gitlab mirror available jenkins maven manual manual manual

swe-generic-services SWE-XI gitlab mirror available jenkins maven manual manual manual

SWE-ISM SWE-III DVD/FTP available jenkins maven manual manual manual

SWE Registration Portlet SWE-III DVD/FTP available jenkins maven jenkins&gitlab-CI manual manual

Evolution Maintenance SWE-III 3.4 DVD/FTP not available manual manual manual manual manual

Evolution Maintenance SWE-III 3.5 DVD/FTP not available manual manual manual manual manual

Tools

ESA Gitlab

SSA-NEO

Neo Portal COM-V.1 DVD/FTP available manual manual manual manual

SSA Jenkins Neo Tool BE NEO-VI DVD/FTP available jenkins maven manual manual manual

SSA Maven Neo Tool FE NEO-VI DVD/FTP available manual manual manual manual

Docker NEODECS via bridging gitlab available

D3 COM-V.1 DVD/FTP available manual manual manual manual

SSA-SST

SST Cataloguing Tool SST-I WP 1 DVD/FTP available manual manual n/a manual

Analisys Tool SST-I WP 1 DVD/FTP available jenkins manual n/a manual

SST Visualization Tool SST-I WP 2 DVD/FTP available GitLab CI GitLab CI n/a manual

SST Web Portal Applications COM-V.2, SST-XIX DVD/FTP available jenkins manual manual manual

DPC COM-V.2, SST-XIX DVD/FTP not available not available manual manual manual

Planning System COM-V.2, SST-XIX DVD/FTP not available not available manual manual manual

SSIM COM-V.2, SST-XIX DVD/FTP not available not available manual manual manual

General Services COM-V.2, SST-XIX DVD/FTP to be created not available manual manual manual

Gendared FTP to be created not available manual n/a manual

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 19

• Whether the team is building the right thing in the first place

• Whether the operational requirements can be captured as executable specifications

• Whether you can add value by working with the team (ESA Team, Industry Partners, Developers,

Operators) to identify and express acceptance criteria, scenarios and examples

• Whether processes are working or can be improved

• Where the waste is and how it can be removed

• How to approach non-functional testing, performance, security, accessibility etc.

• What data and configuration are required across all of the environments in your build pipeline

• What does make sense to focus upon, what does make sense other teams focus upon

Evolution

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 20

• Critical analysis of current processes

• Avoid the syndrome “we are doing like this

because of our peculiarity and constraint”

• IPR, Security, Quality, Asset, Financial

management are all independent processes

• Practices adopted quite heterogeneously across different

activity /projects/programme

• Last (and worst) relying on human (manual) processes

• Often not linked together and scattered around

• Evolution

• Link the specialised management (Git based)

• IPR Management

• Security Management

• Quality Management

• Corporate asset management

• Continuous screening

Evolution

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 21

DEVOPS https://en.wikipedia.org/wiki/DevOps

Is a cultural shift and a collaboration concepts (between developers and

operators) reflective of key aspects of SW Development and Delivery

processes

• Code — Code development and review, version control tools,

code merging;

• Build — Continuous integration tools, build status;

• Test — Continuous testing tools that provide feedback on

business risks;

• Package — Artifact repository, application pre-deployment

staging;

• Release — Change management, release approvals, release

automation;

• Configure — Infrastructure configuration and

management, Infrastructure as Code tools;

• Monitor — Applications performance monitoring, end–user

experience.

Evolution

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 22

Conclusions

• No Conclusions but we can certainly report our initial feedback

• Piloting Phase successful

•Quick results Further impulse

• Raised the awareness of the process complexity

•Helped to focus on next steps

• Tools selection and their proper usage is a key factor

•But avoid getting paranoid

• Communicate (within the Team, with the Customer, with Industry)

• New approach open, not imposed to the Industrial partners, in the pilot

Phase

• Will became applicable in the upcoming procurement

ESA UNCLASSIFIED - For Official Use ESA | 01/01/2016 | Slide 23

Conclusion

and we are open to questions

… and cooperation