Continuous integration - stability, reliability and speed in software development
-
Upload
computaris -
Category
Technology
-
view
253 -
download
0
Transcript of Continuous integration - stability, reliability and speed in software development
Table of Contents
Introduction......................................................................................................................................................................................................3
Why continuous integration?...........................................................................................................................................................................4
Benefits of continuous integration....................................................................................................................................................................6
Challenges and Solutions................................................................................................................................................................................9
Achieving full test automation-gradually.........................................................................................................................................................10
Managing complexity......................................................................................................................................................................................11
Introduction
Stability, reliability and speed are just
some of the words used to describe the
advantages of a software development
practice known as Continuous Integration
(CI). It has transformed systems
development. Martin Fowler, a thought
leader on CI describes1 it as the practice
of integrating code into a shared repository
regularly with automated check-ins
authenticating the build to identify issues.
This procedure allows teams to detect and
resolve problems early and quickly –
before they become major downstream
issues.
CI’s key benefit could be summed in one
word. Quality. Developer teams at
Computaris use CI for one aim - to achieve
excellence. Some projects are more
demanding than others. Larger projects
require change often during the
development lifecycle.
The CI process allows Computaris to
deliver outstanding results – faster.
Computaris has an excellent track record
of delivering successful projects.
Customers come to Computaris for both
new projects and to seek counsel on
existing projects expecting swift
turnaround times combined with excellent
quality. This whitepaper explains best
practice methodology for CI and the
benefits it delivers.
/
Continuous integration: stability,
reliability and speed in software
development
page 3
Why continuous integration?
In today’s dynamic programming environment, software development is getting increasingly complex. For example, systems being built on established languages2 today could easily be outdated3
in a matter of a few years. As such, IT teams need to stay ahead of the curve in software development best practice to create systems and programs that operate effectively not just in the short term, but also continue to deliver results in the future. Teams that adopt CI can ensure continuity.
By using CI, development teams can continuously integrate and modify code within a shared repository several times a day. An automated build verifies each check-in and this allows development teams to detect problems early4. This is an important step to minimize the cost of fixing anomalies and to maximize build quality.
The technique that underpins CI is the integration of code being constantly developed and added to a code base or a source code control system5. In the past, most developer teams relied on a daily build. Now it is at least once a day with each significant change. This enables the developer team to have constant feedback on the status of the software and anomalies are easier to manage.
These are the basic tenets of CI:
> Use a single repository for source code control: This is the developer team’s fail-safe and hub for automation6.
> Self-testing the builds: Have a suite of automated tests to examine the code base for bugs regularly7.
> Validate each commit: Select and stick to a consistent coding standard. Regularly monitor the repository for oversights and
check each commit8.
> Commit often: Team members need to commit regularly within an agreed timeframe to reduce the likelihood of conflict changes9.
> Test accurately: A realistic test environment that reflects the production environment will contribute to obtaining a higher-quality of the final build10.
> Efficient and transparent processes: Having good API documentation with robust processes can ensure that everyone, including customers, are in-sync with the build to avoid needless and costly rebuilds11.
> Deployment strategy: When the business is ready, use a script to deploy the application into production via an automated live test server that's accessible to key stakeholders12.
page 4
2Different Programming Languages
3Fabien Potencier blog
4Mike Jackson, Software Sustainability Institute
5C. Aaron Cois, Software Engineering Institute
6Martin Fowler blog
7Martin Fowler blog
8Paul Jones, Why Coding Standards Matter
9Seth Robertson, Blog
10John Overbaugh, TechTarget
11David Flanders and Malcolm Ramsey, Joint Information Systems Committee
12Eric Reiss, Radar
/
Continuous integration: stability,
reliability and speed in software
development
Benefits of continuous integration
Faster time-to-market
In the past, development teams were
bogged down with paperwork and had to
write deployment codes for other teams to
follow. This was a silo approach. Such
time-consuming process can mean that
time-to-market for new products and
services could take months – even
years13. CI has transformed and
accelerated time-to-market.
An automated pipeline of continuous
delivery ensures that only good quality
code makes it all the way through14. As
such, automation plays a fundamental role
in CI and helps to build a robust
development work flow where fixing bugs
and improving quality is at the heart of the
system. This level of quality control speeds
up time-to-market and gives businesses a
competitive edge15.
Faster development lifecycles
The usual developer process, without CI,
can often comprise of the following steps:
1. Develop new code OR perform bug
fixing a. This includes Unit tests or
manual smoke tests
2. Commit code in version control
3. Wait for QA teams to raise issues or
verify fixes
4. Receive issues from QA teams and
return to Step 1
Some developers write code, run a few
tests and pass the code to someone else
to verify it16. That’s inefficient, it damages
quality and hampers knowledge sharing
within the team17. Steps 3 and 4 usually
take a long time, measured in hours if not
days. By the time a developer received
feedback, he could have moved on to
different tasks and prioritized something
else. It could take more time for the
developer to familiarize himself with the
code again and retrieve that context. If the
duration for steps 3-4 took even longer
than just a few days, it is very likely that
another developer could have taken over
the fixes and the original developer might
not even be available. All this adds
unnecessary delays and is a setback for a
business hoping to deploy a solution to the
market.
page 5
13Sven Malvik blog
14Florian Motlik, Codeship
15Jason Tee, TheSeverSide
16Scott W Ambler
17Jason Cohen, SmartBear Software
/
Continuous integration: stability,
reliability and speed in software
development
Benefits of continuous integration
CI provides a new lease of life for the developer process:
1. Develop new code OR perform bug fixing
a. This includes adjusting the automated tests to verify the new code
b. For large test suites, nominal bugs and glitches should be addressed at this step
2. Commit code in version control
3. CI server (e.g. Jenkins) detects the new commit and runs a new build
4. Team members are notified via e-mail if the CI build found issues Some developers write code, run a few tests and pass the code to someone else to verify it16.
That’s inefficient, it damages quality and hampers knowledge sharing within the team17. Steps 3 and 4 usually take a long
time, measured in hours if not days. By the time a developer received feedback, he could have moved on to different tasks and prioritized something else. It could take more time for the developer to familiarize himself with the code again and retrieve that context. If the duration for steps 3-4 took even longer than just a few days, it is very likely that another developer could have taken over the fixes and the original developer might not even be available. All this adds unnecessary delays and is a setback for a business hoping to deploy a solution to the market. CI provides a new lease of life for the developer process.
The main difference is that steps 3 and 4 are automated and happens in a window of just a few minutes – not days - from the time the developer finished coding. The developer will not have time to lose context, any issue detected by the CI build will be fixed much faster.
Other CI applications referenced by this White Paper aim to have 1 to a few builds per day, a build lasting one to several hours. We try to design our builds to be faster, in the range of 10 to 15 minutes with a test suite having over a thousand full end-to-end automated tests (see “4.4 Parallelization and configuration”).
Quality: from good to great
CI makes the integration process efficient. Without CI, Quality Assurance (QA) engineers could spend more than 50% of total project time running nominal tests for integration. This will shorten the time the QA team spends looking for bugs and other subtle issues18.
page 6
18Gautham Pallapa, blog
/
Continuous integration: stability,
reliability and speed in software
development
Benefits of continuous integration
The CI process ensures that the QA team receives code with integration done and the nominal test cases are successful. This frees the QA team to dive deeper and focus on problems, test more scenarios and yield better overall quality and results19 .
Stable codes and demo ready
The fast development lifecycle described in section 3.2 will allow a large part of the code to be stable even when under active development. Agile development methodologies20, which rely on incremental development and getting constant feedback from stakeholders, ensures that the project is on the right track. Regardless of the project management techniques used, by using CI the team can be ready at any time to demo the system. This allows for quick feedback during customer meetings and ensures requirements are well understood and the
project is moving in the right direction. As such, customer-led improvements can be incorporated within a matter of a few hours – and not days21.
Smarter risk mitigation
During a build's lifecycle, there is always the possibility of probable mistakes22. If there are suitable parameters and metrics in place, the status of the build can be constantly monitored accurately. CI enables developer teams to discover and resolve flaws promptly. Furthermore CI provides intelligence for teams to eliminate potential vulnerabilities too. This is especially the case when they have access to historical data which allows QA teams to build test cases to ensure that historic mistakes are not repeated.
CI mitigates risk because check-ups and trials are run frequently - in some cases every few hours - so there's a higher
probably of detecting issues and fixing them before launch. This ensures that businesses have robust solutions they can deploy with confidence. Joe Green, a developer and champion of CI wrote: "Having a good CI system is akin to having a fire alarm in your house. It won't fix your error for you but you can be rest assured that it will flag up the error as soon as it is integrated and that the build containing the error will not make it into your staging or production environments.23
Flexibility
CI underpins flexibility and can foster creativity in the development team. Thanks to the automated nature of CI, development teams have the flexibility to change code quickly and without fear. Fast CI builds give developers the opportunity and freedom to innovate and try new techniques to enhance the system. This is one of the key benefits of CI.
page 7
19Gautham Pallapa, blog
20Agile Alliance
21Duval, Matyas et al. Tech Target
22Francis Adanza
23Joe Green, Smrtr
/
Continuous integration: stability,
reliability and speed in software
development
Benefits of continuous integration
Faster onboarding for new team
members
Sometimes onboarding new team
members, especially within a large project,
can take up valuable time. If key members
of the developer team are not available,
bringing someone up to speed on a build
can be challenging. CI makes onboarding
more efficient. It generates API-based
documentation to make the software
architecture easily accessible to new
members on the developer or clients’
teams. You only need to browse the test
cases to have a quick overview of the
functionality of the system.
Having a CI process allows you to have
the tools to automatically build, deploy,
and test the system. These are the same
tools needed for a new developer/QA
engineer to hit the ground fast25.
24Manuel Weiss, Codeship
25Darragh Curran, Inside Intercom
/
Continuous integration: stability,
reliability and speed in software
development
Challenges and solutions
While there are many benefits to CI,
developer teams and customers can face
a number of challenges from the build
phase to deployment.
Achieving full test automation
gradually
Automating an entire build at the start is
not easy, so start simple26. Manage the
process continually by keeping
deployments consistent and have a well-
defined process that can be constantly
automated and developed over time. This
continuous automation methodology can
help developers automate the entire build.
Even with the 80/20 principle27, achieving
100% automation is obviously more
difficult than 80% automation. Some
developers have found that fixing 20% of
the bottlenecks could give you 80% gain in
speed28. Yet coding automated tests is still
coding that requires careful handling.
Some teams fail to build a reliable CI
process because they consider it is easy
and do not pay enough attention to detail.
Here are some common mistakes:
> Poorly-coded automated tests will cause
team members to spend more time
investigating failures related to the test
suite itself, rather than investigate real
errors of the product that’s being tested.
> After a while, frustration grows, and the
entire test suite is abandoned or rewritten
from scratch.
> Poorly-managed infrastructure will cause
environmental problems and will probably
impact reliable CI builds. These problems
can include: malfunctioning machines,
patchy networks, disks with limited space,
and sluggish server and CPU
performance, which can cause CI builds to
run slowly.
26Eric Reiss, Radar
27Pareto Principle
28Naresh Jain, Key Principles for Reducing
Continuous Integration Build Time
/
Continuous integration: stability,
reliability and speed in software
development
Managing complexity
Software development best practices must
be carefully applied to test code not only to
production code. Especially for
large/complex test suites with
hundreds/thousands tests, poorly
implementing automated tests will greatly
increase cost of maintenance and may
even lead to completely abandoning the CI
test suite.
It is important for companies to ensure that
their developer/quality assurance teams
adhere to best practices, including Don’t
Repeat Yourself (DRY) and Keep it Simple
Stupid (KISS). In their book, the Pragmatic
Programmer, Andy Hunt and Dave
Thomas coined the phrase DRY29. DRY is
about reducing complexity to manageable
units by dividing a system into pieces30. As
systems are fragmented into components
and subcomponents, complexity could be
reduced to a single responsibility.
Keep It Simple Stupid or Keep it Stupid
Simple (KISS) is another best practice
concept in software development which
means – the simplest explanation is likely
to be the best one31. In KISS, developers
aim to find a resolution as easily as
possible and produce code to solve
complex problems in fewer lines of code –
faster and with remarkably better quality.
Best practice guidelines for complex
coding
> Conduct tests on asynchronous I/O
(Input/Output) for faster and accurate
results instead of relying on “sleep” mode32
> To manage test data efficiently, select an
accessible tool or language so team
members can retrieve data efficiently33
> Test scenarios must produce log files
easy to read and follow, this is mandatory
for easy tracking of issues when the CI
build fails34.
29Andy Hunt and Dave Thomas, The Pragmatic
Programmer
30Chris Peters, Code
31The Apache Software Foundation
32Mixu, blog
33Cohen, Hunter et al., Association of
Computational Linguistics
34Tamir Korem, Cloudify
/
Continuous integration: stability,
reliability and speed in software
development
Managing complexity
Maintaining build speed
For improved productivity, builds need to be fast. Along with best practice concepts, the developer teams also need to consider the following:
> Ideally a build should be fast enough to be able to run and validate each commit a developer makes in the version control system early and regularly35. Of course this is hard to achieve especially for large projects with teams of many developers where a commit might happen every few minutes
> It is difficult to find an issue with 10 commits and 200 failures out of 1000 tests. Investigating each commit takes time. A fast build exposing problems in each individual commit would minimize costs of debugging/investigating
> Black box testing, where testers are only
aware of what the software is supposed to do and not how it does it, runs much slower than unit tests. Unit tests verify the functionality of a specific section of code. All this can be complex36.
> For black box test, all components of the system under test must be built, deployed and configured fast. Assertions on system behavior, where tests can be run on your assumptions, are black box. They verify the system’s output so they must wait for disk I/O, network I/O, caches to fill up, messages to pass through queues and all other type of events that must happen until a noticeable output from the system can be tested. All these operations take time and making the build fast becomes a challenge.
Parallelization and configuration
The solution to make the test suite faster is to use parallelization, which means
running tests on multiple components or subcomponents simultaneously. One of Computaris’ test suites can run approximately 1000 tests in parallel with corresponding sub-tests concurrently. A rough estimation would be 40 processes each with an average of 100 threads running in parallel. Of course this requires a fair amount of hardware.
Most systems have system-wide configurations which are changed very rarely. A CI build must test various configuration options and this creates a new problem, how can you run parallel tests with different configurations? The solution is yet again parallelization. Computaris has the capability to carry out multiple system tests that can run simultaneously, each instance with its own configuration.
35Wikipedia, Revision Control
36Black Box Testing
/
Continuous integration: stability,
reliability and speed in software
development
Managing complexity
This approach requires the following:
> The need for robust hardware: multiple
instances of parallel system runs tend to
require a great deal of hardware
resources.
> The need to automate the infrastructure:
Computaris has a regression suite with
over 1000 tests running on 16 parallel
systems being tested and uses tools such
as Chef to manage infrastructure
automatically37. This also helps to solve
the need for many hardware resources.
Having automation allows the team to run
CI builds on commodity hardware that may
fail. Recovery from these failures is
extremely fast using “infrastructure as
code” which eliminates the need for
expensive server hardware38.
Make tracking down real problems easy
and fast
Ultimately, finding and resolving problems
faster is why CI has been used for the
build. For large test suites (e.g. 1000 tests)
if you have 200 tests that fail, the simple
action of checking the failure message of
each of the 200 failures can be time
consuming.
In this instance, a “Root Cause Analysis
Report” which looks at the origin of the
problem and the corrective measures
required can be helpful for the developer
team. The Root Cause Analysis Report
can group tests by cause of failure and
provide an overview of the problems that
caused the build to fail in the first place39.
Decide on your tests
For the CI build to be successful, it is
important for the developer team to settle
on a testing regime. Part of Extreme
Programming (XP), Test-Driven
Development (TDD) uses automated unit
tests to “get something working now and
perfect it later40.” After each test has been
evaluated, similar tests are performed
again until it reaches a satisfactory result.
A collaborative practice, Acceptance Test
Driven Development (ATDD) involves
customer being part of the test design
process to define criteria41. ATDD ensures
that all stakeholders are project members
and they understand precisely what needs
to be done and implemented42.
37 JT Gray, Safari Blog
38Hummer, Rosenberg et al, Distributed Systems
Group
39Alon Linetzki, Root Cause Analysis for Software
Testers
40Narayana Maruvada, Agile Record
41Scott W Ambler 42Gabo Esquivel, blog
42Gabo Esquivel, blog
/
Continuous integration: stability,
reliability and speed in software
development
Managing complexity
Behavior-Driven Development (BDD) is, as
the name suggests, behavior-based build
which incorporates the practices of TDD
and focuses on language and
interactions43. One benefit of BDD is that it
provides the developer team and customer
a clearer indication as to what the system
should do.
TDD can be more technical and complex
that requires developer teams to become
conversant with the detailed object
model44.
Continuous deployment
Continuous Deployment or continuous
delivery (CD) is described as the practice
of releasing only good build to the final
user45. This concept is based on CI and is
a continuation of this principal to the 'last
mile' for customer delivery46. With CD,
teams can quickly adapt to business
requirements and user needs which in turn
enabled greater co-operation between
operations and delivery, championing best
practice and driving real change47.
Software developers deploy their software
into production each night and this
placement provided a number of benefits:
written software was not wasted,
developers responded quickly to problems
and fast turn-around times led to valuable
partnerships between teams and their
customers48. Similar to CI, CD mitigates
risk and detects bugs fast. Crucially, it also
enables the developer team to progress on
to fully functioning software even faster.
Fine-tuning
As we've seen, CI is one of the most
effective concepts for software
development. However for it to work
perfectly, it requires discipline in the team.
However, developer teams are human
after all. Sometimes, the bigger the team
and the code, the more likely it can get
broken and the lengthier the process will
be to fix it because other business
priorities take precedence49. Team
members can ignore the red flags from
Jenkins and overtime flawed software can
get passed on to the end customer. The
solution? Forbid team members from
including anything into the main repository
and make a script that anyone can access
easily. The script should merge, test, and
commit without exceptions. The developer
of each code should be responsible for
broken tests and the team should raise red
flags before a code gets into the master
repository.
43Agile Alliance
44Gabo Esquivel, blog
45Carl Caum, Puppet Labs
46Martin Fowler, blog
47Florian Motlik, Blog
48Sarah Goff-Dupon, Atlassian Blog
49Yegor Bugayenko, Dev Ops
/
Continuous integration: stability,
reliability and speed in software
development